Are you tired of running endless A/B tests with minimal insights? Do you find yourself drowning in data but thirsting for actionable results? If you're nodding your head, you're not alone. In the fast-paced world of digital marketing, A/B testing has long been a go-to strategy for optimizing campaigns and improving conversions. But let's face it – traditional A/B testing can be time-consuming, resource-intensive, and sometimes, frankly, hit or miss.
But what if there was a way to supercharge your A/B testing? A method that could not only speed up the process but also provide deeper, more accurate insights? Well, buckle up, because that's exactly what we're diving into today.
And who better to guide us through this AI-driven revolution than our guest today, Nicole Leffer. Nicole is a seasoned marketing leader and a widely recognized expert in deploying generative AI technology to improve marketing efficiency, quality, and outcomes. Her expertise in leveraging AI for A/B testing and optimization is unparalleled, and she's here to share her insights on how we can all test smarter, not harder.
✨ Efficiency Boost: How AI dramatically speeds up the A/B testing process, allowing for faster iterations and improvements.
✨ Deeper Insights: The ability of AI to uncover complex patterns and relationships in test data that humans might miss.
✨ Personalization at Scale: How AI-powered A/B testing can lead to more personalized user experiences across different segments.
Learn more about Nicole Leffer
- Connect with Nicole Leffer on LinkedIn
Resources & Brands mentioned in this episode
- NicoleLeffer.com
- RICCE Framework for prompting
- AI Work Buddy
- The Bottleneck Diagnostic Quiz: A free 2-minute quiz to pinpoint your biggest business bottleneck. Find it at theaihat.com/diagnostic.
- Follow Mike Allton on Instagram and get more AI tips
- Explore past episodes of the The AI Hat Podcast podcast
Full Transcript
(lightly edited)
Test Smarter, Not Harder: AI's Impact on A/B Testing in Digital Marketing with Nicole Leffer
[00:00:00] Nicole Leffer: What you wanna be telling the AI is like, what is your goal, right? Like your goal is to get the highest conversion list of that landing page that you possibly can. You wanna tell it what that conversion is so it knows what it is, but then you wanna ask it like, based off of the page you're looking at, I've given you a screenshot of this page.
[00:00:18] What is the, like, you know, I want you to look at this. Explain to me what you're. Seeing and then tell me, based on all of this, what are the different things that could lift the conversion rate? And then analyze all of those things in the context of this specific page in the context of your expertise.
[00:00:37] And tell me which of those things that, if I did it, do you think is going to make the biggest impact, the biggest difference, move the needle the most and see what it comes back with.
[00:00:49] Mike Allton: Mike's got The Hat, the AI kind. Unlocking power in business, you'll find. Cut through the hype, explore the tech. Every week, new ideas to check.
[00:01:14] Allton leads us on a digital quest. AI secrets, he'll always invest. Tune in tight, don't miss a beat. Knowledge, groove, keep you on your feet. Listen up! To the AI flow. Business growth, here we go. Cuttin' edge insights, that's the plan. With The AI Hat you'll understand.
[00:01:58] Greetings program. Welcome back to The AI Hat Podcast, where I selfishly use this time to pick the brains of experts at harnessing the power of AI across all areas of business. From streamlining operations and boosting productivity to getting a competitive edge and driving explosive growth. Oh, and you get to learn, too.
[00:02:15] Subscribe to be shown how prepare yourself and your brand for this AI revolution. Come on ahead now. Are you tired of running endless AB tests with minimal insights? Do you find yourself drowning in data but thirsting for actionable results? If you're nodding your head, you're not alone In this fast-paced world of digital marketing, AB testing has long been a go-to strategy for optimizing campaigns and improving conversions.
[00:02:38] But let's face it, traditionally, AB testing can be time consuming, resource intensive, and sometimes frankly, hit or miss. But what if there was a way to supercharge your AB testing, a method that could not only speed up the process, but also provide deeper, more accurate insights while buckle up? Because that's exactly what we're diving into today.
[00:02:57] And who better to guide us through this AI driven revolution than our guest today? Nicole Leffer. Nicole is a seasoned marketing leader and a widely recognized expert in deploying generative AI technology to improve marketing efficiency, quality, and outcomes. With over three years of hands-on experience using AI tools and strategies, Nicole has been at the forefront of integrating AI into various marketing domains, including performance and growth marketing.
[00:03:21] Nicole has worked with numerous companies to implement AI solutions that boost efficiency and drive results, and our expertise and leveraging AI for AB testing and optimization is unparalleled. And she's here to share insights on how we can all test smarter, not harder. Hey, Nicole, welcome to the show.
[00:03:37] Nicole Leffer: Hey Mike, it's so great to be here. Thank you for having me.
[00:03:41] Mike Allton: My pleasure. I'm so excited to talk to you. I wonder if you could start by just giving us kind of a, a quick overview of traditional AB testing. I know a lot of the marketers listening have done it and a lot of marketers know what it is, but they haven't done it for whatever reason, and things have changed. There's limitations there. What do you think?
[00:04:02] Nicole Leffer: So let's just go with like the simple definition of what I'm talking about when I talk about AB testing, and that is you have an A and a B. You could have a c de and F version of an asset and you put them head to head with each other. So you put that, maybe that's ad content or a landing page on a website.
[00:04:22] You have them, and it is randomly determining which of your audience is going to see each version of that asset, and you are measuring for a specific outcome. So usually that is some type of a conversion when it's ad copy or ad creative, that could be a click through rate. AB testing could be measuring for different things, but you're comparing two or more versions head to head until you get to a statistically significant point where you can see that this version performs better statistically.
[00:04:54] So more people are more likely to react when they see that version of whatever you're giving them than if they see the other version and then you rinse some repeat. So you take, you take that winner. And you put something that is basically designed to try to beat the winner and you keep testing until eventually your content or your asset, whatever it is, converts or generates that behavior at the best possible level.
[00:05:20] The thing that's amazing about AB testing is it could go on forever because you can always try to beat that winner.
[00:05:26] Mike Allton: Right. Yeah. As a, as a long time organic content marketer my familiarity with AB testing was mostly in the realm of email marketing where we would ab split test subject lines. Right.
[00:05:38] And, uh, the beauty there was the tools would have to have that built in for starters, but Right. They would determine how many and what segment of your audience should change the, to send the A subject versus the B subject and then they'd monitor that. And at a certain point, the tool, if it's working well and if it's a good test, would say, you know what, B is performing better.
[00:06:01] Let's send B to the rest of our audience. And Right. That was nice. I didn't have to pay for any additional software and I didn't have to learn statistics because that was something I struggled with in school. That's why I shifted to history and not computer science, but I'm wondering how AI applies to this, how is this transforming this AB testing process?
[00:06:23] Nicole Leffer: So let's get this outta the way I wanna talk today about how the AI tools that you already use, like a Chat GBT or Google Gemini or Claude, could assist you with your AB testing rather than going down the rabbit hole of custom-built AB testing AI tools because there are tools that exist to help you do that.
[00:06:45] But I think that like a bazillion tools in your toolkit can get really overwhelming for marketers, and I wanna make sure people understand that they have tools that can assist them in this process. If you already have any kind of an LLM. To begin with that is like there to help you with this process that maybe isn't purpose built for AB testing.
[00:07:07] So let's just start there. Yes, tools exist that can like automate and do a lot of this a lot more intensely, but that's not what we're gonna talk about today. I want everybody to know what's already in their pocket without adding anything to their budget. So is that good for how we direct this conversation, Mike?
[00:07:21] Mike Allton: Absolutely. I love it. That is my exact philosophy. I want to use Gemini for everything if I can.
[00:07:27] Nicole Leffer: Yes. Awesome. Because the thing about this technology is you have so much that it can do, including your AB testing, so. I'm just gonna give you a few things off the top of my head and then we could dive in from there on some of these different areas.
[00:07:41] One of them is you can use it to help you write the variations, right? Like it can help you brainstorm and write variations of your assets, or even give you creative, like new creative, because these image generators are getting so much better. So it can help you actually create the AB test variations that you're gonna use.
[00:08:01] It can also look at what you're doing and help you identify what you should be testing, so it's not just about like giving you different variations of ad copy. Maybe you are using vision in your tool and having it do a deep analysis for conversion rate optimization, and suggest the most impactful AB tests that you can possibly run so that you're skipping a lot of that guessing, and it's going from its massive corpus of training data to help it understand how you might be able to move the needle the fastest. Um, there's, there's lots of different ways that you can go with that kind of thing where it's coming up with your strategic plans. And then the other thing you can do with these tools, depending on which one you're using, you can also give it some of your data.
[00:08:48] Now, I would make sure you're using this in a data safe environment. But you can also have it look at your data and help you look for the opportunities where maybe you're going to move the needle the best with your AB testing. And again, like AB testing is kind of broad, like you brought up. That could be, uh, subject lines on an email.
[00:09:06] It could also be the content of your email. It could also be ad copy, ad creative. It could also be landing pages. It could literally be pretty much anything. You could do direct mail and you could be ab testing your direct mail response too, so it's not even all digital. So what you're doing with AB testing is going to be very, very, very directive of what you would exactly want to do with the tools and how you'd wanna leverage them.
[00:09:35] Mike Allton: Got it. So I, I definitely wanna dive into how to use the tools in some of those examples that you mentioned in a moment. Yeah. But I think maybe we should start with, if we're going to use Chat GPT or Gemini yeah for this purpose, longtime listeners on the show should know that Jet GT doesn't come outta the box ready to do what you want it to do. You have to prompt it and, and train it and tell it. What, where would you start if you were working with, you know, a small business who hadn't been using one of these models yet for this purpose?
[00:10:03] Yeah. What, how would you start to prepare it?
[00:10:05] Nicole Leffer: So the first thing is I would make sure for something like this, that you're definitely in a paid account, like a teams or an enterprise account, despite depending on the size of your organization. So obviously a small business is probably not using Enterprise Chat GBT, because that's large numbers of licenses, but you can be using a team account and I would really recommend that because you are starting to get into a use case where you're putting some of your data and your results into the AI.
[00:10:32] But to start with, you wanna identify exactly what are you AB testing, so like at least bring that to the table. And I don't even mean like exactly what. Like the test is gonna be as much as like, are we testing Google ad copy? Are we testing landing page? Are we testing ab, you know, uh, subject lines?
[00:10:52] Like what is that test? And then you want to be giving your AI at the beginning of your prompt. You want to give it this skills that it needs to do a good job. So you are a conversion rate optimization specialist who that specializes in X, Y, Z type of businesses. Maybe it's e-commerce, e-commerce, product landing pages.
[00:11:14] And you have all of these skills and expertise to help you to generate the highest conversion rates via AB testing. And then you wanna be giving it all the context. Who is your customer? Who is the person coming to this landing page? Like, where are you? Where are they coming from? What do they have as their context to begin with?
[00:11:37] Did they see an ad before they got to that landing page? Then you wanna give the AI all of that information. Maybe you even wanna tell it what has converted, like what your best performing ad is in the past that you are wanting to test, or like your landing page that you use right now. What that looks like that you want to test against.
[00:11:57] And maybe you wanna give it a screenshot of that landing page and then ask the AI to use all of this expertise that it has to evaluate and analyze this page and tell you what is going to be the highest impact test that you can run. Because there's all kinds of things you could test on a landing page, right?
[00:12:17] Like there could be imagery, there could be button placement, there could be how many fields on a form there could be headline copy, like there's a million different things you don't necessarily wanna be getting into testing. Like all of those things simultaneously, like that's gonna get really complex.
[00:12:35] But you wanna give the AI all of the context that you can. And also a lot of this is gonna be directed by, are you B2B or are you B2C? Because you may have a lot more traffic if you're B2C. And so you also wanna be telling it, you know, how much traffic are you expecting? How fast can we get these results?
[00:12:54] And all of that context is going to use to help the AI to build you out a plan based off of what you have, of what you should be testing first. What is going to make the highest impact? Because our goal is to help identify what are the highest impact tests that we can run that are the most likely to move the needle for our business.
[00:13:16] Mike Allton: Okay. There's two huge things that I love so much about what you just said. The first, as you were outlining how to prompt the ai, you were essentially using the RICCE framework that I've talked about on this show previously. I dunno if that was conscious or not, but you gave the AI role not on framework, gave instructions.
[00:13:32] Yeah. Right. And this is what I tell people all the time, you know, use the frameworks if you're not familiar with how to really communicate with the AI, and they're like training wheels after a while, you don't need them anymore. You just know I need to give the AI all, all this information. But you were giving it a role.
[00:13:46] You were giving instructions, you were giving it context and constraints, and you were giving examples. And so for folks who haven't really done that kinda thing before, I'll link to the RICCE framework in, uh, the show notes below so that you can use that as a guide for building this prompt, but Nicole was just outlining it perfectly.
[00:14:02] You want to give the AI all that information upfront. So that it knows what it wants from you, how it's supposed to act and what it's supposed to look at. The second thing that I've really loved about what you said was a very subtle point, and that was actually gonna be my next question, which was that you don't have to as the user come up with what the test is going to even be. Yeah. You can give the AI at a high level, we're testing email or we're testing a landing page or whatever, and let it help guide you on what to test specifically. And I'd love if you could share a little bit more about how that might work, what some directions you might give to the AI to really help it hone in on what might be the most valuable things for a business to test in that scenario.
[00:14:48] Nicole Leffer: Yeah, so I think one of the things is, let's use this example of a landing page, because a landing page, there are like a lot of different things that could be tested. Yeah. What you wanna be telling the AI is like, what is your goal, right? Like your goal is to get the highest conversion lift of that landing page that you possibly can.
[00:15:07] You wanna tell it what that conversion is so it knows what it is, but then you wanna ask it, of the page you're looking at, I've given you a screenshot of this page. What is the like, you know, I want you to look at this, explain to me what you're seeing, and then tell me, based on all of this, what are the different things that could lift the conversion rate?
[00:15:28] And then analyze all of those things in the context of this specific page, in the context of your expertise. And tell me which of those things that, if I did it, do you think is going to make the biggest impact, the biggest difference, move the needle the most. And see what it comes back with, because it's gonna be very, very different.
[00:15:49] And there is no one answer, right? Like I can't tell you what the one thing you should do on your landing page is, 'cause I haven't seen your landing page, right? Yeah. So for some people listening, that thing might be social proof and the AI might look at this and go, given your persona and your landing page and all of the things that you have given me and all of this context and information, the number one thing that you could do to, I believe.
[00:16:15] I being the ai, I believe the number one thing that you could do is add social proof to your LAN page, but somebody else might already have really good social proof on their landing page. So the number one thing that for them might be to limit the number of form fields or the number of clicks that are required, or the color and call to action, the color of the call to action along with the call to action, right? Different pages are gonna have very different things, and that's one of the things that's so powerful about using a tool like Chat GPT or like Gemini to do this, especially with the like high level reasoning models.
[00:16:53] I would not be doing this with a non reasoning model. I would be using the smartest reasoning, A model that is available to you in the tool you're using to think through this and help identify what is gonna have the biggest lift for you.
[00:17:08] Mike Allton: I love that because you're, you're reflecting what a classically trained conversion optimization rate specialist would do, which is to have an hypothesis.
[00:17:17] I think if we change this element, whatever it is, on a landing page and an email, whatever, then we would get X or Y, and that's important to have. We don't wanna do this kind of stuff at random. You also made a point, you just kinda tossed this in, but it's so true. Only run one test. At a time don't test four different elements on the page.
[00:17:38] Nicole Leffer: Yeah, and I mean that for the vast majority of people, especially if you are not like a super expert in a b testing and conversion rate optimization and all of this stuff, now there are ways to go and start running these tests that are multi-variate and like way more complex. We are not talking. If you are that person, then this is not necessarily the conversation for you on that.
[00:17:59] Uh, but yeah, for the average marketer who is not doing a ton of AB testing already, that is not crazy with this. Stuff. Yes. Just pick one element and start with that. But then what you do is you can ab test that one element. Once you have the winner, you take the winner and you go to the next element, right?
[00:18:17] So your goal, it's not a one and done. AB testing should never be one and done. AB testing should be to the point where maybe you're starting out with big things, like does adding a testimonial on this page lift the conversion rate? Or adding, you know, social proof or changing my form fields? And you're gonna get to a point when you're doing AB testing where it might be that the AI is like, Hey, change that period to an exclamation point, or take the punctuation off completely.
[00:18:47] And to be honest, that could actually be the first. Thing it recommends. Like that could be the thing. And don't laugh at it. Like if Chad GPT or Gemini came back and they were like, you need to change that punctuation, believe it and test it and you know, don't go not test it. Right? Remember you are doing this to find out what to AB test.
[00:19:07] But I have seen the craziest things make dramatic differences. No. I should probably step back and tell people like my perspective of this. I actually grew up in marketing in e-commerce and both B2B and B2C e-commerce. I started a online chocolate ingredients company in the early two thousands, so quite a long time ago.
[00:19:32] And I learned from a guy who's one of my mentors, a man named Perry Marshall. I learned, uh, Google AdWords from him and grew my business on Google AdWords, and it was very big back then. It was AB test, AB test, AB test. So like I was like crazy, AB tester, everything. That is my framework. I have been AB testing since.
[00:19:53] The early two thousands, you know, so I've been doing this for a lot of years. That's kinda where my perspective is. So when I tell you I have seen crazy things make a difference in what a conversion rate is, I'm talking, I have seen really seemingly minor, you would never believe it things take something from being a 2% to an 8% conversion rate of how many people act in the way you want on a page.
[00:20:17] So if Chat GBT, or Gemini is telling you some really seemingly simple things now. Don't be begging it to ask, you know, don't be like, tell me what period is gonna get me from 2% to 8%. But if it's telling you that, like give it a chance to see if that is correct.
[00:20:35] Mike Allton: Folks, I hope you see why I brought Nicole on the show.
[00:20:38] 'cause she isn't someone who's just used AI a couple times to split test a subject line. She's been doing this for a long time. Yes. And she's got a wealth of traditional marketing experience that she's brought in and understand how. AI can help. So when she talks about, you know, a 6% increase when you've got a page that's getting a thousand or 10,000 or more visitors, that's huge improvement.
[00:21:00] That's serious money folks.
[00:21:02] Nicole Leffer: Even going from 1% to 1.5%. In your mind that might not seem like, oh, it's, what's the difference between 1% or 1.5%? Depends on your traffic, right? Like if you have very little traffic, it may not make a big difference.
[00:21:15] If you're getting a lot of traffic, a half a percent is gonna be a huge difference in how much revenue you're driving and how much of a difference. You know, that's 50% more so like realizing that like sometimes the numbers look little in this. The other thing that I wanna make sure, especially if anybody is newer to this, is you need to be looking at statistical significance.
[00:21:39] It is really easy to like go. You start a test, you see a result, it ha it seems to happen really quickly and you're like, oh, it's obvious. But if you do not wait until you have enough traffic and views and everything for this to be statistically significant. It's not actually valid. So you could even like have the conversation with, uh, the AI and being like, in this context, like, how much traffic do you think I need?
[00:22:04] Like, how do, if you don't know how to measure statistical significance, I'm not gonna teach you. Now I'm gonna tell you, tell cha GBT, you don't know how to measure statistical significance and ask it what you need to do to be making sure that the test, that you know the winner before you declare a winner.
[00:22:20] Because like now a lot of the stuff, you know. Email subject lines. It does that for you. It's built in. We did not use, it's funny, you're like, you know, you did it like you do it. The AB test on the automated, I remember when we split our list up manually and we would go, okay, this hundred people is getting this subject line.
[00:22:40] This hundred is getting this, and the rest are getting, and then we go, did it get to statistical significance or do we need to do another hundred? And then we would blast it on ourselves. If you don't know how to do it and like for whatever tools you're using, it's not automatic to tell you a winner.
[00:22:57] Definitely have that conversation with the AI because that's the amazing thing about this technology is it can coach you through as well.
[00:23:05] Mike Allton: I could not agree more. Folks. We're talking with Nicole Leffer about the many ways AI can help marketers, but test ideas and options. In a moment, we're gonna get into some of the challenges and the opportunities for small businesses.
[00:23:16] But first, a quick word from our sponsor.
[00:23:18] This episode of AI for Business Leaders, the AI Hat Podcast is brought to you by Magai tired of juggling multiple AI subscriptions and trying to figure out which tool to use and when? Magai simplifies AI for businesses by offering access to all the leading language models and image and video generators for one low price.
[00:23:39] With Magai, you can create custom instructions to personalize AI for every business need. And use them across all LLMs, including check, GPT, Gemini, Claude, and even Deep Seek. No more building custom GPTs or gems that are limited to just one platform. Magai also lets you create customized workspaces for your entire team, giving everyone in your organization access to the same AI assistance and tools and props ready to streamline your AI workflow and empower your team.
[00:24:08] Tap the link in the show notes to visit Magai today to learn more and now.
[00:24:16] So Nicole, I love the point you were just making that you know a lot of. The tools today have have split testing built in. 'cause you're right, we didn't, we didn't have that back in the day, you know, so you've got that probably with most, if not all email marketing solutions. You should have it with virtually, I think all of the, uh, like advertising platforms, they, they build that capability in where you can split test headlines, images, and so on. So you got that With ads, with social media, there is no such thing. All you can really do is share different posts at different times, I think really, and just test to see which one performed better.
[00:24:53] It's not a true split test, but it's the best we can do with the platforms. They don't have that capability built in, but websites. Just to help the folks listening who maybe have not done this before, my understanding is you'd have to have some kind of software installed. Like if we're at WordPress, you have to have a plugin that that does that.
[00:25:09] Is there one that you recommend?
[00:25:11] Nicole Leffer: I try to get, I keep away from specific plugins and stuff, just like recommendations, just because there's so much out there that is, you know, different and depending on your needs, on your website and type of company, it just gets too complicated to make specific recommendations.
[00:25:28] But like, there's tools like that are ab, like AB testing, conversion rate optimization testing tools too. And I'm, this is not necessarily a specific recommendation, but tools like Optimizely are like, have this kind of functionality built in for landing pages. But again, it just depends on exactly what you're doing, what your tools are.
[00:25:49] Like some website builders do have this built in and others you're gonna need external tools. So it's very, very different. And also. That's not, I'm really more on the AI side of things, like, you know, the ai, how to leverage that. So I have, I'm not up to date on like the latest conversion rate optimization tools as much as how can you use the AI to make those suggestions?
[00:26:12] Mike Allton: Yeah. I'm, I'm in the same boat. I've used Optimizely, uh, I was a CMO for, for a global SaaS company, but that was. Eight years ago. Yeah. I couldn't have told to optimize it until around, I have no idea.
[00:26:25] Nicole Leffer: Yeah, it was really great when I last used it. Don't, I don't always practice what I preach anymore. I used to be like very, very deep in doing all of this.
[00:26:33] Now I tell other people how to do it, but if you look at my own website, I'm like very, I have not conversion rate, optimized it at all. Don't judge that my website is thin, has been optimized in any way in that way.
[00:26:46] Mike Allton: Yeah, so I mean, I guess the SMB, you know, and, and marketing folks listening, turn to perplexity, turn to Gemini and say, Hey, I have a WordPress install and I want to, you know, start split testing the advertising, uh, or split testing, you know, my landing pages, what tools are out there, what do you recommending?
[00:27:03] Nicole Leffer: I, you know, if you don't wanna. Spend money on an extra tool. Like there are great tools out there that are really, really good for that. But you can also just duplicate the page. Like I'm gonna tell you how we used to do it in the early two thousands, and it's the easy war ground. You just make two versions of the page.
[00:27:18] You make it so that the second one is like not indexable, so it doesn't mess with your search rankings or anything like that. And you can direct in the ad. The ad is sending to two different versions. So usually in the ad platforms you can just give it different landing page URLs and make that part of the split test that's going on.
[00:27:36] So there are ways around it that you can do old school. I mean, we used to, when I got started with Google ads. We used to just have, you know, you'd have two of the exact same ad copy running and then you would be, or maybe we'd have four, 'cause we were testing for click through and landing page. And so you'd have two of every ad copy running at the same time and each one would have a different landing page.
[00:27:59] And so you could measure both of those things. But it was very manual. It was just, we made two versions of that page. And it doesn't always have to be complicated.
[00:28:09] Mike Allton: I love that. I hesitate to call it a hack. It's not really a hack, but I love that, that approach. Do you know? You know,
[00:28:16] Nicole Leffer: You know the funniest thing is that I feel like the fact that we used to have to hack how to do all of this stuff back in the early days of like e-commerce and the internet and digital marketing has been like the greatest asset in ai because it's like the same stuff you have to do with learning AI is like, you gotta just figure out how to get the tool to do what you want it to do, even if it's not made for that. So when I first started using AI in 2021, goodness, it was like almost four years ago at this point. The tool I was using was a writing tool. It was made for writing at the time, like I think Facebook ad copy, maybe Facebook post copy.
[00:28:54] We were using it on my team. It was like, okay, this is what it's made for, but I bet you we could hack this and we could get we could get this to write our email copy. We could write this to write our email subject once. And you know what? It works like you could totally do it. So sometimes you just gotta think outside the box and go.
[00:29:12] If you've got unlimited budget, then by all means go get the tools. To be able to do it with the unlimited budget. But if you are a small business and take this from somebody who has owned a business and has been responsible for that, there is always a way that you can do so many of these things. If you just get creative and go, what's the workaround?
[00:29:32] And if you have no budget and you have no idea, what's the workaround? Ash, Chad, GPT, what's the workaround? Like how, let's think through this creatively. I have no budget. This is what I do have. How can I do this in a way that will work? For me,
[00:29:47] Mike Allton: IL love that I am in that boat and I know many of the folks listening appreciate that tip as well.
[00:29:53] I'm wondering, Nicole, can you share any of the potential challenges or, or roadblocks that a business might run into a marketer when they're trying to use ai for split testing?
[00:30:04] Nicole Leffer: Yeah, so when you are doing split testing, if you don't have enough traffic, it can take a very long time or be very, very challenging to get to the point where you have statistical significance. So, oh, like, you know, it's really gonna depend if you get a lot of traffic, then. Lit testing is a really powerful thing.
[00:30:22] Um, if you don't get a lot of traffic, then it's maybe not the best use of your energy and time because you have to be able to get enough traffic to, for it to be statistically significant. I'm not saying like you shouldn't still try to go for it and like do some big, big tests, but just know it's not gonna go at the same speed and the same pace.
[00:30:44] The other piece of it is gonna be budget AB testing is a lot easier when you have money. And so a hurdle that a lot of marketers might run into is not having a budget for ads or like advertising is a very fast way to split test and get AB test results. So if you don't have that budget, then it can be a challenge.
[00:31:06] Mike Allton: Yeah, that makes a lot of sense. Everything's easier when you've got a lot of. Budget and it
[00:31:12] Nicole Leffer: doesn't necessarily have to be like huge budgets. Lemme give you guys, okay? I'm gonna give you guys a secret hack of like how you can do AB testing. I haven't done this. Like, this has nothing to do with hang ai this's.
[00:31:21] Just like now I'm getting, I'm putting on my total other like, old school geek hat. You, let's say you like ChatGPT is telling you that if you change this headline, you're gonna get a higher conversion rate, right? And it's like, but you're going, I'm not getting enough traffic to AB test this, like on that landing page.
[00:31:41] Like, but the traffic I get is really high value, so I want it to convert. And you wanna make a really educated guess whether this is true. That's our scenario, right? I wanna just make an educated guess. Should I actually just update that headline instead of running the AB test? It's getting a little harder 'cause maybe like your audience might not be on Facebook anymore, but if you run meta ads with the copy of the A, like you head-to-head, testing your audience with the headline, drive it to that page that you wanna drive people to get to statistical significance on the click-through rate on your app.
[00:32:17] On your ad image. So now I have seen very few times where it doesn't translate, right? Mm-hmm. So it's much easier to get a thousand people in your audience to view a Facebook ad than a thousand people to view your landing page. Those are two very, very different things. Also, way cheaper to get a thousand people to vote.
[00:32:40] View your Facebook ad or your meta ad, your Instagram ad, whatever. I'm sure you could do this with like LinkedIn too, but. A little more money to do it at LinkedIn, but if you could get a thousand of your target audience to see the variations that you wanna test and you get a statistically significant difference, and what your ad copy is, is that headline, you're not even really doing it as an ad.
[00:33:05] The fact that it is an ad is kind of irrelevant. You're just seeing what resonates more with your audience. Magic when it comes to AB testing.
[00:33:14] Mike Allton: I love that idea. I'm definitely gonna steal that. In fact, I, I just, I would add onto that, folks, if you're not limited to only a US audience mm-hmm. Then target an international audience with that ad.
[00:33:27] Target Canada. Target the uk. Target Australia. Yeah. You know, pick a, pick like another English speaking. Country, if, if that's your language that's not the US because the ad spend is gonna be dramatically cheaper to target.
[00:33:39] Nicole Leffer: Just make sure who you're targeting is your actual like audience because like, you don't what you don't wanna do.
[00:33:46] And it is a pitfall with AB testing. It doesn't matter, like if I am selling to 65-year-old women in the, like in the country, right? Like if that's my target audience. It does not matter if Mike clicks on my ad or like responds to it, right? Like Mike is irrelevant to what I'm actually selling. So you do wanna be careful like, 'cause you could do real cheap ads if you wanna send them to like India or Pakistan or the Philippines or something like that.
[00:34:15] Like you might be able to get real cheap ads. But if that's not remotely your target audience, like what those people do does not matter. And what Mike is gonna respond to and my, you know, 80 year olds. Woman in the, like in the country might not be the same thing in any way, shape, or form. So just do make sure if you are running those kind of tests, you are very hyper targeted on it being the correct audience.
[00:34:40] Otherwise you could go down some rabbit holes.
[00:34:43] Mike Allton: For sure. Terrific advice. I'm wondering. Nicole, if, if you were just looking ahead, do you see any emerging trends with AI powered AB testing and and optimization? How do you think this field might evolve in the next just year or so?
[00:35:01] Nicole Leffer: I think we're gonna have a lot, like there are already purpose-driven tools for this, but we are gonna have a lot more like data-based tools that are like really just kind of putting this on autopilot and doing it on autopilot. But, so I would say that's probably a big trend with this.
[00:35:17] I think that you're gonna see a lot more B2B that is doing AB testing, testing, and really taking advantage of this like conversion rate optimization with the tools like chatt, PT, like, you know, AB testing. And this has usually historically been a lot bigger in like B2C and E-commerce. And even though I did it on all my business pages, and I've always done it in a B2B context, that really was a factor of me coming out of this B2C and B2B E-commerce world, and it's just not as big in B2B. So I think that we are gonna see with it getting easier because of ai, because of the technology. Also AI just opening up more time in our lives in general to in our work in general, so that we're able to focus on more things we always should have done, but never got around to.
[00:36:09] I think you're gonna see more AB testing happening.
[00:36:13] Mike Allton: Absolutely. I've also heard a lot of folks talk about how, as we move over into more Agentic ai, you'll be able to have agents that, you know, instead of doing an AB test or even an a b, C test, they'll be able to test like a hundred different headlines at once.
[00:36:28] Yeah. Come up with all that copy and run it.
[00:36:31] Nicole Leffer: The coming up with and running it is great, but again, it goes back to your audience size. So like that's great. Like if you have a hundred different headlines, you can test like a hundred different, you know, like you can, but only if your audience is big enough to do that, right?
[00:36:48] You still have to have a statistically significant enough sized audience to be able to make it make sense. If you're selling, you know, a lot of my clients are like enterprise level. Like selling into the enterprise. They don't have the, they could never test a hundred different headlines because there's just nowhere near the kind of traffic that would be necessary to test a hundred different headlines.
[00:37:12] So keep in mind like how big is your traffic and does that make sense for you?
[00:37:18] Mike Allton: Terrific advice. Any other last piece of advice you'd like to share with the marketers listening?
[00:37:23] Nicole Leffer: I think don't overthink it and just make Chat GBT or Gemini or Claude a thought partner in how you can start doing this stuff.
[00:37:32] Learn how to communicate with it. Learn how to give it skills, expertise, and realize there is no one size fits all advice for this because every company, every scenario, every group of tools you have, like, it's all gonna be different. And so you've got to find the tools and the strategy and the the path forward that is ideal for you.
[00:37:52] Do not try to make yourself and your testing and your strategies fit a mold that isn't right for your company or for you. Make sure you're following the advice that works in your specific context. And that's what's so cool about this technology is it can give you that hyper tailored advice if you can give, if you take the time to give it all the context, it needs to be able to do that.
[00:38:14] Mike Allton: Your, your point about making AI a thought partner is so spot on for folks. If you've not done something quite like that, if this is gonna be part of your role going forward, definitely use the AI Work Buddy method that I've got. I'll, I'll link that below in the show notes. It's a, basically a questionnaire that the AI will ask you and then you'll, it'll use those answers to create a custom jam or a custom GPT that knows everything about your role.
[00:38:37] Responsibilities, including split testing and making sure that you're improving the optimization and conversion rates of your ads or your social copy or whatever it is you're running. Nicole, this has been such an insightful conversation. Thank you. Appreciate you. For folks who want to ask you more questions or reach out and connect, where should they go?
[00:38:57] Nicole Leffer: LinkedIn is the best place to actually like connect. I post tons of content on there, not at all. Like, I mean, some occasionally is about AB testing, but it's not all I talk about. So definitely connect on LinkedIn. Um, if you send me an actual connection request, please tell me where you came from so that I know you're not just trying to spam me and sell me something.
[00:39:18] And then, and I, I'll actually accept you if you do that and for anybody else also, if you like, need to get in touch with, you're interested in working together for your business or you're interested in my foundations of generative AI for B2B marketing course, my website is nicole leffer.com and I have all of that information there.
[00:39:39] Mike Allton: Brilliant. Thank you Nicole. Thank you, all of you for listening. We will, as always have all those links in the show notes below.
[00:39:44] Until next time, welcome to the grid.
[00:39:47] That's a wrap on another episode of The AI Hat Podcast. I hope you've gained some valuable insights and actionable strategies to boost your business with ai. If you enjoyed this show, please subscribe, leave a review, and share it with your network. And don't forget to visit the ai hat.com for more resources and to connect with me.
[00:40:05] Until next time, keep innovating and remember, the future of business is human and it's powered by AI.
