Ep 106.- A/B Test PITFALLS! and CRINGE Stuff! Ask Us ANYTHING

Welcome to do this not that the podcast for marketers. You'll walk away from each episode with actionable tips you can test immediately.

You'll hear from the best minds in marketing who will share tactics, quick wins, and pitfalls to avoid. Also, dig into life, pop culture, and the chaos that is our everyday. I'm Jay Schwedelson. Let's do this, not that.

We are here for Ask us anything from do this not that. This is a short episode where all week long we get in work questions and we get in ridiculous questions and we try to answer one of each.

And if you want to submit a question, it's real easy and you just go to jschweddelson.com podcast and there's a big button that says ask us anything. And you could submit any question. It may be totally off the charts. Doesn't matter. We will try to tackle it here.

So first, let's get into the work question. We got a question in from Dallas from St. Louis. Well, that's, that's totally confusing.

Dallas, it's got to be rough to have your name be the name of a city because it's like, is that where you're from? Is that your name? Like, if your name is like what, like Austin or Savannah, is that annoying? I don't know. I feel like that might be annoying.

But anyway, Dallas, what's up? We are working on doing more AB tests for our marketing programs. What are some pitfalls to avoid when setting up these A B tests?

A B tests are everything. So first off, if you don't know what an A B test is, they're also called split testing.

But basically an A B test is comparing two versions of something to figure out the better performing variation. Right. So if you're going to send out an email, two different subject lines, you have an A subject line and a B subject line.

You want to know which one's going to get more people to open it up so that way you can then use that moving forward. That is an A B test. So what are some pitfalls? And there are a lot of pitfalls to avoid when setting up an A B test.

The first one that I would think about is, and marketers make this mistake, you want to make sure that your A B test, that the two versions are different enough. Sometimes when you set up your A B test and let's say it was a subject line test, for that matter, right.

And you change, you know, one word in the subject line, that's not really enough.

You want to make sure that the Tests are different enough, so that way the results of the test are going to result in a very different outcome and then you'll have something that you can then move forward with. I get us all the time.

Since I do talk a lot about email, I get asked all the time, what is the minimum quantity that you should do a split test, a 50, 50 test on.

If you're doing an email of some kind with 50% getting version A, 50, getting version B, I like to have a minimum of 5,000 names in total, no less than 2,500 names in each group when you do your test, because otherwise you're going to get statistically irrelevant information. Some other pitfalls I would think about is know what you're going to do with the outcome of the test before you run the test so many times.

We'll run an AB test, okay, and then you'll have a winner. Doesn't matter. Not just email. Could be a social media post, it could be a direct mail program. It could be anything. Okay?

You run this AB test and then you actually have a winner. Okay, One did a lot better than the other. And then you sit back like, well, now what do we do? No, that can't be the plan.

You need to have a plan that says, okay, once we establish the winner, we're going to do this with the winner. So think about knowing what, what the outcome may or may not be and then saying what you're going to do next.

I think the number one pitfall when anybody does a B testing is not leaving rollout potential. So let's say you were doing an email campaign. Okay, So I want to do this a B test.

See what does the best right for these two different versions of the email you're going to send out, and you go through your database and you try to come up with a segment of your database where you're going to run this test. Say, Great, I got 11,000 names from this test. We're going to run 5,500 names here and 5,500 names there. See which creative does better. Okay, great.

You send it out and version A crushes it. Here's the problem. You have nobody else to send it out to.

In order for you to roll this out, you have to send that creative that just won the AB test to a new population of people. And I don't care if it's similar.

If it's not being pulled from the same overall population of people, then it's not really utilizing the data from your a B test. So when you're setting up your test, you want to say to yourself, okay, is the population of people that we are testing just a small subset?

So that way if we do have a winner, we can then roll out. There is nothing worse than running a test, having a clear winner, and then having no way to scale that test.

That is an epic waste of time and an epic waste of money, in my opinion. Okay, before we get into the ridiculous question, I want to let you know that this podcast is sponsored by Marigold.

Marigold is my email sending platform. They are amazing. I send out billions of emails and they have a new piece of content that you got to download.

It is called the seven Tips to Reduce Spam Complaints. That's a big deal. That's important.

And part of it, it says, how can you determine which contact lists you need to sunset which ones you need to put off to the side and archive?

If you want to get this free piece of content, the seven Tips to Reduce Spam Complaints, which is a great piece of content, all you got to do is go to jschweddelson.com Marigold that's my full name, jschwettleson.com Marigold and you could download the seven tips to reduce Spam complaints. Marigold is awesome. Check them out. All right, let's get into the ridiculous question. We got a question from Sarah from Springfield, Illinois.

You know what goes. I bet you don't know this. Springfield, Illinois, I know this. They have a food that you eat when you go there.

It's like, you know, you go to New Orleans, you eat the, the beignet or whatever and you go to Chicago, you get deep dish. Well, Springfield, Illinois, I know this. They have something called the horseshoe sandwich. I wonder if Sarah knows that. It's actually disgusting.

In my opinion. It's this like open faced sandwich that is like got meat and cheese sauce and french fries. It sounds good, but it's really gross.

But whatever, that's what they got going on there. So what's the question, Sarah? She says on a recent episode of your podcast, you said the phrase low key.

Do you use cringy terms like that in your regular life? Come on now, that's not nice. Well, sort of, I guess. So I've been banned.

So I have teenage kids and anytime I say anything that's cringy, they tell me that, that I'm a loser and I need to stop talking. And I try to step to date with certain words, but then I always feel like whenever I use them, I'm like six months to a year behind the time.

So, like. Like. Like, what do. What are words right now I'm not allowed to use? I'll give you examples. So the other day I said something to my son.

I go, oh, that's fire. Which, you know it means. Like, you know, that's good. He's like, you. Nobody says fire anymore. That's not. You don't say fire anymore. And.

And then I said, I was like, where do you want to order from your order on Uber Eats? I'm like, why don't we get from this pizza? He says, why don't we get from this pizza place? I go, bet. Which means, like, I agree. Like, let's go.

And he goes, you cannot. You. You can't say bet. You're. You're an old loser. So I don't say bet. I don't say slay. I don't say no cap.

I tried saying, this slap's different to my daughter the other day, and she was like, looked at me like, you, you're such a tool. I don't even know if I can say tool. Who knows?

You know what one term is I don't know how to use, and I know that this sounds really stupid, but I don't know how to use it. When people say that they're gaslighting somebody, I always get confused on what that actually means. Like, oh, you're gaslighting me.

I'm like, I know it means like, I turned it around or you turned it around or somebody turned it around. I. I never know how to use that phrase properly. I know. That's like the dumbest thing you've ever heard. Anyway, I shouldn't be allowed to talk.

It's kind of ironic that I have a podcast because I am totally not someone that should be saying words verbally. It's unbelievable. Anyway, thanks for being here. Leave this thing Review. Register for the Free Guru Conference.

Gurucomics.com we're going to run out of virtual seats. Our email marketing conference. 20,000 people. What's wrong with you? Go register. You're awesome. Leave a review and don't say no cap or whatever later.

You did it. You made it to the end. Nice. But the party's not over.

Subscribe to make sure you get the latest episode each week for more actionable tips and a little chaos from today's top marketers. And hook us up with a five star review. If this wasn't the worst podcast of all time.

Lastly, if you want access to the best virtual marketing events that are also 100% free, visit GuruVents.com so you can hear from the world's top marketers like Daymond John, Martha Stewart and me, Guru Events.com check it out.