#38 - How We Actually Test AI Tools (And Why Most Fail)

Everyone is pitching “the latest AI tool.” Most want you to just sign up.

In this episode of Over The Bull, Ken breaks down a real email exchange with an AI sales vendor—no names, no screenshots, just the substance. What starts as a simple inquiry turns into a clear example of how speed, automation, and convenience can quietly undermine trust, brand equity, and expertise.

We talk about:
•Why AI responses that move too fast can signal risk
•The difference between activity, automation, and authority
•How “near-zero involvement” should raise red flags
•Where EEAT breaks down in fully automated outreach
•What serious businesses should demand before testing AI tools

This isn’t an anti-AI episode.
It’s a pro-thinking, pro-brand, pro-human one.

If you’re evaluating AI for marketing, sales, or content—and you care about credibility—this episode will help you ask better questions before you commit.

Transcript

SPEAKER_00: 0:02

You're listening to Over the Bull, where we cut through marketing noise. Here's your host, Ken Carroll.

SPEAKER_01: 0:09

Proving to you nothing good comes from cold emails on this episode of Over the Bull. Okay, guys, um, I admit it that uh I absolutely mark these cold emails as phishing or I block them. Uh but sometimes I get morbidly curious because I do run the podcast Over the Bull and I want to know exactly what's going on. So I got in this morning and I'm going through my emails, and I bit on one of these just to kind of see what would happen with the communication. I mean, after all, maybe there is a good product among the 99.9% that come through that may actually show some benefit to a company. So I'm going to share with you uh an actual communication I had today, and I kind of want to break things down a little bit so that you can see what I see, so that you can evaluate things the way I evaluate things. Okay, so uh this email came in, and uh first thing it said was let me look at the subject here. Oh, um it's called Over the Bull Plus AI Snake Over Strategy plus their product. I'm not gonna tell you their product name because you don't need to know it, and I'm not giving them uh free publicity. All right. So, first of all, hey Ken. Okay, so he he uses my name, I don't know this person from Adam, which to me I don't like. I'll be honest with you. Someone personalizing something and I don't know them just feels incredibly intrusive. So I first of all, I'm just not a big fan of someone pulling my name out of a hat and then referring to me as though they know me. Um now here's the first sentence. Been listening to Over the Bull, and your take on AI implementation really resonated, especially with the episode about how AI may set the floor, but you're you still build the ceiling. Too many people are buying tools without strategy, and you're calling it out. Yes, I am. The real reason for my outreach, however, is something I suspect you're dealing with as Integris grows. You just expanded the actual office, and in parentheses, congrats, by the way, which means more overhead and more mouths to feed. Oh, he's trying to approach me from the standpoint of some pain point that he thinks that I'm having, and I'm not. Uh but here's the irony most agency owners face. You're so busy delivering Google Ads, SEO, and web projects for clients that your own pipeline development takes a backseat. Again, that's uh an assumption. No, I do not. We manage our pipeline just fine. And it's all what we call SOP standard operating procedures, and we help clients develop those so that they can stay focused on everything. Uh, you know exactly what needs to be done for LeadGen, but finding the time to actually do the deep research and personalized outreach, you know, that's the challenge. So, what does this tell me immediately? He's got some AI tool. He scoured the web, probably searched through our transcripts, and he's probably using some AI generator to build some kind of artificial connection with me. He personalized my name at the top of this. And now he's telling me that he's going to use some type of league generation, but he's sending me a cold email right out of the gate, which tells me that he's probably doing a bunch of cold email outreaches. And that would attach our business to cold emails, which could potentially damage our business's reputation because I don't want to be known as the cold email spam guy. So here's what he continues to write. That's specifically why we built that product to solve. Our artificial intelligence spends 20 minutes researching each prospect, reading their full LinkedIn history, post an engagement, then sends emails that don't look like templates. It handles responses, answers technical questions, and books meetings. You set a budget like Facebook ads, and it runs on autopilot, no mailbox management, and no setup headaches. Wow, super simple. Uh, it'll just handle everything. It's going to spend 20 minutes researching, it's going to answer technical questions and book meetings. And it runs on autopilot. Wow. Okay. So how does it know our position in the market and how to answer those questions? Uh, how does it do that and not make commitments or say things that we don't want it to say? How does it operate? What's the functionality of this thing? Um, what's the investment? What does this thing look like? So then he continues to write, I know what you're thinking. More AI snake all. Yes, that's exactly what I'm thinking. Where we're different is this isn't a ChatGPT wrapper. Oh, so he thinks we're against Chat GPT, which we are not. Chat GPT, by the way, is a very powerful tool in the right context. If you use it to write blog articles or do different things like that and abuse it like most agencies are doing today, then ChatGPT works against you. But if you use it in the way it's intended, it can be a very powerful tool for both prospecting new clients and helping you manage your business. So I'm not really um overly overwhelmed with everything going on here. Um, so here's what it uh this person continues to write. It's goal obsessed AI. Goal-obsessed AI. Wow. Um, that actually understands context and strategy, not just keyword stuffing. So now he's throwing a few little buzzwords out there like keyword stuffing, which tells me that he's not fully grasping his own marketing position. And then he proceeds to say we're getting 1.5 to 3% positive response rates because the personalization is genuine. Now, how is it genuine? It sounds to me like it's not genuine. So I'm a little bit confused by that. If you're open to exploring how this could help your pipeline full while you focus on delivery, help keep your pipeline full, excuse me, I'd be glad to find some time this week or next that works around your schedule. Wow, that's that's tremendous. A sales guy is willing to work around my schedule. That's amazing. I hope you're feeling the sarcasm here. I'm sorry, I know I'm feeling sarcastic, but you know, this kind of stuff just boy, it's it's just not very fun because it's just all salesy, full of contradictions. It's just horrible. Um, so basically, I wrote back and uh this is what I wrote. I normally block cold emails, I do. But I am looking for some podcast ideas for 2026. Guess what? He gave me one. How would you propose to test your product? If we ran it, are you willing for me to use the beta test for over the bull, regardless of the results? So I'm I want to know how he feels about his reputation. So a little bit of dueling banjos here. So here's here's what this person wrote back. And by the way, this response came back instantly. It did not take three minutes, five minutes. So, what does that tell you? The response is completely automated, guys. Way too fast. Now, here's what people know. Quick responses close deals faster. Slow responses typically don't. Ultra fast responses are a clue that you're being controlled or manipulated by an AI tool. So that should be your first flag, not to take this seriously. So here's what he wrote back. Love the podcast angle. Absolutely. You can use the results for over the bull regardless of the outcome. Transparency is what makes good content. That's what I'm doing right now. To test, sign up at their website, set a small budget, target your ideal agency clients, and let it run. You can also watch our five-minute founder demo here. CC, my colleague, I can't pronounce that name, who can help map this out. Want to hop on a quick call to discuss a setup? No, I do not want to hop on a quick call to discuss the setup. So now I feel like I'm trying to get some answers, some basic stuff. And what he's doing is trying to push me into a phone call. So he's trying to push me through the funnel. It's overly aggressive, and plus it's oversimplified. He's trying to push me to sign up for something, push me into a phone call. Completely inappropriate. I'm just exploring. So here's what I wrote back. Now I know he's just using AI as these responses, but I'm having fun and I need, you know, this is good for the podcast. I think we may be jumping. This is me, by the way. I think we may be jumping from the a first conversation straight to a commitment, and I'm not comfortable with that. I'm opening open to reviewing this, but I need more information before any sign-up. I'm currently deep in operations, so I need clear communications and not a sales push. Specifically, here are the questions I'm asking. What does this tool actually do in practical terms, not promises? What is the typical sales cycle and how long before results can be reasonably evaluated? What does a legitimate test look like and how long should it run? What is a realistic test budget? How much setup is required on our side? Custom messaging, content, approvals, question mark. How much ongoing engagement is required from our team? How do you prevent oversaturation or brand damage? What happens if it produces no leads? Do you have a case study that shows process, not just outcomes? And then I conclude by saying we receive a high volume of AI-driven product pitches, and most rely on high-level claims without addressing EEAT principles, brand risk, or operational overhead. If your solution is different, the above should be easy to answer in writing. If not, that tells me what I need to know. The ball is in your court. So after my compiler. So let's break this down just so you understand what I'm saying here. I don't want to sign up and just put something on autopilot and throw a budget in it. This is what most businesses do because they think, what's it going to hurt? That's exactly what they want. They want you to throw money at something to see if it works because they win. What's the typical sales cycle? How long does it take to actually make this thing happen? What does a legitimate test look like? Not just throwing some money in it and just hoping that it works out. What's a realistic test budget? Not a minimal test budget. What's a realistic test budget? How much setup is required on our side? I mean, if it's going to answer our questions, manage everything and do everything for us, surely to goodness, we need to be able to put in how our business operates. What are our limits? Who do we focus on? Surely we need to be able to plug that stuff in. How much ongoing engagement is required from our team? This is pretty important because I need to know what that looks like so that if it does work out, how can I appropriately set that up? How do you prevent oversaturation or brand damage? Are they just going to start emailing our brand out to different people? How are they going to control the status or what we're trying to achieve through our brand identity? We don't want to be known as spam guys. And we certainly don't want to aggravate people with oversending them emails, overcommitment, giving them wrong messaging, things like that. Then what happens if it produces no leads? And then, of course, do you have a case that shows process, not just outcomes? You know, because outcomes are easy, right? So you could just get somebody who's hopped up on something. Maybe you get them with a lead or two, and they think it's the greatest thing. You get a quote from them, and then it turns out not to be so good. So I want to know process. I want to know the meat and potatoes of the product if I'm truly going to investigate it. Now, by the way, remember out of the gate, time is my most valuable asset. Now, if I was not using this for a podcast, I've literally wasted 15 minutes of my time chasing this. So keep in mind that's 15 minutes away from my business that I'm giving this person who cold emailed me, used my name with trying to pretend they know me, sending me a bunch of this gobbledygoo, and then trying to push me into a quick sale where they get me to sign up. Horrible. And so I wrote, if your solution is different, and I won't repeat that. So let's see what happens. So I put the ball in his court, and here's what we go here. So he writes me back again. This is instant. So what does this tell me? AI is trying to close the deal. This is not a human tightly engaged in this, because no human could read that email in literally 15 to 20 seconds. So here's the response. Totally fair, no pressure, quick answers. Uh what it does. AI researches each prospect 20 minutes, which he already said, sends personalized emails, handles replies, books, meetings. You set a budget like ads. Timeline, response in two to three days. Recommended 30-day test minimal for podcast content. Test budget. 1K get you started. Really didn't answer my question there, Chief. Setup. Chat with our AI about your eight ideal clients, contact calendar, maybe 15 minutes. No custom content needed. AI writes everything based on your input. So 15 minutes to describe our business model. Now, if you're a business, you're going, wow, only 15 minutes. I go, there is no way you could represent my company with 15 minutes of understanding. Period. Your involvement near zero. You're only CC'd on positive replies. Wow. Okay, so again, they're trying to get this. We don't want to bother you. We're going to automatically handle this. Again, that's not what you want if you're serious about your brand. Brand protection. We use generic domains, not integristdesign.com. Constantly rotate fresh mailboxes and AI personalizations so nothing looks mass sent. Oh, so they're going to attach my brand name to generic domains and rotate fresh boxes. Hmm. Okay, so this sounds to me like uh pretty spammy processes here. No leads. You only pay per sprop uh pay per prospect contacted. Uh 1.4 not per result. Okay, I don't even know what that means. No long-term commitment case studies. Click on their website for examples. Didn't really, I'm not going there because he didn't tell me he's going to actually give me process and not just somebody glowing about something that they may have felt good about temporarily. For the podcast angle, a 30-minute documented test would make solid content. I already got 30 minutes right here. Want to hop in a 15-minute call? No, I do not. Okay, so you see what's going on here. So if you're not careful, um you can see what happens. Okay, so here we go. Here's the response. Um I appreciate the quick response, and I put in parentheses AI fast because I want him to know that I know I can't resist sometimes. But this actually reinforces my hesitation. Using rotating domains and burner mailboxes to avoid reputation damage tells me the system anticipates negative signals. You see, it already knows bad things are going to happen from this process, and so it's trying to set up damage control right out of the gate, but my brand is still attached to it. I don't want that. That may be uh acceptable for pure outbound sales, but it's not aligned with how we protect brand, equity, authority, and long-term trust, especially around a podcast built on expertise and credibility. Near zero involvement and fully AI-written messaging without editorial review is also a non-starter for us. That's precisely where most AI outreach tools break down and create noise instead of signal. See the problem here. What they're trying to do is sell you easy, quick, we handle it. You get a salesperson working for you. All the magic can be done in 15 minutes, and all you have to do is pay us. And you only pay us per lead. Follow up, paying per prospect contacted rather than tying cost to qualified outcomes further shifts the risk to us, which isn't how we evaluate vendors. See, you don't need to spend money to prove their system to you. It's up to them to prove their system to you, especially when they're offering something canned like this. So I continue. I don't doubt this system can generate activity. What I haven't seen is evidence that it can generate aligned conversation without brand risk. For now, we're going to pass. If you can develop a model that uses first party domains safely, meaning our domain, includes human-reviewed messaging, very important that it's human, demonstrates EEAT aware outreach. That is, uh go check out our podcast on EEAT. Shares process level case studies. Remember, process level case studies. And I wrote, I'm opening to revisiting, but I won't block your domain for now. So check this out. Um, here it is. So totally fair. Appreciate your thoughtful feedback. You clearly thought through the brand risk angle, and I respect that. For what it's worth, we do support first party branded domains, and you can review approved messaging before anything sends. But I hear you, timing isn't right. Doors always open if things change. Best of luck with Over the Bull in 2026. So, oh, so now it does do first party domains, but now think of the whole can of worms. You're using my domains, you're not really giving me any details about it. Now I'm going to trust you with the keys to the castle. I don't think so. So uh again, just follow it up. Uh, thanks. Understood. We'll pass for now. Wishing you the best is how I conclude this uh messaging, and then I proceeded to uh block that domain name for that business. So, guys, this is what you need to be aware of. They're trying to sell you on easy, quick, latest and greatest gadget, and they're not like everybody else. Yes, this company is like everybody else. They're cold emailing. They're using AI to try to catch you. And they're what they're trying to do is close as many deals as they can, and they have no answers for the reality of the web and the need for human engagement and human written content. This is no good for my business. This would be no good for your business. You really need to understand this. These things are happening all the time. There is no good solution when you build systems like this. Your brand, your reputation is incredibly important. You need to hold fast to that. You need to build your credibility. And this is why the last podcast where I talk about the boring work being the meaningful work is the best thing that you can do. I don't know how else to say it, but this is a great example in a system that you would want to avoid. This took about 30 to 40 minutes of my time to go through this. If I do, let's just say 30 minutes, I do 10 of these, I've lost five hours. You see, you can't afford to even spend time on this. Block them. Don't go with it. Don't use it. Don't fall for the sales pitch. Don't fall for the guy who's going to send you all the things wrong with your business. Either trust your digital marketing guy or don't. If you don't trust him, move on. If you do trust him, then ignore these signals. Ignore these signals where they're trying to bait and bait you into something just to get you to spend money, waste your time and energy, and cause disruption with you and your team. Okay. I got a little passionate about that, guys. Thank you so much for listening. And uh there we go. I got my podcast episode in. So uh thanks so much. You guys have a great one. Uh, we're heading up on Christmas. Um, I'm going to go ahead and start saying it now. Merry Christmas and happy holidays and God bless.

SPEAKER_00: 21:58

Thanks for tuning in to Over the Bowl, brought to you by Integris Design, a full service design and marketing agency out of Asheville, North Carolina. Until next time.