At my son’s birthday party some weeks ago I was talking to a few of the other parents while kids were playing, and, as it does these days, the subject of AI came up. Of the two dads I spoke to, both are using AI tools in their work. One is in advertising/marketing (“Mark”) and the other is in customer service (“Gus”). I thought their perspectives were quite interesting, and illustrate some of the different approaches to using these tools, so I thought I’d summarize them here, along with some more general thoughts.

Use 1: Advertising/Marketing

Mark said he’s used tools like Claude and ChatGPT to come up with sets of ideas for marketing proposals. In each case he reviews the output and makes corrections before finalizing the proposals. In some cases he has the model re-do proposals based on adjusted specifications, before re-evaluating the output.

He also has had the models produce code to run specific advertising/marketing campaigns. He said in many cases there are significant errors in the output which requires fixing before the code will run correctly. So this is a definite area for improvement.

A final use case is “vibe coding”, particularly in delegation. He said with interns he’s given them a specification for a task and then given them access to a model and asked them to produce a program that completes the task. These are interns with no coding experience, and after a week they produce a running program. It’s not perfect, but it’s a learning experience and usually works without much modification.

Use 2: Customer service

Gus said much of his work involves managing a team of (mostly younger) customer service associates. Their job primarily revolves around answering customer queries only after bots have done the triage. Increasingly, he says, over 70% of queries can be handled by a chatbot, and for the remainder his team is actively developing ways to answer and respond automatically. They use AI tools to develop these workflows, and agents (via Claude) to implement them.

His company offers these services to others and he said that most of what they do involves customizing for niche industries. They can develop a workflow for automating responses to customer queries for the majority of Qs that businesses receive (depending on the industry), but customization is the special sauce. He also highlighted that most of his Gen-Z team is perfectly comfortable using AI to assist with these tasks.

Assessing output

While both Mark and Gus were pretty impressed with the tools overall, they were also a bit cautious for two general reasons related to the outputs: quality and time.

Caution 1: Quality

The question of quality involves whether the tool is giving you what you want. Are you getting information that addresses your query (Gus)? Are you getting a proposal that meets your needs (Mark)? A recent study showed that workers want AI that assists them, not that completely does things for them. The difficulty is that the tasks we are now trying to automate are rather complex.

When you train a human to do a complex job, you spend some time with them, instructing them on the task, and once they’ve been trained the output is generally the same, or at least within an acceptable range. AI takes a lot longer to train, and the outputs can vary wildly. This means you need a clear set of criteria for assessing output, criteria which are often very specific to the context.

Caution 2: Time

This is brings us to the second caution regarding time. The reality is that assessing output takes time. Although an AI might produce a result relatively quickly, this depends on the model you’re using and the complexity of the task. You also have to factor in the amount of time it takes to assess the result, which also scales with the complexity of the task.

A key observation that Mark and Gus both made is that they “feel” more productive but they also feel more busy. Which begs the question: Are they actually more productive or are they just spending more time on the same tasks? They are doing less of the actual work (ideation, pointing/clicking) but more checking. Does this make us more productive or just more busy?

The future will be automated

These two anecdotes reflect a trend: the continuation of the process of automation. The early adopters of the cotton gin probably had a lot of challenges related to getting their machinery to produce the correct output (usable or non-seeded cotton) without breaking down. This reflects my own challenges with building AI models. I train something that works for (ideally) 90% of cases in a particular domain, and then have to write code (rules) to handle the other 10% of (often critical) edge cases. The result does seem to save time, but is not ideal, and in some contexts the model only works 20% of the time, which means it’s not worth it. I’ve written about some of my testing in a previous post, but suffice it to say that we should all do some cost-benefit analysis.

Even in those cases where an AI model works 90% of the time, it’s not necessarily clear how it works, and it’s a bit hard to replicate. For example, I might identify problems relevant to those 10% of remaining cases and develop/tailor a new dataset intended to address the problems. When I then train on the tailored dataset, sometimes the new model is actually less usable than the previous version, which leaves me scratching my head.

Despite these issues, however, I do think that AI is here to stay in one form or another. Especially as people get more familiar with how the technology works, what it is capable of and what it’s NOT capable of, it will become normal to use these tools to get things done. Right now I’d say we’re in the negotiation stage, with the opportunity, as with all technology, to decide what we want from it.