Scapegoat in the loop

As a software developer, I have spent the last three or so years reading, viewing, maybe even being bombarded with the advent of this new age of Artificial Intelligence and, to be kind to its proponents, its apparent values to all of us as technology. Now, I am no stranger to AI and ML technology or even the standard methodologies of creating and working with it. A year or so prior to the release of ChatGPT I even made sure to take time to read up and play around with training some small models as one does when they’re trying to improve their skills and stay aware of what tools are used in or around their trade. I, myself, was impressed with ChatGPT on release and felt like that period of study was worthwhile in that I could mostly understand what was enabling it to function.

In the time since, like many of you, I have seen the rise of “AI”… well, everything. Suddenly, one of the most difficult challenges regarding input processing in software seemed to be solved and everyone wanted to leverage that and more. Multimodality, fine-tuning, search algorithms, vector based search, a wide range of implicit “knowledge”, and other concepts have started to come along as well. Oh! And we can’t forget: vibe coding. I know, I know. This sounds like the intro to an anti-AI blog post and, to be quite honest with you: I detest using any of these new AI products for just about anything, but I will defer on outlining why in order to talk about something a little more insidious, something a little more relatable to all of us. I’m not talking about the idea that AI will take our jobs (I doubt it) nor am I talking about the high costs of running them (far higher than the value they provide, IMO). Instead, I want to talk to you about the idea of a “human in the loop” and how it might fuck workers, software developer or otherwise, in both plain and severe ways.

What is a human in the loop? Allow me to explain by example. Often, in software, we make plans and designs, then we try to identify risks or trade-offs in order to either account for them (if we can) in the design or revisit other options for the design (if we can’t). One such type of conversation can go as follows:

Person 1 (P1): We have a technology that can generate a ton of code for us. The more processes we run, the more code it can generate. All we need to do is provide it instructions and context.

Person 2 (P2): Ok. But it can’t be 100% accurate, unfortunately. We’ve run tests and we can see that plenty of the code is inaccurate even with vast amounts of context, instructions, and clear prompts. How do we deal with that?

P1: Hm. There’s really no way to make this technology more accurate in its current state. We’d have to revisit the entire basis of it. That could take years of further research and effort.

P2: We don’t have that kind of money or time. There is demand for this technology as it is today. Is there any compromise we can make or some way to support it through human effort? (ed. note: this is the classic “throw bodies at the problem” solution)

P1: Well, we need someone to start the process, so it should be easy for them to check the results and correct any minor errors that come up. We could include a static note that always reminds people this stuff is not perfect and to check the responses they get.

P2: That’s true, we can say that these products are intended for use by experts to help them do their work. We’ve also been hearing that people don’t like the idea of having their jobs taken over or not having input on their own “work”.

P1: Yes, we can keep them “in the loop”! Then the output is accurate and there’s always someone who can fix it if it’s wrong. Two birds with one stone, as they say.

So, everything is solved, right? Humanity created the generative AI machine. We as users will always check its work and then we can all benefit from the fact that it can do the bulk of the “work”. Except that’s not quite the end of the story. We all need to work to some degree, to have income, to shop for groceries. Most of us will have a boss or a manager and teams of people we work with. Life goes on and the world continues to function, if a little differently. So what can we imagine happens in that world if we expect to function as above, using AI tools to generate… stuff (code, marketing, emails, images, and more), verifying or modifying it as needed, and then submitting it as our contributions to those jobs, those careers, those teams and products?

Before answering that explicitly, let’s look at another situation first: autonomous driving. Tesla is in the news again, as it is from time to time, due to reporting on the high accident rate of their vehicles even though they come with advanced “self-driving” features they proclaim to be safe. Now, granted, their cars are not the most dangerous and there are other vehicles higher ranked or surrounding them on this list. The danger of cars in general is not in question. Driving is still, unfortunately, often very dangerous. The reason this is news is because of the juxtaposition of claims regarding how safe these vehicles are supposed to be as compared to the actual results found in real world data.

Tesla, of course, is not the only company attempting to produce and improve some version of self-driving technology. Another example is Waymo, owned by Alphabet (basically Google). They operate fully autonomous, but often supervised, taxis in California.

Consider this: self-driving is a similar product to ChatGPT and its competitors, based in AI and rife with inaccuracies and need for feedback, adjustment, retraining, and the input of more curated data for it to infer responses from. The more input, the better. The more time spent researching and developing the product, the better the final product will be. The more shortcomings it has, the more we need to make up for it in some way whether it be caution or safeguards or… accountability.

Waymo and Tesla comparisons are nothing new, for sure. They are often compared due to Tesla’s decision to rely purely on camera technology and to forgo additional points of input such as lidar and other sensors which self-driving companies with much safer results, like Waymo, choose to use. That decision to forgo additional points of input, to shortcut either via time or via cost, needs to be made up. How do they account for it? At every turn (pun not intended) Tesla will be more than willing to remind us that drivers are required to keep their hands at 10 and 2, or whatever specific verbiage they use, to pass off the blame in case anything happens.

They require a human in the loop.

They require… someone else to blame.

And that is what I want you to consider. This “human in the loop” idea is a way to find accountability for when a computer system can’t be blamed, per se, for an inaccuracy, an accident, or a death. Except, we missed something earlier in that example scenario. The premise I sold you wasn’t complete. There was one consideration that was skipped and is often skipped because it has to do with culpability of the company for which the participants in that conversation work. Another option would be to release a product that is inaccurate and allow the blame to fall on the company.

This has never been an option, however, because how could any company ever survive by releasing an often ineffective product, for which they are responsible, that can do harm to their customers? They can’t. It’s the antithesis of a company, in fact. This is why products of any industry have thorough testing, thorough feedback sessions with potential users, regulations around safety, recalls, and so, so much more in place. Unless… the promise of that product is so high that it would seem nearly irresistible to release especially if it is new, generally untested, and there has been little proof of damage or negative impact in the world yet (a common logical paradox for new products and markets).

So with the advent of LLMs and generative AI, businesses around the world have a product that they would love to make money on, that they know is risky, they know is often inaccurate, but works just often enough that they think they can sell it for a profit (laughable currently if you look at most numbers, honestly). But they can’t release that product and do as companies have always done: stand behind it and take the blame for inaccuracies, failures, and damages. Instead, they need someone or something to be accountable. They need someone to blame. They need a scapegoat.

They need: you and me.

If I, a professional software developer, submit code to my employer’s product which introduced a bug because a deterministic linter process updated a small bit of syntax and it functioned a little differently: we blame the linter. We’d probably say, “there’s a bug in the linter, file a report with reproduction steps and get it fixed.” I’m free of blame or accountability. I still take responsibility and get it fixed, along with the code/product it affected, but no one judges me or my skills.

Conversely, if I, the same software professional, submit AI generated code to my employer’s product which introduces a bug because the AI’s inherent probability based inference and generation produced something stupid and not real: I will be blamed. After all, it’s my responsibility to check the code I am submitting! There’s no bug in the AI model. We’d say, “that’s how these models work! They’re sometimes inaccurate! You need to check it and not trust it!” Even if the issue was a simple flipping of a boolean from false to true. Even if it’s something hard to spot to a trained eye. I still take responsibility to fix the issue in the product, but I will go home considering how much my reputation has taken a hit. Very likely, my team members will ask why I didn’t catch the issue myself. Or maybe worse: my team members will wonder about their own abilities to spot issues if they reviewed the code themselves.

If AI models for coding were considered to be purely experimental or something we didn’t necessarily pay for, then these issues would fall by the wayside. We’d consider these tools to be more promotional or kind of like a trial or a toy even. Similar to less feature rich IDEs and text editors: good to use sometimes, but not reliable enough for the real work. Except they’re already involved in the real work.

They’re involved in our work. Our life. Our safety. And they’re not good enough products to stand on their own. Companies are not willing to stop and do the difficult task of making them as safe as the average baby stroller, or the average deli product, or the average consumer electronic device. Rather than doing the hard work of making products that are safe, companies are jumping at potential, and so far unseen, profits while trying to find a way to avoid the responsibility of managing the harm they are also actually providing through those incomplete products.

Instead of doing the hard work, the companies are shifting the blame to us, the consumer. Similar to the classic (and cowardly) Surgeon’s General Warning on cigarettes. As a reminder, that warning states:

SURGEON GENERAL’S WARNING: Smoking Causes Lung Cancer, Heart Disease, Emphysema, and May Complicate Pregnancy.

In other words… use at your own risk because when you die early: you can’t blame the company who sold you the product.

At least, that’s what they want you to believe. They want you to blame yourself, or your coworker, or your boss, or your friend, or anyone except the companies creating these products which encourage people towards suicide or towards mental instability. They want you to blame the driver of a vehicle, rather than the car that is supposed to be able to drive more safely than any human. They want you to blame your peers that reviewed your code or other developers who aren’t “good enough” at using these tools. They want to make us all out to be their scapegoats.

So, the next time you use one of these products, or you see a recurring charge for your monthly subscription to them, I want you to ask yourself: are you willing to take the blame for these companies when they fuck up your work, your life, or that of someone close to you?

Because I’m not.