What exactly is AI generating?

The technical term is "bullshit".

There’s been a flood output on social media on chatGPT — a machine learning¹ Large Language Model (LLM) from OpenAI designed to create natural sounding responses to queries almost as an interactive search engine or, say, the Enterprise D’s main computer. People have used it to produce mediocre punditry, mediocre business news interviews, or even code that you could likely get away with if you were giving a presentation to some business-types. It’s the kind of output you could use to try to talk about something you don’t really understand. The kind of output we usually refer to as “bullshit”.

I mean bullshit in the technical sense of Harry Frankfurt’s 1986 article [pdf] and subsequent 2005 book. AI in general and chatGPT in particular seem to have discovered an infinite font of bullshit. Why is it bullshit in the technical sense? Because the AI does not care about the truth value of what it produces — that which distinguishes bullshit from lying. As Frankfurt put it:

[The bullshitter’s] statement is grounded neither in a belief that it is true nor, as a lie must be, in a belief that it is not true. It is just this lack of connection to a concern with truth — this indifference to how things really are — that I regard as of the essence of bullshit.

An AI system only “cares” (i.e. has its objective function / hyperparameters tuned) such that the output it produces gives the researchers the feeling of a natural intelligent human response. To that end, the AI only cares about the truth inasmuch as it helps convince the researchers building it that it is producing human-like responses. Or, as Frankfurt put it:

His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says.

Why do people bullshit? Frankfurt again:

Bullshit is unavoidable whenever circumstances require someone to talk without knowing what he is talking about. Thus the production of bullshit is stimulated whenever a person’s obligations or opportunities to speak about some topic are more excessive than his knowledge of the facts that are relevant to that topic.

In a sense the entire endeavor of producing an LLM is forcing an AI’s neural network into the position of having to talk about subjects it does not actually know anything about. So it bullshits until it eventually persuades the researchers that it’s doing a good job. Per wikipedia:

The liar cares about the truth and attempts to hide it; the bullshitter doesn't care if what they say is true or false, but cares only whether the listener is persuaded.

The AI only “cares” (i.e. its parameters are tuned) about whether the researchers building it are convinced it is working, not whether it is producing true or false information (except inasmuch as that helps persuade the researchers).

It’s kind of wild from a science perspective — you are building a computer model whose sole purpose is to persuade (trick?) you into believing it is working. This is exactly the opposite of Feynman’s dictum:

The first principle is that you must not fool yourself and you are the easiest person to fool.

However this is also why I don’t think AI bullshit is an existential problem — humans have long bullshit each other and we’re often good at detecting it when we want to. In fact, reducing the cost of production may well drive the price of bullshit to zero reducing the return on the human-generated variety.

1. This is a couple of AI posts in a row, but this is in part because I think certain types of AI have a lot of formal similarities with markets and understanding one helps with understanding the other.