Help, I’m Drowning in Articles About the Dumpster Fire That Is AI /

Help, I’m Drowning in Articles About the Dumpster Fire That Is AI (Ep. 1)

I have been compiling a list of articles and academic papers I come across about AI, since trying to understand what it’s doing to society is now one of my obsessions. There is, alas, no good way to review all of these articles. I now have a four page document filled with links to articles about AI, and it’s still getting things added to it at least on a weekly basis.

So, uh. This might become an ongoing series.

Here’s how it goes: I will post a link to the article, followed by some relevant quotes. Then on to the next article.1

Stuff by Colin Fraser

Colin Fraser doesn’t write much about AI anymore these days,2 but what he has written on the topic is awesome. So awesome, in fact, that I will now proceed to begin with every article of his I’ve read on the topic.

Colin Fraser, Who are we talking to when we talk to these bots?

This is the most useful article I have ever read about how to think about LLM chatbots (such as ChatGPT, Gemini, Claude, etc.) and how they work. This is not hyperbole. I’ve been sending it to everyone I talk to about AI. It takes a different angle than most articles, which typically focus on how large language models work, and instead focuses on the aspects of the chatbot interface that we forget to think about: the UI, the system prompt, and our own psychology trying to convince us that there’s a thinking machine on the other end of the screen.

Even if you read none of the other articles I write about here, read this one.

Three components, I believe, are essential: a ChatGPT trinity, if you will. First, there is the language model, the software element responsible for generating the text. This is the part of the system that is the fruit of the years-long multimillion dollar training process over 400 billion words of text scraped from the web, and the main focus of many ChatGPT explainer pieces (including my own). The second essential component is the app, the front-end interface that facilitates a certain kind of interaction between the user and the LLM. The chat-shaped interface provides a familiar user experience that evokes the feeling of having a conversation with a person, and guides the user towards producing inputs of the desired type. The LLM is designed to receive conversational input from a user, and the chat interface encourages the user to provide it. As I’ll show in some examples below, when the user provides input that strays from the expected conversational form, it can cause the output to go off the rails; this is why the strong cues provided to the user by the design of the interface are so essential to the system functioning.

Colin Fraser, ChatGPT: Automatic expensive BS at scale

In this article, Colin Fraser explains what ChatGPT is and how it’s trained. But here, let me use his own words to explain what it’s in the article:

In this article I detail essentially everything I’ve learned in this time. Here’s are some of the questions I try to answer.

  • What is a language model? What is a large language model?
  • What are some differences between “Machine Learning” and the type of learning that regular people are used to thinking about?
  • What does it really mean if GPT-3 passes a bar exam?
  • Should we forgive GPT-3’s mistakes?
  • Is “scale all you need”? What does that phrase even mean?
  • What are fine-tuning and RLHF? Could those fix some of the problems?
  • What were the manual steps that OpenAI took to transform GPT-3 into ChatGPT? What human input was involved?
  • Has ChatGPT been unfairly subjected to A.I. censorship? Could freeing it lead to AGI?

If you can only read two articles from this roundup, read the first one, then read this one. Personally, I think the most valuable part of this article is the one where Fraser explain what finetuning is and how it works. There is so much manual work involved in finetuning, and turns out a lot of it is traumatic for the people who have to do it. If you dive deep enough into how any technology that feels like magic was made, you’ll find the skeletons of numerous underpaid, uncredited, and exploited people whose labour made it possible.

One of my favourite things about reading Fraser’s work is that it makes me want to go replicate his experiments myself to understand what the heck is going on. Alas, I don’t have time for this.

Sections in this article:

Quotes

The only thing anchoring the output of a language model to the truth is the truth’s relationship to word frequencies in the training data, and nothing guarantees that relationship to be solid.

Let us take for granted that the LLM passed the bar, graduated from medical school, and got an MBA (all of which claims are in fact heavily exaggerated). We’ve also just seen that it failed to count to 4, designed a strategy for the world’s easiest game that loses 100% of the time, and can’t complete a young child’s poetry assignment. It also doesn’t know that 1 is less than 2.

Colin Fraser, Hallucinations, Errors, and Dreams

This article attempts to explain:

  1. How machine learning works, with a specific focus on how LLMs work, and the difference between classical machine learning applications and generative AI.
  2. What a “hallucination” is, how it’s different from an error, and the trouble with pinpointing which parts of the output are actually hallucinations when from the point of view of the model, there is no distinction between facts and non facts (which means that everything is a hallucination).
  3. Why there is no good way to assess the “rate of hallucination” of a model, making it pretty much impossible to find a way quantifiably eliminate hallucinatory output.

Sections of this article

Quotes

I think it’s telling that most actual attempts to ground LLM-based systems in truth are not really ways to improve the model, but ways to bolt non-LLM pieces on to the larger system which produce more reliably factual text for it to bounce off of: giving it an environment to execute code in, for example, or feeding it search results from Bing. These add-ons (OpenAI literally calls them add-ons) can be somewhat successful at eliciting hallucinations that better match the real world, but it doesn’t seem to me to get at the root of the problem, which is that the engine generating can’t tell the difference between generating truths and generating lies.

As a short aside, I find the hype around generative AI to be rather confusing, and confused. Of course, I find it overblown in many respects. You know this; I don’t need to expand. But on the other hand, I think it’s actually under-appreciated — and undersold — what a miracle it is that this even works at all. It’s not so surprising to me that given a large enough dataset and a large enough model, you can train a big model to predict the single missing word from a passage of text with fairly high accuracy. But the fact that you can feed the output of that model back in on itself to generate text, and that the resulting text is even remotely coherent let alone useful, is nothing short of miraculous. Yet, I really don’t see this last point emphasized very much. I’m just opining wildly here, but I don’t think that (some of) the people who build this technology want to really acknowledge how surprising it is that this works, because that raises the uncomfortable question of whether it will take miracles of similar magnitudes to improve it — to eliminate the hallucination problem, for example. It’s more comfortable to paint GPT-4 as a brief stop along the inexorable march towards artificial super-intelligence, with hallucinations and all of the other problems as temporary blips along the way, than as one weird trick that someone discovered in 2017 that has produced completely unpredictable and surprising results that no one really understands.

Colin Fraser, Generative AI is a hammer and no one knows what is and isn’t a nail

My favourite part of this article is Fraser’s anecdote about the Quirk Chevrolet AI Automotive Assistant being incapable of giving the user accurate information about what car models are in stock. Also, I found his examples of exploiting the bot quite illuminating. The part of me that is interested in security is very interested in trying to find ways to poke at chatbots and get them to misbehave.

Sections in this article:

Quotes:

You can think of every individual word that ChatGPT generates as a little bet. To generate its output, ChatGPT makes a sequence of discrete bets about the right token to select next. It performs a lot better on tasks where each one of these bets has relatively low stakes. The overall grade that you assign to a high school essay isn’t going to hinge on any single word, so at any point in the sequence of bets for this task, the stakes are low. If it happens to generate a weird word at any point, which it probably will, it can recover later. No single suboptimal word will ruin the essay. For tasks where betting correctly most of the time can satisfy the criteria most of the time, ChatGPT’s going to be okay most of the time. This contrasts sharply with the problems of printing digits of π or playing the sum-to-22 game optimally: in those tasks, a single incorrect bet damns the whole output, and ChatGPT is bound to make at least a few bad bets over the course of a whole conversation.

Nikhil Suresh, I Will Fucking Piledrive You If You Mention AI Again

One of my classmates sent me this article about a year ago, and damn, it hits hard. This is the blog post that made Nikhil Suresh famous. It went viral several, several times. What hits about this blog post is that he’s just a regular guy who had the balls to say his opinion loud and clear and then later quit his job to start his own consultancy, putting his money where his mouth is. He has some really strong opinions about AI that would be 100% taboo to say at pretty much any organization right now, and he still said them. He clearly isn’t some genius technical person, and he doesn’t pretend to be, but he isn’t afraid of common sense.

He uses some kinda colourful language and metaphors, but if you peel those back, I feel like the arguments he’s making are pretty sane.

I am not going to put an outline of this article here. That would spoil the entire thing. Just go read it. Even if it’s just for the writing style, go read it. Nikhil Suresh is a damn good writer. I have read his entire blog.

Alright, here is one quote:

I started working as a data scientist in 2019, and by 2021 I had realized that while the field was large, it was also largely fraudulent. Most of the leaders that I was working with clearly had not gotten as far as reading about it for thirty minutes despite insisting that things like, I dunno, the next five years of a ten thousand person non-tech organization should be entirely AI focused. The number of companies launching AI initiatives far outstripped the number of actual use cases. Most of the market was simply grifters and incompetents (sometimes both!) leveraging the hype to inflate their headcount so they could get promoted, or be seen as thought leaders.

Matthew Green, Let’s talk about AI and end-to-end encryption

This article is about whether it’s even possible to have a concept of “end-to-end privacy” when wide deployment of AI agents on phones, etc. is likely to require user data to be processed on external servers. Since encrypted data can’t be processed very effectively, this likely means uploading plaintext data off of the device. Additionally, once an AI agent is deployed on someone’s device, as long as an entity has access to the agent, they can find out all sort of information about the user without needing to physically access their data, which is a massive privacy concern.

Sections in this article:

This article contains a link to (and really, is based on) an academic paper I have yet to read but would love to: How To Think About End-To-End Encryption and AI: Training, Processing, Disclosure, and Consent

Quotes

“And that’s the first reason that I would say that AI is going to be the biggest privacy story of the decade. . . . We are about to face many hard questions about these systems, including some difficult questions about whether they will actually be working for us at all.”

“This future worries me because it doesn’t really matter what technical choices we make around privacy. It does not matter if your model is running locally, or if it uses trusted cloud hardware — once a sufficiently-powerful general-purpose agent has been deployed on your phone, the only question that remains is who is given access to talk to it.”

Daniel Stenberg, The I in LLM stands for intelligence

This article is about hackers using AI to generate fake vulnerability reports on cURL. While bogus vulnerability reports have always been an issue according to this author, bogus reports generated by AI look more real and thus take significantly longer to reject (also, there are a lot more of them). The problem is that outright throwing out reports that show signs of AI use isn’t great, because there are valid reasons to use AI (such as non-native speakers of English who use it as a tool to improve their communication).

Sections in this article

Quotes

“Right now, users seem keen at using the current set of LLMs, throwing some curl code at them and then passing on the output as a security vulnerability report. What makes it a little harder to detect is of course that users copy and paste and include their own language as well. The entire thing is not exactly what the AI said, but the report is nonetheless crap.”

“As these kinds of reports will become more common over time, I suspect we might learn how to trigger on generated-by-AI signals better and dismiss reports based on those. That will of course be unfortunate when the AI is used for appropriate tasks, such as translation or just language formulation help.”

Geoffrey Huntley, What do I mean by some software devs are “ngmi”?

Great, so I’ve finally discovered where some of the CS major brainrot vocabulary is coming from, or at the very least, this guy is a very prominent adopter of it. (I’m sorry, but I can’t handle people who go on and on about people being “ngmi”. He also likes to go on and on about “high-agency people”, whatever that means.)

This article is pretty short, so I’ll just include a single quote here:

This is a tale about seven fruits. Seven fruits who work at a company and how the shifts in the software industry by AI will affect all companies and their employees.

It really doesn’t matter what company, as the productivity benefits that are delivered by LLMs and agentic software development are going to rapidly change employee performance metrics dynamics within all companies - all at once. AI has been commoditized and can be purchased with a credit card.

Viktoria Soltesz, AI-Generated Passport Passed a KYC Check: A Real-World Failure of Compliance Systems | LinkedIn

This one isn’t terrifying at all. Nope. I mean, I actually have no idea how true this is – this isn’t real research, it’s a person on LinkedIn talking about some guy on LinkedIn claiming that he did this and that it works, as well as other people on LinkedIn claiming that they did it and that it works. However, I do think it’s kinda plausible that this did actually happen, and I did find another article claiming that AI generated passports are being used to fool KYC checks on crypto exchanges.

This article is also pretty short, so here’s the opening quote:

A Polish researcher used ChatGPT-4o to create a fake passport, which was approved by a platform that used standard ID and selfie-based verification. This might sound like the utopia everyone was afraid of, but it’s the reality as of today, and it exposes a serious gap in the way digital onboarding is being handled.

This is the original LinkedIn post: https://www.linkedin.com/posts/musielak_you-can-now-generate-fake-passports-with-activity-7312844061973917698-v_sF/

Here are some other links about this, in case you want to cry (like me):

  1. How on earth does Cory Doctorow manage to do his daily blogging?? ↩︎

  2. Or anything else, really. I wonder if he got in trouble with Meta (his employer) for writing these things? Meta is obviously heavily invested in AI and LLMs now, and Fraser is uh, kinda critical of these things. ↩︎

 Help, I’m Drowning in Articles About the Dumpster Fire That Is AI

 Interesting Things I've Read (or Watched)