Large Language Models make me nervous

Date created: 24-08-2025 Date finished: xx-xx-xxxx State: Open

GenAI has been the thing everyone's been going on about for the past couple of years in a gradually increasing pitch, and for good reason. The capabilities made possible by LLMs are diverse, composable, and well-aligned with the direction that the software development industry was calibrating for: to optimize for software developed in the large. All the patterns build into the development ecosystem for programming languages and tooling that are the go-to tools of the era are focused on development velocity over everything else, with variations on which parameter of the tooling is tweaked to bring in more speed. Programming languages, cloud service design patterns, deployment strategies, fucking... microservices, all aimed towards shipping. Fuck yeah, let's keep shipping.

I don't quite know if we're in that phase of tool adoption where anything beyond the first-order effects of adoption is not in consideration at all. I think we are, but I don't know for sure. It's hard to make blanket statements about an entire industry, especially one where we now have celebrity devs, high-profile programmers who (willingly or otherwise) hold actual influence over the general populace of programmers. It's insane.

I am worried about AI the same way I am worried about social media in kids' lives - the consequences of giving people tools that exceed what they would be capable of without said tools beyond a certain threshold, much the same as letting a newbie motorcyclist ride a machine that's beyond their skill level, or giving teenagers admin access to a computer without firewalls and ad-blocking. It's not about the AI being competent at generating code or whatever, far from it. It's about the user ending up in a situation that they can't get out of.

It's interesting to read all the examples of the above that have turned up since I first started writing this stupid little rant. It's also interesting, among other things, to see that at least one LLM Coding Tool provider has decided to call their tool's most risky mode of operation YOLO Mode. It's fascinating to have programs that can work with as much fuzzy input and work with such flexibility, I suppose it's only fair to look at the programs' ability to outright lie to their users as a bonus too.

Of late, it feels like the interaction paradigm that's winning for programmers right now, "agentic coding", is essentially based off of pair programming where humans are permanently in some version of the navigator role. And there's a flipped around version too, I've heard - AI-guided pair programming as a learning mode for new programmers, but I'm not sure if bean counters would find it interesting to spend all that money on licenses for juniors - we're already hearing a lot about fewer junior openings in companies now, I don't see the average senior exec figuring that out without coming across and plagiarizing a LinkedIn post about it first.)

But overall, the expectation that AI tooling will increase productivity globally (hard emphasis here) doesn't hold water to the extent that most LLM peddlers keep shouting about, simply because the vast majority of programmers aren't as skilled as the biggest proponents of these tools to evaluate the code-gen output in a meaningful sense. And on my side of the software industry, the unglamorous, unappealing software consultancy side of things, there are few people who are interested in (never mind capable of) scrutiny of any order at all. You'd think all the improved tooling you throw at these teams would improve their output, in some way raising the floor of code quality for them. But the space in which this floor exists isn't two- or three-dimensional. We're simply creating a whole new form of incomprehensible code - code that was generated by AI, reviewed by AI, debugged by AI.

Your government's tax management systems aren't developed by the celebrity developer you see advocating/testing/hailing/pushing the codege- pardon me, the "agentic coding" tool of the hour - are you prepared for that code to be incomprehensible to the people maintaining it? Are you prepared for it to be entirely incomprehensible? Are you sure that the contractor employed by the consultancy that has a billing model based entirely on small-quote-upsell-later is going to have those PR's reviewed, those tests vetted? Do you have any fucking idea how much more code there is going to be out there? I don't know if anyone's drawn a parallel with the advent of disposable plastics yet, but that's what it looks like from where I am. Some rich but stupid fucks saw that the quality of work is generally proportional to labour and decided that this would not stand, so hey, I guess thirty years from now I'll have a pacemaker that has eight gigs of RAM and runs firmware with a FastAPI front-end that'll give me an arrhythmia every few days because it keeps opening DB connections to a SQLite instance but not closing them so the pacemaker eventually crashes and reboots because the whole thing runs on a single Docker container and the FastAPI service is what runs with pid 1 or something.

I know folks' usual retort to this: government services have already done a bad job in the past, might as well try to shake things up and see what sticks. Innovation! Evolution! Synergy! Forgetting entirely that opening things up to change does not guarantee the change will be for the better. Are you prepared for the systems we built society atop to get unprecedentedly worse? And not just for you, the well-informed aero-press-sipping 60-percent-board-touch-typing home-row-inhabiting free-of-the-matrix operator. Are you prepared for things to get measurably worse for everyone else?

There's a funny thing I've noticed in the places I've worked since LLMs came into the mainstream, when it comes to development practices. There's a much stronger focus towards using tools that maximize the ease of writing code for the beginner, in the interest of productivity. I'm not suggesting this is anything new — I chose to learn Python as my first proper programming language because I liked the syntax and thought it made me more productive. The difference here is that the difficult part when actually implementing something was not accelerated by Python very much; barring some syntactic sugar, any other high-level language could've been a good fit for whatever it was I was working on.

I've been trying to think of a way to conclude this piece of writing but it's a rant about an ongoing train wreck so there's no way to end it in a coherent fashion so I'll stop here and maybe try to write about something else next time.