AI Doesn't Replace Thinking About Code (Thankfully)
DHH claimed his AI agents could port Campfire from Rails to Rust, Elixir and Go in one shot, but Piotr Sarnacki's analysis shows the agents quietly made product and architecture decisions, introduced bugs, and made the benchmarks unreliable. Przemek and Marcin explain why models trained to max out a grader tend to overengineer, and why it helps to tell your agent what you don't need (Sean Goedecke's point). They close with Addy Osmani's case for verification loops and where developers should place their bets now.
The episode audio is in Polish. The transcript below is an English translation.
Also available on
Sources discussed in this episode
Episode transcript
Automatically transcribed and translated from Polish - it may contain minor errors.
Back to the player ↑Welcome to the Opanuj.AI podcast, where we analyze the most important changes and news from the world of artificial intelligence. You’re joined by Marcin Czarkowski and Przemek Smyrdek. In this podcast, as a counterpoint, like we mentioned in the last episode, we’ll talk about challenges that are not that easy to untangle when it comes to building with AI and using agents for programming. Specifically, we’ll talk about what happens when the agent decides for us what implementation to use, what decisions to make, where accidental complexity and problems pop up. Where are we starting from here? We’re starting from the story of the famous DHH, creator of Rails, who recently became known for his Rails World talk about the end of the classic programmer. But today he’ll in a sense pay for that, because we have to look at his migration of the system to other programming languages.
The Polish hussars ride in here with an answer.
The Polish hussars ride in with an answer, not even us, but our Polish representatives on X specifically. We’ll talk about DHH’s migration; he was recently describing on X how he migrated Campfire ONCE, a chat app they’re building within Basecamp, from Rails to various programming languages—Rust, Elixir and Go. All of this was basically supposed to be done one-shot. These low-level details were supposed to no longer matter. Agents were supposed to handle the implementation details for us. But here hussar Piotr Sarnacki shows that not so fast; we’ll talk about his analysis, about what goes wrong in such a naively executed migration. We’ll give examples of where agents still make mistakes, where they make suboptimal decisions. We’ll also move on to where the real gaps in agent behavior are. Gaps that come from the very process of preparing these models. We’ll talk about why these models are calibrated this way and not differently. And finally, a few recommendations from Addy Osmani. Also in a sense as a follow-up to the last episode, about what to focus on in teams, what the bottlenecks are, what to optimize for as a programmer.
Yes, we’ll talk a bit about what we can still expect to get paid for as programmers, if it’s no longer just cranking out code. It so happens there are still plenty of areas in software engineering that you can collect a paycheck for. So, let’s talk about this Campfire case, for which DHH rightly got some serious backlash. I don’t know how to say it in Polish.
A beating.
A beating, which he rightly got. He has now turned into a turbo AI enthusiast. A year ago he resembled part of our competition on the local scene. You could say he was a developer of the previous era. Now he’s not just a developer of tomorrow, he’s already a developer of the day after tomorrow. And so, he wanted to prepare a case showing how easy it is now to re-create a system in different languages, that it doesn’t matter at all and that we can do this kind of thing one-shot. Trying to prove to people that reading code no longer makes any sense at all, and that we now have full ability to port ourselves between different technologies, when in fact that’s really not the case at all. That group also very often declares that they hate both reading and writing code in Rust. And as we know, Rust is supposedly the ultimate programming language, at least this year. That’s the current trend. Everyone should write everything possible in Rust, because of course it has the best performance. Fast and safe. Granted, you can’t really read it or write in it. And it’s quite a demanding and unpleasant language for most people. And it just so happened that this test was quite problematic, as noticed by Piotr Sarnacki, who wrote a nice analysis on his blog, he dug into that code, and noticed that the comparison itself was pointless from the very beginning. We were comparing apples to oranges, basically skipping what actually matters most. Namely, whether these systems are at all similar to each other, or whether they have important hidden flaws. And it turns out they did. If we create a simple prompt, a simple skill that’s supposed to do some task for us, then by definition we have to accept that part of the decisions will be made by the agent. That’s why I so often push back against the approach that skills are unnecessary, that planning is unnecessary, because supposedly we can just chat with agents. “Make me Campfire, done.” Maybe we’ll just add “make it blue” or “make it fast.” A lot of people are kind of selling that vision, because these agents are supposedly so omnipotent now that they’ll do everything better than us. This is also being reinforced from the labs’ side, which are releasing new model versions, where the devrels write that they no longer put that much effort into specifying the environment, because things just happen. But it turns out we’re standing firm here, that you still need to plan, you need to make those decisions consciously, and you also need to steer the model so that it hands that decision-making back to us. And this came out nicely in Piotr Sarnacki’s comparison, where he rightly noted that many decisions don’t have a single correct answer and depend heavily on the product. For example, do we keep backward compatibility with the existing client of our application? There’s no clear answer here as to which approach is correct. Both options carry specific consequences. Next, what matters more to us in the system? Is it response time, or its throughput? There’s also no clear answer to what really matters. How much memory can we allocate under load? That also depends on what machines we have, what budget we have, and what our peak load looks like. If some failure occurs, do we accept losing data? In some systems we can accept that, for example by requiring a retry on the user side. But if we have a critical system that manages, say, health, we’d probably prefer that no data disappears from that system, and we add extra complexity to handle that. There are no obvious yes-or-no answers to these kinds of questions. If we just tell an agent, write me a little system, then all these kinds of decisions, and there are tons of them, will be made for us. And this also showed up in those Campfire copies. We had a version in Rust that cut out the CSRF token, because that made it easier to cache responses. And it also replaced Redis from the original version with an in-process memory queue. Then we had a version in Elixir that was closer to the original architecture, but it also didn’t work perfectly. So we can sit and compare which is better, Rust or Elixir. Such a comparison was also made by DHH, saying that Rust rules here. It just so happens that Elixir at least worked according to the assumptions, while in the case of Rust, in that initial version, not really. Let’s maybe go into even more detail about what came up there in those implementation details that an experienced programmer would probably notice quickly.
I think that to understand this, it’s best to start from the end, meaning from how these results were interpreted, which many people do according to their own ideas about the best languages and best technologies. One such context is of course the discussion about where things run faster, better, more intensively. It turns out that the conclusions drawn here after this migration, that this language was simply worse, just cannot be presented that way. In this case Claude also prepared an interpretation of this analysis for me, because I admit I’m not an expert in either Rust or Elixir, but we’ll quote it for you to show just how many points there are here where you can trip up. So in Rust concurrency can be, if we don’t have, in many different technologies, implemented in many different ways, but here, in this one-shot implementation, some database operations were blocking calls, even inside async tasks, so we thought we were doing something non-blocking, yet we still had these blocking operations. There were also locks, which didn’t exist in the reference implementation, so we kind of agreed to this blocking more or less accidentally, exactly, we sort of agreed to it without really even having a chance to agree, it just appeared. There were similar problems in Elixir; here all SQL queries were going sequentially through a single process, even when we could have parallelized something, like reads that could run independently, that just didn’t happen, so you can see that this architecture was just, I don’t know, naive, you could say, like kind of something rolled in some casino somewhere, it definitely wasn’t a 1-to-1 match with what we had in the reference implementation, and that unfortunately started to be interpreted in many different ways, even the benchmarks that were comparing these solutions turned out to be not really all that good, and here again our hussar Piotr, whom we keep mentioning, also challenged a certain Mr. Zach Daniel, when they were comparing notification delivery in this system, where Rust was supposed to deliver 1% of notifications under a given load, and Elixir 100%, but it turned out a completely different load was being generated in these systems, Rust then had to handle 4 times more requests, and in general the policies differ depending on the technology, these systems, under different load in each language, behave a little differently, so actually it’s not like the language was the only parameter and everything else was the same, there were really many policies hidden underneath there that just got ignored.
Yes, these systems cope differently with a steady inflow of, say, 100 requests per second, and differently when there are spikes where suddenly there are a lot of requests and then that number drops, and on top of that they’re really not actually equivalent systems that work in the same way, and the bugs that occurred in both the Rust and Elixir implementations, as Piotr rightly pointed out, are really bugs that a backend-specialist programmer would have noticed without much trouble, and if we take the approach of not reading that code at all, then we don’t give ourselves the chance to catch that kind of issue. And significantly boosting the capacity of that system written in Rust also came down to really simple changes at the level of stream capacity, meaning a bigger buffer, and that already raised the delivery of those requests from 14 to 90%. Such a simple change that again as an experienced Rust programmer we could introduce quickly. So here you have a huge number of purely technical problems, plus a totally random approach to product decisions, which were not faithfully recreated compared to the original at all, but were the result of some kind of lottery. And now we have to go back again to how important that engineering judgment is, and to how we should consciously make decisions, because even if we have a perfect machine for implementing code, we still have to accept that some of the decisions are not obvious at all.
Here the nature of the traffic that concerns a given product plays a huge role. We can have two apps in Elixir that behave completely differently, because they have to process totally different traffic somewhere. We can have a classic web app that, say, is used by accountants from 8 to 4. We can have a streaming app where there’s this one event a month, some match, and then everyone, a million subscribers, just launches the app and it just catches fire. It’s a web app and it’s a web app, but the traffic pattern alone means that completely different mechanisms have to be wired in under the hood. I even remember one of the job interviews I once went to, where the interviewer was trying to convince me why it was worth joining, because that traffic pattern itself is so interesting that it’s like a roadmap on its own, that we sort of know what to expect. In that case it was exactly this situation with those periodic events and those spikes, but it’s not like we’ve solved it. We know how it behaves, we know that those millions of subscribers just pile into the app. So come on in, man, and let’s just think it through. Let’s solve these problems. We’ve got a few options here. One option could be cutting off the clients who didn’t manage to get into that pool that’s being processed somewhere, just a simple fail fast. You didn’t manage it, you didn’t get into some process here, try again. We could have a bigger buffer, like a little spinning wheel, some loading that can show up here, so a completely different user experience. Do we want to keep someone waiting on some action that’s being processed, or do we want to tell them to try again? That doesn’t come from the language at all, it doesn’t come at all from the library used in a given product, in a given solution. It’s just something at the intersection of business, product, and user experience, it also comes from this kind of engineering taste and is also subject to analysis, because each of these ways of reacting to these problems can just produce different side effects. Some churn, drop-off of users, some support tickets. I think this is also a great kind of problem for people with that product mindset, because you can literally fill your whole backlog with error codes, edge cases, and just focus on that. Because often in practice the happy path is solved quickly, and the more interesting things live out on the edges. What happens under load, that’s where the real meat is.
Absolutely. And let’s also remember how important it is to create benchmarks that actually make sense. Most benchmarks don’t. A great example, closer to home for us, was when front-end frameworks were being compared back in the good old days, like how fast tables render in React, Angular, or Vue. React did best there, so the automatic conclusion was that it’s simply the fastest.
The best overall, not just the fastest.
Yeah, yeah, yeah. Based on the fact that it renders a table with ten thousand rows the fastest in the browser. Is that a good benchmark to decide whether it’s worth using a given framework in our company on the front end? I don’t think so. Even if we really care about performance itself, it still doesn’t give us the right information about whether that’s actually true for the specific app we’re going to build. Unless we’re rendering ten thousand rows in an HTML table, but I don’t know if such apps are the kind that generate big profits and that we actually want to build. So keep that in mind. And also remember how important deep knowledge is, understanding what the system should do on the back-end side. Here you’ve got lessons about compatibility, latency, throughput, memory limits. There are good books. There’s that famous book with the boar on it, Designing Data-Intensive Applications. It’s still worth reading, probably even more than ever, because it’ll just keep being useful in our everyday work. Never before have we been able so easily to ship systems that are meant to handle these kinds of problems. The decision whether we go into queues or into some kind of synchronous processing requires a team and architectural deep thinking. Today you can prompt out some first V0 version, but the consequences are of course on our side and we have to deal with them. So not everything got solved. When agents appeared, it turned out that this migration DHH talked about created a lot of buzz, but it turned out the devil is in the details. So let’s talk a bit about why these agents, in a sense, have blinders on, why they can’t read our minds. Let’s talk a bit about what we actually don’t know at the moment a project starts, or what the agent doesn’t know.
Right, and before we move on to that topic, so no one misreads this part of the conversation, and in case they don’t scrub to the end, this is not an analysis saying that agents write crappy code. This is an analysis saying that agents write crappy code in the first iteration, especially when they’re making random decisions. We talked a lot in previous episodes about how important planning is and that it’s worth sticking to that process, so we encourage you to catch up if you haven’t listened to that episode. There you’ll understand how we approach the planning process and why it’s still important and most likely always will be. And now let’s move on to what good planning actually involves, because you have to watch out for one important problem, namely that LLMs are no longer diligent interns. That was a nice metaphor about 2–3 years ago. Now they’re diligent geniuses, but they still don’t really understand the specifics of our business, our application, and what the costs are of shipping their brilliant ideas to production. Very often, and you’ll probably see this more and more, when you try to do something ambitious with an LLM, it’ll really try to overcomplicate your solutions. I just ran into this recently. I was trying to set up synchronization between two platforms and I’m like, damn, this is a mess, there are so many things I haven’t thought through. What if I lose internet on one side of the system and a message gets lost? And I sat on it for a week, fried my brain like five times, and then I realized, damn, most of those mental gymnastics were totally irrelevant. In the past I wouldn’t have had time to even think about this, I’d just build something simple that works, but here I got pulled by the LLM into all these overthinking rabbit holes, which might make sense if it were a distributed system keeping an entire hospital alive, and not a sync between two educational platforms.
Yeah, 100%, I see this again in my own benchkit backyard. We have an open-source solution that’s still at 0.x in semver. It’s not some widely deployed version. Even though there are instructions and the environment is calibrated, there’s still this huge obsession with backward compatibility. Meaning, I want to introduce some big change because, for example, I’ve got some feedback from participants, something could be simpler, something could be more complex. I don’t know, I started from GitHub Actions, now we’re moving more toward these runners, just self-hosted ones now. And there the agents all the time, whether it’s Codex or Claude Code, keep talking about maintaining backward compatibility. Because listen, those two million users who have this benchkit, they’re not ready for what you’re doing.
Exactly.
They’re not ready for the revolution we’re bringing in here. And if you, for example, want to go down, I don’t know, I’m doing a case like this now, from seven skills to three, because I’d like to simplify this loop, then let’s introduce a deprecated skills document and we’ll document everything we’ve abandoned there. And even if you tell it three times that no, we’re not maintaining that compatibility, it’s still all about backward compatibility.
Listen, where does that come from? It comes from how the models are trained. What’s going on here, Przemek? What is this process that’s causing us more and more problems?
Well, in reality the model focuses on pleasing the grader. There’s this kind of obsession with completing the task in a way that will satisfy as many components as possible. There’s some kind of score-maxing, you could say, going on. And we basically don’t have those nuances. The model has no nuances at the training stage around the idea that I’ll give you 25% of the solution, because 75% is probably irrelevant. It will always give you 100%, because it gets the max for that 100%. And that just translates into the everyday experience of using these agents.
And you can also see what that 100% is from how the bar is being raised over time. These labs are most likely constantly expanding these graders and they have higher and higher expectations for these models, for these agents. Which makes them more and more sophisticated. Because they are, we admit that. Of course, their level of intellect, their ability to implement in highly complex systems is growing. Which most likely comes from the fact that the graders used for training are getting better and better and these companies are checking more and more factors and fine-tuning to them. But that still means we have to protect ourselves more from how those unnecessary requirements get injected into our systems.
We also have this regression-to-the-mean effect here that we mentioned before. Usually it’s averaged in some way. Now it’ll be averaged toward the maximum of that score. It’ll most often be pushed toward that max score. In day-to-day work, obviously not every feature we implement has to hit that max somewhere, has to represent it, for a variety of reasons. We’ll talk about that. Products are at completely different stages of maturity. Different number of users, the level of quality we want to offer here, and so on, and so on. A startup is one thing, an enterprise is another. Completely different environments.
100%. So what should you watch out for now? You should watch out for those best practices in your technology, in your language. Here we’re also basing this on Sean Goedecke’s analysis. I hope I pronounced that name correctly. He programs a lot in Go and says there’s an overproduction of mutexes, which are often used in Go and generally considered good, but AI code usually adds too many of these mutexes. Next there’s this constant checking whether it actually went through. Whether we don’t have some race condition. Defensive programming. Very much so, but taken to absolute extremes. I’ve bounced off this a few times already, so you really have to be careful whether our solution isn’t overly defensive. And don’t let the LLMs drive you crazy, because they’ll often sell it to you in a way that makes you think, damn, maybe you really should. Not only do you feel like you were a noob as a programmer because you never did it. On top of that, it even sounds very rational. And then a PR that could have been 1,000 lines is 15,000 and you say fuck.
Exactly. We’ve also written down these out-of-this-world scenarios. What if someone loses admin privileges, the power goes out, you jump in halfway through a call, you suddenly lose your battery, the computer starts shutting down. We only need 13 modifications at every stage just to test this query. Brilliant idea.
Exactly. Some kind of state storage, it has to be stored in five different places. There’s tons of this. When you work with these models, especially frontier ones, with these frontier agents, you can see they do way more than a human would usually need. We also sometimes let these kinds of redundant changes into production, just so it’s clear.
100%.
We’re learning this too, but we’re raising awareness because that particular DHH case was loud. A lot of people looked at it. And here, while we’re at it, we’re trying to understand how exactly those training fundamentals cause all this.
Yes, and again we’re coming back to this topic of engineering judgment. We have to understand what our system, our business that we work for, actually needs. And that’s both on the positive level—what should be introduced—and also what should not be introduced. Because any significant gaps in this area will be filled by the LLM either automatically, or it will sneak them in to us in a devious way during planning. And what about this thing here, my friend? Yeah, yeah. And we can’t let ourselves be fooled by this apparent genius, which will then cost us more expensive code reviews, probably because there’ll be so many lines that no one will really check it well. And also maintenance, accidental complexity, and so on, and so on.
And what’s even worse, for example, when you don’t store the context. And the next change takes it into account as an established fact.
Right, why bother with context, why a plan, why a spec. Sure, you can just look and see, aha, if someone used to just keep up the tradition with that spec, it wasn’t there at all. Accidental, non-accidental complexity and so on, and so on. That’s why it’s so important to store and archive that context in the repositories, because that lets us understand whether it’s accidental or no longer accidental, or even what all of it actually comes from.
Listen, there are also lots of examples that show it’s worth remembering that we don’t have to strive for perfection; many systems consciously don’t aim for that ideal. Przemek, what nice examples do we have?
Well, we’ve noted down, for instance, Unicorn, a Ruby app server that deliberately ignores the slow-clients problem so it can queue requests directly on Unix sockets. We have AWS S3, which doesn’t support atomic updates of many keys at once, in return it’s easier to scale those batches. We have SQLite, which gives up concurrent writes in favor of simplicity: one file, almost zero operations. And here Goedecke, as we said, mentions that well-designed systems turn their weaknesses into strengths. I’m thinking that practices like ADRs and so on also draw their power from the fact that they always include the non-goals, what we are not doing. It shows that, okay, a group of people discussed some topic, but decided we’re not going there at all, we don’t need it, it’s not really a competitive advantage. And from that also comes a kind of continuity, a continuity of learning within a given project, of what we are not doing, where are those, let’s say, directions we’re not heading in, because we don’t want to allocate resources there and so on, and so on. That’s what this discussion is really about.
Absolutely, and remember that for each of these systems that Przemek mentioned as an example of something that consciously works maybe not perfectly, an LLM will try to push you to do both things at once, and we always pay a price for that in some way. So let’s stay aware that everything has its cost, and it’s probably better to have a simpler system that meets the goals than to keep adding bars we don’t even want to jump over.
It reminded me of that grill PM skill that the guys from AI Product Heroes showed on a livestream, which we actually ran together, where you grill someone who’s bringing in requirements not just for what’s there and what doesn’t add up, but for what’s missing, which way we’re going and which way we’re not. There’s also this Definition of Ready pattern. While Definition of Done is more associated with Code Review, Definition of Ready is important here too. Are we even taking this task on, and is it written there what not to do? And again, this shouldn’t depend on whether you got out of bed on the right or left foot, but everyone who picks up a task in this project has to check. Okay, do we know what we’re not doing? Because if we don’t, and I’m supposed to do everything, there are probably 100 directions where I could overbuild something, and that one optimal one that might randomly happen, I could go there, but I really don’t have to.
And you also have to step a bit into the role of a complexity guardian. The five whys rule: why do we need this, are you sure, because someone will come to you, an LLM will hammer out some very nice explanation for them of why all of this is needed.
Especially remotely, when you don’t see each other in person. Stop grenade, boom, off it goes.
And you have to ask somewhere along the way: what for? What for? You need to be a bit of an annoying guy, because now, again, that’s what we’re going to be paid for. Or some people just won’t pay and then they’ll have a lot of trouble. And if we want to stay good engineers all the time, who bring incredible value, more than an LLM, then let’s not race at writing little bits of code, but at making decisions about what should be built.
A quick observation, literally from today: I had the pleasure or displeasure of putting together a LinkedIn post that blew up and got quite a lot of reactions and comments, and when I see how many people in those positions, not even C‑level but execs, middle managers, use LLMs to reply, to share some simple reflection, it makes my teeth hurt, because I imagine, I don’t know, I picture it like there’s a post and someone writes, hey, ChatGPT, what do you think? And the replies are things like: you’re right, great observations.
Let’s not be mid‑proxies; of course we use LLMs, they support us in the writing process, but not in the process of thinking and forming ideas.
I mean the contrast between the senior role you have and how much effort you put in. Again, you can go back to the idea that there’ll be no competition if everyone uses LLMs, because only the person who does critical analysis will stand out.
Thinking is still valuable; of course LLMs can support us in that thinking process, but we should be at the core of that process, so it’s hugely important. Remember your specs, your skills, and other ways you communicate with agents. Make sure there’s a section on what we don’t require, what we don’t do, and stuffing in as much information there as possible is also crucial.
Any example of what you can skip in a typical web app, either from the notes or from what we decided to do?
For example, that data in our system might be stale altogether, or for some longer period, if it’s not a critical application. That, for instance, we don’t have to guarantee concurrency in every system, because the last write can win and we’ll somehow survive that in our CRUD app, say for submitting comments on lessons. That we don’t need to maintain backward compatibility for an API that only me and my buddy use. It’s enough to say, “Dude, here’s the new endpoint, just switch over to it.”
Exactly.
Another topic, also important from the perspective of what we as engineers can focus on if we’re not busy writing little bits of code for our buttons or REST API endpoints. Verification, something people talk a lot about lately; Addy Osmani talks about it too, another of my buddies from Google I/O.
Ex‑Google, now Anthropic, Claude Code.
He points out how important that verification loop is, and that often it brings more value than a better model, a better harness, because it gives you the eyes and hands that are necessary to actually deliver something in reality, even if it’s digital reality. So it’s about knowing whether the code we deploy is correct, or in fact knowing that it’s not correct, because that’s valuable information at the verification level, since then we know there’s still something to fix. So it’s very, very important today, something we often didn’t do in the past because there was no time or resources, and now it may turn out to be a key responsibility, a core task of a programmer. Tests, benchmarks, specs, good end‑to‑end tests; you can create these, it’s worth creating them, obviously in cooperation with an agent, but at the design level we humans are crucial here.
Yes, and these are exactly the low‑hanging dimensions that weren’t obvious before, like bundle size as the first example that comes to mind, the platform you’re building on, whether it builds on all platforms at all, some autodiscovery of new environment scenarios that can appear, all the stuff you do in a CLI, like adversarial code review: “find me the holes where this can crash, where this can leak, and now let’s do everything so it just doesn’t happen.” This is also a bit of a new game we’re playing.
Static analysis, linters—you can really go wild now, add lots of rules that check the code in great detail. We didn’t want to do that before because fixing all of it was a pain, now we can raise the bar a lot, turn on errors in many places to squeeze the absolute maximum out of the code the agent writes, because the cost is low. The agent will quickly iterate through those errors, something we didn’t want to do earlier because we were bad at TypeScript.
Exactly, and speaking of those loops, Anthropic also recently shared a post about how they optimize the frontend, where they say that the end goal is very important, that those details which sometimes matter aren’t always that important, because you also get this hill‑climbing effect. So if we give a nice north star like “make sure the page loads under 300 milliseconds, half a second, or whatever metric,” and we provide an environment where you can run those iterations, where it accumulates, where some learning effect appears over time, then that /goal really yields great results today. And combined with some agent in the cloud, it’s a kind of self‑optimizing box where we just point at a fragment of our project, we have those indicators, those metrics that are super important and come from the engineer’s taste and intuition for what needs to be optimized. And there you basically have this “/goal, Enter” effect; it’s not that we have to sit by the model and say, okay, now check this button—instead we give the final goal and things just happen. Here we really benefit from agent sessions getting longer, and in that context you can also take advantage of it.
100%, and we also have to watch what these agents do well and what they mess up over time, and add more and more automatic verification methods, for example whether they stick to the design system; we can also handle that in between, or whether all the actions exposed to humans actually work. Some end‑to‑end test that clicks through the app, through the main flow, so if that doesn’t happen, if something goes wrong, we again adjust the verification method so that we as humans get more information about what we care about. You could say, “Damn, this way I’ll lose my job.” In the short term, if you keep optimizing for dragging your feet as much as possible, maybe that’s an effective tactic, but sooner or later people will realize it’s not a truly productive approach. So the question is whether we’re the kind of person who introduces problems and is a bit of a rebel in the company. I don’t know if that’s good; we’re assuming our manager, our boss, isn’t fully aware of what’s going on. I think sooner or later they’ll figure it out, and we’ll not only be weak at the engineer’s job as it is now, because we did everything just to avoid learning what we needed to and bringing value, we’ll just end up unemployed. The only thing we’ll be able to put on LinkedIn is that we know Vue 2.5 from five years ago, and have no idea how code actually gets shipped to production these days. Honestly, I see quite a lot of people with “open to work” who write about lots of things, just not about what currently matters, and I kind of feel sorry for them. So I think it’s worth shifting our focus a bit to things that will be taken into account today and tomorrow. Of course knowing various technologies is valuable, but knowing the topics we’ve talked about today or in the previous episode really matters now, and I think more and more companies will be focusing mainly on that.
Here I think what’s tricky, but can be interpreted more easily, is agent onboarding and building those environments for agents. Because when we think about a junior joining the team and we want to make their day‑to‑day work pleasant, that’s easy for a programmer to accept. People even say that if you’re looking for a chance to climb the corporate or career ladder, help the juniors. We kind of have to accept that today the newbies are agents, who just have huge potential, but the environment still isn’t compatible with how they work, and you can really move up by helping those agents do their job, just by unblocking all those processes. I don’t think we said this in the last episode where we talked about changes in the work environment, but Anthropic says that the amount of change, or the general focus that used to be on agent infrastructure work, has gone from 10 to 22%. So basically almost a quarter of what a developer does is unblocking agents, and maybe at the quality level of the things we talked about we won’t be tweaking every single property anymore, but if you look at how you unblock those agents overall, that’s where work opens up for you, that’s where opportunities open up for you. Again, we have to accept that it’s no longer me writing the code, it’s some agent writing it, some kind of Claude Code or OpenCode—Claude Code, because someone recently said I keep talking about cloud in the cloud sense, so better Claude Code—but damn, there’s plenty to do. I think if you talk to your boss about unblocking things, getting stuff done faster, rolling out those agents, very few bosses will say, “No, damn it, let’s just do everything the old way.”
Right, and how do we provide those verification methods? Once again, you don’t have to reinvent the wheel at all, over 23 years lots of tools and techniques for effective code verification have already been invented. All kinds of automated tests: end-to-end, unit, integration. Their various, more specific subtypes, property-based testing, which Addy Osmani points to here, where we feed in a ton of random inputs and check how the system behaves. Mutation tests, where we change some code, add a test, break that code, and check whether the test fails together with that modified code. If it doesn’t, the test is weak and we need a different one, or the code doesn’t actually do what we intended. These are those old classic books everyone supposedly read and always wanted to apply. Most people neither read them nor applied them. Now again we have a chance to show our love for engineering. Catch up if we’re behind. Learn how to verify code in a really great way. And enjoy the fact that we’re building a high‑quality system, with high stability. And that, I think, is again job security 101 nowadays.
Right, about tests, I don’t think this has come up yet, and it really should. I think coverage itself also matters. There’s a trap you can fall into here, because we’re talking about these different loops, these factors that are supposed to ensure quality. It’s insanely easy to hit 100% coverage again with agents. It’s easy to write tests that confirm that what was written works. But it’s definitely harder to test what actually matters. For example, in our 10x tests we go out, that’s also…
With 100% coverage our system can still have lots of risks that tests don’t cover. Exposed, vulnerable.
Yeah, because we’re just checking whether lines are hit by some test, not whether the test actually checks what should be checked. So a technique I once noticed as well, learned from people smarter than me, is that we mostly work based on risks. We think about what the user is doing in our application, what can blow up and, let’s say, have huge consequences, and what can blow up and, I don’t know, I can fix it between Friday and Monday and there won’t be any fire from it. These are exactly the risks that come out in that kind of audit that’s not focused on tests.
And yeah, the overlap between coverage and testing what actually matters can be very, very small. And this is also not a conclusion from the AI world, from the world of agents, it’s a conclusion that many companies reached long before agents, and many companies optimized themselves around this metric and we had to have 80% coverage, otherwise you couldn’t even open a PR. It didn’t make sense 10 years ago, and it still doesn’t make the slightest sense today. That’s not really what this is about. We need to understand again, thinking from first principles, just like we recently talked about why we were doing code review in the first place. So why do we test? Not to have 80% coverage, but so that our system works and is safe.
100%. And I’m wondering how this might look, for example, in bigger companies. Let’s say in bigger companies, or companies where you don’t have contact with the end user, with the person receiving your work. How is that developer supposed to imagine where the real risks are, that’s what I mean. I’m wondering what they should do to get it, to realize that just closing tickets and writing tests for them might not actually be the real end goal. That grill PM thing comes to mind again. Talk to your people.
Exactly, talk to your people. It was also this illusion that we were doing some real engineering work when we were taking tickets we didn’t understand, delivering even perfect code that reflected that ticket, shipping it to prod and feeling like we were engineers.
All good.
All good in practice and in theory. That’s not the point, you have to actually understand, you have to talk with the team, with that kind of universally hated “business”. Right now we simply don’t have a choice. And again, last week someone wrote that, damn, this won’t be that important. I don’t know.
Undoubtedly. I always like to say in situations like this that time will be the best judge. For now it seems like that judge is on our side, although of course, we’ll see how it all plays out.
So listen, a lot of people have doubts and worries about whether there will even be work for programmers, for software engineers. What can you actually bet your chips on these days, Przemek?
Definitely this broadly understood sense of taste. Being able to understand the boundaries of what we’re doing. What’s needed and what isn’t. What the user is actually using. Where they see value and what they don’t use. We talked about building this infrastructure, so that’s definitely a roadmap that’s going to keep evolving. We hear this in many companies, that these core AI teams are being set up. Some labs are being created that basically, with two, three, four people, are trying to redefine the processes from scratch. Trade-offs and also failure modes, ways things can break. That’s hugely important. To even find that out, we have to step outside the happy path. We have to stop accepting tickets like, do this, here’s the design, here’s the copy, and when you click this button, something gets sent. Cool when you were a junior, but once you’re mid-level, not so much. You need to build a habit of challenging all those requirements. And assuming that something will simply go wrong. Especially when we work with distributed systems. Systems that aren’t always available. Where the server and client are, for example, separated. The client won’t always see that server. The server won’t always be as available to the client as we imagine. For example, when using the web ChatGPT or Claude. You can clearly see when America wakes up, when you use these apps. And building those loops that give us evidence. I think that’s super important. When we can say and prove to our boss, look, it used to crash every Wednesday and Friday, and now it crashes one Friday a month, for example. And also being part of the answer to the question, that search for an answer to the question, what even needs to be built, what we actually need. and not waiting for a finished spec, for a ready-made answer, the most important one right now. So being part of that process. We as programmers have a very deep understanding of how technology works, how it should work, what users like and what they don’t like. So we can be a really valuable part of that process and a great discussion partner for the product manager, project manager, the business. And depending on the company, the designer too, we can really contribute a lot to this whole process, because we live inside these applications. Most likely more than any of the other groups mentioned, or at least just as much. So we really are partners in the discussion. Let’s use that knowledge we have, those intuitions.
That’s the famous product thinking, which supposedly is bullshit. Jeff Bezos thinks otherwise, and many other people do too. We’ll see how it all plays out. It’s not called Domain Event Design, it’s not called Event Storming. Other people turn this into trainings in this country and want to make money on it. The main point is the very idea of being this end‑to‑end involved in the software delivery process. And whether we use this or that technique to get to that goal. Because Event Storming or Domain Event Design also are…
They can deliver fucking amazing results.
Great, and they also lead to that. It’s one technique out of fifteen we have at our disposal. It’s worth knowing as many as possible, because then in the right situation we’ll be able to use exactly what we need.
Yes, but you also pointed out something hugely important. I think that’s the kind of difference between, say, the US and Poland, or those countries where we mostly just execute some tasks from overseas. So this sort of engineer’s ethos. Who is this engineer in the team? Just like you said, that you were challenging yourself, that he might have the most, or almost the most, knowledge about production. And he actually has the most, right? He has the most knowledge of how this product really works. I don’t mean the product in the sense of, that next quarter there’ll be these business needs. He might not feel that, right? His boss might feel that. But where are the opportunities that come from us using this particular technology and not another? Where are the fuckups? What’s going on in monitoring, in analytics? You know that, right? You’re damn well first in line among the people who know this. And it seems to me that that kind of internal respect is much more common across the ocean, and here it’s still like, damn, I’m just this developer. We of course decided that we’re this fire mage who simply knows all the incantations, the syntax of this one framework that’s the best for building this specific system. And we dug ourselves in around that. Well, unfortunately, that sort of collapsed into rubble somewhere along the way. And now everyone’s like, what now? Well, you can just become a software engineer in general and that opens up a huge number of cool opportunities and reasons to be proud. Even if it’s not about how we name classes in Java.
Exactly. Listen, make sure you leave comments. We know a lot of people are wondering what’s next in their career, what’s next with their job. One topic I didn’t want to skip, and it’s also come up recently, this question also comes up: will you give me a guarantee that this will work? Will you give me a guarantee? I’ll say, I won’t give you a guarantee. Just like throughout our careers we always placed some bets on a direction of growth, this is another bet we’re putting on the table now. But it just seems to me that the decision is kind of binary in a sense. There are some people who decide this is no longer for me, that someone else will code and I really don’t want to continue here. That’s a good bet. You can also place that kind of bet somewhere. Yes, but the other bet is that if you want to continue in IT, be useful to that boss. Be useful to the business, be useful to that money-making engine behind this product. Maybe it can make more money, maybe it can bring it into the company at lower cost, but damn it, just be useful in that whole process. And why are we placing this other bet? Because we kind of understand how these LLMs work. We see that the ability to write code and pass tests is already at an incredible level no human can reach. No one will convince me they’re a better programmer than Opus 5.5, especially across a range of different technologies, if the spec is well prepared, but Opus’s ability to understand business requirements and talk to people still isn’t impressive. It’s definitely better than GPT-4’s, but the gap between GPT-4’s coding skills and Opus 5.5’s is many times larger. Here, the progress in soft skills and product skills is much, much smaller, so that’s why we’re making this particular bet. Do we guarantee this will give you job security for the next 20 years and 20k a month, no matter whether you’re standing or lying down somewhere? No, but if we’re going to bet on something, we’re betting on this. And we were betting on it even back when these agents didn’t exist, because in our view that’s what the job was about: making good technology, and code used to be the biggest bottleneck, just a means to that end. It isn’t anymore, and we can do, in our opinion, more interesting things. And with that, we’re wrapping up this next, already November episode, because it’ll be published in November, in these new conditions. Episode two recorded here in the Daj Głos studio. Thanks to the crew from Katowice for preparing this material. Listen, leave comments, leave feedback, thumbs up, likes, subscriptions, follows, stars, wherever you’re listening to us. Our observability, monitoring and analytics need to see your reactions somehow, but also, quite honestly, it’s just nice to see this community grow. Quite a few people give us feedback that we touch on topics that aren’t actually discussed that widely. And it’s great that you say that. For us, that’s fuel to keep going. Every Monday we send out the Opanuj.AI newsletter. Three handpicked pieces of content about machine learning and artificial intelligence, also product-and-business-focused ones related to programming with AI as a kind of supplement. We encourage you to subscribe to that too. So, what else would you recommend?
Yeah, opanuj.ai/newsletter, or Przeprogramowani slash newsletter. That’s our second newsletter, the technical and personal-development one. The technical links are usually about AI too, while the self‑development ones are broader. And there’s also more personal stuff from us there, so if you’re into that, we publish that newsletter regularly as well. It has many loyal readers. Apparently it’s really good, so we encourage you to follow us there. And thank you. See you in the next episode.