Let's assume that coding agents are able to generate most of the code, even in a large software system. There are two extrapolations which I think are reasonable to make under this assumption:
- The cost of code generation trends towards zero
- Systems start to be treated much more like black boxes.
What does it mean for software engineering? Will software engineers have anything to work on instead of writing code? What does it mean for the overall cost of producing software? What does it mean for software architecture? Let's try to consider the possibilities.
This post is a bit meandering - I include both extrapolations and counterarguments. There are many different possible paths.
Cost of code generation trends towards zero
I am assuming that it becomes essentially free to produce not just any code, but correct code. Does it mean that useful working software becomes free as well? Probably not. I think two large problems remain:
- Getting the software to do what we actually want, not just what we asked the agent for
- Complexity in all its guises - to take a simple example, if we ask the agent to add a new feature and it does it correctly, it doesn't mean that the feature interacts well with all the other features.
To understand these problems better, let's consider the likely second order effects of zero-cost code generation:
- The volume of code expands
- Rewriting becomes potentially easier than dealing with technical debt
- Refactoring becomes easy so tech debt can be controlled better.
As the volume of code expands, I think both its essential and incidental complexity are likely to increase.
Growing essential complexity is kind of a given. In my experience, no matter how hard we try to factor out independent subsystems, they still interact. The real world is also a complex place, meaning that the requirements are entangled as well. This drives complexity.
Incidental complexity is more of a mixed bag. On one hand, the easy management of technical debt drives down incidental complexity. On the other hand, even if it's easy to deal with tech debt, it's too much of an assumption that the system architecture is just as easy to modify continuously, so it may not happen, and eventually the system will accumulate choices which are out of step with what the n-th iteration of functionality requires.
But what if we let go of the traditional additive model of writing software, where we keep adding bits to the code base and slowly moulding it as the requirements change? Zero-cost code generation might allow regenerating whole systems with one new requirement baked in. That could deal a serious blow to incidental complexity as it grows via accretion of code over time. If we start from a blank canvas every time, a pristine code base can be laid down (as far as non-essential complexity goes).
Aside: This approach hinges on having a very clear spec to regenerate the code from - this dovetails with the black box discussion in the next section.
Another implication here is that this approach changes the definition of maintainability. Instead of allowing us to make additive changes with bounded effort, a maintainable code base is one where the coding agent can regenerate parts of the system in a contained manner to incorporate new requirements.
Whether incidental complexity can be eliminated or not, free code provides the incentive to solve ever more problems with software, meaning that it becomes more complex. But can software continue to become more and more complicated without limit? Probably not. It’s more likely that we get to some kind of plateau where neither AI nor humans are able to change the code effectively. This is like what we have now - the cost of changing and maintaining systems goes up nonlinearly as complexity grows, and at some point everyone has to run just to stand still. The difference will be that people may stand little chance of untangling the piles of complexity built up by AI.
However, there are more drastic ways to chop up the code into independent chunks, thus dividing and conquering the volume/complexity problem. Currently software tends towards bloat in part because there are certain economies of scale that an existing system provides. Adding in some extra bits to satisfy the demands of a new customer is much easier than building a wholly customer-specific system. In fact, we go to remarkable lengths with plugins, extensions, and integrations in order to leverage already-built software.
But this logic may no longer apply when code is free. In that kind of world, everyone might get their unique snowflake of a software system that is hyper-specialised to their needs.
Contrariwise, it’s worth keeping in mind that there are reasons most software isn’t built in-house. Running the software still requires operational expertise – and besides, organisations love outsourcing responsibility. Having an external party to blame for outages, compliance failures, or data loss is a great incentive to continue using third-party software.
One way to reconcile these two competing motivations is that software is heavily customised but operated by third parties (much like what the cloud provides today). In that case, even more complexity shifts towards infrastructure, orchestration, monitoring etc.
Systems start to be treated much more like black boxes
If the code is generated and the AI is able to explain what it does, then there isn't much incentive to read it. It's similar to assembly in pre-GPT days - it's there, very occasionally you might have to dive in for an obscure reason, but most of the time you deal with higher level representations.
Aside: This isn’t entirely new. We already have examples of this mode of interaction with code: extremely large code bases. When working on software made of millions of lines of code, people can only know tiny slivers of it in detail. A lot of it is therefore a black box. We can take lessons from such projects.
What would software engineers focus their efforts on then? I think that the remaining work is about defining what the system should do and characterising those requirements in much more detail than is common today. It's also about observing and verifying the behaviour of the system.
To break it down a bit more, the tasks would include:
- Working out what the software needs to do
- Defining a set of constraints which the coding agent must adhere to when generating code; this would consists of specifications (maybe formal specifications?) and tests
- Defining tests (the test code can also be generated from high level requirements of course)
- Extending tests to be more and more comprehensive - property-based tests, mutation tests, chaos tests, deterministic simulation tests, performance tests etc.
- Analysing various aspects of the system - including code, functionality (eg. various interactions of the features), architecture, usage analytics etc.
- Expanding observability in various ways - this can get quite complex so assuming a lot of AI assistance as well.
All of this could make the software a lot more robust, usable, performant and so on. However, a less positive projection is that the economic incentives prevent this kind of optional work, and instead we just have a much smaller number of overworked engineers continuing to churn out low quality software (today, most software is low quality) with coding agents without all the niceties like extensive tests or good observability.
Another consequence of code becoming a rarely examined artefact is that the existing engineering specialisations (frontend, backend, DevOps) might dissolve to a large extent, and engineers become generalists, much like in the early days of software. If you spend your days looking at code, there is not that much point in being proficient in a specific programming language or framework, even though it might still make sense for multiple languages and frameworks to exist for different purposes, and to continue to evolve for richer and more precise encoding of behaviour. On the other hand, everything might just become React, and that's that. Regardless, software engineers would be dealing with AI output, not the underlying details.
What about debugging? I think it would be similar to how we debug in production when working with large and complicated system. It can’t really be done by eyeballing the code and figuring out what’s wrong. Instead, you have to look at metrics and logs, attempt to narrow down the reproduction scenario, create failing tests to match it, then roll out a fix and monitor the results.
Architecture: Focus on replaceable components, clear interfaces, and observability
Finally, let's put the previous two sections together and consider the effects on architecture. As new requirements come in, we might want to frequently regenerate components or whole subsystems to avoid accumulation of incidental complexity. The system is also a black box, by and large. It's likely to be more complex.
Therefore, the architecture has to emphasise:
- Very well factored subsystems/components
- Clean interfaces between them and externally
- Very high degree of observability
- Exceptional testability
These are important properties even today, so I think it’s more of a shift in emphasis than a tectonic shift. For example, microservice architecture, which I've always considered to be unfit for 99% of situations, might start to make more sense in this world.
Conversely, we no longer have to worry about:
- Readability
- Discoverability
- Consistency
- DRYness (AI can deal with repetition)
These are perhaps aspects of code organisation but they do spill over into architecture. This is definitely a significant departure from how we organise things today!
Conclusion
Overall, it seems to me that generating most of the code with AI means that priorities shift towards testing and verification, and that working on software systems will resemble most how engineers currently interact with very large, very complex systems. We should be looking at such projects for lessons. A really novel aspect that AI brings is wholesale replacement of software components instead of gradual, mostly additive evolution that tends to happen today.