What AI Is Actually Good At in Code, and What It Costs

The work it genuinely shortens

Boilerplate first, because it is the least interesting and the most reliable. A Dockerfile for a stack with ten thousand public examples, the fortieth CRUD handler in a service that has thirty-nine. You know the shape of the text before you read it, and can tell in seconds whether it is right.

Translation is the same trick in costume: Python to TypeScript, a schema into a migration, a table of sample data into a test fixture. The structure is fixed by the source, the source is in front of you, and checking the output is comparison rather than recall — a model does this the way a very fast typist with broad syntax knowledge does it.

Reading is where models are strongest, because the code stays as ground truth. Ask what a two-hundred-line function does and treat the answer as a hypothesis you can check. Tests for existing code fit the same logic: the implementation pins the behaviour down, so the model describes rather than decides. It will also cheerfully write tests that assert your current bugs, which is why these cannot be your only tests.

Then the incantations: regexes, jq filters, the git command you can never remember. Write-only syntax with an enormous public corpus and a five-second check: run it on sample input. The alternative is a scroll through the man page, so the saving is real even at a mediocre hit rate.

None of this is about difficulty; it is that verification is cheap, and in a domain you cannot evaluate you are not saving time, you are postponing it.

Where it starts costing more than it saves

The first weak case is code that has to fit a codebase the model cannot see. It does not know you already have a retry helper, so it writes code that is locally plausible and globally wrong: a duplicated utility, a database call in a layer that is not allowed one. Bigger context windows help, and so does indexing the repository, but retrieval gives the model a sample of your codebase, not your codebase; the conventions in reviewers' heads are in no file it can read.

The second is anything depending on a library newer than the training data. A model's picture of an API is frozen roughly at its cutoff. A tool that can read your lockfile or the installed source can find out which version you actually have; a model answering from weights alone is guessing, and the answer looks identical either way. The tone is identical whether the method exists in your version or was removed two releases ago.

The third is subtle concurrency and security: both are properties of the whole system, not of the snippet on screen. A race condition is a claim about interleavings, and a model reading one function cannot see who else calls it or which lock the caller holds; whether interpolating a string into a query is exploitable depends on where the value came from, three frames up. It can spot a textbook flaw in a diff, but not give you an argument that the code is safe. And an authorisation check placed after the side effect it was meant to guard reads, on the page, exactly like one placed before it.

The failure that costs the most: an API that does not exist

Name it plainly: it is the single most expensive thing these tools do. A model will invent a function, a flag or a package name, give it a signature that looks exactly right, and describe what it returns in a confident sentence. It is producing the most plausible continuation of your code, and a method called parseTimestamp on a date library is plausible whether or not anyone ever wrote it.

What makes it costly is that it fails late and reads like your own mistake. The flag is rejected, so you check your version, then the changelog for a release you might have missed; that doubt is the expense, not the typing. Invented package names are worse: a name that does not exist today can be registered tomorrow by someone reading the same suggestions.

The defence is mechanical and takes seconds: any API name you do not recognise gets checked against the installed source or the real documentation before you build on it. Jump to definition, grep the vendored package, run the command with --help. Do that consistently and the worst failure mode becomes one of the cheapest.

A workflow that holds up

Review every diff as if it came from a competent contributor who has never seen your codebase, because that is precisely what it is. What happens on empty input, on a timeout, on the second call? Keep the change small enough that you actually read it; large generated diffs are where plausible-but-wrong survives review.

Run the code instead of trusting the explanation: the paragraph describing what the function does is a separate generation from the function, and the two can disagree without either looking wrong. A test run is evidence; the prose underneath it is a claim. When something breaks, paste the actual output — the whole stack trace, the compiler message with its line number, the failing assertion with both values — because the literal text carries the file, the line, the exception type and the offending value, and paraphrasing deletes exactly that.

Which is why "it compiles" is not "it is correct". Compilation checks types and syntax and has no opinion about intent; a green suite says the tests passed, and the model may have written those tests from the same misunderstanding that produced the code. Everything that actually hurts compiles fine: the off-by-one at a boundary, the condition inverted in the branch nobody hits until quarter end.

Why the productivity numbers do not settle the question

Published speed-up estimates disagree enormously, because they are not measuring the same activity. The most-cited large number comes from a 2023 experiment run by researchers at GitHub and Microsoft: 95 freelance developers were asked to write an HTTP server in JavaScript from scratch, and those with Copilot finished about 56% faster. Read the task before the percentage: greenfield, self-contained, no architecture to fit, correctness visible in a minute. METR's 2025 trial is the sharpest counter-example: sixteen experienced open-source developers, working 246 real issues in repositories they averaged five years on, took 19% longer with AI tools allowed, and estimated afterwards that they had been 20% faster. Sixteen people and one generation of tooling is not a law of nature, but the gap between the measured 19% slower and the felt 20% faster should worry you.

Other variables swamp the effect: how well you know the codebase, how much public code exists in its language, whether review time is counted, and whether "done" means merged or still fine six months later.

So treat any single percentage as marketing. The honest version is conditional: large gains where verification is cheap and public precedent abundant, small or negative gains where the difficulty is understanding a system rather than producing text. If you want a number for your team, measure your own tasks over a few weeks; nobody else's transfers. We build one of these tools ourselves, Firas Code, and nothing above is softer because of it: the review step it puts in front of you is the whole point, and it is the step people skip.