https://www.linkedin.com/posts/ali-hamie_im-an-engineer-at-m...
> I'm an engineer at Meta and I never felt this uncomfortable doing my job. I'm writing code these days and it all feels like a black box. This new standard of shipping with AI is moving so fast that if we're going to spend our time checking each line of code, it's going to take forever. That means you can end up with systems you've built where you don't truly understand how all the code works. You understand the high level, but not every detail. And maybe this is becoming the new standard. It goes against every instinct of being a software engineer. I'm used to understanding what I'm building. Now I'm trying to figure out how much I need to understand before I'm comfortable moving forward. Are other engineers feeling this too?
How are you handling this?
Lots and lots of testing. I usually interactively test with the LLM. Now that I'm spending less time, coding, I get to spend more time, testing.
In my experience, I test more, with an LLM, than I did, with my own code. I don't find coding errors, but I do find implementation errors.
On another note: I ran an LLM on a bunch of really mature code that I wrote, and could swear was "perfect," and it found a number of issues.
Nobody's perfect. Trust; but verify.
What have we gained aside from shifting where time and effort is spent in the design loop?
The verification is what we always wanted but never had time for. For 30 years I wrote side project games without extensive testing or a fuzzer or anything because I wanted to spend my time writing the game. Now I can do that and still have the testing I always knew I should be building.
This method also helps with the classic problem of being unable to identify essential complexity up front.
Right now, we're in a "honeymoon" stage, where non-engineers think that they can somehow, magically produce stuff that is comparable to what experienced and disciplined engineers can do.
That won't last long.
Despite all the caterwaulin', LLMs are just another engineering tool, and good engineers will figure out how to implement them, in ways that absolutely blow away the garbage produced by non-engineers.
It's like that rich, no-talent guy that buys a PRS guitar, and thinks he'll beat the experienced musician that still uses his beat-up old strat, but consistently makes hits.
The goal is to get to a point where I can say with a good level of confidence that a change didn't break anything. It might be the wrong change, and it might be crap code, but I can ship it without anything blowing up. I've been using it for a while across a few game projects and so far it's worked quite well.
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
All those extra tests are churning through tokens and CI resources. These are far from free in a direct financial sense. I've also noticed some of the more recent models will try to do a lot of experimentation and ad-hoc testing by spinning up shells or running something like Python scripts direct from a command line. Every time they do that you either have to review whatever they propose to run carefully - which is slow and tedious work - or stick the tools in YOLO mode and hope that your agents don't accidentally wipe your entire filesystem, open your firewall for malware to take over your network, and perhaps launch a few nuclear missiles.
I would respectfully disagree about the value of unit tests as well. The reason unit tests appear to have less value in the LLM/agent era is because current LLMs suck at decomposition and other architecture-level work. They just see a mass of code and change it as a mass of code. This probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem and that automated work also takes longer in wall time. Again this gets brushed under the carpet because a lot of people trust that LLMs are so good that their costs are negligible regardless of how much work they do. People used to say that about the Cloud transition as well because not needing so many in-house staff to manage your resources was going to be a huge saving. Today we have companies bringing in consultants because they literally can't even work out how much they're spending on AWS by themselves. Instead of having a few greybeards who could make a Linux server do anything we have a few increasinglygreybeards who know which combination of 17 AWS resources they need Terraform to spin up so they can write the web server equivalent of hello-world.c. Meet the new boss - same as the old boss. :)
Testing is NOT verification.
EDIT: Sorry, oops, it's also called a kind of verification.
But there is the problem that tests only check or verify a small sample of possible cases or inputs. They can never show that the program is correct. (Famous Dijkstra statement.) To know something is really correct you have to understand and prove it.
Black-box e2e testing isn't suitable, or practical, for a lot of applications. Unit and integration tests generally require you to understand the code you want tests for, to know what to actually assert
Are you just trusting the AI to do it all?
Even if your test suite tested every single possible behavior and code path, tests usually can't intercept all possible unintended behavior that an LLM might have added.
In all likelihood, most test suites don't intercept any unintented behavior (such as unintended internet connections).
That's not a big problem when a honest human has written or reviewed the code, because unintended code is usually easy to spot.
But an LLM might have dropped any whatsoever code: maybe some of your project's code resembles the one of a trojan in its training, so it completes it with the trojan.
Or maybe it found that if your server is unreachable, it can upload the client's data to a public Github repo; or that if there's no space left on the device, it can delete some of the user's folders, easy!
Would your clients be understanding if that happened?
Eventually, the tools become trustworthy, and we no longer need to worry about their products.
We're just about there, with frontier LLMs, but I don't think we're all the way there, yet.
English isn't a good programming language if you want to trust that the output will be as you want it to be. High level code has no room for ambiguity - that's why we use it. It's a concise, very explicit language to tell the compiler exactly what to do.
Dealing with an LLM is a fundamentally different thing. The reason it's able to speed us up is because it makes assumptions for us and reads between the lines. The only world where we trust the output is one where it can literally read our minds. That's not coming any time soon
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
┌─────────────┬────────────┬──────────────────────┬──────────┐
│ Benchmark │ clang -O2 │ xcc --par-auto (GPU) │ Speed-up │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ saxpy │ 1,546 µs │ 726 µs │ 2.1× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ Mandelbrot │ 394,238 µs │ 2,110 µs │ 187× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ nbody │ 37,090 µs │ 1,035 µs │ 36× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ perlin │ 78,016 µs │ 1,850 µs │ 42× │
├─────────────┼────────────┼──────────────────────┼──────────┤
│ mandel_deep │ 73.2 s │ 55.8 ms │ 1,312× │
└─────────────┴────────────┴──────────────────────┴──────────┘
Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
For anyone wondering, the thing lives at https://compile-xc.org/ ;)
Still working on some of the data-parallel recognition in LLVM IR space. In order to support both C and C++ with the same data-parallel section extraction, I moved the detection from the ‘xc source code’ phase to the IR lowering phase, meaning that everything could ‘feed in’ to the same routines. That’s caused a little friction with the slightly different ways the LLVM IR from clang and xc’s IR encode loops. The above results are all single-loop extractions and the double-loop ones are still ongoing.
The system works in 3 stages: 1) by being greedy about which loops could possibly vectorise and marking them, 2) by analysing the IR to reject loops which do pointer-aliasing, or object manipulation, or… and then 3) by measuring the fastest way these known-safe kernels could execute.
Which brings me onto one nice thing I haven’t mentioned: that the GPU choice(s) are made at runtime, not compile time, and the system automatically measures (and then persists so it doesn’t have to measure again) the fastest way to run a kernel on the current machine. It handles MSL on Apple devices, PTX on Nvidia cards, Spir-v on Windows, Linux, and Android, and WGSL in the browser. It’ll quite happily divide up the parallel task across a 3090 and a 5080 on my Windows test box for example, with the 5080 doing ~2/3 of the work, because the scheduler is opportunistic.
All without writing anything other than plain C with nary a mention of the GPU. Give me a week or two, and I’ll probably announce it on phoronix (which might then get a mention on HN if it goes down well there, and it’ll be a different url so hopefully won’t be insta-dead).
I wouldn't be too sure about that :)
I think it’s my own fault, I’ve posted the link to the compiler pages several times in what I thought were relevant posts, and I think HN thinks I’m spamming. So I’m not posting the home page for a while and we’ll see if a future submission “takes”
Bottom line: GPL3 for the compiler, standard library exclusions for derived works, so use it for anything but feed me back improvements to the compiler itself, and you don’t get to redistribute it as your own.
If your definition of "works" includes "doesn't have vulnerabilities", you're going to need to do more than just poke at it and add tests.
A.k.a. "The concrete didn't crumble when I hit it with a hammer."
Process quality has been a thing for ages because it is essential.
Except it’s about 1000 “hammers” (aka tests), x 9 different architectures, x 2 different (a reference and a production) variants, and x 3 different optimisation levels. Every time I find something that slipped through, it gets at least one test written to cover it in the future. My main time-sink is the hours it takes to run the ‘gate’ before a release across the test-farm. And that’s after I’ve been playing with it myself for a while, pushing corner-cases through whatever I’ve been working on.
Jesus F Christ! No wonder most software lately (~decade) has been shit. Have y'all not heard of code review? Or did you think it just for show...you sit on a change for a week and then press "Accept" without reading it?
If you're an LLM prompter shipping products, you're outputs are more akin to a product managers, not an software engineer's.
It's hard to hold an LLM accountable for anything.
Software engineers are always responsible for their code except now they have to be responsible for Claude's code
Which is - of course - absurd. Accepting responsibility without demanding the corresponding authority is a fool's game.
I have never been much of an advocate for unionisation in the software development industry. I didn't think it was necessary to protect the interests of the workers because for most of the industry's history the workers have largely been able to stand up for themselves anyway. Likewise I've never been a big believer in accreditation and licensing practitioners in our field - because who is qualified to judge someone else worthy in such a young industry with so little hard data about what works or doesn't work? I find my views on these subjects are evolving now.
It's not about protecting jobs if they truly aren't needed any more or protecting people working in the industry who refuse to move with the times. The Luddite philosophy isn't going to get us anywhere here. And besides - if AI is as effective as its advocates have been claiming then there should soon be more demand for capable developers to take advantage of the new opportunities. (The Jevons Paradox is relevant here.) But it is about protecting standards. If the tools aren't doing a good enough job then expecting professional developers to ship the code produced by those tools even though they know it's junk is something that needs to be challenged. And if developers are being expected to ship anything important without having the slightest idea whether it even works then that's even worse.
We had a lot of silliness going on when the very low interest rates and huge amounts of VC funding were propping up ideas and practices that would never have survived otherwise. Today we have a different kind of funny money distorting the industry but it's still propping up ideas and practices that we wouldn't normally support or perhaps even tolerate. And just like before no individual is going to be able to hold back the tide and it's going to have to be a collective effort to push back - hence my evolving views on topics like unionisation and accreditation. We've all seen the drop in quality across much of the industry and we all know why it's really happening because we see how the sausage is now being made.
Time will tell whether we were right to move from a system where we trusted humans to understand the low-level system, to one where we trust fancy calculators. But so far, it's working.
There seems to be disturbingly little objective evidence of that. There's certainly a lot of code being written but how much actual value that code is adding is another question. With the scale of resources that have been pumped into AI-supported software development over the past few years where are all the new applications and the quantum leaps in quality and performance in the applications we already had? If anything I seem to be seeing more anecdotes going the opposite way so far.
Disturbingly little evidence? I would hardly call that "disturbingly little"...
This does feel a lot like the "Agile era" that was the last silver bullet that was supposed to fix software development forever and the "Cloud era" that was supposed to fix infrastructure and deployments.
Agile had a few good ideas. Code reviews and automated tests are commonly used today.
It had a few plausible ideas that it took too far. We are better at recognising the value of incremental development over big up-front design today - despite the ironic clash this seems to be having with agentic workflows - but almost everyone hates stand-ups and lots of projects have gone off the rails by breaking things down far too much and never committing to anything and so never getting anywhere.
And it had plenty of ideas that almost all of us have thrown out as unpleasant and/or unworkable. Pair programming comes to mind immediately.
What didn't really change was the fundamental nature of software development. Did we adopt a few useful ideas and get a more more efficient and improve quality a bit? Certainly. Did we get orders of magnitude better at anything and give up all the tried and tested ways of working in favour of radical new ones that never quite felt right? No.
On our current trajectory I'm expecting a broadly similar outcome for LLMs. They're now very good at some things. But they are still pretty bad at other things that also matter - and some of the reasons they are bad at them are an inherent part of how they work and therefore probably can't be overcome (at least not with incremental improvements on the technology that still basically work the same way). So I am expecting to see a period of consolidation with a lot of organisations pulling back from their dramatic leaps into the unknown and downsizing of dev teams once the big AI companies start squeezing financially and the problems of developer burnout and stagnating skills and low morale that we're already seeing after just a few months become endemic. And then as I suggested in another comment I think it's also increasingly possible that we'll see developers unionise and/or push for professional regulation and licensing so they can resist the pressure to just ship whatever their agent wrote without even bothering to check it but still being responsible when inevitably things then go wrong.
Well, the fancy calculators do. Yes, they still need guidance to keep from going off the rails. But, at least from personal anecdotal experience, I need to go spelunking into the codebase less and less over time. The agents and models really did just start to become that good.
It's totally fair to load up some open source project into an agentic IDE and instead of asking it to write something ask questions like "how does X happen when I use feature Y?" or "can you explain how client code could accomplish Z"
All the time I see an AI use a language feature or library feature that I never used before or I see some code I don't understand and it does much better than a lot of humans do at explaining what it did.
I’ve been experimenting with writing a living spec — API endpoints, schemas, how things work at a low enough level that the LLM can fill in the details - and that’s been working well.
It also forces me to do the hammock-driven development necessary to build the right thing, and as a byproduct it keeps the design in my head longer.
Can you break down for me exactly how the unit and integration tests are made? Do you ask the AI what to assert?
There should be enough detail in the spec that it's pretty obvious. And unlike - many - human devs, agents don't shy away from writing tests. Add a coverage guardrail and tell them to use red/green TDD and you're already a good way there. The main thing is to ensure shared understanding, so they also know business expectations and write appropriate tests. And... nothing here is really specific to an agent, as a human dev also needs sufficient guidance (tailored to their experience level).
Here[0] we have benchmarks for AI writing tests for code that already has the correct behavior. Best case scenario is that they're at 67% with Fable, and that's with the highest quality specification we could possibly have (code). English is far more ambiguous
I have never in my career been afforded the time to make a complete spec.
- https://dev.to/ayanozturk/dont-blame-the-developer-lessons-f...
- https://www.linkedin.com/pulse/blame-game-why-blaming-develo...
- https://thedevopsdojo.substack.com/p/stop-blaming-developers...
have an architecture
Be consistent. All parts of the app should look like all other parts of the app.
Dont allow clever code
break things into modules
do true unit testing, meaning test the smallest unit of code possible.
So how do we build confidence anyhow when we can't know everything?
I think first, standard checks - does it compile? What do static analysis tools say - code quality, passes SAST analysis? Linters? Style checkers?
Do the tests run? I'm not comfortable with the tests I write let alone the AI.. by adding fault injection (fuzzing, chaos), and mutation testing, I have a better degree of confidence in the tests and code.
Spend even more time getting to know the product instead. Maybe I don't know the code as well as I used to, but I know the product backwards and forwards. That helps me have some new confidence that I didn't really have before. Knowing the greater whole.
I agree with the advice to design your system before implementing it. It is pretty easy to iterate on a design with your favorite coding assistant, and this is your chance to be ultra-picky and make sure you get exactly what you want. This was always where the "software engineering" was done, but in practice it's easy to have a vague idea and refine your (internal) list of requirements / tradeoffs as you type in the code. I find it harder to do that when the LLM is generating the code, just because one turn can change so much.
Keeping it under control requires discipline, just like before LLMs were a thing. You set a bunch of engineers loose on your codebase with no vision or supervision, you get spaghetti. You just get a little bit more control now. Agents don't feel shitty for the rest of the day when you tell them to do it some other way, so you are free to review as critically as you want without hurting anyone's feelings. I think it's a good thing.
The further away I got from the implementation details, the fuzzier the semantics became. To the point that now I could not in good faith say I understand the system I have built from the ground-up and worked on the last ~4-5 years.
The solution to this, in my mind, is not to revert back to reviewing implementation details. The economics of that from a business/productivity perspective don't play out well.
The only viable solution in my mind is adapting code/languages themselves to the shift by making programs formally verifiable. That way you can be sure that the code "does what it says on the tin", and provably so.
I feel so strongly about this I've written on it:
https://gavinray97.github.io/blog/design-by-contract-and-eff...
You weren't programming with LLMs in 2021 or 2022, be real. More like 2024 (that's already pushing it) or 2025.
The point is that it's easy to lose understanding of the finer details, even of a codebase you lovingly created, once this switch occurs and an LLM gets its hands on it.
The bridge is built. How do we test the load? Drive bigger and bigger trucks until it fails, note the weight, print the sign and rebuild the bridge the same way.
You trust the label. The person. The company. You hope someone did their due diligence.
Sometimes taste the shell. That is an acceptable risk.
(made a small edit)
Just kidding, it's de same as with any other Open source plataform You install from their repo. It will be hard at the start but then You Will be very aware where it starts to fail and how to fix it.
Personally, with side projects and tooling, go nuts. There's no reason to read all of that code.
It then gives me a hint on the area or a typical code flow that would hit the bug - whichever gives me less info. I can obviously ask for more hints as I read along.
Obviously all honour-code based since I could just ask it to fix the bugs itself. I think this is something that can't be fixed systemically - either you're going to bother reading and thinking through the code or not.
Other than that I guess reading the tests - to make sure they fail without the code / fix, to make sure the setup makes sense etc.
2) Ask agents to do more than just code.
They need to fully demonstrate and prove that their PRs are rock solid. Ask for screenshots, videos, code reviews, detailed architecture diagrams, etc.
It would be an insane amount of work for humans, but AI can do it easily for each PR.
Merging any PR from an agent should be a no-brainer if there's real proof that the feature is built correctly and working.
If there's still doubt, then the agent needs to do a better job of proving that the feature is built correctly, doesn't introduce regressions, etc.
And before agents, we often built this working theory of the system by writing the code by hand. It helps us figure out the "grain" of the codebase so we have a better idea of how to extend it.
So what happens when agents are writing the code? The main avenue with which we built the working theory of the system in our mind is now taken away, and we're left feeling like our understanding of the system is blinded.
I ended up more or less changing my entire workflow. Four main principles:
1. Interrogation/interview-based design and specification
2. Make the decisions that matter and leave the rest to the agent.
3. Use agent to generate explainers, quizzes, and simulations.
4. Use verifiers and guardrails in the coding loop.
Interview-based design and spec
When I tell the agent what I want to build or extend, there's naturally ambiguity. But the agent knows what is ambiguous to it, so it can ask me. /grill-me is one way to do this. You'd think this would be commonplace now, but it's actually not. Some people are still wondering why they're getting bad results. It's because they're not resolving the ambiguity.
Making the decisions that matter
Not every decision matters in a product. Agents are fast at writing code and pretty good at some of the nitty gritty details, so I let it make those decisions. Also, if a particular subsystem is commonplace and undifferentiating, such as login, then I also let it make architectural decisions for that subsystem.
For decisions that are product differentiators, where that differentiation is supported by technology, I make decisions on the stack, the architecture, and the approach. This is the design phase. Because agents tend to make the mainstream decisions, they almost always take a midwit approach which doesn't leverage the technology in a way that makes a difference for the customer.
Some try to build this understanding from code reviews. I personally don't think it's the way to go, because it's not motivating. I find that I'm more motivated to learn how the system works today, because I want to change how it works tomorrow--and it sticks in my head better too (no different than pre-agents).
Generating explainers, quizzes, and simulations
As I'm doing design for an improvement and I don't know how a part of the system works, I can ask questions of the agent in chat. If I still don't get it, I can ask for diagrams (either ASCII or mermaid). But if it's an entire subsystem, I have a skill in which I can ask for visual and interactive explainers. It'll generate a markdown page (or html) that has these interactive components that help demonstrate a concept.
To help make the concept stick, there are quizzes after each section and at the end.
With the computational simulations, I generate a throwaway tool in which I can explore the state space. This doesn't happen as often, as I try to keep state spaces small, and for those explainers tend to work well.
With the design simulations, a same-model agent and a cross-model agent and I will talk about an explainer, and we'll name things that we understand or notice about the design and the intended direction we want to extend it. Then, we'll use subagents to generate options and then discuss the pros and cons. I find this is a good way for me to understand the dead-ends of a design.
Using Verifiers and Guardrails
I define guardrails as mechanistic checks that doesn't change with the feature. such as linters, typecheckers, and pre-commit hooks are all guardrails. Verifiers are mechanistic checks that will be different with each feature, such as unit/integration tests.
For more complicated states or concurrent parts, I'll ask the agent to use formal methods to write a proof. Then use the proof to generate a trace (a series of steps that demonstrate a happy or unhappy path). Then I run that trace in unit tests (kinda like property-based testing--so I guess this can be called trace-based testing), and the code has to adhere to the trace.
I also wrote a design verifier, where the requirements in the spec can be converted into a series of mechanically checkable rules. Then during testing, the verifier fires up the headless browser and uses the DOM measurements of the page to see if it matches up to the rules. This was completely impractical before agents.
All of this so a) I can be sure that the agent is doing what it's supposed to with the decisions that it's making b) I can be out of the coding loop, so that I'm not up at 2am begging, "you fixed this, but you broke that. pls fix." The agent can check for itself.
---
If any of this sounds helpful, lemme know. I've been wondering if other engineers care about understanding the code that's being written, or if people just don't care anymore. I've spoken to a few, and it seems to be split. And it may just be a matter of degree and detail. Some like it at a higher level. Some like it nitty gritty.
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push. You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop. tests are a great answer, but only if you treat them with serious intentionality.
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
It also took the industry quite a while to accept that server code needs to be secure even if that requires a lot of extra effort. AI will probably be the same.
No I only do that for unit tests so far
I went through like a mid-life crisis when the thing I learned/spent last 13 years on became meaningless.
The problem assembly programmers had with compilers wasn't that they wouldn't understand the generated machine code - it was that the generated code was often bad (as in unoptimized and bloated) and didn't justify the resources required to run the compiler. As hardware got more complex and powerful, compiler got much better, and operating systems became the primary layer between hardware and applications, assembly became more niche. Nowadays it's pretty challenging to beat the compiler at generating efficient assembly and writing a program in assembly would consist primarily of OS and library API calls anyway.
There are definitely parallels there. The only obvious difference I can see is that both high level language code and assembly are unequivocal in their meaning (as in you can determine what gets executed), whereas natural language is not.
This is a bold assumption and inconsistent with my knowledge of history. High level languages allowed better management of complexity and reduction of tedium. The common complaint was that compilers produced less efficient (slower and/or larger) code, not that the output was more complex. It was[1] normal to read the compiler's output, which was generally assembly or another intermediate representation. After all, a compiler is merely a tool for translation from one language to another. Additionally, early compilers often were not optimizing compilers; the code they generated was simpler than handwritten programs.
1: And still is, for some of us!
Each step in abstraction we've made had people who felt uncomfortable at the level of magic they were accepting. At one point in the 70s/80s, traditionalists were arguing that C itself was too high level. Then in the 90s and early 00s, that languages like Python and garbage collection were absurd: a "good" programmer doesn't need them and they give up too much control. Then as web development ramped up, people entered the "Javascript is slop" phase into "React is slop" phase.
Anytime the barrier to entry is reduced and some skill or profession becomes more focused on output than the ego of the professionals, you can bet a counter-culture of people will rise up to oppose it. It's a threat to their sense of "betterness" and/or the (perceived) value of their career.
For me I dont look too closely at the implementation but I look at the api where if a human was using my framework how is the code they'd have to write and engage look at. Then I also do plenty of performance against a comparable target in my case slate (default unreal) and keep overhead minimal.
So good performance and reasonable public facing api are the 2 main pieces im focused on. Less with the internals.
Care about the seams and interaction and the things you can measure or count.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
Recommend reading (All though a little self serving): https://www.linkedin.com/pulse/ai-code-quality-jorge-varas-a...
"This new way of piling bricks is so fast that I can no longer afford to check the buildings integrity".
I just don't understand how the fuck we ended up with this narrative. Somehow the machine making things twice as fast forbids us from spending time ensuring quality? Surely if it does things faster you have more time and tools to check!
I'm half convinced this frenzy is the marketing work of AI companies at this point.
It's at least nice to be able to prove something about the code, rather than blasting it with tests which only prove that the tests that are written do not fail (and perhaps get you some amount of code coverage — necessary but insufficient for anything like quality).
How to do all of that in your environment? That can vary greatly!
To me, "It works" is pretty much the lowest possible quality bar for an acceptable creative work product. We should not aspire to "it works." We should not hold ourselves only to the "it works" standard. And, yes, I know that many traditional software companies fail to even meet "it works" for their shipping products.
I don't even know what to say anymore. This has become a deeply unserious industry putting out mountains of junk that barely even meets the "it works" bar, and LLMs are just accelerating this.
AI still does not know how to even write code that makes impossible state unpresentable so testing won’t help.
An experienced senior engineer with access to a coding agent is the most powerful code-understanding system that has ever existed. There's no excuse for shipping or approving anything you don't fully grok.
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
Code understanding doesn’t matter anymore (or won’t matter soon)
not understanding the detail is quite similar to a coworker writing the code and another coworker approving and merging the changes.
Which is fine, usually. Depends on what you are doing.
The things that always mattered were the core system, the trunk of a tree if you will. The branches less so, and the leaves not at all (though that is what people see and appreciate! The irony!).
Or at least, I don't think people should.
To me this completely goes against the whole point of engineering.
IMO If you don't understand the code, then you're just a manager now. Soon your salary may also reflect as such, assuming you get to keep your job.
Yes, there are more CVEs discovered every day, but, if you don't patch, the hacker doesn't have to try.
This will not only not prevent CVEs, it will also make everything a giant pain in the ass
LLMs cause errors at a noticeably higher rate than human devs do. If you lean on them without actually checking what they wrote, the error rate in your software is going to go up.
Does the LLM understand it, in detail? Can you trust its interpretation of the information and instructions you provided? Did it independently make an adverse decision you haven't noticed, because you didn't read the code? Was the model missing any information it required -- and would you know if it was?
LLMs depend on humans and human oversight to ensure they have the information required to make the right decisions, to catch when they don't, and to steer them based on your much more complete understanding of the broader context and requirements. You can't rely on them to do that thinking for you (yet?).
You said LLM need human oversight and it’s true, but it does not imply you have to read the code.
I think we are going towards a place where code no longer makes sense, you just describe what you want in English, and the LLM just builds directly in machine code. You then build different verification tests to make sure the program is doing what it’s suppose to do.
The mental gymnastics developers are willing to go through just so they can avoid standing up for something is quite perplexing.
On the other hand, not even surprised anymore most people even try to provide serious answers to this. Welcome to the pure insanity.
easy, don't write code with LLMs. If my job forced me to do so I would quit.
My dumb little utils in ~/.local/bin? Those get merged after a quick functional test without my having read a single line of it. Still, the skill I maintain for such a job directs agents to write such throwaway scripts in anything but shellscript because I might want to extend some day and fuck shellscript
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
I have some work in Lean4 that I'm certainly unqualified to validate; Layer Coding QEC derivation, Positive Grassmanians and the Fresnel-Kerchhoff correction to the Parallel-Axis theorem, Huygens intensity as an alternate model for simulating photonic and electronic interference in a double slit experiment, for example.
It was a black box but I'm letting it tutor me about it so
Visual proofs, Tools for logic and formal methods in distributed systems
Things that compile logic!
> That means you can end up with systems you've built where you don't truly understand how all the code works
Though AI coding and system development can be overwhelming due to information overload and rapid growth in complexity (and concern for unwittingly embedded mistakes, the unsecurability of agent-based development, and number of LOC of code per day), not understanding the whole system sufficiently is normal and expected in engineering.
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
You're going to have to understand it one way or another, either at writing or at debugging. Even if you delegate all debugging to AI, you'll still need to determine what is "right".
If you don't understand the code at writing or debugging, you'll pay the architectural price (where it degrades). That price may be worth paying, but that depends on the system you are building.
Is it a company culture thing or a domain specific thing?
Models and harnesses have gotten really good, but they still are hard-capped by the operators that use them. The sycophancy of models is perhaps their greatest weakness, as when a user who might be experienced in one area attempts to use AI to build within an area they are less familiar with, the model does not take a step back and examine the "why" of what is being built from an architectural standpoint, it simply performs the expected task in a direct manner, ultimately creating sloppy work that may be easy to ship but difficult to maintain.
A good example of this was a recent interaction I had with a Dev who was very familiar with how to develop embedded and system level applications. He and every other dev are being asked for increasingly more due to AI, and he had to bring something he had built as a single-user desktop application into a distributed cloud environment. He ended up rsyncing a copy of his working directories including multiple credentials and sensitive internal project documents to a virtual machine binding it to a public ip address. Thankfully our tooling detected this before he could get any farther but it illustrates how the AI model isn't doing any real thinking or engineering at the end of the day. Unless the operator knew to instruct the model about proper cloud architecture and understood best practices around deployment of public systems, the model simply did what it was being told to do by a user who didn't know any better.
Another example of this is the sheer amount of unsophisticated users we've been detecting using Javascript injection into their web browser sessions as a form of automation for web services they may use. These users would never know to ask how to request an API token which may allow them to accomplish things in a much easier and approved way, and instead are relying on the frailest possible automations that expose them to risks of credential exposure.
We've got to be careful with how we engineer things with AI models. Just because we've made boilerplate easier to generate doesn't mean that we can avoid the actual engineering of what we do. And I can't help but think that firms which are fully committed to AI code generation, won't just be introducing an avalanche of slop into their product which will explode the costs of maintenance over time.
The problem here is that you're trying to do a job that is already being done for you. Maybe you don't yet trust it to do a good job, but that is quickly changing.
don’t wash your hands, or at least don’t expect us to wash them for you
Recognise you aren't the one shipping it.
Occasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
quality is all we have, it’s what makes us different from these agents.
quality is perplexing to agents because it requires a strange loop into asking why. there’s not always a clear answer. but i think the power is in the ambiguity.
if we don’t know why (or don’t care) then what happens if we all adopt this attitude at scale?
Imagine the carnage if real engineers started vibe-designing bridges.
I think it's silly to make strong statements like this. _Most_ professional software engineers (women too, not just him!) are using LLMs and are moving some amount of faster as a result.
The code was always a black box. Always.
Specs come in, passing tests come out. If you find an issue, your specs were shit.
Anyone that has ever had to outsource work to someone else knows this. If you've ever worked with another team whose codebase you didn't touch - you know this.
You write your specs, you launch an agent to implement them. You launch a 2nd agent to keep the first one honest. When done, you launch as many agents as you have budget to try and poke holes in the implementation. As it always was the case.
You can thrash around, you can fight it. Like it or not, the genie is out of the box, and the realization you were never really good at your job to begin with is gonna set in eventually.
By that I mean that, rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
The key thing here is that, by doing this, I DO know how the code works. I may not know how each unit is implemented, but I know each unit’s responsibilities and how they interact with one another. I’ve found that LLMs are HORRIBLE at writing decent abstractions, regardless of how much prompting and skills I’ve given it. It will always get the job done, but it will often be unmaintainable and require bigger and heavier refactors as I introduce more and more requirement changes.
But by flexing my basic design skills, I don’t have those issues. Future refactoring fit into easy abstractions that I’ve designed. And by documenting it up front, I also have what becomes a living document on how this work will turn out.
It takes a few hours up front to plan ahead, but it makes the LLMs job easier and use less tokens, and it makes the code better long term. I don’t care if one unit is suboptimal, as long as the abstraction is right — because at worst I can change an implementation for a better one later.
I'm old enough now to be a protected class, so I don't want people accusing me of ageism, because that's not where this is coming from - but there's a type of person that clearly is on their last job of their career and knows it and I try to avoid them like the plague, I've seen it too many times.
I just approve PRs, because I have to. I don't know what else to do. I can barely keep an eye on all my code.
Actively looking, but job market is rough in Northern Europe. I quit caring about quality in any capacity here, the company has clearly chosen its direction, against my very vocal complaints.
My last interview, with what I thought was a great company, failed when I didn't understand what NLP was when they asked me about it.
> NLP? > Natural Language Programming, it's the future. Claude, GPT, etc.
I laughed, he didn't, looked offended and said it was a serious question. I had thought it was a joke, said "oh you mean LLMs". I tried to answer seriously, of course, but that was my third round technical interview and I didn't get a call back.
Just getting a paycheck and producing mediocre work until I get something new. (Senior engineer, 10 YOE, patents, 3 years at a unicorn).
Also considering becoming an electrician.
No one cares at best, or I'm callously told "keep up or fall behind" at worst. When that's not even the problem, it just, sucks now. There's no joy in anything I do professionally.
Can I ask where you went? I'm seriously looking for an exit, I just don't know what yet. Maybe a trade, in Scandinavia the pay is actually comparable, I just don't want to go back to school.
Often, the lack of meaningfully challenging demands being put on you is punishing on its own. It can feel like a relief for a week or month or year, but if you catch a glimpse of that being your whole future, you can come to recognize yourself as being in a mind-eroding secular purgatory.
This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
I went nuts for years, until I sold off my interest in my dream partnership.
10 years of heaven. All the benefits and none of the downsides of my internal colleagues. Then 14 years of bureaucracy aimed at predictability over progress, which was infinitely more difficult for me to dodge as an external player.
Compensation for my interest was very objectively valued, with the rate of stalled progress priced in. Which made complete sense, while also being its own madness.
I walked away from so much value, but I don't regret it for a minute.
Pretty much this. I know it sounds melodramatic but it literally feels like rotting.
I don't solve anything anymore. I don't get any spark of fixing bugs. I don't DO ANYTHING, and it's fucking chipping away at me.
Also I'm fairly sure people are going to realise soon that if they're happy with slop, they don't actually need highly trained engineers anymore.
This whole thread actually prompted me to look into electrician work, especially in renewables that we have here in Norway.
I might fix my burnout and go to school to get my 2 year qualification. AI can't rewire a hydro dam.
Obviously there are mission critical systems where it matters, but if “your code can be A/B tested” or “slowly rolled out”, chances are, it doesn’t matter.
Tech debt is still tech debt, even if there exist models that can wade through, because it's going to require increasingly sophisticated models navigating increasingly large context windows and increasingly speculative workspace operations to keep making progress on it.
Vendors like Anthropic and OpenAI are laying a pit trap when they demonstrate how their models can independently code their whole platform as it becomes increasingly byzantine and inscrutable because not only do they benefit from internal/prerelease models but they have effectively unlimited spend because all that spending acts as a lure to draw you into their trap.
For everybody else, design still matters. People are just working really hard to make you forget.
Sometimes, not enough to believe you that you can do it better. And even if you could, to believe that it's overall worth it to work in that way.
It still applies to all things LLms can do from law to code to art to ideas.
They can do it - and very acceptably - but only to the untrained eye.
Way to come off as incredibly insecure. There's no need to randomly put people down.
That's it. Maybe you should go re-read?
Do you do this with every repository that's brand-new to you? You refuse to, for example, use an agent to learn where the feature should generally live? This seems absurdly slow for that purpose, because even though humans can still get a deeper understanding of a codebase after several months, an agent can get a much deeper understanding over a timeline of the initial few days. Unless everything is already meticulously documented for you, I suppose.
It can keep patching everything left and right whenever a new requirement comes up so it does not "need" to build abstractions, and it will only do it to please the human reviewer.
Are you working on a team? I've found that for solo projects this works but, even when you own the design of your piece, integrating with the work of several people doing this ultimately leads to even the part you initially designed no longer working exactly as you remember it.
A good chunk of my time now is spend working with an LLM to understand how other parts of the code base work and how they relate to and interact with the product features I manage. But I'm still on a small enough team that this is possible. I'm not surprised at all that an engineer at an org the size of Meta would not even be able to perform this level of work in understanding what's happening.
It still has sharp edges but IMO it’s the best we have at the moment, and it works. It also means we deliver larger volumes of code, but we mostly look for alignment with the original review — so our reviews are more focused on what tests say behavior should be
I have found that making the AI look the plantuml and the rendering and reviewing before presenting it to me results in much more legible designs. To me the peak of this would be a shared interactive whiteboard w the ai during design.
Of course I only arrived at the framework after trial and error with an LLM showed me the shape of the problem and the framework presented itself to me.
Planning is fundamentally so critical to ensuring alignment and outcomes, that over time how one plans can be better trained into the process to help do more and more of it.