8090 works on production systems for large, often regulated, enterprises.
Vibing isn’t tolerated because these are the systems that run western society - banking, power, healthcare, insurance etc.
Over the last few quarters, the gains that we got from using frontier models inside of our Software Factory on these systems started to shrink but the costs kept doubling. This makes sense I guess, as in hindsight, we were initially asking the model to do mostly light work (generate basic PRs) and now we were asking it to do more complex work (mitigate dependencies across systems).
Unless you grow context massively, be willing to run many A/B tests and iterate massively (ie use massively more tokens) complex tasks stay roughly unfinished by the model and requires the engineer to largely act alone.
In other words, we find the last 5% (ie where a model is truly equivalent to a reasonable engineer) extremely difficult to achieve and extremely expensive to such a degree that the fully loaded cost of the model + the engineer will not pay for itself.
So I asked our CTO to start thinking about other ways. We need our engineers to have access to the best tools BUT we also need to educate them to think even more for themselves - not less - in this last mile.
At the same time, we need to find solutions that decrease our token costs by 90% - especially because these bleeding edge tokens are not nearly as cost effective as the tokens before it and are creating a big OpEx bill for us.
I wonder how many engineers, in all orgs, are running amok right now by using the latest frontier models as a kind of slot machine. Increasingly turning their mind off, largely keeping productivity flat while their CEO and CFO deals with a massive token bill?
My advice to you is that when you encounter this last 5% of very hard technical challenges in getting a complex system into production, be circumspect.
The challenge of the last 5% is actually getting harder - especially as hundreds and thousands of code generation model runs run amok adding all kinds of random cruft into codebases that eventually need to be rationalized.
Quoted context
These are the quoted author’s words.
Chamath reveals his company's AI token costs are doubling every 45 days but productivity is only up 5%
"I sat down with my CTO today, I said how are we doing on token spend. And he said the most incredible thing, he said right now, our token costs are doubling every 45 days. I said well what is the downstream productivity? And he said maybe 5% max."
"So my costs are doubling every 45 days, my upside is essentially flat. He said honestly, what we're finding out is that you need to use a lot more tokens to get to this next iteration of improvement because we've effectively already asymptoted."
"We're going to take a step back and try to figure out what to do. I don't know how many other companies will actually go through this reckoning now, but the point is everybody in the next three or four years will for sure go through it."
"I suspect that if you can get out now, you should get out now before all of that starts to seep into the water table. Because I think that's probably what allows you to get out at a huge price and raise a huge amount of money."
dnapway ↗