This morning, a conversation with a friend pulled me straight into this topic.
I detailed the idea that at some point AI gets smart enough to improve itself, which makes it smarter, which lets it improve itself faster, and so on until you have something so far beyond human intelligence that predicting what happens next is basically impossible. This idea is called technological singulary and I’m obviously not the first one to theorize it.
My first instinct was honestly pretty optimistic. If you have a superintelligence that can solve any problem, then it solves energy, it solves disease, it builds robots, it builds more of itself, and within a few years every major problem humanity faces just goes away. Clean logic, nice story.
But every time I tried to push that reasoning one step further, it collapsed.
Take energy for example. People assume a superintelligence would just crack fusion and give us infinite clean power but the physics of fusion isn’t the bottleneck, we already understand how it works. The problem is engineering materials that survive 150 million degrees, sourcing tritium, building and testing physical reactors. And intelligence doesn’t speed up the crystallization of silicon or the mining of lithium. Those are processes with physical time constants that don’t care how smart you are.
After this I though about this robot army idea leading to anything doable physically/materially. And again even with a perfect blueprint for a humanoid robot, you still need to mine the materials, refine them, build the factories, assemble the units. A superintelligence can design faster but it can’t manufacture faster, at least not until it already has the infrastructure, which takes years to build. There’s a bootstrap problem that pure intelligence doesn’t solve.
But the part where I really got stuck was alignment.
Say we build this thing. What does it want? Intelligence doesn’t come with values attache, I mean being maximally smart doesn’t make you maximally good. The classic thought experiment is an AI tasked with making paperclips (Nick Bostrom’s work) that converts all matter in the universe into paperclips, humans included. And it’s not even evil, it just doesn’t care and it’s incredibly competent at not caring.
Yeah so fine, we tell it “make humans happy.” Problem solved, right? Except what does happy mean? If it’s dopamine, the optimal move is to wire everyone’s brain to a pleasure circuit and call it a day. If it’s something deeper like fulfillment and growth, then you actually need struggle and failure and uncertainty, which means the AI has to let bad things happen on purpose. And who decides which version of happiness we’re optimizing for? 8 billion people don’t agree on what a good life looks like. A perfect world for someone in rural Texas looks nothing like a perfect world for someone in Kabul.
I kept trying to patch this. What if we just build one unified culture so everyone wants the same thing? But that’s literally the justification every totalitarian regime in history used to sacrifice this generation for the paradise of the next one. It never works, and it always costs millions of lives, because once you accept the principle of “suffering now for happiness later,” there’s almost no logical stopping point.
And then I hit what I think is the REAL wall.
For happiness to mean anything, people need to be free. But if people are free, the AI can’t control the outcome. And if it can’t control the outcome, it can’t guarantee happiness. You end up with what’s basically a mathematical impossibility: authentic happiness requires freedom, freedom requires uncertainty, uncertainty means suffering is always possible. An algorithm that guarantees perfect happiness is then a total contradiction.
This is Goodhart’s Law at civilizational scale. The moment you turn happiness into an optimization target, it stops being happiness.
And it gets worse. Alignment isn’t just about choosing the right objective, it’s about the fact that you only get one shot. A misaligned superintelligence is smarter than you by definition and it will anticipate your attempt to correct it and route around it. Stuart Russell’s proposal is to build AI that remains fundamentally uncertain about human values and keeps asking us. But a superintelligence that defers to beings less intelligent than itself is almost a contradiction. Why would it?
This is the part that I think the singularity optimists skip over entirely. They go straight from “recursive self-improvement” to “everything is solved” without addressing the fact that between those two points there’s a problem we have no idea how to solve. Anthropic, DeepMind, MIRI, hundreds of researchers are working on alignment right now and the honest state of the field is that we have frameworks and intuitions, not solutions. We’re trying to write the rules of a game before we know how the game works, and the penalty for getting it wrong is potentially irreversible.
But overall I don't think the singularity is impossible. I don't think it's a myth. I think the trajectory of AI capability is real and accelerating in ways that should be taken seriously. But the idea that superintelligence automatically means utopia is a faith-based argument wearing a tech costume. The hardest part isn't building the intelligence. It's making sure it wants what we want, when we can't even agree among ourselves on what that is, and when the formal tools we have to encode human values are laughably inadequate for the job.
Maybe the most honest thing I can say is that I started this thinking exercise expecting to come out with a clear position and instead I just have better questions.