Source item

Quoted context

These are the quoted author’s words.

Freedom in the post-AGI world means building political superintelligence with tireless, brilliant political agents who represent us, the people—not governments or companies. In a special July 4th issue of System Check, I get into what this might mean. Several forces tilt the post-AGI world toward totalitarianism: the concentration of resources required to train frontier models, AI's obvious uses for surveillance and control, and existential risks that could justify extreme security states. But AI is also the biggest opportunity to upgrade democracy since the printing press. Most of our governance failures happen because citizens are too busy to pay attention, so a small group of highly motivated wackos drives the process. (See: NIMBYism.) What if that changed? What if AI could give every person a super effective political agent that represents them all the time? @gwern 's new "Guardian Angels" essay is the most serious technical sketch I've seen of that agent—one that learns you deeply, remembers everything, and can carry out "direct democracy on unprecedented scale." His most vivid example: official GAs for every member of Congress, able to simulate a roundtable debate among hundreds of politicians within minutes, or convene an emergency session at 4AM while every human is asleep. I see two big open questions. First: the agent has to be more than a digital twin. It should share your values without freezing your less-considered opinions in amber — willing to push you on topics it has studied more deeply than you have. On contested political questions, AI models don’t seem to possess that capability yet. Second: who governs the guardian angels? Gwern proposes a startup with dual-class founder shares. Sensible for the development phase. But can we build democratic infrastructure on private rails that one company controls forever? Which is why the recent attention back toward open-weight models and orgs that own their own models matters (see the great interview between @satyanadella and @ypatil125 below). The same logic driving firms to want their own models applies to democracy, too. If we're going to own our agents—agents that answer to us, and can't be secretly commanded from afar—we may need models we can run ourselves. The counterarguments to the open model idea (RSI leaving open models behind, safety pressure on open weights) are really big though, and I really have no idea how this is going to play out, or even how it should play out. How are we going to run a democracy if every citizen's agent is built on a single closed model with a single point of control? That's the question to think about this Independence Day. Check out the full piece here: https://t.co/ZyAZlTr9Zs

ahall_research ↗