{
  "site": "at://did:plc:ngokl2gnmpbvuvrfckja3g7p/site.standard.publication/3mjpv6cwinz2f",
  "$type": "site.standard.document",
  "title": "limits of ai",
  "updatedAt": "2026-04-27T03:05:21.067Z",
  "publishedAt": "2026-04-27T03:05:21.067Z",
  "textContent": "Henry Farrell reviews Ben Recht's book The Irrational Decision, arguing that AI has a \"sweet spot\" -- and that many AI people can't see the limits because their tools are designed to hide them.\n\nThe core argument: Mathematical rationality (optimization, statistics, machine learning) is powerful in its domain but has structural limits. The problem is that the people most invested in it are the least able to see those limits, because their evaluation methods systematically exclude the kinds of problems their tools can't solve.\n\nThe sweet spot: ML works best in the messy middle -- problems too complex for clean mathematical solutions but regular enough for statistical approximation. It fails at the extremes: problems so clean you can just code a solution, or so messy that no statistical regularity holds.\n\nThe measurement trap (the key insight): The Common Task Framework (shared dataset, clear metric, competitive evaluation) is how we judge ML performance. But tasks that can be measured this way are definitionally the tasks that machines will be good at. If you can cleanly articulate outcomes, data, and metrics, you can automate it. If you can't, you can't -- but you also can't measure it, so it disappears from the evaluation landscape. The tools of measurement confirm the biases of the measurers.\n\nHistorical context: Recht traces mathematical rationality from 1940s-50s optimization (linear programming, optimal control, game theory, RCTs). The \"one weird trick\" is recomposing complex problems into optimizable form. When that works (chip design -- simplifying into \"standard cells\" + simulated annealing), it creates extraordinary feedback loops. When it doesn't, it simplifies human beings instead (poker solvers optimizing players rather than the game).\n\nPolitics resists optimization: When you have incommensurable tradeoffs (playground vs fire station), there's no cost function to optimize. Herbert Simon and James Scott both make related arguments -- societies aren't chips, and treating them as \"standard cells\" to be rearranged is the core error of technocracy.\n\nThe two unhelpful positions:\n\nAI Rationalism (Nate Silver's \"River\"): assumes computers make better decisions than humans, launches into fantasy from ridiculous premisesAI-Con Thought (Bender & Hanna): claims AI is mostly hype, curdles into denialism as the technology finds more uses\n\nRecht's position is neither:\n\nML is a powerful statistical tool with a sweet spot, and the limits are real but hard to see from inside the framework."
}