The numbers that get thrown about don’t mean what the people throwing them around think them to mean

That describes a common rationalist failure mode. They reach for a false sense of quantification by throwing lots of numbers at things, but the numbers are already approximations of much more nuanced, complex, and/or continuous things, so by overemphasizing the numbers, they actually get further from properly understanding. See for example… fixation on IQ; slapping probabilities everywhere; extrapolating trend lines (METR task length); and prediction markets.