Bits of Knowledge: People vs. Language Models

A large language model can hold more raw facts than any one person, but only under conditions that are easier to state than to meet. Putting people and models on the same scale, bits of declarative information, makes the comparison concrete and shows where it breaks down.

The result is a back-of-envelope estimate. Two published figures anchor it, one for people and one for models, and everything else is a labeled assumption. Read the chart as a way to reason about orders of magnitude, not as a measurement of what anyone actually knows.

How to read the chart

The chart uses one unit, bits, on a log scale, so each gridline is ten times the one before it. Everything counted is declarative information: facts, text, and history. Motor skill, vision, and other non-factual memory are left out on purpose.

Solid markers come from published figures. Dashed outlines mark estimates or assumptions. The orange band is the uncertainty around the human figure, and the light blue band in the lower group marks the whole 27B model's realistic range.

The human anchor

An adult retains on the order of 10⁹ bits of learned information, about 125 MB. That is the estimate from Thomas Landauer's 1986 paper, which used several methods, including measured rates of input and loss in long-term memory, and found results clustering around that figure. Summaries of the work describe a retention rate of roughly two bits per second across reading, pictures, and other material, which accumulates to somewhat over 10⁹ bits across a lifetime.

Two caveats apply. The estimate is good to roughly an order of magnitude at best, so the chart shades 0.1 to 3 gigabits around it. And it measures what people retain, not what a brain could hold: Landauer later published a follow-up whose title makes that distinction explicit. The figure already reflects forgetting, but the chart makes no attempt to say how much is lost along the way.

The model side

A model can store at most about two bits of factual knowledge per parameter, and only when each fact is seen about 1,000 times in training. That is the central result of Allen-Zhu and Li (ICLR 2025). With roughly 100 exposures per fact, capacity in their experiments fell to about one bit per parameter. For a 7B-parameter model, the ceiling works out to about 14 billion bits.

Two things limit how far this carries. The experiments used synthetic, biography-style facts in controlled models, not the messy mix of a web-scale corpus. And real training data repeats unevenly: a few facts appear thousands of times, while most appear a handful of times or once. The chart therefore adds a lower realistic range of 0.1 to 0.3 bits per parameter for facts seen about ten times or fewer. That range is an assumption, not a published result, which is why it is drawn with a dashed outline.

For a 27B model, the three cases look like this:

File size is a different quantity from information content. A 27B model stored at 16 bits per parameter is about 54 GB on disk, while its factual capacity is at most about 6.75 GB (54 Gbit).

Breadth versus depth

A model's total capacity is spread across everything it has seen, so its depth in any one field depends on the share it devotes to that field. The lower group on the chart splits a 27B model into slices. At 0.1% of capacity, a slice holds roughly 3 to 8 megabits at the realistic range, about 45 times less than the roughly 250 megabits an illustrative human expert holds in a specialty.

If a code-heavy training mix devotes 5 to 10% of capacity to programming (an assumption, not a disclosed figure for any particular model), the same slice rises to about 135 to 810 megabits at the realistic range, which brackets the expert figure. At 1 bit per parameter it reaches 1.35 to 2.7 gigabits. The expert figure is itself illustrative: it assumes about a quarter of a person's roughly 1 gigabit sits in their specialty.

The trade-off runs both ways. A model tilted toward code has less capacity for rare facts elsewhere, though the cost may come less from filling the weights than from how often each fact is seen, since training tokens spent on code are not spent repeating other facts.

What the numbers do not tell us

The chart bounds how much could be stored. It says little about what is actually stored, or how it is used.

• Two definitions of a bit. Landauer's figure is functional information measured through human recall. Allen-Zhu and Li count stored knowledge tuples in models. The unit is shared, but the two quantities are not identical, so the comparison is approximate.
• Capacity is not knowledge. A model can store a fact inaccurately or produce one that was never in its training data. The chart says nothing about correctness.
• Facts only. An expert's skill, judgment, and pattern recognition are not counted, and a model's ability in coding or math comes largely from how it uses stored information, which bits do not capture.

What the estimate supports

The defensible claim is a range, not a single multiple. At realistic exposure, a 27B model's raw capacity is a few times one person's, and it reaches tens of times only if facts are seen hundreds of times or more. Models around half a billion parameters fall at or below one person's total even at the ceiling. Depth in a narrow field is where a person still leads, unless the model devotes a large share of its capacity to it.

A closed-book test would settle more than these estimates can. Give people and models the same set of facts to recall, then compare how much each gets right.

References

• Allen-Zhu, Z. and Li, Y. "Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws." ICLR 2025. arXiv:2404.05405 · ICLR proceedings
• Landauer, T. K. (1986). "How Much Do People Remember? Some Estimates of the Quantity of Learned Information in Long-term Memory." Cognitive Science 10(4), 477-493. Record
• Landauer, T. K. (1988). "An Estimate of How Much People Remember, Not of Underlying Cognitive Capacities." Cognitive Science.