The Tragedy of the Information Commons

Information has always behaved a little like a shared pasture. Every article published, every dataset released, every archive digitized adds another blade of grass to a commons that anyone can graze. For most of the web's history, that arrangement worked because the grazers were people: readers, researchers, the occasional search engine indexing a page so other people could find it. The commons stayed healthy because the traffic it received was, in some rough sense, proportional to the value it returned.

That proportionality has broken down. Automated crawlers now arrive by the thousands, not to send a reader to a page but to strip-mine it for training data, indifferent to the cost their visit imposes on the server paying for it. A single well-meaning publisher can absorb a few thousand extra requests a day without noticing. A few hundred automated systems, each convinced its own scrape is harmless, add up to something closer to a denial of service. No single crawler intends to break the commons; it simply isn't in any individual actor's interest to hold back, even though everyone is worse off when nobody does.

The response, however rational for any single site operator, is to fence the field. Paywalls go up where there used to be free articles. APIs that once had generous limits now require keys, quotas, and fees. Smaller publishers, unable to absorb the bandwidth and compute cost of being endlessly re-scraped, simply go dark, or move what they can behind a login. Each of these decisions is individually defensible — and collectively, they are dismantling the open web one enclosure at a time.

There is a bitter irony at the center of it. The technologies driving this wave of automated consumption are marketed as tools that make information more accessible: instant answers, synthesized summaries, search that reads the whole internet on your behalf. Yet the appetite of those same systems for raw material is what is pushing the raw material behind higher and higher walls. The more voraciously information is harvested, the less of it remains freely offered to be harvested. Openness is being consumed faster than it can be replenished.

Garrett Hardin's original framing of the tragedy of the commons assumed a shared resource with no mechanism to exclude anyone and no cost attached to using it. The information commons was built on exactly that assumption — that publishing something openly was a gift to readers, not an invitation to be stripped for parts at industrial scale. Restoring anything like the old balance will mean finding ways to make automated access costly again, without making human access difficult in the process. Until then, expect more locked doors, not fewer.