“If Twitter were to go in the morning, let’s say, all of this, all of the first-hand evidence of potential atrocities or war crimes, and all of that potential evidence, would just disappear,” says Ciaran O’Connor, senior analyst. of the Institute for Strategic Dialogue (ISD), a global think tank.Information gathered through open source intelligence, known as OSINT, has been used to support war crimes prosecutions and acts as record of events long after human memory fades.
Part of what makes Twitter’s potential collapse a unique challenge is that the “digital public square” has been built on the servers of a private company, says O’Connor’s colleague Elise Thomas, an analyst ISD OSINT senior. It’s a problem we’ll have to deal with many times over the next few decades, he says: “This is perhaps the first big test of it.”
Twitter’s ubiquity, its adoption by nearly a quarter of a billion users in the past 16 years and its status as a de facto public archive have made it a goldmine of information, says Thomas .
“In one sense, this represents a huge opportunity for future historians: we’ve never had the ability to capture so much data about any previous era in history,” he explains. But this enormous scale presents a huge storage problem for organizations.
For eight years, the US Library of Congress was responsible for maintaining a public record of all tweets, but stopped in 2018 and selected only a small number of posts from accounts to capture. “It never, ever worked,” says William Kilbride, executive director of the Digital Preservation Coalition. The data the library was expected to store was too large, the volume coming out of the fire hose was too large. “Let me put it in context: It’s the Library of Congress. They had some of the best knowledge on this subject. If the Library of Congress can’t do that, that tells you something pretty important,” he says.
This is problematic, because Twitter is full of important content from the past 16 years that could help tomorrow’s historians understand the world today.
