The ethics and mechanics of archiving internet culture properly

The internet is often described as a place where nothing is ever deleted, but the opposite is closer to true.

Most of what was posted is gone; most of what remains is unfindable; accidents unrelated to what mattered selected the surviving fraction. Existing archives are heroic and partial, so knowing which archive type you hold separates use from being misled.

This page describes the kinds, what each keeps and loses, how to cite from them without overclaiming, and the ethical problem of preserving what was meant to pass.

What to take away

The four kinds

The gaps are complementary, so serious work uses all four and trusts none alone.

The four kinds, by name

Two things sit underneath all four. Captures are usually stored in WARC, the standard archive file format, rather than as ordinary files, and Memento is the protocol that lets one request check several archives at once.

Webrecorder, the browser-based capture tool, records what a page does rather than only what it returns, which is why it can hold interactive material that a crawler sees as an empty frame. Perma.cc does the narrowest job of all: a permanent, citable address for a link in a document.

A crawl archive has a capture of a page but not the conversation around it. A community wiki has the conversation but asserts an origin it cannot support. A personal collection has the one file nobody else saved and no way to date it.

What survives, and why it is not random

The problem is usually described as technical, and it partly is: formats become unreadable, media degrades, software stops running. Archivists call that cluster of problems digital preservation. But the pattern of loss has a shape, and the shape biases everything built from the remains.

Text survives best. It is small, it is easy to copy, and it migrates between formats with almost no loss. A plain message from decades ago is as readable now as the day it was written.

Images survive moderately, with degradation. Each copy is re-encoded, cropped or captioned, and the version that survives is usually several generations from the original, with metadata stripped along the way.

Anything interactive or tied to a specific runtime survives worst. Script-built pages, plugins nobody runs, games, and animated formats are often captured as shells: the shell stays, the contents vanish.

The fear of a digital dark age, a period whose record successors cannot read, is a fear about this layer specifically.

Popular things were copied more, so they survived more. For this site's subject, that bias matters most: the surviving record over-represents what was already widely seen.

A history built from that record says the period was dominated by what happened to be saved. That proves nothing. The same circularity shapes any account of how a format spread, described under memes and virality.

And private things left nothing. The communities that were closed by design, which includes most small ones, produced conversation no crawler reached. A history of online community written from archives is a history of the public and the large.

Citing an archived item without overclaiming

An archived copy supports a narrow claim, and most citations stretch it. What a capture actually establishes:

A citation that respects these limits names the address, the capture date, the archive, and the date the citation was checked. It says which of the four kinds of archive it came from. That exceeds most citations yet is the minimum that lets a later reader reproduce the check.

A formatted citation reads like this, with the bracketed fields filled in: Title of the page — address — captured [capture date] — [archive name], for example Wayback Machine — checked [date you looked]. Take the capture date from the archive's own record, not from the page's own text or from memory, and never present it as the date the page was written.

The reasons the check must be reproducible and the ways archived material fails when built into a chronology are set out under platform timelines.

Provenance in a personal collection

The most valuable material is often the fourth kind: the personal collection, the file somebody saved that no crawler captured. It also has the least provenance.

A saved image carries no record of where it was saved from. Its file date is when it was written to that disk, possibly a copy of a copy. Its metadata may be original, stripped, or rewritten by whatever service it passed through.

What can be done: record the collector's memory as recollection, and record what the file itself says as file metadata. Look for the same item in one of the other three kinds to anchor a date.

What cannot be done is treat the file as dated. The early social networks page describes why so much of that period's record exists only in this form, and why its dates are the least reliable.

The ethics of keeping what was meant to pass

Much of what archives hold was posted by people who expected it to vanish, in places they thought were small, under names they have since abandoned. Preserving it makes a decision on their behalf, and the decision is not obviously right.

The argument for preservation is that the record of a period belongs to everyone who comes after. The people who posted are part of the history, and the alternative is a period with no record at all.

The argument against is that a message to a room of thirty was not a publication. The person who wrote it did not consent to a permanent public copy, and an archive can turn a discarded persona into a permanent identity for someone who moved on.

The field has not resolved this, and this site does not pretend to. What it does is decline to reproduce material from small or closed spaces, describe patterns rather than people, and treat visible as different from published. An archive is a tool, and the people inside it were not consulted about being in it.

What a person can do about material they want removed depends on where they are and who holds the copy. In the United States, a copyright holder can send a takedown notice under the Digital Millennium Copyright Act, and the Internet Archive accepts exclusion requests from site owners who do not want their pages replayed. In the European Union, the General Data Protection Regulation gives individuals a right to erasure. A site can also place a robots.txt exclusion, which crawlers that honor it read as an instruction not to capture. None of these routes resolves the ethics; they only decide who has to answer the request.

Common questions

If a page is not in the crawl archives, did it exist?

Very possibly. Coverage is uneven, skewed toward the public and popular, and the absence of a capture is the normal condition for most of what was ever posted.

Which kind of archive is most reliable?

For dates, the crawl archives, within their coverage. For context, the community catalogs, with their origin claims discounted. For material nobody else has, personal collections, with no reliable dates. Institutional deposits for stability. None for everything.

Can an archived copy be wrong?

A capture may show a page as served to a crawler, not as a person saw it, or a shell with content missing. It may show a version that was live for an hour.

It is accurate about what it captured, and silent about what it did not.

Should I archive material from a small community I belong to?

Ask the community. The answer will tell you something about the community, and it is their record.