Rules
Part of The ethics and mechanics of archiving internet culture properly
Five recognizable gaps in internet culture archives, and how to spot them
Five archive gaps and what each looks like from outside: the login wall, the assembled page, missing media, an overwritten capture, a closed community.
An archive gap does not look like a gap. It looks like a result: a page that loads, a search that returns things, a record that appears complete. Recognizing the shape of what is missing is a skill, and the internet culture archives examples below — drawn from the Wayback Machine, archive.today and Common Crawl — are where you learn it.
The five gaps below each get a signature and a case, and the cases are the ones you will meet first: GeoCities neighborhoods, MySpace profiles, LiveJournal's friends-locked entries, Vine's hosted video, and the Facebook groups where so much fan activity ended up. They recur across the Wayback Machine, archive.today, Common Crawl, Archive Team's rescue projects and the national web archives run by bodies such as the British Library. The tool changes; the shape of the hole does not.
What to take away
- Five gaps cover most of itthe login wall, the assembled page, the separately hosted asset, the overwritten capture, and the closed community.
- Each produces a distinctive appearance in the record, and four of the five look like presence rather than absence.
- Writing honestly about a gap means naming which one you hit, not writing "no records were found".
- A gap is evidence about the collecting system. It is almost never evidence about the past.
The five signatures
| Gap | What you see | What is actually missing | What you can still say |
|---|---|---|---|
| Login wall | A sign-in page, or a stub with a title and nothing else | Everything the account holder saw, which was most of the site | That the address existed and required an account by that date |
| Assembled page | An empty frame, a loading message, a navigation bar with no content | The content, which was built in the reader's browser and never sat in the file | That a page was served at that address, and nothing about what it showed |
| Separately hosted asset | Text intact, images and media broken or absent | The media, which lived on a different host the crawler did not follow | That the text referred to media, and what the surrounding text says about it |
| Overwritten capture | A capture that looks fine and is from the wrong time | The earlier state, which the newer crawl replaced under the same address | What the address held on the capture date you actually have |
| Closed community | Nothing at all, or a members-only notice, or a name in someone else's post | All of it, permanently | That the space existed, from external mentions, and nothing about its contents |
Only the last row looks like absence. The other four return something, and the something is routinely written up as though it were the thing.
Four gaps that return something
Login wall
- What you see
- Sign-in page or stub
- What is missing
- Everything behind account
- What you can say
- Address existed, account required
Assembled page
- What you see
- Empty frame or loading message
- What is missing
- Browser-built content
- What you can say
- Page served, content unknown
Hosted asset
- What you see
- Text intact, media broken
- What is missing
- Media on other host
- What you can say
- Text referred to media
Overwritten
- What you see
- Capture looks fine, wrong time
- What is missing
- Earlier state replaced
- What you can say
- What address held on capture date
Reading each one
Login wall. The tell is a captured page far shorter than the surrounding pages, often just a title and a form. LiveJournal entries locked to friends, the members-only communities on the same site, and the posts inside early Facebook groups all sit here: readable to a signed-in member, invisible to a crawler. Any account of a period built from crawls is therefore an account of its public surface, and the surface was never where most of the activity was. What that does to a history of a group is set out on our page about internet communities.
Spotting each gap type
- Login wallcaptured page far shorter than surrounding pages
- Assembled pagemenus and footers present, middle empty
- Separately hosted assettext refers to something you cannot see
- Overwritten capturecapture looks fine but is from wrong time
- Closed communitynothing, members-only notice, or external mention
Assembled page. The tell is structure with no substance: menus and footers present, the middle empty. The content was fetched by script after the page loaded, and the crawler saved the shell. It is the standard failure for single-page apps and for feeds that load on scroll, and it grew as that technique became standard, so the record thins in a way that has nothing to do with how much was posted. A crawler receiving a page and a person receiving a page are getting different things, and the raw crawl files that Common Crawl publishes show exactly what the crawler got: the shell.
Separately hosted asset. The tell is text that refers to something you cannot see. Images, audio and video were commonly served from a different host, and the crawler's scope stopped at the boundary; the Photobucket and ImageShack links that filled 2000s forum posts, and the embedded players that pointed at another site, are the classic cases. The text often describes the missing item well enough to work with, and using that description while saying it is a description is the honest move.
Overwritten capture. The tell is that the content does not match the date you expected, or that a page shows a later design than the period you are studying. Where an archive keeps several captures this is recoverable by looking at the list: the Wayback Machine's calendar of captures and its CDX capture index both let you compare one date against another. Where it keeps only one, the earlier state is gone. Recording the capture date rather than only the address is what protects against reporting this as the period's state, which is part of the reference discipline set out on our page about internet culture archives.
Closed community. The tell is external evidence with no internal evidence: a name that appears in other people's posts, an address that never resolved for outsiders, a description with no primary material behind it. Invite-only forums, boards behind a members' login, and the group chats and mailing lists that most small communities actually used all land here; Yahoo Groups, which shut down and took its member archives with it, is the largest single example.
This one is total, permanent, and much more common than the others, because closed was the ordinary condition of a small group. The consequence, which is worth stating flatly, is that the surviving record over-represents whatever was open, and there is no correction for it. The way that skews any origin claim is covered on our page about memes and virality.
How to write the gap down
Replace "no records were found" with something that says what you looked for and what stopped you: the tool or crawler, the capture date, the address, and whether the crawl's scope or a robots exclusion kept the material out.
Rewriting absence as method
Instead of
- Login wall
- No records were found
- Assembled page
- No records were found
- Separately hosted asset
- No records were found
- Overwritten capture
- No records were found
- Closed community
- No records were found
Write
- Login wall
- Pages captured as sign-in prompts
- Assembled page
- Structure without browser-assembled content
- Separately hosted asset
- Text survives, images outside crawl
- Overwritten capture
- Only capture postdates the period
- Closed community
- Not publicly reachable, external mentions only
Each of those tells a later reader what to try next, which "no records" does not. It also stops absence from being read as evidence nothing happened, the error these gaps most reliably produce.
The general form, treating the surviving sample as the population, is survivorship bias. It is the single most common fault in writing about this period.
What a gap can and cannot support
A gap supports a claim about the collecting system: this material was not reachable by this method at this time. That is a real finding and worth stating.
A gap does not support a claim about the world. It cannot show something did not exist, a community was small, a practice was rare, or a term was not yet in use.
From an absence, a careful writer gets only a dated statement about their own search, with the method named: the crawler, the capture date, the scope. The limits that no amount of care removes are set out on our page about platform timelines.
Common questions
Can any of these gaps be repaired later?
Sometimes, from the other kinds of collection. A personal collection may hold what a crawl missed, and a participant's account may supply context nothing else has. Neither is dated reliably.
Which gap is most often mistaken for evidence?
The assembled page, because it returns a real capture with a real date. It is a capture of a container, and it is regularly cited as though it were a capture of what the container held.
Is there a way to tell an overwritten capture from a complete record?
Look for a list of captures rather than a single one, and compare designs across dates. In the Wayback Machine that means opening the calendar of captures for the address and stepping through them; in other archives, it means checking whether more than one snapshot exists at all. If the archive holds a single capture, you cannot tell, and the honest citation says how many captures you saw.
Do these gaps get better or worse over time?
Both. Coverage of open, static material has improved. The share of material that is behind a login or assembled in the browser has grown, which moves more of the record into the first two rows.
