Back up the open-model world.

Strategy: three concentric archives

Windows machine and storage design

• Launch set: 20 TiB usable is comfortable.
• Canonical org mirror: provision only after live inventory.
• All formats: use an expandable storage server/JBOD, not a fixed workstation disk.

Windows runbook

Download the four attached files from the Discord reply into C:\ModelArchiveTools: Setup-OpenModelArchive.ps1, archive_windows.py, discover_official_orgs.py, and manifest_windows.json.

Setup and launch set

# Put the attached files in C:\ModelArchiveTools
Set-ExecutionPolicy -Scope Process Bypass
cd C:\ModelArchiveTools
.\Setup-OpenModelArchive.ps1

# Inspect, download, and resume the audited 77-repository set
python .\archive_windows.py --manifest .\manifest_windows.json inventory
python .\archive_windows.py --manifest .\manifest_windows.json download
python .\archive_windows.py --manifest .\manifest_windows.json status

Full official mirrors

# Every canonical/original repo from trusted official organizations
python .\discover_official_orgs.py --output .\official_org_manifest.json
python .\archive_windows.py --manifest .\official_org_manifest.json inventory
python .\archive_windows.py --manifest .\official_org_manifest.json download

# Literal everything: official quants, conversions, and duplicate formats too
python .\discover_official_orgs.py --all-formats --output .\official_everything_manifest.json
python .\archive_windows.py --manifest .\official_everything_manifest.json inventory
python .\archive_windows.py --manifest .\official_everything_manifest.json download

Targeted operations

# Tier A first
python .\archive_windows.py --manifest .\manifest_windows.json --scopes A download

# One model
python .\archive_windows.py --manifest .\manifest_windows.json --repos moonshotai/Kimi-K2.7-Code download

# Stop on first error instead of logging and continuing
python .\archive_windows.py --manifest .\manifest_windows.json download --stop-on-error

Re-run unchanged after a reboot or network failure. Completed revisions are skipped and partial Hub/Xet chunks resume.

The audited 77-model launch set

Tier A is frontier/current/unique; Tier B adds strong breadth and size tiers; Tier C is large overlap or poor bytes-per-capability. The full instruction downloads all tiers—the letters only control order.

Sizes include every file in the pinned repository, not only tensor shards.

Everything mode: trusted official organizations

The crawler dynamically inventories every public model repository from these organizations. That is the durable way to capture new releases and long-tail specialists that no static list can keep current.

Default crawler

All first-party repos except obvious deployment repacks.

--all-formats

Every public model repo, including official GGUF, AWQ, GPTQ, FP8, ONNX, and MLX duplicates.

Outside Hugging Face

Preserve upstream-only releases separately; never replace them with unaudited community uploads.

Notable upstream-only release: xAI Grok-1 official source and weight instructions.

Gated models and licenses

A read token is not sufficient when click-through or manual approval is required. Open the model page while signed in, accept terms, wait for approval if needed, and rerun the same command.

• Meta Llama: manual Llama license approval.
• Google Gemma: Gemma terms.
• FLUX dev: gate plus non-commercial terms.
• Stability: community-license click-through.
• Aya: gated non-commercial variants.

Verification and auditability

1. Every launch model is pinned to a full immutable commit SHA.
2. hf download uses content-addressed Hub/Xet transfers and resumes interrupted chunks.
3. Each completed snapshot receives .open-model-archive.json with source, revision, expected bytes, and completion time.
4. status reports completed and pending bytes after any interruption.
5. Copy manifests and logs off-machine; they are the reconstruction map.

python .\archive_windows.py --manifest .\manifest_windows.json status
Get-Content "$env:OPEN_MODEL_ARCHIVE_ROOT\failures.json"
Get-ChildItem "$env:OPEN_MODEL_ARCHIVE_ROOT\models" -Recurse -Filter .open-model-archive.json | Measure-Object

Long-term operations

Monthly refresh

1. Rerun official-org discovery to a date-stamped manifest.
2. Diff repository IDs and revisions against the previous run.
3. Preserve old commits when main moves; a mutable sync is not an archive.
4. Download new revisions and protect any snapshot whose upstream disappears.

Failure policy

• 401/403: accept terms or refresh the read token.
• Disk full: add storage, then rerun; do not purge incomplete chunks mid-transfer.
• Network failure: rerun unchanged.
• Repository disappears: isolate and duplicate any completed snapshot immediately.

The honest definition of “everything”

“Every remotely good model” is not a stable finite list. The 77-row table is audited on the report date; the crawler is the reproducible long-term rule: mirror every public model released by trusted first-party labs, optionally including every format.