Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

File storage & resilience

DataPrism accepts any file, any size. To do so it combines content-defined chunking, convergent encryption, erasure coding, content-addressing, and multi-cloud dispersion. The result: providers only ever see encrypted fragments, identical data is stored once, and a file survives the loss of any minority of providers.

This is the conceptual model. The File pipeline page documents the concrete TypeScript API.

Upload

   file (any size)
        │  ① CHUNKING (content-defined, FastCDC by default)
        │     boundaries defined by content → variable sizes
        │     (inserting bytes shifts only one chunk → robust dedup)
        ▼
   [chunk 1] [chunk 2] ... [chunk N]
        │  ② CONVERGENT ENCRYPTION
        │     chunk key = KDF( hash(plaintext chunk) )   (deterministic)
        │     → ciphertext  (same content ⇒ same ciphertext)
        ▼
   [ct 1] [ct 2] ... [ct N]
        │  ③ ERASURE CODING (Reed-Solomon), on the CIPHERTEXT
        │     each chunk → k data shards + m parity shards
        │     (e.g. 6 total, quorum 4 → any 4 of 6 reconstruct)
        ▼
   [shard 1] [shard 2] [shard 3] [shard 4] [shard 5] [shard 6]
        │  ④ CONTENT-ADDRESSING + DEDUP
        │     shard key = hash(shard content)
        │     if the shard already exists at the provider → skip upload
        ▼
        │  ⑤ DISPERSION
        │     each shard → a provider (round-robin over dataprism.providers)
        ▼
   S3 ◄─ shard1   GCP ◄─ shard2   Azure ◄─ shard3   R2 ◄─ shard4  ...
        │  ⑥ MERKLE MANIFEST
        │     the chunk list (hash + len + shard hashes) forms a "manifest";
        │     if it is large it is stored ITSELF as a blob (encrypted + RS +
        │     dispersed) → rolled up to a tiny ROOT
        ▼
   ⑦ NETWORK WRITE (the root only)
        dataId = hash(filename)
        value  = encrypted root index (~a few hundred bytes, CONSTANT size
                 regardless of file size)
        sent via the execution layer (direct account OR delegated relayer)

Why each step matters

  • Content-defined chunking (FastCDC). Chunk boundaries are chosen by the content via a rolling hash, so inserting or removing bytes shifts only one chunk instead of re-aligning the whole file. This keeps deduplication robust across file versions. Chunk sizes are therefore variable, so the index records the length of each chunk. The chunker is pluggable: FastCDCChunker (default) or a fixed-size chunker.

  • Convergent encryption. Each chunk is encrypted with a key derived from its own content, KDF(hash(chunk)). The same plaintext always yields the same ciphertext, so providers only ever see ciphertext and deduplication still works. (Caveat: like all convergent schemes, it is deterministic, so it is vulnerable to a confirmation-of-file attack when the attacker already holds the exact plaintext.)

  • Erasure coding (Reed-Solomon). Each chunk's ciphertext is split into k data shards plus m parity shards. Any k of the n = k + m shards reconstruct the chunk. The quorum (for example 4 of 6) sets the fault tolerance.

  • Content-addressing plus dedup. Each shard is stored under the hash of its content. Before uploading, the SDK checks whether the shard already exists through an operation-scoped signed HEAD URL; if so, it is not re-uploaded. Two files, or two versions, sharing identical chunks share their shards: zero duplication. It also gives integrity: any tampering changes the hash and is detectable.

  • Dispersion. Each shard goes to a different provider, round-robin over the ordered provider list. The file is never in one place.

  • Compact index (Merkle manifest). The routes are not all written to the network. The chunk list is treated as a blob (encrypted, erasure-coded, dispersed, content-addressed), and only a root goes to the network. For a 1 TB file as for a 1 KB file, the stored index is a few hundred bytes. The real chunk list lives at the providers, deduplicated and resilient, and everything is verifiable from the root.

The index structure

FileIndex = { name, size, chunkSize, quorum, providers[], root: BlobRef }
BlobRef   = { level, size, chunks: [ { hash, len, shards[] } ] }
  • level 0: the chunks are file data (small files are inline).
  • level > 0: the chunks reconstruct a manifest; the tree is descended on download.
  • providers[]: the ordered list of provider IDs used, frozen at upload. A shard is located by providers[position % providers.length] plus its content hash. Only the IDs are frozen, not the location, bucket, or signer endpoint, which come from the live dataprism.providers block at download time.

Download / reconstruction

   filename
        │  ① hash → dataId
        ▼
   read the ROOT from the network (via RPC) + decrypt it (prism key)
        │  ② DESCEND THE MERKLE TREE
        │     if root.level > 0: reconstruct the manifest(s) from the
        │     providers, until the full data-chunk list is obtained
        ▼
   data-chunk list: for each → hash + len + shard hashes
        │  ③ fetch the shards from the providers (in parallel)
        │     provider = providers[position % n]  (list frozen in the index)
        ▼
   gather at least `quorum` shards per chunk
        │  ④ RECONSTRUCT (Reed-Solomon) → the chunk CIPHERTEXT
        │     if a provider is down (e.g. 1 of 6 KO), move to the next;
        │     as long as quorum is reached, the chunk is recovered
        ▼
        │  ⑤ CONVERGENT DECRYPTION
        │     key = KDF( chunk hash, read from the index ) → plaintext chunk
        ▼
   reassemble the chunks
        ▼
   original file reconstructed

This is what makes DataPrism resilient: the failure, or the censorship, of one provider does not lose the file, as long as a quorum of healthy providers is reachable.

Migrations (provider override)

Because shards are content-addressed, their keys are stable. A migration is therefore:

  1. Copy the shards (by their content hash) to the new provider.
  2. Download with an override that points to the new provider for that ID.

No re-encryption, no index change. The SDK exposes this as download(index, { providers }), where the override takes precedence over the default provider for a given ID. See File pipeline.

Deletion

A file lives in two places, so deletion is a two-layer operation:

  • Unreference (delete): remove the sealed index from the prism's state. The file disappears from the prism and nobody can list or fetch it anymore. The encrypted shards stay at the providers as harmless orphans.
  • Erase (purge / purgeOrphans): physically delete the shards from the buckets. This is the only real deletion, for a reason worth understanding:

Because shards are deduplicated by content, a shard may be shared by several files, or by several prisms writing to the same bucket. Blindly purging one file's shards could therefore corrupt another file. The SDK offers two safe paths: purge only shard keys you know are unshared, or run the garbage collector (purgeOrphans), a mark-and-sweep that recomputes the set of live shards from every prism's indexes and deletes only what nothing references. See File pipeline for the API.

Copyright © 2026 DataPrism.