Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

File Storage & Resilience

DataPrism accepts any file, any size. To do so it combines content-defined chunking, convergent encryption, erasure coding, content-addressing, and managed object storage. The result: storage only sees encrypted fragments, identical data is stored once, and missing shards can be reconstructed when the configured quorum is available.

This is the conceptual model. The SDK File Pipeline page documents the concrete TypeScript API (FilePipeline, ReedSolomon, chunkers).

Upload

   file (any size)
        │  ① CHUNKING (content-defined, FastCDC by default)
        │     boundaries defined by content → variable sizes
        │     (inserting bytes shifts only one chunk → robust dedup)
        ▼
   [chunk 1] [chunk 2] ... [chunk N]
        │  ② CONVERGENT ENCRYPTION
        │     chunk key = KDF( hash(plaintext chunk) )   (deterministic)
        │     → ciphertext  (same content ⇒ same ciphertext)
        ▼
   [ct 1] [ct 2] ... [ct N]
        │  ③ ERASURE CODING (Reed-Solomon) - on the CIPHERTEXT
        │     each chunk → k data shards + m parity shards
        │     (e.g. 6 total, quorum 4 → any 4 of 6 reconstruct)
        ▼
   [shard 1] [shard 2] [shard 3] [shard 4] [shard 5] [shard 6]
        │  ④ CONTENT-ADDRESSING + DEDUP
        │     shard key = hash(shard content)
        │     if the shard already exists in storage → skip upload
        ▼
        │  ⑤ MANAGED STORAGE
        │     each shard → its own content-addressed object
        ▼
   managed storage ◄─ shard1, shard2, shard3, shard4, ...
        │  ⑥ MERKLE MANIFEST
        │     the chunk list (hash + len + shard hashes) forms a "manifest";
        │     if it is large it is stored ITSELF as a blob (encrypted + RS +
        │     stored) → rolled up to a tiny ROOT
        ▼
   ⑦ ON-CHAIN WRITE (the root only)
        dataId = keccak256(filename)
        value  = encrypted root index (~few hundred bytes, CONSTANT size
                 regardless of file size)
        transaction sent via the execution layer (wallet OR api/relayer)

Why each step matters

  • Content-defined chunking (FastCDC). Chunk boundaries are chosen by the content via a rolling hash, so inserting or removing bytes shifts only one chunk instead of re-aligning the whole file. This keeps deduplication robust across file versions. Chunk sizes are therefore variable, so the index records the length of each chunk. The chunker is pluggable, FastCDCChunker (default) or a fixed-size chunker.

  • Convergent encryption. Each chunk is encrypted with a key derived from its own content (KDF(hash(chunk))). The same plaintext always yields the same ciphertext, so storage only ever sees ciphertext and deduplication still works. (Caveat: like all convergent schemes, it is deterministic, vulnerable to a confirmation-of-file attack.)

  • Erasure coding (Reed-Solomon). Each chunk's ciphertext is split into k data shards + m parity shards. Any k of the n = k + m shards reconstruct the chunk. The quorum (e.g. 4 of 6) sets the fault tolerance.

  • Content-addressing + dedup. Each shard is stored under the hash of its content. Before uploading, the SDK checks whether the shard already exists with a signed HEAD → if so, it is not re-uploaded. Two files (or versions) sharing identical chunks share their shards: zero duplication. It also gives integrity: any tampering changes the hash and is detectable.

  • Managed placement. Each shard is a separate content-addressed object. The SDK requests a short-lived, project-scoped URL for every storage operation.

  • Compact index (Merkle manifest). The routes are not all written on-chain. The chunk list is treated as a blob (encrypted, erasure-coded, stored, content-addressed), and only a root (BlobRef) goes on-chain. For a 1 TB file as for a 1 KB file, the on-chain index is ~a few hundred bytes. The real chunk list lives in managed storage, and everything is verifiable from the root.

The index structure

The on-chain index has the shape:

FileIndex = { name, size, chunkSize, quorum, providers[], root: BlobRef }
BlobRef   = { level, size, chunks: [ { hash, len, shards[] } ] }
  • level 0: the chunks are file data (small files are inline).
  • level > 0: the chunks reconstruct a manifest; the tree is descended on download.
  • providers[]: the ordered storage adapter IDs used at upload. The managed path records dataprism_managed; object access is resolved by the API from the user, project, and content hash.

Download / reconstruction

   filename
        │  ① hash → dataId
        ▼
   read the on-chain ROOT (via RPC) + decrypt it (prism key)
        │  ② DESCEND THE MERKLE TREE
        │     if root.level > 0: reconstruct the manifest(s) from the
        │     storage adapter, until the full data-chunk list is obtained
        ▼
   data-chunk list: for each → hash + len + shard hashes
        │  ③ fetch the shards from managed storage (in parallel)
        ▼
   gather at least `quorum` shards per chunk
        │  ④ RECONSTRUCT (Reed-Solomon) → the chunk CIPHERTEXT
        │     as long as quorum is reached, the chunk is recovered
        ▼
        │  ⑤ CONVERGENT DECRYPTION
        │     key = KDF( chunk hash, read from the index ) → plaintext chunk
        ▼
   reassemble the chunks
        ▼
   original file reconstructed

The quorum describes how many shards are needed to reconstruct each encrypted chunk. With the default managed-storage setup, all shards are authorized through the same platform service.

Current limits

Safe deletion of shared files (reference counting / garbage collection) is not yet implemented. An overwrite or delete may leave orphan shards behind (planned work).

Copyright © 2026 DataPrism.