Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

File Pipeline

The file pipeline is the SDK's implementation of File Storage & Resilience. It combines content-defined chunking, convergent encryption, Reed-Solomon erasure coding, content-addressing (dedup), and a Merkle manifest (a compact on-chain index). The orchestrator is FilePipeline.

Chunkers (pluggable)

interface Chunker { cut(data: Uint8Array): { offset: number; length: number }[] }
 
new FastCDCChunker({ avg, min?, max? })   // content-defined (default), variable sizes
new FixedChunker(size)                     // fixed size (fallback)

FastCDC uses a Gear rolling hash with normalized chunking, so inserting bytes shifts only one chunk, robust deduplication across file versions. Because chunk sizes vary, the index records each chunk's length.

Convergent encryption

contentHash(bytes): Hex                                      // 0x + sha256
convergentSeal(algorithm, plaintext): { hash, ciphertext }  // key = KDF(hash), deterministic
convergentOpen(algorithm, hash, ciphertext): Uint8Array

Same content ⇒ same ciphertext ⇒ deduplication works and shards are encrypted at rest. The algorithm is taken from the prism's dataprism.encryption config.

ReedSolomon

Systematic erasure coding over GF(2⁸) (Vandermonde, MDS). Any dataShards of the dataShards + parityShards total reconstruct the data.

const rs = new ReedSolomon(dataShards, parityShards);
const all = rs.encode(dataShards);                  // k → n = k + m
const k   = rs.reconstruct(shardsWithNullsForLost); // n (with nulls) → k

FilePipeline

new FilePipeline({
  providers: StorageProvider[],     // ManagedStorageProvider in browser applications
  standard: CloudStandard,          // holds the key (algorithm + sealing)
  execution?: ExecutionLayer,       // to write the index on-chain
  cloudId?: CloudId,
  source?: SlotSource,              // to detect "file already exists"
  manifestThreshold?: number,       // default 8192 bytes
})

Upload

upload(name, data, options): Promise<UploadResult>

UploadOptions = { chunkSize, dataShards, parityShards, meta?, overwrite?, chunker? }.

The pipeline: chunk → convergent-encrypt → Reed-Solomon → address each shard by contentHash(shard), uploaded only if missing (dedup) → disperse over providers[pos % n] → build a ChunkDesc list → roll it up into a Merkle tree (putBlob) while it exceeds manifestThreshold → produce a compact BlobRef root → seal the FileIndex → write it on-chain (if execution + cloudId were given).

Up-front validation: the name must be non-empty, must not hash to a reserved slot, and the file must not already exist (unless overwrite: true, which routes to modifyData). It warns if providers.length < dataShards + parityShards (shards would share a provider, reducing real fault tolerance).

const providers = [new ManagedStorageProvider({
  apiUrl: "https://api.dataprism.co",
  projectId: cloudId,
  getAccessToken,
})];
const pipeline  = new FilePipeline({
  providers, standard: std, execution: client.execution, cloudId, source: client.reader,
});
 
const { index, slot, write } = await pipeline.upload("report.pdf", bytes, {
  chunkSize: 65536, dataShards: 1, parityShards: 0,
});
await write?.wait();   // index committed on-chain

UploadResult = { index: FileIndex; slot: SlotWrite; write?: ExecutionResult }. If you omit execution/cloudId, upload returns the index and slot without writing, you can persist them yourself.

Download

download(index, options?): Promise<Uint8Array>          // DownloadOptions = { providers? }
downloadFromPayload(payload, options?): Promise<Uint8Array>

Descends the Merkle tree (getBlob): reconstructs manifests down to the data chunks → fetches shards in parallel (outages → null) → Reed-Solomon reconstruct → convergent-decrypt → reassemble. As long as a quorum of shards per chunk is reachable, the file is recovered.

const view  = std.reader(client.reader, cloudId);
const index = await view.file("report.pdf");   // read + decrypt the index
const bytes = await pipeline.download(index);   // gather ≥ quorum → reconstruct

downloadFromPayload(payload) is standard.readFile(payload) followed by download.

Limit: safe deletion (reference counting / garbage collection) is not yet implemented, so an overwrite or delete can leave orphan shards behind.

Copyright © 2026 DataPrism.