Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

File pipeline

The file pipeline is the SDK's implementation of File storage & resilience. It combines content-defined chunking, convergent encryption, Reed-Solomon erasure coding, content-addressing (dedup), and a Merkle manifest (a compact index). The orchestrator is FilePipeline.

Where file operations live

File upload, download, and deletion run in your application through this SDK. The dashboard's Storage tab is a read-only view of the file indexes recorded for a project; it does not receive file contents or perform storage operations.

For one end-to-end setup, including all three operations, use the Node.js example.

Chunkers (pluggable)

interface Chunker { cut(data: Uint8Array): { offset: number; length: number }[] }
 
new FastCDCChunker({ avg, min?, max? })   // content-defined (default), variable sizes
new FixedChunker(size)                     // fixed size (fallback)

FastCDC uses a Gear rolling hash with normalized chunking, so inserting bytes shifts only one chunk: deduplication stays robust across file versions. Because chunk sizes vary, the index records each chunk's length.

Convergent encryption

contentHash(bytes): Hex                                      // 0x + sha256
generateScopeKey(): Uint8Array                              // persist client-side per prism
convergentSeal(algorithm, plaintext, { scopeKey }): { hash, ciphertext }
convergentOpen(algorithm, hash, ciphertext, { scopeKey }): Uint8Array

The key and nonce are derived from the plaintext hash and a random 32-byte scopeKey. Same content means the same ciphertext only for prisms that share that key, so deduplication works inside the chosen trust boundary without letting a provider test guessed files globally. The scopeKey is client-held: the SDK never writes it to the index, object names, the network, DataPrism, or storage providers. The caller must distribute and back it up with the prism's other secrets. The algorithm is taken from dataprism.encryption.

ReedSolomon

Systematic erasure coding over GF(2⁸) (Vandermonde, MDS). Any dataShards of the dataShards + parityShards total reconstruct the data. A shard is what the product pages call a fragment.

const rs = new ReedSolomon(dataShards, parityShards);
const all = rs.encode(dataShards); // k → n = k + m
const k = rs.reconstruct(shardsWithNullsForLost); // n (with nulls) → k

FilePipeline

new FilePipeline({
  providers: StorageProvider[],     // built from dataprism.providers via initProviders
  standard: CloudStandard,          // holds the key (algorithm + sealing)
  scopeKey?: Uint8Array,            // required for uploads and scoped v2 downloads; 32 bytes
  scopeKeyResolver?: FilePipelineScopeKeyResolver, // customer-owned historical resolver
  scopeKeyVersion?: number,         // default 1; recorded in chunk descriptors
  execution?: ExecutionLayer,       // to write the index to the network
  cloudId?: CloudId,
  source?: SlotSource,              // to detect "file already exists"
  manifestCommitter?: ManifestCommitter, // direct optimistic #65 updates
  onRegistryOperation?: FilePipelineRegistryObserver, // customer-side collection hook
  manifestThreshold?: number,       // default 8192 bytes
})

The pipeline captures the provider order at construction. Editing the original array does not change shard placement or custody facts; create another pipeline to use a different configuration. Provider instances and their capabilities stay customer-owned and are not copied or frozen.

Upload

upload(name, data, options): Promise<UploadResult>

UploadOptions = { chunkSize, dataShards, parityShards, custodyPolicy?, meta?, overwrite?, chunker?, scopeKeyVersion?, onProgress?, onShardOperation? }.

custodyPolicy defaults to "attested". Before writing, the SDK requires that placement survives loss of the most-loaded provider and that no provider holds a quorum. A 7+3 upload passes with four or five providers, but fails with three. Set custodyPolicy: "relaxed" explicitly only to accept weaker placement, such as a single-provider smoke test. It warns when either condition fails; it does not establish either guarantee. UploadResult includes the policy and custody facts. Uploads using publicIndex return PublicUploadResult: custody facts are present when the public index has root chunks and omitted for an empty upload. Existing files keep their recorded placement and remain readable.

The pipeline: chunk, convergent-encrypt, Reed-Solomon, address each shard by contentHash(shard), uploaded only if missing (dedup), disperse over providers[pos % n], build a ChunkDesc list, roll it up into a Merkle tree while it exceeds manifestThreshold, produce a compact BlobRef root, seal the FileIndex, and write it to the network (if execution plus cloudId were given).

onRegistryOperation is awaited after writeData, modifyData or deleteData returns. Its observation contains the operation, cloudId, dataId and unchanged ExecutionResult, including an API request ID or transaction hash when available. If the callback fails, FilePipelineRegistryObservationError retains that observation so the caller can reconcile the submitted operation without sending it again. After deletion, the error also retains the collected shard map and provider IDs so the caller can still purge the objects. A pipeline without this callback behaves as before.

onShardOperation is an optional customer-side collection hook. When present, the pipeline assigns a correlation ID to each shard has and put, passes it through the storage options, and awaits one file.upload.shard observation after the operation. The observation carries the file and shard coordinates, effective provider IDs, bounded outcome and bounded provider failure. A signed provider's separate StorageOperationReceipt uses the same correlation ID. Callback handoffs are serialized even though shard storage requests remain concurrent.

If the callback rejects, FilePipelineUploadObservationError retains the failed observation. For a missing-object check, the shard write is not started until the observation succeeds. Uploads without the hook leave provider correlation-ID generation unchanged.

Up-front validation: the name must be non-empty, must not hash to a reserved slot, and the file must not already exist (unless overwrite: true, which routes to modifyData). Custody checks use the shard counts and actual provider placement, not simply whether every shard has a different provider.

const providers = initProviders(
  await std.reader(client.reader, cloudId).providers(),
  signedUrlFactories({
    getSignerHeaders: async () => ({
      authorization: `Bearer ${await getStorageToken()}`,
    }),
  }),
);
// Application-owned helper: load exactly 32 bytes from the prism's secret
// store. Generate once with generateScopeKey() when provisioning the prism,
// then persist, distribute to authorized members, and back up that same value.
// This read path must fail if absent; it must never generate a replacement.
const scopeKey = await loadPrismScopeKey(cloudId);
const pipeline = new FilePipeline({
  providers,
  standard: std,
  scopeKey,
  execution: client.execution,
  cloudId,
  source: client.reader,
});
 
const { index, slot, write } = await pipeline.upload("report.pdf", bytes, {
  chunkSize: 65536,
  dataShards: 4,
  parityShards: 2,
});
await write?.wait(); // index committed to the network

UploadResult = { index: FileIndex; slot: SlotWrite; custodyPolicy: ProviderCustodyPolicy; providerCustody: ProviderCustodyAnalysis; write?: ExecutionResult }. The publicIndex overload returns PublicUploadResult, with a PublicFileIndex and optional providerCustody: PublicProviderCustodyAnalysis as described above. If you omit execution / cloudId, upload returns the index and slot without writing them; you can persist them yourself.

If a provider write or the index commit fails, upload throws FileUploadIncompleteError. Its ordered attempts list records the provider, content-addressed shard key, data or manifest level, operation, and whether the shard was already present, stored, or has an unknown outcome. A failed request is unknown because it may have reached the provider. The error exposes only a bounded failure class, never the provider's raw error message, credentials, or signed URL. On retry, the presence check skips a content-addressed shard whose write completed before its response was lost. After an index-phase failure, read the slot first and use overwrite: true if the commit landed.

New uploads write file-index version 2 and record scope-key-v1 plus the scopeKeyVersion in each chunk descriptor. Unversioned legacy indexes remain readable with the old content-hash-only derivation; they are never produced by new uploads. An unsupported future index version fails closed.

For versioned customer self-custody, pass a bound TenantScopeKeyring instead of a raw key:

await keys.create();
const pipeline = new FilePipeline({
  providers,
  standard: std,
  scopeKeyResolver: keys,
});
 
const first = await pipeline.upload("first.bin", firstBytes, {
  chunkSize: 65536,
  dataShards: 4,
  parityShards: 2,
}); // version 1
 
await keys.rotate();
const second = await pipeline.upload("second.bin", secondBytes, {
  chunkSize: 65536,
  dataShards: 4,
  parityShards: 2,
  scopeKeyVersion: 2,
});
 
await pipeline.download(first.index); // resolves version 1
await pipeline.download(second.index); // resolves version 2

scopeKey and scopeKeyResolver are mutually exclusive. The resolver is asked for the exact descriptor version and never falls forward to a newer key. A resolved version is cached once per operation; resolver-owned and operation copies are cleared afterward. Custom resolvers must therefore return a fresh, caller-owned 32-byte array. Missing, destroyed, invalid and failed resolutions use FilePipelineScopeKeyError without retaining the adapter error or key material.

This resolver bridge preserves version 2 indexes; it does not activate public file-index version 3. Resolver-backed health checks require an explicit raw key and version in the method options. Repair requires a pipeline configured with the exact raw key and version.

Download

download(index, options?): Promise<Uint8Array>          // DownloadOptions = { providers?, onProgress?, onShardOperation? }
downloadFromPayload(payload, options?): Promise<Uint8Array>

Descends the Merkle tree: reconstructs manifests down to the data chunks, fetches shards in parallel (outages become null), Reed-Solomon reconstructs, convergent-decrypts, reassembles. As long as a quorum of shards per chunk is reachable, the file is recovered.

When onShardOperation is set, every attempted shard get receives a caller correlation ID. The awaited file.download.shard observation identifies the file, sealed provider placement, object key and manifest coordinates, then records loaded, hash-mismatch or a bounded provider failure. It can be joined to a signed provider's separate response receipt by correlation ID. Callback handoffs are serialized even though reads remain concurrent. If the callback rejects, FilePipelineDownloadObservationError retains the completed operation. Downloads without the hook leave provider correlation-ID generation unchanged.

const view = std.reader(client.reader, cloudId);
const index = await view.file("report.pdf"); // read and decrypt the index
const bytes = await pipeline.download(index); // gather a quorum and reconstruct

downloadFromPayload(payload) is standard.readFile(payload) followed by download.

Shard health

verifyHealth(index, options?): Promise<FileShardHealthReport>

The existing metadata and full checks are unchanged. Set onShardOperation to receive one awaited file.health.shard observation for each completed check. It carries the data ID, optional cloud ID, the provider IDs recorded in the index and the existing ShardHealthEvidence result. Each provider attempt, including retries, has its own correlation ID; the evidence retains those IDs in order and records whether the provider operation was has or get.

The IDs join individual attempts to signed-provider receipts. A missing provider configuration has no ID because no request was made. Callback handoffs are serialized while checks retain their configured concurrency. A rejected callback throws FilePipelineShardHealthObservationError with the completed observation. Without the hook, provider correlation-ID generation is unchanged.

Migrations

FileIndex.providers freezes only the ordered provider-ID list, not the location, bucket, or signer endpoint; those come from the live providers passed to the pipeline. Because shards are content-addressed, their keys are stable, so a migration is:

  1. Copy the shards, by their content-hash key, to the new provider.
  2. Download with an override that replaces a provider by ID:
const bytes = await pipeline.download(index, { providers: [newAwsProvider] });
// override > default, matched by provider id

No re-encryption, no index change.

Repair and rebalance

repair uses a complete full shard-health verification, authenticates every manifest and data chunk, reconstructs the committed shards, writes and reads back the selected placement, and only then advances the #65 manifest version with the expected lineage and version. The previous manifest remains authoritative until that transaction commits.

const pipeline = new FilePipeline({
  providers: allCurrentAndReplacementProviders,
  standard: std,
  cloudId,
  source: client.reader, // must expose getDataEvidence
  manifestCommitter: client.manifestCommitter, // fails closed outside direct account mode
});
 
const receipt = await pipeline.repair("report.pdf", {
  selectProviders: ({ index, failedProviderIds }) =>
    chooseApprovedPlacement(index, failedProviderIds),
  minimumToleratedProviderLoss: approvedMinimum,
  verifyContentCommitment: ({ bytes, contentCommitment }) =>
    verifyCustomerCommitment(bytes, contentCommitment),
  onRegistryOperation: (observation) => repairCollector.capture(observation),
});

Provider selection, the minimum loss margin, and content-commitment derivation are deliberately caller-supplied policy. The SDK does not silently select a customer risk posture. A placement that reduces the known margin requires an explicit riskAcceptance.reason. Relayed EIP-712 updateManifest is not implied; the bundled committer is the direct AccountExecution path.

The receipt links the unchanged lineage/content commitment, old and new manifest versions, provider mappings, shard hashes, timestamps, and transaction evidence. It contains no plaintext, encryption key, provider credential, or sealed manifest. Repeating a completed repair is a no-op; partial object writes are content-addressed and safe to retry, while a stale worker is rejected by the optimistic on-chain version check.

Shard writes use bounded concurrency (eight provider operations by default). Set concurrency from 1 to 128 to tune that limit; receipt evidence and write progress remain in deterministic manifest order.

Pass onShardOperation to correlate repair work with provider receipts. Health, reconstruction, write and read-back operations receive caller correlation IDs; the awaited file.repair.shard observations retain those IDs in retry order, along with the phase, provider, key, outcome, effective placement and shard coordinates when known. Callback handoffs are serialized without reducing provider concurrency. If the callback rejects, ShardRepairOperationObservationError retains the completed observation.

When provider placement changes, onRegistryOperation is awaited after the manifest update is submitted and before its confirmation is awaited. The observation includes the returned execution result, the previous and expected manifest versions, and both provider placements. If the callback rejects, ShardRepairRegistryObservationError retains that submitted result so the caller can follow it without starting a second repair. No-op and in-place repairs do not call this hook.

Cleanup is separate because content-addressed objects may be shared:

await pipeline.cleanupRepair(receipt, {
  canDelete: (candidate) => referenceIndex.provesUnreferenced(candidate),
});

Cleanup first confirms that the receipt's exact lineage, version, and index are still authoritative. canDelete is required for every object; without a complete reference/liveness view the old shard is skipped rather than deleted. Reference checks, deletes, and delete verification are bounded by requestTimeoutMs; provider failures use the same retries and retryDelayMs controls as repair, and cancellation stops before the next destructive step. Custom providers must honor the optional abort signal passed to delete so a timed-out cleanup cannot complete later against newer authority. cleanupRepair accepts the same onShardOperation hook for provider deletes and presence checks. Missing provider configuration is reported without a correlation ID because no provider request occurred.

Deletion

Deletion mirrors upload in reverse and comes in three levels of radicality. The concept page explains the semantics (why unreferencing is not erasing); this is the API.

delete(name, options?) removes the file from the prism. It first reads the index and walks the manifest levels to enumerate every shard key (data and manifests), then deletes the network entry via execution.deleteData.

const deletion = await pipeline.delete("report.pdf");
await deletion.write.wait();
// deletion.shards: ShardLocation[] = { providerId, key }[]

Save deletion if you may want to purge later: the index was the only map of the file, so once it is deleted the shard list cannot be recomputed. Pass { collectShards: false } to skip enumeration and delete the index only (no source required).

deleteById(dataId, options?) accepts a known 32-byte entry ID, such as one returned by getDataKeys, and does not derive the file name. Pass { collectShards: false } if the scope key needed to traverse sealed manifests has already been destroyed.

purge(shardsOrDeleteResult, options?) physically deletes specific shard keys. It is tolerant and replayable: every key is attempted, failures are collected instead of thrown, and provider deletes are idempotent, so retrying is always safe.

const result = await pipeline.purge(shards, { onProgress: (done, total) => ... });
// { deleted: number, failed: { providerId, key, failure, error }[] }
if (result.failed.length > 0) await pipeline.purge(result.failed);   // retry later

Pass the complete deletion result to keep each provider operation attributable:

const result = await pipeline.purge(deletion, {
  onShardOperation: async (observation) =>
    customerCollector.append(observation),
});

Each awaited file.purge.shard observation includes the data ID, optional cloud ID, provider placement, shard location, outcome and bounded failure classification. A configured-provider delete also has a caller correlation ID that matches its StorageOperationReceipt. Provider deletes run concurrently; callback handoffs run in the original shard-list order. FilePipelinePurgeObservationError retains the complete purge result and the observations that still need to be handed over if the callback fails. The array-only form remains available when no observer is used.

failure is a bounded machine-readable reason such as unauthorized, rate_limited, timed_out, unknown, or provider_not_configured. error is an SDK-generated summary; raw adapter messages, signed URLs, and credentials are never copied into the result. A failed delete does not prove the object remains reachable.

collectShards(index, options?) exposes the enumeration alone, with no deletion, for inspection or for building your own tooling.

Garbage collection: purgeOrphans

A mark-and-sweep over one or more prisms. Mark: iterate each prism's data keys, decrypt its file indexes, and accumulate the set of live shard keys. Sweep: list() each provider and delete every shard-shaped object that is not in the live set. This is a server-side maintenance operation: use a customer-controlled connector that implements list() and holds the required administrative access. SignedUrlProvider deliberately exposes only object-scoped operations and cannot list a bucket. The truth is recomputed from the prism indexes on every run, so there are no counters to drift out of sync.

The sweep starts only after every non-empty, non-reserved data entry has been opened and all nested manifest shards have been collected. An unreadable key, malformed index, or unreconstructable manifest throws PurgeOrphansIncompleteMarkError; provider objects are not listed or deleted. Prisms that mix FilePipeline indexes with other non-reserved data are not a supported garbage-collection scope, because the collector cannot safely prove that an unknown entry references no shards.

Each prism can supply a scopeKeyResolver, such as its customer-owned TenantScopeKeyring, instead of a raw scopeKey. Nested manifests resolve the exact versions they record, even after rotation. Keys are cached only for one file traversal and cleared on success or failure. A missing, destroyed or unavailable manifest key aborts the entire mark with PurgeOrphansIncompleteMarkError before any provider listing or deletion. Flat indexes and legacy unscoped manifests do not need scope-key resolution; collecting shard references is not a check that the file remains readable.

Do not set both scopeKey and scopeKeyResolver. Raw-key callers can continue to supply scopeKeyVersion (default 1). The resolver must return a fresh caller-owned copy for the requested version; the SDK clears that copy after copying it into its operation-local cache.

import { purgeOrphans } from "@dataprism/sdk/files";
 
const report = await purgeOrphans({
  prisms: [{ standard: std, cloudId, scopeKeyResolver: tenantKeys }], // every prism using these buckets
  providers, // must implement list()
  source: client.reader, // SlotSource + iterateDataKeys
  minAgeMs: 6 * 3600 * 1000, // grace period (default 6 h)
  dryRun: false, // true = report candidates, delete nothing
  onSelection: async (observation) => {
    await customerAuditSink.record(observation);
  },
  onDeletion: async (observation) => {
    await customerAuditSink.record(observation);
  },
});
// { filesScanned, liveShards, scannedObjects, orphans, deleted, failed, skippedRecent }

The optional onSelection callback receives one frozen observation for each shard-shaped object returned by a provider. It preserves the provider ID and exact object key, and distinguishes these decisions:

  • retained-live: the object is referenced by an index in the supplied prisms.
  • retained-recent: its modification time falls within the grace period or is in the future.
  • retained-unknown-age: no modification time was supplied.
  • retained-invalid-age: the supplied date is invalid.
  • selected-candidate: it has no reference in the supplied prism set and passes the age check. This decision is also emitted in a dry run.

With minAgeMs: 0, every unreferenced shard-shaped object is a candidate, regardless of its age. At a positive threshold, an object exactly as old as minAgeMs is eligible. The existing skippedRecent result count includes recent, unknown-age and invalid-age objects; the callback keeps those reasons separate.

Each observation includes the supplied prismRefs, minAgeMs, dryRun and observedAt. The last field is the SDK clock at the start of selection, not a qualified timestamp. Callbacks are awaited serially in provider listing order. Provider IDs must be unique: ambiguous provider instances are rejected before marking, listing or deletion, whether or not callbacks are configured. If one throws or rejects, PurgeOrphansSelectionObservationError retains that observation and the original cause; discovery stops before any deletion. Earlier callbacks may already have completed. The SDK does not persist these observations or certify that a customer sink retained them.

A selection decision is not a deletion result. The optional onDeletion callback records one settled provider delete call per candidate, with the exact provider/key, supplied prismRefs, operation: "delete", SDK completion time (observedAt) and either outcome: "deleted" or outcome: "failed" with a bounded failure code. Success describes the call, not erasure of every version, replica or backup. Dry runs and retained objects produce no deletion observations.

All bounded-concurrency delete calls settle before outcome callbacks are awaited serially in discovery order. If a callback fails, PurgeOrphansDeletionObservationError retains the completed sweep in result, the failed observation, its original cause, and unreported observations whose callbacks have not run. Retry recording from that retained evidence, not by repeating the physical sweep. Check the sink first after an uncertain append: failure to receive an acknowledgement does not prove the append failed.

The operation snapshots its prism and provider inputs, source entry points, callbacks and selection settings before awaiting work. Callbacks cannot turn a dry run into deletion by changing the options or alter the frozen decisions. Provider implementations still own their internal state and listing behavior.

The sweep can reclaim shards left by deletes, interrupted uploads and overwrite. Its live set covers only the supplied prism indexes, and its observations cover only the objects returned by list(). Neither is an independent check of complete bucket ownership or listing completeness. Concurrent writes can change the referenced or listed objects while the scan runs. The grace period retains recent objects but does not make marking and listing an atomic snapshot; coordinate maintenance with writers when that boundary matters.

Copyright © 2026 DataPrism.