Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Storage providers

Storage providers connect the SDK to the object stores where encrypted file shards live. The file pipeline drives them; each provider only ever sees ciphertext under hash-shaped keys.

The StorageProvider interface

interface StorageObject {
  key: string;
  size?: number;
  lastModified?: Date;
}
 
interface StorageReadOptions {
  signal?: AbortSignal;
  correlationId?: string;
}
 
interface StorageWriteOptions {
  signal?: AbortSignal;
  correlationId?: string;
}
 
interface InspectStorageConfigurationOptions {
  signal?: AbortSignal;
}
 
interface StorageProvider {
  readonly id: ProviderId | string;
  readonly writeCapabilities?: {
    abortSignal: true;
    putIdempotency: "none" | "object-key";
  };
  put(
    key: string,
    data: Uint8Array,
    options?: StorageWriteOptions,
  ): Promise<void>;
  get(key: string, options?: StorageReadOptions): Promise<Uint8Array>; // throws if absent
  has(key: string, options?: StorageReadOptions): Promise<boolean>; // existence check used for deduplication
  delete(key: string, options?: StorageReadOptions): Promise<void>; // idempotent
  inspectConfiguration?(
    options?: InspectStorageConfigurationOptions,
  ): Promise<StorageConfigurationInspection>;
  list?(prefix?: string): AsyncIterable<StorageObject>; // optional
}

Shard repair writes only through providers that declare abortSignal: true and honor the supplied signal. A timed-out write is retried only when the provider also declares putIdempotency: "object-key"; otherwise repair checks whether the first attempt produced the expected content-addressed object and fails without issuing a second write.

SignedUrlProvider defaults to putIdempotency: "object-key". Set putIdempotency: "none" when the customer signer cannot guarantee that repeating the same object-key write is safe.

has(key) powers deduplication: before uploading a shard, the pipeline checks whether it already exists and skips the upload if it does. With SignedUrlProvider, this uses an operation-scoped signed HEAD URL.

list(prefix?) is optional and powers the garbage collector (purgeOrphans), whose grace period relies on lastModified. A provider without list still works for upload, download, and targeted purge, but cannot be swept.

SignedUrlProvider

SignedUrlProvider connects the SDK to customer-owned storage without exposing cloud credentials to the browser or to DataPrism. Before each put, get, head, or delete, it asks a customer-controlled signing endpoint for a short-lived URL scoped to that operation and object key.

import { SignedUrlProvider } from "@dataprism/sdk/storage";
 
const provider = new SignedUrlProvider({
  id: "aws_s3",
  signerUrl: "https://storage.example.com/aws/sign",
  getSignerHeaders: async () => ({
    authorization: `Bearer ${await getStorageToken()}`,
  }),
  onOperation: async (receipt) => {
    await customerReceiptStore.append(receipt);
  },
});

The same connector works with AWS S3, Google Cloud Storage, Azure Blob, Cloudflare R2, and Scaleway because provider-specific signing stays behind the customer's endpoint. The SDK never implements cloud signing and never receives an access key.

The optional configuration inspection reports bucket versioning and object lock without reducing either control to a boolean. Versioning can be enabled, suspended, disabled, unsupported, or unknown; object lock can be enabled, disabled, unsupported, or unknown. Unsupported and failed checks remain visible.

Signing endpoint contract

The SDK sends an authenticated POST request:

{
  "providerId": "aws_s3",
  "operation": "put",
  "key": "0x...content-hash...",
  "contentLength": 65536,
  "correlationId": "550e8400-e29b-41d4-a716-446655440000"
}

The SDK generates a new correlation ID for every object operation unless the caller supplies one through StorageReadOptions or StorageWriteOptions. A supplied value must contain 1 to 128 letters, digits, dots, underscores, colons or hyphens. Existing signing endpoints may ignore this additive field; endpoints that retain it can later match the URL they issued to the SDK receipt.

The endpoint returns one operation-scoped URL. headers and expiresAt are optional.

{
  "url": "https://customer-bucket.s3.eu-west-1.amazonaws.com/0x...?X-Amz-...",
  "headers": {},
  "expiresAt": "2026-07-31T12:01:00.000Z"
}

The endpoint must authenticate the caller, validate that key is a DataPrism content hash, bind the HTTP method and content length, restrict access to the configured bucket or prefix, and use a short expiry. It must never return permanent cloud credentials.

When onOperation is configured, it receives a StorageOperationReceipt for each object response. The receipt contains the operation identity, HTTP status, correlation ID and SDK observation times. It copies only known response fields: S3, GCS or Azure request IDs, ETag, version ID, Last-Modified and a valid content length. Signed URLs, request headers and arbitrary response headers are not retained. A transport failure produces a receipt with status: null.

The callback is awaited. If it fails, the storage call rejects, including when the provider request may already have completed. Receipt storage should therefore be idempotent by correlationId. A signing-endpoint failure does not produce an object receipt because no provider request was issued.

For configuration inspection the SDK sends a separate request with no object key:

{
  "providerId": "aws_s3",
  "operation": "inspect"
}

The endpoint returns provider-reported facts directly:

{
  "profile": "dataprism.storage-configuration.v1",
  "outcome": "succeeded",
  "source": {
    "kind": "provider-reported",
    "coverage": "complete",
    "providerId": "aws_s3",
    "reportedAt": "2028-12-31T23:59:58.000Z"
  },
  "bucketVersioning": "enabled",
  "objectLock": "enabled",
  "limitations": []
}

The SDK adds source.collectedAt. Both timestamps are observations, not established time. Limitation values are exported machine-readable codes rather than free-form provider errors, so reports cannot carry signed URLs, response bodies, or credentials. A versioning or object-lock state does not by itself prove deletion.

The endpoint may use kind: "provider-reported" only for facts it read from the underlying storage provider's control plane. Desired settings copied from application or IaC configuration are not provider-reported evidence.

Initializing from dataprism.providers

initProviders builds the provider array from a decrypted providers block. ProviderFactories maps each provider ID to the factory that builds it.

initProviders(config: ProvidersSlot, factories: ProviderFactories): StorageProvider[]
initProvidersWithInspection(config, factories, options?): Promise<{
  providers: StorageProvider[];
  inspections: StorageConfigurationInspection[];
}>
signedUrlFactories(options?): ProviderFactories
import { initProviders, signedUrlFactories } from "@dataprism/sdk/storage";
 
const view = std.reader(client.reader, cloudId);
const providers = initProviders(
  await view.providers(),
  signedUrlFactories({
    getSignerHeaders: async () => ({
      authorization: `Bearer ${await getStorageToken()}`,
    }),
  }),
);

The provider block contains only non-secret location metadata and the signing endpoint URL. Authentication for that endpoint is supplied at runtime through getSignerHeaders; it is not stored in the prism.

Use initProvidersWithInspection when configuration facts are required. It keeps the provider order and returns one result for every configured provider. An adapter without inspection support, including a configuration without a registered factory, returns outcome: "unsupported" with unknown states instead of guessing.

ProviderFactories is { [P in ProviderId]?: (config) => StorageProvider }. An ID present in the configuration but with no matching factory is skipped with a warning, so an application can supply only the providers it supports.

Custom factories

ProviderFactories maps each provider ID to a factory, so you can extend or replace the bundled set:

import {
  signedUrlFactories,
  type ProviderFactories,
} from "@dataprism/sdk/storage";
 
const factories: ProviderFactories = {
  ...signedUrlFactories({
    getSignerHeaders: async () => ({
      authorization: `Bearer ${await getStorageToken()}`,
    }),
  }),
  gcp_storage: (config) => new MyGcsProvider(config),
};

Anything that implements StorageProvider works: a connector in front of your storage service, an IPFS pinning adapter, or a local disk connector for tests. The pipeline only requires put, get, has, and delete. Custom providers selected for shard repair must additionally declare their write capabilities and abort put when its signal is cancelled.

MemoryProvider

An in-process reference implementation for tests and demos.

const provider = new MemoryProvider("aws_s3");
provider.offline = true; // simulate an outage

The offline flag makes it easy to test quorum behaviour: configure several memory providers, take some offline, and verify that download still reconstructs the file.

Copyright © 2026 DataPrism.