Scientific lineage and AI datasets

Accepted requirements; complete lineage and dataset experiences are planned. A bounded OSDU smoothing experiment verifies one narrow pinned-input provenance path.

Capture scientific intent in the application

Our own applications and domain APIs can record the operation directly. Store exact input and output revisions, operation and algorithm version, parameters with units, code/build identity, coordinate transformations, execution environment, actor, time, quality checks and uncertainty.

Record semantic manual edits with before/after references. Keep interpretations, labels, rejected candidates and review decisions versioned and distinct. A rejected candidate is not automatically a valid training label; retain its reviewer context and permitted use.

External host APIs may expose only part of the history. Mark that gap explicitly; do not infer the original scientific recipe from a file or a click stream.

Preserve the derivation durably

Persist the run intent and its terminal outcome. Link output references to publication, recover interrupted writes and show incomplete runs. Scientific lineage is durable customer project data and must never be sampled away. A diagnostic trace identifier may link to it, but cannot be its only record.

An example is source case revision + timestep + aggregation + sampling resolution → scalar map → contour interval → vector output. Projection and coordinate transformations belong in that history too.

Lineage relationships require authorization: names and references can disclose restricted inputs even when payload access is refused.

Build a task-specific view

Preserve originals where permitted, scientific metadata, rights and uncertainty. Govern records and suitable domain payloads through supported OSDU APIs in OSDU mode. Derive a versioned view suited to the task:

Tables: Parquet or Arrow are candidates.
Large arrays: appropriate chunked representations.
Documents: text plus source references.
Domain data: use the appropriate domain-service formats.

There is no single universal AI-ready format. Normalization must not erase the original evidence or conceal conversion loss.

Freeze the training and evaluation selection

The planned immutable dataset manifest records:

Exact source versions and content hashes.
Transformations, code versions, features and label origins.
Missing-value and unit handling, spatial and temporal meaning.
Training, validation and test assignments by well, field or time as appropriate.
Preprocessing fitted only on the training split, to avoid leakage.
Approved rights and intended use.

Live assets, training datasets, model artifacts and inference outputs have separate identities and lifecycles. A source update must not silently rewrite an existing dataset. Lineage supports reproducibility and review, but does not establish label quality or model accuracy by itself.

Keep execution under customer control

Use approved customer accounts and compute with scoped access and a chosen model provider. Retrieval, inference and training are different uses. Access to data does not confer training rights, and telemetry consent is not training consent.

No cross-customer training without explicit agreement. Customers retain usable, permitted artifacts subject to their licenses. Revoking an API token cannot recall previously exported data or undo training that has already occurred.

Keep the history portable

Export permitted assets, interpretations, labels, lineage, schemas and dataset manifests together. Independent tooling should verify meaning and hashes and reproduce a selected transformation. Essential lineage and data exit must not depend on renewing a paid UI.

Explore the illustrated example · Understand customer diagnostics.