Use files in a storage bucket

A storage bucket is a folder tree in an S3-compatible object store, such as Amazon S3, Hetzner Object Storage or MinIO. Once your organisation has connected one, a project member can browse it, choose a folder, and add its files to the project as an upload does — without downloading them first. Ophiolite reads the bucket with your own key; it never writes to your storage.

Connect the bucket (organisation)

On your organisation’s Connections page, someone who may add databases and storage chooses Add storage and gives the service address (https://…), the region, the bucket, optionally a folder, and their own access key and secret. Use a key that can only read and list. Anyone else who administers a project asks for it with Ask for storage, without a key; an approver decides, as for a database.

The deployment checks the address before any request leaves it: https with a certificate it trusts, http only to a loopback address, and never a private address your deployment administrator has not allowed. A redirect is never followed.

Each person who uses the connection adds their own key (Add your key); nobody reads with another person’s key. When adding it, the organisation also says whether projects may keep a copy of its files, and who may receive copies.

Add a folder (project)

In a project, Sources → Set up a new source → Storage bucket, choose the connection, open a folder and choose Check N files. Every file is read once and its SHA-256 recorded. Then choose where the data should live:

Choice	What is kept	When the file later changes in your storage
Read from your storage each time	Nothing but the bytes’ digest and tag	Reads report that the revision is unavailable: no copy was kept
Keep a copy (only when your organisation allows it)	A copy, as an upload keeps one	The copy still reads; the change is not taken

Reading in place covers well logs (LAS 2.0) and GeoTIFF grids; any other kind of file is added only as a copy. A folder of more than 5,000 files is refused: choose a smaller one. Files whose names Ophiolite cannot show safely are left out and counted.

In Python

Name the connection by its id, shown under Technical details on the Sources page:

connection = "managed-…"  # the storage connection's id

Browse a folder with your own key (the deployment holds it; no key is given here). A page lists the subfolders, then the files; page["next_cursor"], given as cursor=, continues a long folder:

page = client.browse(connection, "wells/")
for folder in page["folders"]:
    print("folder", folder)
for item in page["items"]:
    print(item["name"], item["size"], item["profile"], "read in place" if item["read_in_place"] else "copy only")

Add every file below the folder in one upload. Without keep_copy each file is read from your storage each time, pinned at the bytes read now:

report = client.import_storage(connection, "wells/", attribution="NLOG.NL", rights_confirmed=True)
print(report.counts)
report.save("storage-report.csv")

The report is the folder upload’s report: one row per file with its outcome (added, already here, not read, not supported) and the reason. Read in place, two objects with the same bytes are two files, and both are added; kept as copies, the second is already here. Running the same import again adds nothing new.

From the command line

ophiolite sources browse CONNECTION_ID --prefix wells/
ophiolite sources import CONNECTION_ID --prefix wells/ --attribution "NLOG.NL" --rights-confirmed --report report.csv
ophiolite sources import CONNECTION_ID --prefix wells/ --keep-copy --audience PERSON --attribution "NLOG.NL" --rights-confirmed

--keep-copy is refused unless your organisation allows copies, and --audience names only people it allows.

When the storage refuses

The Workspace shows the sentence and what to do; the code is under Technical details.

Code	You see	What to do
destination-refused	This storage address is not allowed by your organisation or this deployment.	Use a storage address your deployment administrator allows.
unreachable	This storage cannot be reached. Try again later.	Check that the service address is right and that the storage is running.
tls-refused	This storage’s certificate could not be trusted.	Ask your deployment administrator to trust the storage’s certificate issuer.
sign-in-refused	The storage refused this key. Check the access key and secret.	Replace your key with one the storage accepts.
no-list-right	This key cannot list files. Ask for permission to list.	Ask the storage’s owner for a key that may list the folder.
bucket-missing	This storage has no bucket with that name.	Check the bucket name.
wrong-region	This storage expects a different region.	Give the region the storage names for this bucket.
object-missing	This file is no longer in your storage; this does not confirm it was deleted.	Choose the file again.
object-changed	This file changed in your storage since you reviewed it. Review it again.	Check the file again.
throttled	This storage is busy. Ophiolite will try again shortly.	Try again shortly.
unreadable	This storage answered in a way Ophiolite could not read.	Report the time of the test to your deployment administrator.
no-consistent-read	This storage cannot read large files safely, so large files are not available.	Use storage that keeps a file unchanged while it is read, or keep files small.

Some stores accept any region in a request (Hetzner Object Storage does); with those, wrong-region never appears.

What this is not

Ophiolite does not write, move or delete anything in your storage, does not watch a bucket for new files (add the folder again to take them), and does not read Parquet files in place.