Keep a table and take it out

Not every table is wells or tops. A core analysis, a laboratory sheet or a production table can be kept as it is: the project holds the exact file you gave, the reading choices that read it (sheet, header row, separators, decimal mark) and what you said about its columns (a unit, a missing-value marker, a description). Nothing is mapped to wells and no value is changed. Reading it back gives the same rows, in the file’s order, at an exact revision.

Any table the project holds, and well tops, deviation surveys and points, can also be taken out as one file: Parquet, GeoParquet (points only) or CSV, written by the server from exactly the revision you name and checked before it is saved. client is an ophiolite.Client for your project — see Build on Ophiolite for a key and the SDK. To turn a table into wells, tops or other typed data instead, see A spreadsheet becomes wells and tops.

Keep a table

The examples use the L16-12 core analysis from NLOG (l1612_core.xls, a tab-separated text file named .xls): 111 samples, porosity in percent with -888 where no value was measured, and horizontal permeability in millidarcy.

done = client.imports.from_table(
    table_file, "table",
    columns={"Helium_Porosity": {"unit": "%", "missing": "-888"}, "KHORIZ": {"unit": "mD"}},
    audience=[],
)
unit = done["run"]["units"][0]
print(done["run"]["state"], unit["asset_id"], unit["revision"][:12])

"table" keeps the file as it is; columns says what a column holds: unit, missing (the marker that means no value), quantity and description, each optional. The table is read first and nothing is kept when it cannot be read: a workbook with several sheets, or a file whose header row or separator is not clear, is refused with the question to answer, which you give as reading=. audience=[] keeps it to yourself until you share it.

The revision of a kept table is the SHA-256 of the original file. Keeping the same file with the same choices again adds nothing; the same file with another unit, marker or sheet is kept as another table.

Read it back

kept = client.read_data(unit["asset_id"], unit["revision"])
print(len(kept), "rows;", kept.units)
frame = kept.to_frame(missing="declared")
print(int(frame["Helium_Porosity"].isna().sum()), "porosity values missing")

Values are as the file wrote them: in kept.rows the -888 stays a value. to_frame(missing="declared") turns each column’s declared marker into a missing value and records that in frame.attrs["transformations"]. Columns of a type that is not read are named in kept.not_read and kept only in the original file.

Take it out as a file

kept.download("l16-12-core.parquet")
kept.download("l16-12-core.csv", format="csv")

download saves the file only when its SHA-256 equals the one the server states and the server says it wrote it from this revision; otherwise nothing is saved. client.export_file(asset_id, revision, path, format=...) does the same for any revision you name: a kept table, a table read from a connected database, well tops, a deviation survey, a point set or the curves of a LAS log (curves= chooses curves; omitted, every stored curve). A table read from a connected database is taken out with client.source(source_id).download(path); see Read any table in Python.

Read the file with other tools

The Parquet file holds every row in order, typed, with its units and markers in the field metadata and a record of where it came from in the file metadata (ophiolite.export): the project, the table, the revision, the columns as declared and anything that was lost.

import json
import pandas as pd
import pyarrow.parquet as pq

table = pd.read_parquet("l16-12-core.parquet")
record = json.loads(pq.read_schema("l16-12-core.parquet").metadata[b"ophiolite.export"])
print(len(table), record["revision"][:12], record["columns"][12], record["losses"])

import duckdb

markers = duckdb.sql("SELECT count(*) FROM 'l16-12-core.parquet' WHERE Helium_Porosity = -888").fetchone()[0]
rows = len(pd.read_csv("l16-12-core.csv"))
print(markers, "samples without porosity;", rows, "rows in the CSV")

In the table	In Parquet
integer	int64
float	float64
decimal	decimal128 at the precision the source stated, else the precision its values need; more than 38 digits is text, listed as a loss
date	date32
timestamp	timestamp[us] without a zone (ns when a value needs it)
timestamp with time zone	timestamp[us, UTC]
a date or time Parquet cannot hold	text, listed as a loss
text, boolean	string, bool
a missing value	null

The CSV is UTF-8 without a byte order mark, comma-separated with . as decimal mark. An empty cell is a missing value and an empty text alike; the Parquet file tells them apart. Cells that begin with = are written as they are.

Points as GeoParquet

A point set is taken out as GeoParquet 1.1.0: one geometry column of 2D points (WKB) beside x, y and z and the attributes, its coordinate system as PROJJSON and its bounding box.

client.export_file(points_asset, points_revision, "positions.geoparquet", format="geoparquet")

GDAL with the Parquet driver reads it:

ogrinfo -so positions.geoparquet positions

QGIS opens a GeoParquet file only where its GDAL build has the Parquet driver (ogrinfo --formats lists Parquet). The qgis/qgis 4.2.2 container image does not have it; the GDAL images ghcr.io/osgeo/gdal (ubuntu-full) do.

A point set whose coordinate system is not stated is written with "crs": null, which means unknown. A point set in a geographic coordinate system (latitude and longitude) is refused: “The coordinate system of these points is not known here, so they cannot be written as GeoParquet. Download as Parquet instead.”

From the command line

ophiolite read-data --asset ASSET_ID --revision REVISION --format parquet --output l16-12-core.parquet
ophiolite read-data --asset ASSET_ID --revision REVISION --format csv --output l16-12-core.csv

Limits and refusals

• A kept table holds at most 100,000 rows and 128 columns, and at most 32 MiB once read; a larger file is refused before anything is kept.
• A file taken out holds at most 100,000 rows and 64 MiB once read.
• A table has no positions, so asking for GeoParquet of a table is refused; use Parquet.