Write and run a tool

A tool is a folder that holds one step of your work: reading a file your team uses, fetching data from a system, or turning one kind of data into another. The folder has two parts:

procedure.json says what the step is called, what it reads, what it makes and which settings it takes, in words a colleague can read.
Python code, with one function marked @procedure, does the work.

You write, check and run tools on your own computer; no server or account is needed for that. To share one, add it to your project: it stays private until a project administrator enables it, and then every member can run it on their own computer. Ophiolite never runs a tool on its server. Sending data to another system and checking data against a reference are described in the same way and are checked already, but running them comes with a later release.

The commands below are those of the SDK you install today. From the next SDK release the same commands are also spelled ophiolite tools, and the Python methods tool_register, tool_run and the others; the spellings here keep working.

Start a tool

ophiolite procedures new read-tops --step read --kind well-tops

This writes a start that works as it is: read-tops/procedure.json, read-tops/procedure.py, a sample in read-tops/data/ and the start’s own tests in read-tops/tests/, which pass before you change anything. The steps are read (a file you have), source (a system you connect to), process (data you already have, into new data), send and check. --kind names the type of data, a word such as well-tops, las, grid or deviation; --label is optional and defaults to the step and the kind (“Read well tops (CSV)”). A pair with no start says why, in one sentence, and makes no folder. Each of the twelve kinds has a working process start too; --step process with no --kind writes the two files for you to fill in, and the example below is a complete one.

Edit procedure.json to name what the step reads and makes, and the settings a person chooses. Every label is words, such as “Depth shift”, not a code name such as depth_shift. Edit procedure.py: the marked function receives the inputs and the settings, and returns each output by name, made with a writer from ophiolite.writers (for example write_tops or write_location). Run python -m pytest tests in the folder to check the start still does what its tests say.

If your code imports a package that comes with neither Python nor the SDK, such as psycopg, list it in procedure.json under requires, for example "requires": ["psycopg>=3.1"]; every start writes the list empty. Each entry is a package name with optional versions (>=, <, ==, != or ~=, separated by commas). Before a run or a delivery starts, the SDK checks each entry against the Python that will run it. A package that is missing or too old stops it before anything runs, with its name and the command that installs it, such as python -m pip install 'psycopg>=3.1'. Extras such as psycopg[binary] and conditions after ; cannot be checked and stop a run with a sentence saying so.

The SDK ships a complete example. To copy it next to your own:

cp -r "$(python -c 'import ophiolite, pathlib; print(pathlib.Path(ophiolite.__file__).parent / "contracts/procedures/v1/fixtures/bundles/depth-shift")')" example-depth-shift

Check it

ophiolite procedures validate depth-shift

A tool that follows the rules says, for example, “Valid: Shift tops by a fixed depth, version 1. It reads Well tops (CSV) and makes Well tops (CSV).” One that does not says what to change, for example “Not valid: setting 1 needs a readable label, for example “Depth shift”.” The same rules, in the same words, apply wherever a tool is checked. Add --json to see the field the sentence refers to and the tool’s fingerprint (its digest), which changes when any of its files change. Checking never runs or imports your code.

Run it on your computer

ophiolite procedures run depth-shift --output shifted \
  --input tops=tops.csv --kind tops=well-tops-csv/1 --declared tops=tops.declared.json \
  --setting shift=2.5

--input names the file for each input. Give --kind (the type of data) and --declared (a JSON file with what you know about it, such as the depth unit) for a file of your own. An output of an earlier run already carries both, so --input tops=shifted/shifted.csv is enough.
--setting gives a setting as name=value. Numbers, whole numbers, true or false and choices are checked against the tool’s description; a setting you do not give takes its default.
--system names a local file with a source’s connection settings. They are passed to your code and never written into anything the run keeps.

The run uses a fresh copy of exactly the files that were checked, runs your function in a separate process, and writes a new folder: one file per output, log.txt with anything your code printed, and run.json, which records the tool’s fingerprint, the settings, each input’s fingerprint and what each output is. The output folder must not exist yet. If anything goes wrong, no output folder is written and the command says what to change; the details are under --json.

ophiolite procedures register depth-shift

Ophiolite checks it with the same rules and keeps exactly the files that were checked. It is private: you and the project administrators see it, in Settings → Tools in the Workspace and with ophiolite procedures list. To change it, raise version in procedure.json and register again; each version is kept as it was.

A project administrator enables a version for everyone, in Settings → Tools or with:

ophiolite procedures enable depth-shift --version 1

Members then run the highest enabled version from the project:

ophiolite procedures run --from-project depth-shift --output shifted \
  --input tops=tops.csv --kind tops=well-tops-csv/1 --declared tops=tops.declared.json --setting shift=2.5

Running a tool runs someone else’s code on your computer, so the first time you run a version you did not add, Ophiolite asks first. It says who added it, when it was enabled and its fingerprint, and runs nothing until you answer. Your answer is recorded as your trust in that version only; you are not asked again for it, and a new version asks again. Where nobody can answer, such as in a script, --yes answers it, or ophiolite procedures trust depth-shift --version 1 beforehand; without either, nothing runs.

Before every run Ophiolite reads the version again and checks your local copy against its fingerprint. A copy that changed on your computer is fetched again; a version that was withdrawn no longer runs. An administrator withdraws a version in Settings → Tools or with ophiolite procedures withdraw; whoever added a private version may withdraw it too. Withdrawn versions stay listed, so earlier runs still say what ran.

From Python the same steps are client.procedure_register, procedure_enable, procedure_trust and procedure_run. The notebook Share a tool with your project walks through both sides, yours and a colleague’s. An agent may add a private version or trust one only through a plan a person approved for exactly that request; enabling and withdrawing are always a person’s.

When Ophiolite refuses, the answer says why in a sentence and names its reason; see procedure-refused.

Publish a run

A run on versions kept in the project can be published. Name each input as an exact project version, @RESULT:REVISION (Technical details on the result’s page shows both); Ophiolite downloads exactly that version with the kind and declarations the project keeps for it:

ophiolite procedures run --from-project depth-shift --output shifted \
  --input tops=@<result>:<revision> --setting shift=2.5
ophiolite procedures publish shifted --name "Chalk tops shifted 2.5 m"

publish first reports the run to the project: the version, its settings, every input in order and a checksum of each output. The project checks each claim (the version is enabled, you may build on every input, the settings are the tool’s own, the outputs are the ones it makes) and keeps the run as it was reported. Then each output is published under the run as a new private result. If publishing stops, run publish again on the same folder: it continues the same run, and each output is published once. A run on files from your computer, or an output changed after the run, is not published.

The result’s story says Made with Shift tops by a fixed depth version 1 on your computer; Ophiolite did not run this calculation. That is the point to keep in mind: Ophiolite checks what the run says it used and keeps it unchanged, but the calculation ran on your computer, so Ophiolite cannot vouch that this code made these bytes. A result made by an agent says which agent and for whom. Under Technical details are the version’s fingerprint, the run and the line that runs it again.

Run a result again

Anyone who may read a result a tool made, use its version and build on each of its inputs can run it again on their own computer:

ophiolite procedures again <result> --revision <revision> --output again

The project gives back the version, the settings and the exact inputs; Ophiolite runs them and publishes under a new run. For each output the project compares the new file with the earlier result, by kind:

Same result: nothing is published; the answer names the earlier result.
Same result, newer (or earlier) procedure version: as above, run with another version (--version N).
Different result: a new private result, whose story says Made again from the earlier one; what changed between the two lists the tool versions and each setting in words.
Not compared: a kind Ophiolite does not compare; the new result is published and you compare them yourself.

When you may not run it again, again says why and runs nothing: You cannot read this result., You cannot use one of the inputs of this result. or You cannot use the procedure version this result was made with. (it was withdrawn). From Python the same steps are client.procedure_publish and client.procedure_again.

Older SDKs read a result a tool made as an ordinary declared result (method and settings, without the run). A portable bundle carries the result with its record and method; the bundle’s checksums prove the copy is intact, while the run itself can only be checked by the project that keeps it. Backups of the project include the runs, and the backup check verifies each one.

Send results to your team’s systems

A send tool writes exact versions from the project into a system your team already uses: a database, or a shared folder that other packages read. You run it on your computer with ophiolite deliver. The system’s address and who connects go in a local configuration file that never leaves your computer. Passwords stay in the system’s own password file.

ophiolite deliver "Tops to team DB" --system db=team-db.json --item Tops@3 --setting well=HON-GT-01

--item names the item and its version number. The version you name is the one that is written, even when a newer one exists. Its form is SLOT=ITEM[@N]|SLOT=@ASSET:REVISION: “The item to deliver and its version number (default: its latest when chosen), or exactly as @ASSET:REVISION; repeat per version”. When two items you can see share a name, the name alone is refused and each is listed with its version number and its @asset:revision, so you can copy one. ophiolite procedures run --from-project NAME --item takes the same forms. When the delivery is done, the project’s Activity and the version’s story read, for example, Sent to Team tops database by Tops to team DB on Anna’s computer (reported). A version the tool cannot write, or a missing setting, is refused in words before anything is written. It then reads Not sent to ….

Four send tools are examples in the client kit. Your administrator writes it with ophiolite-ops client-kit (Install); unpack its examples-procedures.zip, and each example is the folder examples/procedures/<name>:

tops-to-team-db writes tops into a PostgreSQL table. Each delivery records its id in the same transaction as the rows, so the database holds a delivery’s tops exactly when it holds its id.
residuals-to-team-database writes a residual result into a PostgreSQL table in the same way, one row per row of the result (Accept results with outcomes delivers through it).
grid-to-surfer-folder writes a grid to a folder as a Surfer text grid.
results-to-vendor-files writes tops and a grid to a folder as a tops file, ZMAP+ and IRAP classic ASCII. Written in the file shapes of a Petrel well-tops export, IRAP classic ASCII and ZMAP+; not tested with Petrel or Kingdom.

The grid headers have no place for the coordinate reference system, the units, what the values mean, which way they are positive or the vertical datum. So each grid is written with <name>.declaration.json beside it, which copies those declarations from the version. Values are written as stored: an elevation above NAP is never turned into a depth. Each file is written under a temporary name and then renamed. A file already in the folder is never replaced; the delivery is refused instead.

If your computer stops in the middle of a delivery, run the same ophiolite deliver again in the same folder. It never writes a second time:

A tool that can look in its system (the database example) answers from there. The delivery reads Sent, or Not sent once the system has recorded that the delivery may no longer arrive.
Otherwise, or while the system cannot say, it reads Outcome unknown … It was not sent again. in the project’s Activity and in the story until a later run knows.

--wait 60 waits for a delivery still running on this computer. After you have looked in the system yourself, --set-aside leaves the outcome unknown in the project and lets this folder start a new delivery.