OxyMake

Next-generation workflow orchestration in Rust. The uv of computational workflows.

OxyMake is a fast, declarative workflow orchestration tool that combines the proven ideas of Snakemake (file-based rules, backward-chaining DAG, wildcards) with modern engineering: content-addressable caching, polyglot execution, in-memory data passing, and first-class support for both human and AI agent users.

Key Features

• Fast DAG resolution: 10K-job DAG resolved in 69 ms on M4 Max, 33.3× faster than Snakemake 7.32.4 (100K-job scaling out of scope for this benchmark wave; cold end-to-end is slower than Snakemake — an honest trade for content-addressable correctness)
• Content-addressable: no phantom re-runs from git checkout or file copies
• Polyglot: shell, Python, R, Julia — each rule chooses its language
• Daemon-free: ox run starts, works, exits. No server to manage.
• Agent-friendly: --json output, structured events, typed API
• Scales: same workflow on laptop, SLURM cluster, or Ray cluster (Kubernetes designed, not yet implemented)

Quick Example

# Oxymakefile.toml
ox_version = "0.1"

[config]
samples = ["A", "B", "C"]

[rule.process]
input = ["data/{sample}.csv"]
output = ["results/{sample}.json"]
shell = "python process.py {input} {output}"

[rule.report]
input = ["results/{sample}.json"]
output = ["reports/summary.html"]
shell = "python report.py {input} > {output}"

ox run                    # build everything
ox run -j 8               # 8 parallel jobs
ox status                 # what's running?
ox plan                   # what would run?

Installation

OxyMake is a single binary called ox, written in Rust. There are several ways to install it.

Install (from source)

git clone https://github.com/noogram/oxymake.git
cd oxymake
cargo install --path crates/ox-cli

This installs both ox and oxymake to ~/.cargo/bin/. Make sure this directory is in your $PATH.

Development setup

For working on OxyMake itself:

git clone https://github.com/noogram/oxymake.git
cd oxymake
cargo build                    # debug build → target/debug/ox
cargo test --workspace         # run all tests
cargo run --bin ox -- --help   # run without installing

With just (recommended):

just build      # debug build
just test       # all tests
just demo       # interactive feature demo
just lint       # clippy checks
just ci         # full CI check (fmt + lint + test + demo)
just --list     # all available recipes

Prerequisites

Required

• Rust 1.85+ (for installation from source)

Optional (depending on your workflow)

• Python 3.9+ -- for rules using lang = "python"
• uv -- for environment = { uv = "pyproject.toml" } (install uv)
• conda/mamba -- for environment = { conda = "..." }
• Docker -- for environment = { docker = "..." }
• Nix -- for environment = { nix = "..." }

Verify Installation

ox --version
# ox 0.6.1

ox init
# Initialized OxyMake project in .
#   Created: Oxymakefile.toml
#   Created: .oxymake/

What Gets Installed

OxyMake is a single binary with no runtime dependencies. All state is stored in a .oxymake/ directory within your project:

your-project/
  Oxymakefile.toml       # Your workflow definition
  .oxymake/
    state.db             # SQLite execution state
    cache/               # Content-addressable cache
    logs/                # Job execution logs

No daemon, no server, no background processes. Each ox run is a self-contained process that reads state, executes, writes state, and exits.

Next Steps

Now that OxyMake is installed, head to Your First Workflow to build something.

Quickstart

Get up and running with OxyMake in under five minutes. This guide covers only features that are tested and working in v0.1.0.

Install

Build and install from source (Rust 1.85+ required):

git clone https://github.com/noogram/oxymake.git
cd oxymake
cargo install --path crates/ox-cli

This installs both ox and oxymake to ~/.cargo/bin/.

Verify:

ox --version
# ox 0.6.1

Create a Project

mkdir my-pipeline
cd my-pipeline
ox init

This creates a starter Oxymakefile.toml and a .oxymake/ directory.

The generated template uses {input} and {output} placeholders for input/output file expansion, plus {config.key} for config substitution.

Your First Workflow

Create the Oxymakefile:

cat > Oxymakefile.toml << 'EOF'
ox_version = "0.1"

[config]
samples = ["A", "B"]

# Default target: require all results to exist.
[rule.all]
input = ["results/{sample}.txt"]

# Process each sample's CSV into a sorted text file.
[rule.process]
input = ["data/{sample}.csv"]
output = ["results/{sample}.txt"]
shell = "sort data/{sample}.csv > results/{sample}.txt"
EOF

Key concepts:

• [config] defines variables. Here samples = ["A", "B"] means OxyMake will create one job per sample.
• {sample} in paths and shell commands is replaced with each value from the config list.
• [rule.all] is the default target. It has inputs but no outputs, so it just ensures its inputs exist.
• Use explicit paths with config variables in shell commands (e.g., data/{sample}.csv), not {input}/{output}.

Create some input data:

mkdir -p data results
echo -e "charlie,3\nalpha,1\nbravo,2" > data/A.csv
echo -e "zulu,26\nmike,13" > data/B.csv

Validate

Check your Oxymakefile for errors:

ox lint
# Oxymakefile is valid (2 rules)

Preview (Dry Run)

See what OxyMake would do without running anything:

ox run --dry-run

Output:

Dry run: 2 job(s) would execute for 2 target(s)
  [process-B] rule=process outputs=[results/B.txt]
  [process-A] rule=process outputs=[results/A.txt]

Run

Execute the workflow:

ox run

Output:

Completed: 2 succeeded, 0 failed, 0 skipped, 0 cancelled (0.0s)

Check the results:

cat results/A.txt
# alpha,1
# bravo,2
# charlie,3

Caching

Run the same command again:

ox run

Output:

Cache: 2 of 2 job(s) up-to-date, skipping.
Completed: 0 succeeded, 0 failed, 2 skipped, 0 cancelled (0.0s)

Nothing ran. OxyMake detected that all inputs are unchanged and all outputs exist. Modify an input and re-run to see only the affected jobs execute.

Build a Specific Target

Build only one output:

rm results/A.txt
ox run results/A.txt

Only process-A runs. results/B.txt is untouched.

Multi-Step Pipeline

OxyMake resolves dependency chains automatically. Here is a two-step pipeline that uppercases text, then counts characters:

mkdir pipeline && cd pipeline

cat > Oxymakefile.toml << 'EOF'
ox_version = "0.1"

[config]
names = ["alice", "bob"]

[rule.all]
input = ["final/{name}.txt"]

[rule.uppercase]
input = ["raw/{name}.txt"]
output = ["mid/{name}.txt"]
shell = "tr '[:lower:]' '[:upper:]' < raw/{name}.txt > mid/{name}.txt"

[rule.count]
input = ["mid/{name}.txt"]
output = ["final/{name}.txt"]
shell = "wc -c < mid/{name}.txt > final/{name}.txt"
EOF

mkdir -p raw mid final
echo "hello world" > raw/alice.txt
echo "oxymake rocks" > raw/bob.txt

ox run --dry-run
# 4 jobs: uppercase-alice, uppercase-bob, count-alice, count-bob

ox run
# Completed: 4 succeeded, 0 failed, 0 skipped, 0 cancelled (0.0s)

cat final/alice.txt
# 12

OxyMake figures out that count depends on uppercase and runs them in the correct order.

Error Handling

If a job fails, OxyMake stops and reports the failure:

cat > Oxymakefile.toml << 'EOF'
ox_version = "0.1"
[rule.broken]
output = ["out.txt"]
shell = "exit 1"
EOF

ox run
# error: job broken failed: exit code 1
# Completed: 0 succeeded, 1 failed, 0 skipped, 0 cancelled (0.0s)
# Exit code: 1

Use --keep-going (or -k) to continue running independent jobs even when one fails:

cat > Oxymakefile.toml << 'EOF'
ox_version = "0.1"
[config]
items = ["ok", "fail"]

[rule.all]
input = ["out/{item}.txt"]

[rule.process]
input = ["in/{item}.txt"]
output = ["out/{item}.txt"]
shell = "if [ '{item}' = 'fail' ]; then exit 1; fi; cp in/{item}.txt out/{item}.txt"
EOF

mkdir -p in out
echo "good" > in/ok.txt
echo "bad" > in/fail.txt

ox run -k
# Completed: 1 succeeded, 1 failed, 0 skipped, 0 cancelled (0.0s)
# Exit code: 1
# out/ok.txt was created; out/fail.txt was not

Static Rules

Rules without config variables produce a single job:

cat > Oxymakefile.toml << 'EOF'
ox_version = "0.1"
[rule.greet]
output = ["greeting.txt"]
shell = "echo 'Hello OxyMake' > greeting.txt"
EOF

ox run
cat greeting.txt
# Hello OxyMake

Alternate Oxymakefile

Use -f to point to a different file:

ox run -f path/to/other.toml

Known Limitations (v0.1.0)

• -j N (parallel execution): All jobs run sequentially regardless of the -j value.
• --set (config override): Does not override config values.

Next Steps

• Read Your First Workflow for a more detailed walkthrough
• Explore ox run --help for all available options

Your First Workflow

This tutorial walks you through creating a simple 3-rule workflow from scratch. By the end, you will understand how OxyMake resolves dependencies, runs jobs, and caches results.

Step 1: Create a Project

Create a new directory and initialize OxyMake:

mkdir my-pipeline
cd my-pipeline
ox init

This creates a starter Oxymakefile.toml. We will replace its contents.

Step 2: Create Some Input Data

Create a data/ directory with two CSV files:

mkdir data

data/alice.csv:

name,score
Alice,85
Alice,92
Alice,78

data/bob.csv:

name,score
Bob,91
Bob,88
Bob,95

Step 3: Write the Workflow

Replace the contents of Oxymakefile.toml with:

ox_version = "0.1"

[config]
students = ["alice", "bob"]

# Rule 1: Compute statistics for each student
[rule.stats]
input = ["data/{student}.csv"]
output = ["results/{student}_stats.json"]
lang = "python"
run = """
import csv, json

scores = []
with open("{input}") as f:
    for row in csv.DictReader(f):
        scores.append(int(row["score"]))

stats = {
    "student": "{wildcards.student}",
    "mean": sum(scores) / len(scores),
    "min": min(scores),
    "max": max(scores),
    "count": len(scores),
}

with open("{output}", "w") as f:
    json.dump(stats, f, indent=2)
"""

# Rule 2: Combine all student stats into a summary
[rule.summary]
input = ["results/{student}_stats.json"]
output = ["results/summary.json"]
lang = "python"
run = """
import json, glob

all_stats = []
for path in sorted(glob.glob("results/*_stats.json")):
    with open(path) as f:
        all_stats.append(json.load(f))

with open("{output}", "w") as f:
    json.dump(all_stats, f, indent=2)
"""

# Rule 3: Default target -- build the summary
[rule.all]
input = ["results/summary.json"]

This workflow has three rules:

1. stats -- computes per-student statistics (runs once per student)
2. summary -- combines all stats into one file
3. all -- an aggregation target that tells OxyMake what to build

Interpolation note. Inside run/shell blocks, OxyMake substitutes the placeholders it recognizes -- {input}, {output}, {wildcards.X}, {config.X}, and so on -- and leaves everything else untouched. It does not treat {{/}} as escaped braces, so write ordinary Python dict literals with single braces (stats = { ... }). The recognized placeholders are listed in the Expression Language reference.

Step 4: Plan

Before running, see what OxyMake will do:

ox plan

You should see something like:

Plan: 3 rules, 3 jobs, 2 source files
Targets: results/summary.json
  1. [stats-bob] rule=stats -> [results/bob_stats.json]
  2. [stats-alice] rule=stats -> [results/alice_stats.json]
  3. [summary] rule=summary -> [results/summary.json]

OxyMake resolved the {student} wildcard from config.students and created two concrete jobs for the stats rule (with the ids stats-alice and stats-bob), plus one for summary.

Step 5: Run

ox run

Output (timings will vary):

  Resolving 3 jobs (3 to run, 0 cached)
  ▸ summary — upstream rebuilt
  ✓ Completed 3/3 in 0.6s (4.8 jobs/s)
    3 succeeded
Completed: 3 succeeded, 0 failed, 0 skipped, 0 cancelled (0.6s)

The last line is the canonical summary: N succeeded, N failed, N skipped, N cancelled. A run is successful when failed and cancelled are both 0.

Check the results:

cat results/alice_stats.json

{
  "student": "alice",
  "mean": 85.0,
  "min": 78,
  "max": 92,
  "count": 3
}

Step 6: See Caching in Action

Run the same command again:

ox run

Output:

Cache: 3 of 3 job(s) up-to-date, skipping.
Completed: 0 succeeded, 0 failed, 3 skipped, 0 cancelled (0.0s)

Nothing ran. OxyMake detected that all inputs are unchanged and all outputs exist with the correct content hashes, so all three jobs are reported as skipped.

Now modify one input:

echo "Alice,99" >> data/alice.csv
ox run

Output:

Cache: 1 of 3 job(s) up-to-date, skipping.
  Resolving 3 jobs (2 to run, 1 cached)
  [1/3] ✓ stats-bob [cached]
  ▸ summary — upstream rebuilt
  ✓ Completed 3/3 in 0.4s (7.5 jobs/s)
    2 succeeded, 1 skipped
Completed: 2 succeeded, 0 failed, 1 skipped, 0 cancelled (0.4s)

Only stats-alice and summary re-ran. stats-bob was cached (reported as skipped) because its input did not change.

Step 7: Add a New Student

Edit Oxymakefile.toml and add a student:

[config]
students = ["alice", "bob", "charlie"]

Create the data file:

echo "name,score
Charlie,76
Charlie,82
Charlie,90" > data/charlie.csv

Run again:

ox run

Cache: 2 of 4 job(s) up-to-date, skipping.
  Resolving 4 jobs (2 to run, 2 cached)
  [1/4] ✓ stats-alice [cached]
  [2/4] ✓ stats-bob [cached]
  ▸ summary — upstream rebuilt
  ✓ Completed 4/4 in 0.4s (10.5 jobs/s)
    2 succeeded, 2 skipped
Completed: 2 succeeded, 0 failed, 2 skipped, 0 cancelled (0.4s)

Only the new student was computed. Alice and Bob's stats were cached (reported as skipped).

What You Learned

1. Rules declare intent -- input/output patterns with wildcards
2. Config drives expansion -- students = [...] determines which jobs are created
3. Content-addressable caching -- unchanged inputs mean cached outputs
4. Incremental execution -- adding data or rules only computes what is new
5. Backward chaining -- OxyMake figures out the dependency order automatically

Next Steps

• The Three Graphs -- understand how OxyMake resolves your workflow
• Content-Addressable Cache -- why caching works correctly
• Execution Modes -- the four ways to execute a rule

Migrating to schema 2

Schema 2 makes unsupported declarations fail visibly. It requires a minimum capable OxyMake binary in every schema-2 file, and catches structural typos. Legacy files keep working, with a warning that their binary requirement is not enforced.

Start with a review:

ox migrate --to-format 2 -f Oxymakefile.toml

The report covers all included files. Generated markers 0.1 and 0.3 become >=0.7.0, because that is the release introducing this contract. Commands and software environments retain their meaning. If the tool cannot justify a requirement or finds unknown fields, resolve those declarations yourself and run the preview again. Do not remove a field until you understand its intent.

Apply the resolved migration explicitly, then inspect the changes:

ox migrate --to-format 2 -f Oxymakefile.toml --write
git diff
ox lint -f Oxymakefile.toml

Use environment = { uv = "requirements.txt" } for the software backend. There is no variables table yet, and rule env is rejected in schema 2. Keep any existing shell assignments in the command. Schema 2 supports named resource classes. Migrate each file that declares or references a class: schema 1 ignores those fields and warns. Overrides preserve the class spelling, so rule cpus = 6 over class cpu = 2 expands both {threads} and {resources.cpu} to 6. Resource exports remain future work.

Migration preserves cache identity. Any future change to execution semantics will state its own cache consequences in that release's notes. Command bytes are preserved, but document formatting, including line endings, may be normalized. Older binaries that predate version enforcement cannot be made to reject a future file retroactively; ensure the installed binary supports schema 2.

Understanding the Output

When you run ox run, OxyMake provides structured feedback about what it is doing and why. This page explains the output formats, using the 3-rule workflow from Your First Workflow (stats for two students, plus a summary).

Terminal Output (Default)

By default, OxyMake prints human-readable progress and ends with a canonical summary line (timings will vary):

  Resolving 3 jobs (3 to run, 0 cached)
  ▸ summary — upstream rebuilt
  ✓ Completed 3/3 in 0.6s (4.8 jobs/s)
    3 succeeded
Completed: 3 succeeded, 0 failed, 0 skipped, 0 cancelled (0.6s)

The last line is the canonical summary, always in the same shape:

Completed: N succeeded, N failed, N skipped, N cancelled (<elapsed>)

• succeeded -- jobs that ran and produced their outputs
• failed -- jobs whose command exited non-zero
• skipped -- jobs whose outputs were already up to date (cache hits)
• cancelled -- jobs that did not run because an upstream job failed

A run is successful (exit code 0) when both failed and cancelled are 0.

Cached Jobs

When outputs are already up to date, OxyMake skips the work and reports the jobs as skipped:

Cache: 3 of 3 job(s) up-to-date, skipping.
Completed: 0 succeeded, 0 failed, 3 skipped, 0 cancelled (0.0s)

On a partial re-run (one input changed), the cached jobs are listed and the summary reflects the split:

Cache: 1 of 3 job(s) up-to-date, skipping.
  Resolving 3 jobs (2 to run, 1 cached)
  [1/3] ✓ stats-bob [cached]
  ▸ summary — upstream rebuilt
  ✓ Completed 3/3 in 0.4s (7.5 jobs/s)
    2 succeeded, 1 skipped
Completed: 2 succeeded, 0 failed, 1 skipped, 0 cancelled (0.4s)

Plan Output

Use ox plan to see what may run without executing:

ox plan

Plan: 3 rules, 3 jobs, 2 source files
Targets: results/summary.json
  1. [stats-bob] rule=stats -> [results/bob_stats.json]
  2. [stats-alice] rule=stats -> [results/alice_stats.json]
  3. [summary] rule=summary -> [results/summary.json]

The header reports the totals (N rules, N jobs, N source files), followed by the requested targets and the concrete jobs, each shown as [job-id] rule=<rule> -> [outputs].

The plan is an upper bound of what ox run executes with the same tree and cache settings. A planned upstream rebuilt job may be skipped at runtime if its producer rebuilds bytes matching hashes recorded under the same cache key and the consumer's normal cache check passes. A producer whose cache key changed still forces its consumers, even with identical bytes.

JSON Output (Agent Mode)

Add --json to ox run for structured NDJSON output -- one self-contained JSON event per line:

ox run --json

{"event":"run_started","total_jobs":3,"to_run":3,"cached":0}
{"event":"job_started","job_id":"stats-bob","executor":"local","reason":"cache_miss"}
{"event":"job_completed","job_id":"stats-bob","duration_ms":209,"outputs":["results/bob_stats.json"]}
{"event":"job_started","job_id":"stats-alice","executor":"local","reason":"cache_miss"}
{"event":"job_completed","job_id":"stats-alice","duration_ms":200,"outputs":["results/alice_stats.json"]}
{"event":"job_started","job_id":"summary","executor":"local","reason":"upstream_rebuilt"}
{"event":"job_completed","job_id":"summary","duration_ms":194,"outputs":["results/summary.json"]}
{"event":"run_completed","total":3,"succeeded":3,"failed":0,"skipped":0,"cancelled":0,"duration_ms":607}

Each event carries an event discriminant (run_started, job_started, job_completed, run_completed). This format is designed for AI agents and scripts to parse programmatically. Use --report-json <path> to write the same stream to a file. See Agent-Driven Workflows for details.

DAG Visualization

Use ox dag to render the dependency graph. The default format is Graphviz DOT:

ox dag

digraph oxymake {
  rankdir=LR;
  "results/summary.json" -> "all";
  "stats" -> "results/{student}_stats.json";
  "data/{student}.csv" -> "stats";
  "summary" -> "results/summary.json";
  "results/{student}_stats.json" -> "summary";
}

Other formats:

ox dag --format mermaid       # Mermaid graph syntax
ox dag --format dot           # Graphviz DOT (same as default)
ox dag --group-by rule        # Collapse nodes by field
ox dag --json                 # Structured JSON

To trace a single target's dependency chain instead, use ox explain:

ox explain results/summary.json

Dependency chain for: results/summary.json

► 1. [summary] rule=summary
     inputs:  [results/alice_stats.json, results/bob_stats.json]
     outputs: [results/summary.json]
  2. [stats-alice] rule=stats
     inputs:  [data/alice.csv]
     outputs: [results/alice_stats.json]
  3. [stats-bob] rule=stats
     inputs:  [data/bob.csv]
     outputs: [results/bob_stats.json]

Error Output

When a job fails, OxyMake reports the failure, cancels the dependent jobs, and ends with a non-zero exit code:

  Resolving 1 jobs (1 to run, 0 cached)
  [1/1] ✗ broken FAILED (exit 1)

  error: job broken failed: exit code 1
    stderr: --- stderr ---
    stderr: boom
  ✗ Completed 1/1 in <0.1s
    1 failed
  Failed: broken
Completed: 0 succeeded, 1 failed, 0 skipped, 0 cancelled (0.0s)
  Failed jobs (showing 1 of 1):
    broken: boom
  Run 'ox logs --failed' for full details.

ox logs --failed prints the full captured output of each failed job. In --json mode, the failure is reported as a job_completed event with a non-success status, so automated tooling can recover programmatically.

Verbosity Levels

Control output detail with -v:

ox run           # Normal output
ox run -v        # Verbose: job start/end, durations, and exit codes
ox run -vv       # Debug: also show each job's stdout/stderr

Next Steps

• CLI Commands -- full command reference
• Execution Modes -- how rules are executed

Rules and Wildcards

What is a Rule?

A rule declares a transformation: given these inputs, produce these outputs by running this command. OxyMake figures out what needs to run based on what you ask for.

[rule.process]
input = ["data/{sample}.csv"]
output = ["results/{sample}.txt"]
shell = "python process.py {input} {output}"

This single rule handles ANY sample. When you ask for results/A.txt, OxyMake matches the output pattern, extracts sample = "A", substitutes it into the input pattern to get data/A.csv, and runs the command.

Wildcards

Wildcards are placeholders in curly braces: {sample}, {cohort}, {model}. They appear in input and output file patterns.

How wildcards resolve

OxyMake uses backward chaining: start from the output you want, find which rule can produce it, extract wildcard values from the match.

You ask for: results/patient_42.txt
                     ↓
Pattern:     results/{sample}.txt
                     ↓
Extracted:   sample = "patient_42"
                     ↓
Input becomes: data/patient_42.csv

Multiple wildcards

Rules can have multiple wildcards:

[rule.analyze]
input = ["data/{cohort}/{region}.parquet"]
output = ["results/{cohort}/{region}/report.html"]
shell = "python analyze.py {input} {output}"

Wildcard expansion from config

When you have a list of known values, put them in [config]:

[config]
samples = ["A", "B", "C"]

[rule.all]
input = ["results/{sample}.txt"]

The all rule has {sample} in its inputs but no outputs — it's an aggregation target. OxyMake expands {sample} from config.samples to request results/A.txt, results/B.txt, results/C.txt.

Expansion modes

When multiple wildcards expand from config lists, the expansion can be:

[config]
samples = ["A", "B"]
conditions = ["treated", "control"]

[rule.experiment]
output = ["results/{sample}_{condition}.csv"]
expand = "product"    # default: A_treated, A_control, B_treated, B_control

Wildcard constraints

Restrict which values a wildcard can take:

[rule.process]
output = ["results/{sample}.txt"]

[rule.process.wildcard_constraints]
sample = "[A-Z][a-z0-9_]*"    # regex: starts with uppercase letter

Existing files and rule ownership

An existing file that matches a wildcard output belongs to that rule when its inputs can be resolved. For example, with raw/{x}.csv → data/{x}.csv, if both raw/manual.csv and a hand-written data/manual.csv exist, ox run can regenerate and overwrite data/manual.csv, just as with make pattern rules.

If raw/manual.csv is missing and has no producer, an existing hand-maintained data/manual.csv is instead treated as a source. The same applies to leading wildcards ({dir}/data.csv) and config-derived directories ({config.outdir}/{x}.csv). Missing intermediate outputs are still rebuilt when their source inputs are available. Cycles, ambiguous producers, and invalid configuration remain errors. A missing source inside another producer is also an error; an existing final output cannot hide a broken dependency chain. Fixed inputs of wildcard rules, such as cfg/settings.txt, are required for every instance even when sample-specific inputs are missing.

Use wildcard_constraints to reserve hand-maintained names, even when matching raw inputs exist. For example, this rule only owns numbered samples, leaving data/manual.csv as a source:

[rule.prepare]
input = ["raw/{x}.csv"]
output = ["data/{x}.csv"]
shell = "cp {input} {output}"

[rule.prepare.wildcard_constraints]
x = "sample[0-9]+"

Outputs recorded in the cache retain their provenance requirements. If their original inputs are missing, only verified cache-import adoption permits them as leaves. ox run --no-cache disables that shortcut and errors until the inputs are restored. A modified adopted output also requires its producer's inputs. An absent cache permits unrecorded manual files, while a corrupt or unreadable cache denies that fallback. CLI, MCP and the Rust API share these rules and check source existence and cache provenance relative to the Oxymakefile directory.

Conditional guards

Rules can apply only to certain wildcard values:

[config]
special_samples = ["X1", "X2"]

[rule.extra_analysis]
input = ["results/{sample}.txt"]
output = ["extra/{sample}_analysis.html"]
when = "sample in @special_samples"

This rule exists only for samples X1 and X2. Other samples don't get the extra analysis — no phantom nodes in the graph, no skipped jobs.

Guards support: in @list, not in @list, == 'value', != 'value', =~ 'regex'.

The Four Execution Modes

Start with shell or run for quick prototyping. Migrate to call when your function stabilizes and you want OxyMake to optimize I/O.

See Execution Modes for details.

The Three Graphs

OxyMake uses three distinct graph representations, each at a different level of abstraction. Understanding them is key to understanding how OxyMake works — and how to debug when things go wrong.

Overview

graph TD
    A[Oxymakefile.toml] --> B["RuleGraph<br/><i>What you declared (abstract, compact)</i>"]
    B -->|"Wildcard resolution<br/>+ guard evaluation"| C["JobGraph<br/><i>What will execute (concrete, optimized)</i>"]
    C -->|"Runtime state annotation"| D["ExecGraph<br/><i>What is happening (live status)</i>"]

RuleGraph — The Logical View

The RuleGraph is what you wrote in the Oxymakefile. Each rule is a node, and edges connect rules whose output patterns match other rules' input patterns. Wildcards are NOT resolved — this is the abstract view.

A single call node represents ALL variant-call instances, not a specific one.

$ ox plan --level=rules

  data ──→ features ──→ call ──→ annotate

What you can learn from the RuleGraph:

• Is my pipeline structure correct?
• Are there circular dependencies?
• Which rules depend on which?

Inspect it: ox plan --level=rules

JobGraph — The Physical Plan

The JobGraph is the RuleGraph after wildcard resolution. Every concrete job instance is a separate node. With 3 cohorts and 4 windows, a single features rule becomes 12 concrete jobs.

The JobGraph goes through optimization passes before execution:

These passes run internally; ox plan reports the resolved jobs after optimization. For the 3-rule workflow from Your First Workflow:

$ ox plan
Plan: 3 rules, 3 jobs, 2 source files
Targets: results/summary.json
  1. [stats-bob] rule=stats -> [results/bob_stats.json]
  2. [stats-alice] rule=stats -> [results/alice_stats.json]
  3. [summary] rule=summary -> [results/summary.json]

The header line summarizes the graph (N rules, N jobs, N source files), followed by the requested targets and the concrete jobs that would run.

What you can learn from the JobGraph:

• How many concrete jobs will execute?
• Which rule produced each job, and what outputs it writes?
• Which jobs are already cached? (re-run after a build to see fewer jobs)

Inspect it: ox plan (optimized, the default), ox plan --no-optimize (skip the optimization passes), or ox plan --level rules to view the RuleGraph instead of the JobGraph.

ExecGraph — The Live Execution

The ExecGraph is the JobGraph annotated with runtime state. Each node carries its status (Pending → Running → Completed/Failed), timing, and resource usage.

$ ox status --group-by stage

  data          3/3 completed
  features      145/3412 running (12%)
  call          waiting (blocked)
  annotate      waiting

What you can learn from the ExecGraph:

• What's running right now?
• What failed and why?
• How long has each job been running?
• Which sessions are active?

Inspect it: ox status

The Relationship

Each graph is a refinement of the previous one:

Vocabulary

To avoid confusion, OxyMake uses these terms consistently:

• Rule = a declaration in the Oxymakefile (unresolved wildcards)
• Job = a concrete, executable instance of a rule (wildcards resolved)
• Pass = an optimization transformation on the JobGraph
• Phase = a stage of the pipeline (parse → resolve → optimize → execute)

Content-Addressable Cache

One of the most frustrating things about traditional build tools is the phantom re-run: you check out a branch, and everything rebuilds even though nothing actually changed. OxyMake eliminates this by using file content as the source of truth, not timestamps.

How It Works

Every time OxyMake runs a job, it computes a cache key from everything that could affect the output:

cache_key = blake3(
    format_version ||
    rule_source_hash ||
    sorted((input_path, input_content_hash) pairs) ||
    params_hash ||
    env_content_hash ||
    shell_executable ||
    clean_outputs ||
    (platform if cache_platform = "exact")
)

Every field is length-framed with a domain-separation tag, so two different job specifications can never hash to the same key. If the key matches a previously computed result, the job is skipped. The key includes:

• Rule source hash -- if you change the shell command, inline code, or function reference, the cache is invalidated
• Input content hashes -- blake3 of every input file's contents, bound to its path; parameter files and (in script mode) the script file itself count as inputs, so editing script.py invalidates the cache
• Params hash -- any parameters passed via --set or [config]
• Environment content hash -- the content of the referenced spec file (requirements.txt, conda YAML, nix expression), every .python-version in the workflow root and its parent directories (for uv), or the container image reference for Docker/Apptainer
• Shell executable -- the same command under /bin/bash and /bin/zsh can behave differently
• Platform -- OS and architecture (a Linux build is not reusable on macOS)

A rule may explicitly set cache_platform = "any" when its outputs are valid across operating systems and architectures. The default is "exact". This is a user assertion: OxyMake records it for audit but cannot prove it.

Three exclusions to know about: call-mode function bodies are tracked only if you declare the module as an input; UV_PYTHON and changes to the installed interpreter set do not enter uv environment hashes (use a checked-in .python-version when this matters); and mutable container tags are hashed as written (pin images by digest -- python@sha256:... -- if you need re-pushed tags to invalidate the cache).

Why Not Timestamps?

Timestamps lie. Here are common situations where they cause phantom re-runs in tools like Make or Snakemake:

Validation Strategies (ADR-006)

OxyMake's cache validation is pluggable — you choose the right speed/correctness tradeoff for your workflow:

ox run                                  # default: mtime+hash (fast + content-verifying)
ox run --cache-validation=mtime         # Make-parity opt-in (no content check)
ox run --cache-validation=hash          # strict mode (CI)
OX_CACHE_VALIDATION=hash ox run         # via environment variable

Configure per project in Oxymakefile.toml:

[config]
cache_validation = "mtime+hash"

Remote caches automatically promote to hash regardless of the configured strategy, because mtime is not meaningful across machines.

Local Cache Metadata

Local cache metadata lives in .oxymake/cache/cache.db. It records the computation key, output paths, content hashes, and validation metadata:

.oxymake/cache/
  cache.db

The outputs themselves remain at their declared workflow paths. Deleting the local metadata makes jobs run again unless you use mtime validation; it does not delete outputs or execution history.

Continuing a DAG on Another Machine

Suppose a Linux box holds a dataset that is too large to move and runs the heavy preparation stages, while a macOS box has the GPU needed for the final fits. The two checkouts must contain the same workflow definition. Mark only the rule whose prepared data will cross the platform boundary:

[rule.merge_counts]
input = ["data/*.parquet"]
output = ["build/counts.parquet"]
shell = "duckdb -c '...your merge command...'"
cache_platform = "any"

[rule.fit_model]
input = ["build/counts.parquet"]
output = ["build/fits.rds"]
shell = "Rscript fit.R"

On the Linux box, run the heavy target and export its cache metadata:

ox run build/counts.parquet
ox cache-export build/counts.parquet -o counts.ox-cache.json
rsync -aR build/counts.parquet counts.ox-cache.json mac-gpu:/path/to/project/

This transfers the prepared product and manifest, but not data/. From the root of the matching checkout on the macOS box, adopt the product and run the fit:

ox cache-import counts.ox-cache.json
ox run build/fits.rds

Import re-hashes every output and rejects the whole manifest before changing the cache if a hash, output set, job specification, platform scope, or reproducibility policy does not match. Once imported, build/counts.parquet is a trusted resolver leaf, so its producer is pruned even though the raw inputs are absent on the Mac. Rules using the default cache_platform = "exact" can only be imported on the producing OS and architecture.

cache_platform = "any" is an assertion by the workflow author, not a fact OxyMake can verify. The surface makes the unsafe case deliberate, local, and enumerable: there is no plural or workflow-wide form, so unprotecting n rules costs n typed lines, and the audit is:

grep -n cache_platform Oxymakefile.toml

It does not make the unsafe case hard. Whether a rule emits machine code or some other platform-specific artefact is invisible to the parser. For example, putting cache_platform = "any" on a cargo build rule can serve a Linux binary to macOS. Keep the default "exact" unless you can defend the rule's outputs as cross-platform data.

OxyMake currently supports a shared filesystem directory as its remote artifact backend. Point --cache-remote at a directory reachable from every machine, for example a team NFS mount:

# Production: everything cached locally
ox run

# Use a shared directory; fetched artifacts are verified by BLAKE3.
ox run --cache-remote /mnt/team/oxymake-cache

--cache-remote promotes validation to hash: timestamps from another machine are never trusted. The directory backend stores each output under its content hash and verifies the hash after every fetch.

The remote backend transports output bytes, not the local cache metadata needed to identify a hit. Copying .oxymake/ between machines is unsupported; use ox cache-export and ox cache-import to transfer that trust record. S3 and GCS URLs are not supported yet.

Cache and Materialization

The cache interacts with the materialization policy:

Outputs with materialize = "never" are kept in memory and never enter the cache. This is a deliberate trade-off: you get speed at the cost of reproducibility. The next ox run will recompute them.

Managing the Cache

# Preview removal of orphaned cache metadata
ox clean --cache-only --dry-run

# Remove orphaned cache metadata
ox clean --cache-only --yes

# Remove all cache metadata
ox clean --cache-only --all --yes

Why This Matters

The content-addressable cache means you can:

1. Switch branches freely without phantom re-runs
2. Add new rules without invalidating existing cached results
3. Share computation across machines of the same platform by default, and across platforms only for rules declaring cache_platform = "any" through ox cache-export and ox cache-import
4. Resume interrupted runs -- completed work is preserved
5. Verify adopted bytes -- import hashes every transferred output before recording it locally

Incremental external datasets

Use per-rule clean_outputs = "never" with the local executor when an idempotent script materializes a remote dataset incrementally. Declare the real files as outputs: completed downloads survive reruns and failures, while successful outputs are still hashed normally. The script owns checking staleness and completeness. See the output cleanup reference and S3 recipe for all three policies.

Materialization Policy

When a call-mode rule produces an output, does it need to be written to disk? Not always. OxyMake lets you control this with the materialization policy, enabling significant speedups for workflows where intermediate outputs are only consumed by other call-mode rules.

The Four Policies

Declaring Materialization

Set the policy on individual outputs:

[rule.compute_features]
output = [{
    path = "features/{sample}.parquet",
    format = "parquet",
    materialize = "auto"
}]
call = "pipeline.features:compute_features"
lang = "python"

[rule.train_model]
output = [{
    path = "models/{sample}.pkl",
    format = "pickle",
    materialize = "always"
}]
call = "pipeline.model:train"
lang = "python"

In this example, the features DataFrame is only written to disk if a non-call downstream rule needs it as a file. The model is always saved.

Setting the Policy per Output

Materialization is declared per output in the Oxymakefile, on the structured output form:

[rule.compute_features]
# ...
output = [
    { path = "data/features.parquet", materialize = "auto" },
]

Valid values are auto (the default — write to disk only when a downstream file consumer needs it), never (keep in memory; no disk, no caching), final (write only leaf outputs), and always (write and cache every output). There is no global ox run flag that overrides the policy today; control it in the Oxymakefile per output.

Guidance for development workflows:

• During prototyping, set materialize = "never" on intermediate outputs to iterate fast
• For production, use the default auto (or always) for full caching and reproducibility
• For presentations or reports, set leaf outputs to final

How It Works with call Mode

When two consecutive rules both use call mode on the local executor, OxyMake can pass data directly in memory:

compute_features  ──[DataFrame in memory]──>  train_model
     (call)                                      (call)

No file is written between them. The format field tells OxyMake how to serialize the data if materialization is needed later (e.g., for caching or for a shell-mode downstream rule).

The Flow

1. compute_features runs and returns a DataFrame
2. If materialize = "auto" and the next consumer is also call mode: pass the DataFrame directly in memory
3. If materialize = "auto" and the next consumer is shell mode: write the DataFrame to disk using the parquet codec
4. If materialize = "always": always write to disk (and cache)
5. If materialize = "never": never write to disk (no cache)

Constraints

Not everything supports non-always materialization:

• shell, run, and script modes always materialize. They manage their own I/O and need real files.
• Distributed executors (SLURM, K8s) force materialization because jobs run on separate machines.
• Non-materialized outputs are not cached. If the process dies or you restart, they will be recomputed. This is an explicit trade-off: speed vs. reproducibility.

The --materialize Flag

The CLI flag sets the floor for materialization:

Practical Example

Consider a three-stage pipeline:

[rule.load_data]
output = [{ path = "data/{s}.parquet", format = "parquet", materialize = "auto" }]
call = "pipeline:load_data"
lang = "python"

[rule.compute_features]
output = [{ path = "features/{s}.parquet", format = "parquet", materialize = "auto" }]
call = "pipeline:compute_features"
lang = "python"

[rule.generate_report]
output = [{ path = "reports/{s}.html", materialize = "always" }]
call = "pipeline:generate_report"
lang = "python"

With ox run --materialize=final:

• load_data output: kept in memory (not a leaf)
• compute_features output: kept in memory (not a leaf)
• generate_report output: written to disk (it is a leaf)

Only the final HTML report touches the filesystem. The intermediate parquet files exist only in memory during execution.

Tags and Filtering

Tags are key/value labels you attach to a rule. They travel with every concrete job the rule produces, so you can group, filter and report on jobs without touching the workflow structure.

Assigning Tags

tags is a table of string keys to string values — not an array. An array of strings is a parse error.

[rule.align]
input  = ["data/{sample}.fastq"]
output = ["aligned/{sample}.bam"]
shell  = "bwa mem ref.fa {input} | samtools sort > {output}"
tags   = { stage = "align", cost = "heavy" }

[rule.qc]
input  = ["aligned/{sample}.bam"]
output = ["qc/{sample}_report.html"]
shell  = "fastqc {input} -o qc/"
tags   = { stage = "qc", speed = "fast" }

Pick your own keys. stage, speed, cost and owner are common, but nothing is reserved. Each job's resolved wildcard values are merged into its tag map too, so a job of align above carries { sample = "NA12878", stage = "align", cost = "heavy" }.

Where Tags Show Up

In the plan. ox plan --json carries each job's tags map, so a script or agent can select jobs by tag before deciding what to do:

ox plan --json | jq '.jobs[] | select(.tags.stage == "align") | .job_id'

In the event stream. The job_queued event carries the originating job's tags, and ox subscribe filters on them with --where KEY=VALUE (repeatable, AND logic):

ox subscribe --where stage=align                 # only alignment jobs
ox subscribe --where stage=align --where cost=heavy

Events that carry no tags — everything other than job_queued — are dropped while a --where filter is active.

What Tags Do Not Do

Tags are a labelling and observation mechanism, not a job selector for execution. There is no ox run --tag or ox run --exclude-tag. To run a subset of the workflow, use the mechanisms that do exist:

ox run results/report.txt      # explicit targets
ox run --rule align            # a single rule
ox run --until merge_vcf        # stop at a rule
ox run --omit-from qc           # skip a subtree

ox status --group-by accepts rule and stage (an alias for rule name); it does not group by tag.

Use Cases

• Observability: subscribe to just the GPU jobs and route their events elsewhere.
• Reporting: post-process ox plan --json to count work per stage.
• Downstream scheduling: an agent reads tags to decide which executor or queue a job belongs to.

Next Steps

• The Three Graphs -- how the DAG is built and viewed
• Execution Modes -- how jobs are executed

Execution Modes

OxyMake supports four ways to execute a rule, forming a spectrum from maximum flexibility to maximum optimization. All four modes coexist in the same workflow -- you pick the right one for each rule.

The Spectrum

shell       Opaque, files only, maximum flexibility
run         Inline script, files only, author manages I/O
script      External script, files only, author manages I/O
call        Pure function, files OR memory, OxyMake manages I/O

As you move from shell to call, OxyMake gains more optimization power (in-memory data passing, task fusion, automatic serialization) -- but you give up direct control over I/O.

Mode 1: shell -- Command Line

The most flexible mode. You write a shell command, and OxyMake interpolates file paths into its command string before passing that string to the configured shell. This is raw shell text, not typed argv passing: validate any third-party-derived path, wildcard, or configuration value before interpolating it. See SECURITY.md.

[rule.align]
input = ["data/{sample}.fastq", "refs/genome.fa"]
output = ["results/{sample}.bam"]
shell = "bwa mem -t {resources.cpu} {input[1]} {input[0]} > {output}"
resources = { cpu = 8 }

Use shell when you are wrapping an existing command-line tool. OxyMake treats the command as a black box -- it just passes file paths and checks that outputs were created.

Mode 2: run -- Inline Script

Write a short script directly in the Oxymakefile. OxyMake interpolates {input} and {output} as file paths.

[rule.analyze]
input = ["data/{sample}.csv"]
output = ["results/{sample}.json"]
lang = "python"
run = """
import pandas as pd
import json
df = pd.read_csv("{input}")
stats = df.describe().to_dict()
with open("{output}", "w") as f:
    json.dump(stats, f)
"""

Use run for rapid prototyping -- when the logic is short enough to live in the workflow file. You manage all file I/O yourself.

Mode 3: script -- External Script

Like run, but the code lives in a separate file. Keeps the Oxymakefile clean when scripts are long.

[rule.transform]
input = ["data/{sample}.parquet"]
output = ["results/{sample}.parquet"]
script = "scripts/transform.py"
environment = { uv = "pyproject.toml" }

The script receives file paths via command-line arguments or environment variables.

Mode 4: call -- Pure Function

The key innovation. Your function receives objects, not file paths, and returns objects. OxyMake handles all I/O outside the function.

[rule.compute_features]
input = [{ path = "data/{sample}.parquet", format = "parquet" }]
output = [{ path = "features/{sample}.parquet", format = "parquet", materialize = "auto" }]
call = "pipeline.features:compute_features"
lang = "python"

The Python function is pure:

import polars as pl

def compute_features(df: pl.DataFrame) -> pl.DataFrame:
    return df.with_columns(
        mean_depth=pl.col("depth").rolling_mean(20),
        depth_std=pl.col("depth").rolling_std(60),
    )

The function never reads or writes files. OxyMake:

1. Reads the input file using the parquet codec, producing a DataFrame
2. Calls compute_features(df) and receives the result
3. Writes the result to disk using the parquet codec (if materialization policy requires it)

In memory mode (when both upstream and downstream are call rules on the local executor), step 1 receives the DataFrame directly from the upstream job and step 3 passes it directly to the downstream job -- zero disk I/O.

Named Arguments

For functions with multiple inputs, use named inputs:

[rule.train_model]
input = { features = "features/{sample}.parquet", config = "configs/model.yaml" }
output = { model = "models/{sample}.pkl" }
call = "pipeline.model:train"
lang = "python"

def train(features: pl.DataFrame, config: dict) -> Model:
    ...

The input keys (features, config) map to function parameter names.

Warm workers and resource observations

ox run --warm-workers fork retains an imported Python template but forks a new child for each call dispatch. The template reaps that child with os.wait4, so the child's peak RSS and CPU time can be written to that dispatch's benchmark output; peak RSS also reaches job history on Linux and macOS. CPU time is local to the child. Peak RSS is the child's high-water mark, but it starts with the warm template footprint inherited at fork. Only memory above that baseline was added by this dispatch; the reported total is not the memory the dispatch would need by itself. For a template that has imported a large library such as NumPy or PyTorch, that baseline can dominate a small dispatch's reported peak.

ox run --warm-workers persistent executes every dispatch inside one retained process. Its RSS is a high-water mark over the worker's whole lifetime, not a per-dispatch peak, and CPU intervals are not isolated. OxyMake therefore leaves both observations absent rather than reporting a delta or sample as job usage.

When to Use Each Mode

The Migration Path

The natural evolution of a rule:

1. Start with run: Write inline code during exploration
2. Extract to script: When the code gets long, move it to a file
3. Refactor to call: When the function stabilizes, make it pure and let OxyMake manage I/O

Each step is backward-compatible -- the outputs are the same files. The cache key changes (because the rule source changes), so the first run after migration will recompute, but subsequent runs benefit from the optimization.

Interaction with Executors

Distributed executors (SLURM, K8s) cannot pass objects in memory between machines, so they automatically force call mode to materialize. Your workflow does not need to change -- OxyMake handles this transparently.

Environments

Real-world workflows need specific software packages, library versions, and runtime configurations. OxyMake supports multiple environment backends that isolate each rule's execution in a reproducible environment.

Declaring an Environment

Add an environment field to any rule:

[rule.analyze]
input = ["data/{sample}.csv"]
output = ["results/{sample}.json"]
lang = "python"
environment = { uv = "pyproject.toml" }
run = """
import pandas as pd
df = pd.read_csv("{input}")
df.describe().to_json("{output}")
"""

The environment is resolved at execution time. OxyMake ensures the environment is set up before the rule runs.

Supported Backends

uv (Python)

The recommended backend for Python workflows. Uses uv to create and manage virtual environments from a pyproject.toml or requirements.txt.

environment = { uv = "pyproject.toml" }

The value is either a project file (pyproject.toml) or a requirements-style file:

environment = { uv = "pyproject.toml" }    # project: uv discovers it
environment = { uv = "requirements.txt" }  # requirements file

For a requirements file, OxyMake runs the command as uv run --with-requirements <file> -- …. For a project file it runs uv run -- … and lets uv discover the project itself; the file is not passed on the command line.

Either way the declared file's bytes enter the cache key — and, for a project file, the bytes of an adjacent uv.lock when one exists. Editing a dependency therefore invalidates the affected outputs.

conda

For workflows that need non-Python packages (C libraries, R, etc.):

environment = { conda = "environment.yaml" }

OxyMake creates or updates a conda environment from the YAML specification.

Docker / OCI Containers

For maximum isolation and reproducibility:

environment = { docker = "python:3.11-slim" }

The job runs inside a container. OxyMake mounts the workspace and handles input/output file staging. The image digest is included in the cache key.

Nix

For fully reproducible builds with Nix:

environment = { nix = "flake.nix#devShell" }

Apptainer (Singularity)

For HPC environments where Docker is unavailable:

environment = { apptainer = "image.sif" }

System (default)

No isolation: the command runs with whatever Python/R/tools are on $PATH. This is what you get by omitting environment — there is no system key, and an environment table naming no backend is a parse error.

How Isolation Works

Each environment backend follows the same lifecycle:

1. Resolve: Determine the exact environment specification (lockfile hash, image digest, flake hash)
2. Prepare: Create or update the environment if needed (uv run --with-requirements, docker pull, conda env create)
3. Execute: Run the job inside the environment
4. Hash: Include the environment specification hash in the cache key

The key insight is step 4: the environment specification is part of the cache key. If you update a dependency in pyproject.toml and the lockfile changes, all rules using that environment will be recomputed.

Mixing Environments

Different rules can use different environments in the same workflow:

[rule.download]
shell = "wget {url} -O {output}"

[rule.analyze]
environment = { uv = "pyproject.toml" }
call = "analysis:run"
lang = "python"

[rule.visualize]
environment = { conda = "envs/plotting.yaml" }
script = "scripts/plot.R"

OxyMake manages each environment independently. There is no requirement that all rules share the same environment.

Environment and Executors

For SLURM, ensure that conda environments or uv projects are accessible from the compute nodes (e.g., on a shared filesystem).

Environment Caching

Environment setup can be slow (minutes for large conda environments). OxyMake caches the prepared environment and only re-creates it when the specification changes:

• uv: Rebuilds when uv.lock changes
• conda: Rebuilds when environment.yaml changes
• Docker: Re-pulls when the image tag resolves to a new digest
• Nix: Rebuilds when the flake lock changes

This means the first run may be slow (environment setup), but subsequent runs reuse the prepared environment instantly.

Executors

OxyMake separates what to run (rules, DAG) from where to run it (executors). The same workflow runs on a laptop or a thousand-node cluster with zero changes -- just switch the --executor flag.

Available Executors

Local Executor

The default. Runs jobs as subprocesses on the local machine.

ox run                # single job at a time
ox run -j 8           # 8 parallel jobs

Best for development, small pipelines, and single-node execution.

Opt into resource admission with --resource-budget, for example ox run -j 3 --resource-budget cpu=6,gpu=1. Two jobs each requesting four CPUs then run sequentially even with -j 3; two half-GPU jobs may run together. An empty budget leaves dispatch bounded by -j alone.

-j caps concurrent jobs, --resource-budget caps declared resources held by those jobs, and --memory-budget separately caps in-memory outputs. A budget is per ox run; concurrent processes each get their own capacity, with no host-wide coordination or capacity promise.

Admission validates the selected graph before cache checks, claims, output cleanup or recipes, including fully cached runs. It reserves only resources named in the budget, using the resource alias and value contract. Reservations cover preparation, execution, finalization and output hashing. Cancellation keeps a running attempt's reservation until its task exits; retries release reservations before backoff and acquire them again for the next attempt. Retry backoff remains responsive to shutdown with or without a budget.

Each scheduler run owns its budget. Concurrent runs each receive their own capacity; claims and output locks do not coordinate resource budgets. These reservations do not constrain actual CPU use or RSS. --memory-budget remains the separate budget for in-memory output materialization. No fairness or preemption policy is promised.

Fatal executor errors and task panics retain the existing cancel/abort path: the scheduler requests executor cancellation and aborts its remaining Tokio tasks. Guards release when those tasks are dropped, including on unwind; this path does not wait for external child processes to exit. A process abort cannot run Rust destructors, and its per-run budget disappears with it.

SLURM Executor

Submits jobs to an HPC cluster via sbatch and polls status with sacct.

ox run --executor slurm

Features:

• Job arrays for wildcard expansions
• GPU scheduling via GRES
• Resource mapping: cpu, mem, gpu map to SLURM --cpus-per-task, --mem, --gres=gpu:N

Ray Executor

Submits jobs to a Ray cluster via the Ray Jobs API. Ray provides elastic distributed execution with a shared object store for fast intermediate data passing.

Setup

Start a Ray head node (or connect to an existing cluster):

ray start --head
# Dashboard: http://127.0.0.1:8265

Run the workflow:

ox run --executor ray

Configuration

Configure the Ray executor in .oxymake/config.toml or Oxymakefile.toml:

[executor.ray]
dashboard_address = "http://127.0.0.1:8265"
working_dir = "/shared/oxymake"
poll_interval_min = "2s"
poll_interval_max = "30s"
max_submit = 10

Resource Mapping

The native Ray DAG driver reserves CPU, GPU, memory and custom resources. Memory is normalized to whole bytes and included in the live-node feasibility check. It is logical scheduling admission, not a hard RSS limit; the per-job Jobs API and array paths do not reserve memory.

Memory Passing

When two consecutive call-mode rules run on the Ray executor, data passes through Ray's object store without disk writes. OxyMake's materialization policies map to Ray behavior:

Execution Modes

The Ray executor supports all four execution modes:

• shell -- commands run as Ray job entrypoints
• run -- inline scripts submitted as Ray jobs
• script -- external scripts submitted as Ray jobs
• call -- Python functions with object store integration

Choosing an Executor

Mixed-Executor DAGs

OxyMake owns the DAG; executors are job-dispatch backends. A future enhancement will allow per-rule executor assignment, enabling mixed-executor DAGs where some rules run locally and others dispatch to Ray or SLURM.

Next Steps

• Execution Modes -- the four ways rules execute
• Materialization Policy -- controlling disk I/O
• Configuration -- project settings

OxyMake × Ray Deep Dive

OxyMake and Ray solve different halves of the distributed compute problem. OxyMake owns the what: which jobs to run, in what order, and what can be skipped. Ray owns the where: which machine, which GPU, how many cores. This page explains how the two systems fit together.

The Three Graphs Meet Ray

Before any executor sees a job, OxyMake transforms the user's declarations through three graph representations. Understanding this pipeline is essential for understanding what Ray actually receives.

Graph Transformation Pipeline

flowchart TD
    A["Oxymakefile.toml<br/><i>Declarative TOML</i>"] --> B["RuleGraph<br/><i>Abstract: wildcards intact</i>"]
    B -->|"Wildcard resolution<br/>+ guard evaluation"| C["JobGraph<br/><i>Concrete: every job instance</i>"]
    C -->|"Optimization passes"| D["Optimized JobGraph"]
    D -->|"Cache pruning removes<br/>up-to-date jobs"| E["Uncached Subgraph"]
    E -->|"generate_driver()"| F["Python Driver Script<br/><i>@ray.remote tasks +<br/>ObjectRef chaining</i>"]
    F -->|"Ray Jobs API<br/>POST /api/jobs/"| G["Ray Cluster<br/><i>Distributed execution</i>"]

    style A fill:#f9f,stroke:#333
    style F fill:#ff9,stroke:#333
    style G fill:#9ff,stroke:#333

RuleGraph — What You Wrote

The RuleGraph is the abstract view: each rule is a node, wildcards are unresolved. A single features rule represents ALL feature instances.

data ──→ features ──→ call ──→ annotate

JobGraph — What Will Execute

After wildcard resolution, each concrete job is a separate node. With 3 cohorts and 4 windows, a single features rule becomes 12 concrete jobs. The JobGraph is bipartite — job nodes and output nodes alternate:

graph LR
    subgraph "Bipartite JobGraph"
        J1["job: align-A"] -->|produces| O1["output: results/A.bam"]
        O1 -->|consumed by| J2["job: sort-A"]
        J2 -->|produces| O2["output: results/A.sorted.bam"]

        J3["job: align-B"] -->|produces| O3["output: results/B.bam"]
        O3 -->|consumed by| J4["job: sort-B"]
        J4 -->|produces| O4["output: results/B.sorted.bam"]
    end

Optimization Passes

Before any executor sees the graph, OxyMake runs optimization passes:

These passes run internally. ox plan reports the jobs that remain after pruning, in the standard plan format -- for a large, mostly-cached pipeline:

Plan: 12 rules, 847 jobs, 1203 source files

Only the uncached subgraph is sent to Ray.

Resource admission

Each shell, script, inline run, and call task receives its checked CPU, GPU, memory and custom resource request. For example, resources = {cpu = 1, memory = "4GiB", metal = 1} reserves one logical CPU, 4 GiB of logical memory and one metal token. A cluster advertising one metal token serializes tasks requesting that token across concurrent DAG runs. Ray memory is logical admission against the memory declared by its nodes; it is not an enforced resident-set-size ceiling. The single DAG driver reserves zero logical CPUs, no memory and no task tokens; it only orchestrates, though it still consumes real CPU and memory.

Before submitting the driver, OxyMake reads /api/v0/nodes?detail=1 from the Ray dashboard and checks every active task against each live node's total capacity. One node must satisfy the complete request simultaneously. A task requiring CPU=2 and metal=1 cannot use nodes {CPU: 8} and {CPU: 1, metal: 1} by adding their capacities. Omitted CPU defaults to 1. Capacity already in use still counts: a busy but capable node queues the task normally.

An impossible request fails before any recipe is submitted, with the rule, requested resources, observed live node capacities and this escape hatch:

ox run --executor ray --ray-allow-pending --follow

Use --ray-allow-pending when a future node will supply the missing resources. It skips node inspection but preserves resource value and name validation. Sites that always permit pending Ray work can set ray_allow_pending = true in the selected [profile.NAME]; an explicit CLI flag takes precedence over a profile value. Without it, inspection has a fixed 10-second deadline, including the response body. Connection errors, timeouts and malformed/unknown payloads have distinct diagnostics. Partial or truncated node lists are rejected. Ray's dashboard must expose the detailed State API. This is a pre-submission snapshot; OxyMake does not monitor later topology changes or manage the autoscaler.

An empty live-node list is not evidence that a request is impossible: the cluster may be scaling from zero or waiting for workers. OxyMake warns once and submits in that case. When live nodes are visible and no single node provides the complete request, submission still fails so misspelled or impossible requests do not remain pending forever.

Token-authenticated dashboards

Set RAY_AUTH_MODE=token when the Ray dashboard requires token authentication. OxyMake then loads the client token using Ray's precedence:

1. RAY_AUTH_TOKEN
2. the file named by RAY_AUTH_TOKEN_PATH
3. ~/.ray/auth_token

Leading and trailing whitespace in environment and file tokens is ignored. If none of those sources provides a token, OxyMake stops before making a dashboard request. The bearer token is used for submission, status, logs, cancellation, and the pre-submission node inspection. It is not written to the generated driver, run metadata, reports, or logs.

A bearer token sent over plain HTTP is readable on the wire; put any Ray dashboard reachable beyond loopback behind TLS.

Native DAG submissions pass RAY_AUTH_MODE=token to the driver, but never the token itself. The driver runs on the cluster and must be able to load its own token using Ray's cluster-local configuration. If a dashboard answers 401, OxyMake reports the observed mode and the token sources it consulted without printing any part of the token.

The token value is case-insensitive. When RAY_AUTH_MODE is unset or has any other value, OxyMake performs no token lookup and sends no authorization header.

Custom names with surrounding whitespace are rejected. Names colliding with Ray built-ins or reserved prefixes are also rejected; see the resource reference. time_min is a SLURM key, not a Ray time limit.

For native DAG submission, mem, memory, mem_mb and mem_gb are normalized to whole bytes and passed as Ray task memory reservations. The pre-submission check compares each request independently with one live node's total memory; it does not sum sibling tasks. Ray queues and serializes simultaneously ready tasks when their combined reservations exceed currently available capacity. An explicit zero-byte declaration remains present in the generated task options, while a task with no memory declaration emits no memory option.

This reservation currently applies only to tasks in the generated native DAG. The older per-job Jobs API path and the array-emulation path do not request Ray memory. On the per-job path, OXYMAKE_MEMORY_LIMIT_BYTES remains a runtime environment hint for the process; it is neither Ray admission nor hard RSS containment. Array submissions likewise keep their existing behavior. Do not rely on either path for memory reservation.

A submission acknowledgment means the driver was accepted. --follow reports its terminal failure, ox status refreshes failed driver state even without a results file, and ox logs JOB retrieves driver logs. ox cancel stops the shared Ray driver when any of its tasks is selected, cancelling the whole DAG, including queued tasks. Interrupting --follow also stops its driver and exits with code 130. The message confirms the driver stopped, or reports a failed cancellation; use ox status to check remote work and ox cancel to retry. Three consecutive poll rounds with errors end follow with exit code 1, naming the dashboard and last error. An error-free round resets the counter. See remote follow for details.

Running the live scheduling witnesses

The Rust test suite includes ignored tests that require an isolated real Ray cluster and a directory mounted at the same absolute path on the driver and workers. They check single-CPU completion, two drivers sharing one metal token with a no-token negative control, busy-node queueing, and cancellation before capacity frees. They do not start or reconfigure a cluster.

export OXYMAKE_RAY_LIVE_ADDRESS=http://127.0.0.1:8265
export OXYMAKE_RAY_LIVE_DIR=/shared/oxymake-tests
# Cluster with exactly one live node and one CPU:
cargo test -p ox-exec-ray --test live_dag_resources single_cpu -- --ignored
# Separate cluster with exactly one live node, two CPUs and metal=1:
cargo test -p ox-exec-ray --test live_dag_resources two_drivers -- --ignored
cargo test -p ox-exec-ray --test live_dag_resources busy_node -- --ignored
# Separate cluster with exactly one live node and at least two CPUs:
cargo test -p ox-exec-ray --test live_dag_resources two_tasks_serialize_when_their_memory_sum_exceeds_one_node -- --ignored
# Against a token-mode head, with a client-side token source configured and a
# cluster-local token available to the submitted driver:
RAY_AUTH_MODE=token cargo test -p ox-exec-ray --test live_dag_resources token_mode_dashboard_and_driver_authenticate -- --ignored

Ignored tests are NOT RUN, not passing scheduling evidence. HTTP contract tests exercise submission and cancellation requests but cannot prove Ray's actual scheduling behavior.

Ray Job Packaging

Why One Ray Job, Not N

OxyMake could submit each task as a separate Ray job. Instead, it generates a single Python driver script that encodes the entire uncached DAG as @ray.remote tasks with ObjectRef dependency chaining.

flowchart LR
    subgraph "OxyMake (Rust)"
        A["Optimized JobGraph<br/>847 uncached jobs"] -->|"driver_script.rs<br/>generate_driver()"| B["driver.py<br/>~500 lines"]
    end

    subgraph "Ray Cluster"
        B -->|"Jobs API<br/>1 submission"| C["Ray Driver Process"]
        C --> D["@ray.remote task 1"]
        C --> E["@ray.remote task 2"]
        C --> F["@ray.remote task 3"]
        C --> G["..."]
        C --> H["@ray.remote task N"]
        D -.->|ObjectRef| E
        D -.->|ObjectRef| F
        E -.->|ObjectRef| H
        F -.->|ObjectRef| H
    end

    style B fill:#ff9,stroke:#333
    style C fill:#9ff,stroke:#333

Benefits of single-job packaging:

Generated Driver Structure

The Rust code in ox-exec-ray/src/driver_script.rs generates Python that looks like this:

import ray
import subprocess
import time
import json

ray.init()

@ray.remote
def run_shell(job_id, command, work_dir, *deps):
    """Run a shell command. *deps are ObjectRefs — Ray waits for them."""
    result = subprocess.run(command, shell=True, cwd=work_dir, ...)
    if result.returncode != 0:
        raise RuntimeError(f"Job {job_id} failed")
    return result.returncode

@ray.remote
def run_call(job_id, module, func_name, *deps):
    """Run a call-mode function with object store integration."""
    # ray.get() inputs from object store
    # invoke function
    # ray.put() outputs back to object store
    ...

# --- DAG encoded as ObjectRef chain ---
# Topological order, upstream refs passed as implicit dependencies

ref_0 = run_shell.options(num_cpus=8).remote(
    "align-A", "bwa mem ...", "/project"
)
ref_1 = run_shell.options(num_cpus=2).remote(
    "sort-A", "samtools sort ...", "/project",
    ref_0  # ← dependency: Ray won't start until ref_0 completes
)
ref_2 = run_shell.options(num_cpus=8).remote(
    "align-B", "bwa mem ...", "/project"
)
ref_3 = run_call.options(num_cpus=4, num_gpus=1).remote(
    "train", "pipeline.model", "train",
    ref_1, ref_2  # ← depends on both sort-A and align-B
)

# --- Collect results ---
results = {}
for ref, job_id in [(ref_0, "align-A"), (ref_1, "sort-A"), ...]:
    try:
        ray.get(ref)
        results[job_id] = {"status": "completed"}
    except Exception as e:
        results[job_id] = {"status": "failed", "error": str(e)}

# Write manifest for ox status
with open("results.json", "w") as f:
    json.dump(results, f)

The Ray dashboard shows this as 1 job with a task-level progress bar:

Ray Dashboard → Jobs → raysubmit_abc123
  Tasks: ████████░░░░░░  127/847 (15%)
  Running: 16  |  Pending: 704  |  Completed: 127

Call Mode and the Ray Object Store

This is where OxyMake and Ray truly complement each other. In call mode, OxyMake manages I/O outside the function — and on the Ray executor, that I/O goes through Ray's distributed object store instead of disk.

Data Flow: Shell vs Call vs Ray-Call

flowchart TB
    subgraph "Shell Mode (any executor)"
        S1["Job A"] -->|"write file<br/>results/A.csv"| SD[("Disk")]
        SD -->|"read file<br/>results/A.csv"| S2["Job B"]
    end

    subgraph "Call Mode (local executor)"
        C1["Job A<br/><i>compute_features(df)</i>"] -->|"Arrow IPC<br/>in-process"| C2["Job B<br/><i>train_model(features)</i>"]
    end

    subgraph "Call Mode (Ray executor)"
        R1["Job A<br/><i>@ray.remote</i>"] -->|"ray.put()<br/>→ object store"| RO[("Ray Object<br/>Store")]
        RO -->|"ray.get()<br/>zero-copy"| R2["Job B<br/><i>@ray.remote</i>"]
    end

    style SD fill:#fcc,stroke:#333
    style RO fill:#cfc,stroke:#333

How Ray Call Mode Works

When a call-mode job runs on the Ray executor, OxyMake generates a wrapper script (via call_mode.rs) that integrates with the object store:

sequenceDiagram
    participant D as Driver Script
    participant OS as Ray Object Store
    participant W as Worker (call-mode task)
    participant FS as Shared Filesystem

    D->>OS: ray.put(input_data)
    Note over D: ObjectRef stored

    D->>W: run_call.remote(job_id, module, func, input_ref)
    W->>OS: ray.get(input_ref)
    Note over W: Zero-copy if same node

    W->>W: result = func(input_data)

    W->>OS: ray.put(result)
    Note over W: ObjectRef returned

    alt materialize = "always" or "final" (leaf)
        W->>FS: write result to disk
    end

    D->>D: Pass ObjectRef to downstream tasks

Materialization Policies on Ray

OxyMake's materialization policies map directly to Ray behavior:

Example rule with materialization:

[rule.compute_features]
input = [{ path = "data/{sample}.parquet", format = "parquet" }]
output = [{ path = "features/{sample}.parquet", format = "parquet", materialize = "auto" }]
call = "pipeline.features:compute_features"
lang = "python"
resources = { cpu = 4, mem_gb = 8 }

With materialize = "auto" on the Ray executor, the features DataFrame lives in the Ray object store. If the next rule is also call-mode on Ray, data passes through the object store with zero disk I/O. If a downstream rule is shell-mode and needs a file path, OxyMake automatically materializes to disk.

The Bridge (ADR-008)

The ExecutorBridge trait formalizes the separation between OxyMake's scheduler and remote executors. It defines three communication directions:

flowchart LR
    subgraph "OxyMake (Rust)"
        S["Scheduler<br/><i>DAG owner, cache, gates</i>"]
        ST["ox status"]
    end

    subgraph "ExecutorBridge"
        direction TB
        SUB["SUBMIT<br/><i>submit_dag()</i><br/><i>map_resources()</i>"]
        MON["MONITOR<br/><i>poll_dag_status()</i><br/><i>fetch_logs()</i><br/><i>sync_results()</i><br/><i>reconnect()</i>"]
        CTL["CONTROL<br/><i>cancel_job()</i><br/><i>cancel_all()</i>"]
    end

    subgraph "Ray Cluster"
        R["Ray Jobs API<br/><i>Driver + tasks</i>"]
    end

    S -->|"uncached subgraph"| SUB
    SUB -->|"driver.py"| R
    R -->|"status, logs"| MON
    MON -->|"DagStatus"| ST
    S -->|"cancel"| CTL
    CTL -->|"ray job stop"| R

Separation of Concerns

State Synchronization

After submission, OxyMake stays connected via the bridge:

1. Submit: submit_dag() generates the driver script, submits to Ray Jobs API, writes meta.json to .oxymake/runs/{run_id}/
2. Poll: poll_dag_status() queries Ray for per-task status, returns DagStatus with job-level completion info
3. Sync: sync_results() writes job results (exit codes, durations, peak memory) back to OxyMake's state database
4. Reconnect: After an OxyMake crash, reconnect() reads meta.json and reconstructs a handle to the still-running Ray job

The meta.json contract:

{
  "executor": "ray",
  "version": 1,
  "submitted_at": "2025-04-01T12:00:00Z",
  "connection": {
    "ray_address": "http://127.0.0.1:8265",
    "ray_job_id": "raysubmit_abc123"
  },
  "run_id": "run-20250401-120000",
  "total_jobs": 847,
  "active_jobs": 847,
  "skipped_jobs": 102582
}

Resource Mapping

OxyMake resources map to Ray resources via map_resources():

Ray's advantage: fractional GPUs (num_gpus=0.5) enable model serving workloads where multiple inference tasks share a single GPU.

This table describes the checked mapper output and native DAG task options. The per-job Jobs API and array-emulation limitations are described in Resource admission.

Philosophy: Complementary, Not Overlapping

OxyMake and Ray solve orthogonal problems:

Why Not Snakemake + Ray or Airflow + Ray?

Snakemake + Ray: Snakemake's file-based cache uses timestamps, not content hashes. It has no materialization policies, no call mode, and its Python DSL prevents static analysis. Adding Ray to Snakemake gives you distributed execution but not the optimization pipeline (task fusion, materialization elimination) that makes the combination powerful.

Airflow + Ray: Airflow is an orchestrator that owns the DAG schedule. Adding Ray as an executor gives you distributed compute, but Airflow's DAG model is runtime-defined Python, not declarative TOML. You cannot inspect or optimize an Airflow DAG before execution.

OxyMake + Ray: OxyMake's declarative format enables static analysis and optimization passes before execution. Ray provides elastic compute and zero-copy data passing during execution. Neither system steps on the other's responsibilities.

flowchart LR
    subgraph "OxyMake Responsibilities"
        direction TB
        A1["Parse Oxymakefile.toml"]
        A2["Resolve wildcards"]
        A3["Check content-addressable cache"]
        A4["Optimize: fuse, prune, eliminate"]
        A5["Generate driver script"]
        A1 --> A2 --> A3 --> A4 --> A5
    end

    subgraph "Ray Responsibilities"
        direction TB
        B1["Receive driver script"]
        B2["Schedule tasks on workers"]
        B3["Manage object store"]
        B4["Autoscale cluster"]
        B5["Report task status"]
        B1 --> B2 --> B3 --> B4 --> B5
    end

    A5 -->|"Ray Jobs API"| B1
    B5 -->|"poll_dag_status()"| A5

    style A5 fill:#ff9,stroke:#333
    style B1 fill:#9ff,stroke:#333

Quick Start

1. Start a Ray cluster

ray start --head
# Dashboard: http://127.0.0.1:8265

2. Configure OxyMake

# Oxymakefile.toml
[executor.ray]
dashboard_address = "http://127.0.0.1:8265"

3. Run your workflow on Ray

ox run --executor ray

OxyMake handles caching, DAG optimization, and driver generation. Ray handles task placement, GPU scheduling, and data passing. Your workflow file does not change.

Shared filesystem requirement

The native Ray DAG driver is generated under the Oxymakefile's .oxymake/runs/ directory and submitted to Ray by its shell-quoted absolute path, with symlinks resolved. For example, -f <symlink>/Oxymakefile.toml submits the path under the real project directory. OxyMake needs write access to the staging directory; staging failures report the affected path and the underlying IO error. The Ray head and workers that execute the driver must be able to see that OxyMake working directory through a shared filesystem. OxyMake does not upload the driver with runtime_env.working_dir, because changing Ray's job working directory would change how workflow rules resolve their relative inputs and outputs. A remote head without that shared mount is not supported by this executor.

4. Monitor execution

ox status                 # OxyMake's view (aggregated)
# or visit Ray Dashboard for task-level detail

Further Reading

• Executors -- all available executors and configuration
• Execution Modes -- shell, run, script, call
• The Three Graphs -- RuleGraph, JobGraph, ExecGraph
• Materialization Policy -- controlling disk I/O
• Content-Addressable Cache -- how cache keys work

OxyMake × SLURM Deep Dive

OxyMake and SLURM solve different halves of the HPC workflow problem. OxyMake owns the what: which jobs to run, in what order, and what can be skipped. SLURM owns the where: which node, how many cores, how much memory. This page explains how the two systems fit together — from job packaging through monitoring to real-cluster deployment.

Running a rule on the submitting host

Set executor = "local" on a rule that must run on the host running ox:

[rule.prepare]
output = ["prepared.txt"]
executor = "local"
shell = "echo ready > prepared.txt"

When the selected, uncached jobs contain this override, ox run --executor slurm uses scheduler dispatch instead of DAG submission and waits for completion, including without --follow. Other rules still use SLURM. --jobs N bounds concurrent jobs, including cluster submissions, and defaults to 1: with default flags, independent cluster jobs run one at a time. Increase --jobs to allow concurrent submissions. The ox process must stay alive until the run completes. A notice at run start explains this switch and prints the current job limit. Native DAG submission and job arrays are not used for mixed runs.

--resource-budget is allowed in a mixed SLURM run and applies only to jobs routed locally. For example, --jobs 8 --resource-budget cpu=2 permits up to eight concurrent jobs while admitting at most two declared local CPU tokens. Remote jobs keep their SLURM resource requests and consume no local budget. Admission validates selected local declarations even when cached; rules excluded by --until or --omit-from do not participate. Pure remote runs reject this local admission flag.

The usual cache, retry, logging and cancellation paths apply. A shared filesystem must make dependencies visible on both hosts; OxyMake does not transfer artifacts between them. --json job-start events and ox history --run-id ... --json report each executed job's actual backend.

The only rule-level value is "local"; backend selection belongs to ox run --executor. Ray's whole-DAG path rejects active local overrides. Overrides excluded by target selection, --until, --omit-from, or cache checks neither block Ray nor switch SLURM away from DAG submission. Forced reruns count as active work. Site placement and multiple remote executors within one run are outside this feature's scope.

The Three Graphs Meet SLURM

Before any executor sees a job, OxyMake transforms the user's declarations through three graph representations. Understanding this pipeline is essential for understanding what SLURM actually receives.

Graph Transformation Pipeline

flowchart TD
    A["Oxymakefile.toml<br/><i>Declarative TOML</i>"] --> B["RuleGraph<br/><i>Abstract: wildcards intact</i>"]
    B -->|"Wildcard resolution<br/>+ guard evaluation"| C["JobGraph<br/><i>Concrete: every job instance</i>"]
    C -->|"Optimization passes"| D["Optimized JobGraph"]
    D -->|"Cache pruning removes<br/>up-to-date jobs"| E["Uncached Subgraph"]
    E -->|"submit_dag()"| F["sbatch scripts<br/><i>Per-job or job arrays<br/>with --dependency chains</i>"]
    F -->|"sbatch --parsable<br/>+ --dependency=afterok"| G["SLURM Scheduler<br/><i>slurmctld</i>"]

    style A fill:#f9f,stroke:#333
    style F fill:#ff9,stroke:#333
    style G fill:#9ff,stroke:#333

Optimization Before Submission

Before any executor sees the graph, OxyMake runs optimization passes:

Only the uncached subgraph is submitted to SLURM. After pruning, ox plan reports the jobs that remain, in the standard plan format -- for a large, mostly-cached pipeline:

Plan: 12 rules, 847 jobs, 1203 source files

SLURM Job Packaging

Two Submission Modes

OxyMake supports two SLURM submission strategies, chosen automatically:

flowchart TB
    subgraph "OxyMake (Rust)"
        A["Optimized JobGraph<br/>847 uncached jobs"]
    end

    A --> DECIDE{"Same rule,<br/>many wildcards?"}

    DECIDE -->|"Yes"| ARRAY["Job Array<br/><i>1 sbatch + N tasks</i>"]
    DECIDE -->|"No"| INDIVIDUAL["Individual Jobs<br/><i>N sbatch calls with<br/>--dependency=afterok chains</i>"]

    subgraph "SLURM Cluster"
        ARRAY --> SC["slurmctld"]
        INDIVIDUAL --> SC
        SC --> C1["c1"]
        SC --> C2["c2"]
        SC --> CN["..."]
    end

    style A fill:#ff9,stroke:#333
    style SC fill:#9ff,stroke:#333

Mode 1: Individual jobs with --dependency=afterok chains. Each job gets its own sbatch script. Upstream dependencies are encoded as --dependency=afterok:JOBID1:JOBID2. Jobs are submitted in topological order so that upstream SLURM IDs are known before downstream jobs reference them. Cached upstream jobs are omitted — their outputs already exist on the shared filesystem, so no SLURM dependency is needed.

Mode 2: Job arrays for wildcard-expanded rules. When a single rule (e.g., process) expands to many concrete jobs via wildcards, OxyMake packages them as a single SLURM job array. One sbatch call submits all tasks. Each task reads its parameters from a JSON-lines file indexed by SLURM_ARRAY_TASK_ID.

Why --dependency=afterok Chains?

Unlike the Ray executor (which generates a single driver script), the SLURM executor submits one sbatch per job (or job array) and lets SLURM's own scheduler enforce ordering:

Generated Job Script Structure

The Rust code in ox-exec-slurm/src/job_script.rs generates bash scripts that look like this:

#!/bin/bash
#SBATCH --job-name=ox_process_j-042
#SBATCH --output=/scratch/staging/run-001/j-042/slurm-%j.out
#SBATCH --error=/scratch/staging/run-001/j-042/slurm-%j.err
#SBATCH --partition=gpu
#SBATCH --account=my-lab
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --gpus=1
#SBATCH --time=01:06:00

# --- Environment setup ---
set -euo pipefail
module load conda 2>/dev/null || true
eval "$(conda shell.bash hook)"
conda activate ml-env

# --- Working directory ---
cd "/data/projects/my-pipeline"

# --- Execute command ---
python train.py --sample=s01 --output=results/s01.parquet

Key design decisions:

• cd to project directory (not staging dir) so that relative output paths resolve to the same locations as the local executor — essential for cache correctness.
• Job name truncated to 255 characters (SLURM's limit).
• set -euo pipefail so failures propagate immediately.
• --time derived from job timeout with a 10% buffer if not explicitly set via the time resource.

Job Array Script Structure

For wildcard-expanded rules, OxyMake generates an array script with a parameter file:

#!/bin/bash
#SBATCH --job-name=ox_array_align
#SBATCH --array=0-4%2
#SBATCH --output=/scratch/staging/slurm-%A_%a.out
#SBATCH --error=/scratch/staging/slurm-%A_%a.err
#SBATCH --partition=gpu
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G

# --- Environment setup ---
set -euo pipefail

# --- Working directory ---
cd "/data/projects/pipeline"

# --- Array task dispatch ---
PARAMS_FILE="$(dirname "$0")/array_params.jsonl"
TASK_LINE=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" "$PARAMS_FILE")

# Export wildcard values as environment variables
export OX_JOB_ID=$(echo "$TASK_LINE" | python3 -c 'import sys,json; print(json.load(sys.stdin)["job_id"])')
export OX_WC_sample=$(echo "$TASK_LINE" | python3 -c 'import sys,json; print(json.load(sys.stdin)["wildcards"]["sample"])')

# --- Execute command ---
TASK_CMD=$(echo "$TASK_LINE" | python3 -c 'import sys,json; print(json.load(sys.stdin)["command"])')
eval "$TASK_CMD"

The companion array_params.jsonl:

{"index":0,"job_id":"j-1","wildcards":{"sample":"A"},"command":"bwa mem -t 8 ref.fa data/A.fq > results/A.bam"}
{"index":1,"job_id":"j-2","wildcards":{"sample":"B"},"command":"bwa mem -t 8 ref.fa data/B.fq > results/B.bam"}
{"index":2,"job_id":"j-3","wildcards":{"sample":"C"},"command":"bwa mem -t 8 ref.fa data/C.fq > results/C.bam"}

The %2 suffix in --array=0-4%2 throttles to 2 concurrent tasks (configurable via job_array.max_concurrent).

The Bridge (ADR-008)

The Executor trait formalizes the separation between OxyMake's scheduler and remote executors. The SLURM executor implements these communication directions:

flowchart LR
    subgraph "OxyMake (Rust)"
        S["Scheduler<br/><i>DAG owner, cache, gates</i>"]
        ST["ox status"]
    end

    subgraph "Executor Trait"
        direction TB
        INIT["INIT<br/><i>init(), health_check()</i>"]
        SUB["SUBMIT<br/><i>submit_dag(), execute()</i>"]
        MON["MONITOR<br/><i>poll_status()</i>"]
        CTL["CONTROL<br/><i>cancel(), cleanup()</i>"]
    end

    subgraph "SLURM Cluster"
        R["slurmctld<br/><i>sbatch, sacct, squeue</i>"]
    end

    S -->|"uncached subgraph"| SUB
    SUB -->|"sbatch --parsable"| R
    R -->|"sacct/squeue"| MON
    MON -->|"JobStatus"| ST
    S -->|"cancel"| CTL
    CTL -->|"scancel"| R
    INIT -->|"sinfo --version"| R

Separation of Concerns

State Synchronization

After submission, OxyMake stays connected via adaptive polling:

sequenceDiagram
    participant OX as OxyMake Scheduler
    participant FS as Shared Filesystem
    participant SC as slurmctld
    participant C as Compute Nodes

    OX->>FS: Write sbatch scripts to staging_dir
    OX->>SC: sbatch --parsable job.sh
    SC-->>OX: 12345 (SLURM job ID)
    OX->>FS: Write meta.json

    loop Adaptive Polling (5s-60s)
        OX->>SC: sacct -j 12345 --parsable2
        SC-->>OX: 12345|RUNNING|0:0|512M|00:05:30|c1
        Note over OX: State change → reset backoff
    end

    alt sacct unavailable
        OX->>SC: squeue -j 12345 -h -o %T
        SC-->>OX: RUNNING
    end

    alt Job terminal (COMPLETED/FAILED)
        OX->>OX: Map SLURM state → JobResult
        OX->>FS: Collect slurm-*.out/err logs
        OX->>FS: Clean staging directory
    end

Adaptive backoff prevents overloading slurmctld:

• Start at 5 seconds (configurable via poll_interval_min)
• Multiply by 1.5× each poll with no state change
• Cap at 60 seconds (configurable via poll_interval_max)
• Reset to minimum on any state change
• Batch queries: sacct -j id1,id2,...,idN — one call for all jobs

The meta.json contract:

{
  "executor": "slurm",
  "version": 1,
  "run_id": "run-20250401-120000",
  "total_jobs": 847,
  "active_jobs": 847,
  "skipped_jobs": 102582,
  "job_mapping": {
    "align-A": "12345",
    "align-B": "12346",
    "sort-A": "12347"
  }
}

Resource Mapping

OxyMake resources map to SLURM #SBATCH directives via resource_mapper.rs:

Mutual exclusion: --mem and --mem-per-cpu cannot both be specified. OxyMake validates this at submission time and returns a clear error.

Timeout derivation: If no explicit time resource is set but the job has a timeout, OxyMake derives --time with a 10% buffer. A 1-hour timeout becomes --time=01:06:00.

[rule.train]
output = ["model/weights.pt"]
resources = { cpu = 8, mem = "32G", gpu = 2, time = "4:00:00" }
environment = { conda = "torch-env" }
shell = "python train.py --epochs=100"

SLURM Job States

SLURM reports over a dozen job states. OxyMake maps them to four:

stateDiagram-v2
    [*] --> Queued: sbatch accepted
    Queued --> Running: Resources allocated

    Running --> Completed: Exit code 0
    Running --> Failed: Non-zero exit
    Running --> Failed: TIMEOUT
    Running --> Failed: OUT_OF_MEMORY
    Running --> Failed: NODE_FAIL
    Running --> Cancelled: scancel / PREEMPTED

    state Queued {
        PENDING
        REQUEUED
        SUSPENDED
        CONFIGURING
    }

    state Running {
        RUNNING_STATE: RUNNING
        COMPLETING
        RESIZING
    }

    state Failed {
        FAILED_STATE: FAILED
        TIMEOUT_STATE: TIMEOUT
        OOM: OUT_OF_MEMORY
        NODE_FAIL_STATE: NODE_FAIL
        BOOT_FAIL
        DEADLINE
    }

    state Cancelled {
        CANCELLED_STATE: CANCELLED
        PREEMPTED_STATE: PREEMPTED
        REVOKED
    }

Failed Node Exclusion

When a job reports NODE_FAIL or BOOT_FAIL, OxyMake:

1. Queries sacct for the failing node's hostname
2. Adds it to an in-memory exclusion set
3. Passes --exclude=node1,node2 on all future sbatch submissions
4. Reports excluded nodes when the workflow completes

This prevents cascading failures from bad hardware without requiring manual intervention.

Monitoring: sacct Primary, squeue Fallback

Status polling uses a two-tier strategy:

flowchart TD
    START["Poll job status"] --> SACCT["sacct -j ID --parsable2"]
    SACCT --> SACCT_OK{"Records<br/>found?"}
    SACCT_OK -->|"Yes"| PARSE["Parse state,<br/>exit code, memory,<br/>elapsed, node"]
    SACCT_OK -->|"No (empty or failed)"| SQUEUE["squeue -j ID -h -o %T"]
    SQUEUE --> SQ_OK{"Job in<br/>queue?"}
    SQ_OK -->|"Yes"| RUNNING["Report as<br/>Running/Queued"]
    SQ_OK -->|"No"| RETRY["Wait 2s,<br/>retry sacct"]
    RETRY --> RETRY_OK{"Found<br/>now?"}
    RETRY_OK -->|"Yes"| PARSE
    RETRY_OK -->|"No"| LOST["Report as<br/>JobNotFound"]
    PARSE --> TERMINAL{"Terminal<br/>state?"}
    TERMINAL -->|"Yes"| RESULT["Return JobResult<br/>(exit code, duration,<br/>peak memory, node)"]
    TERMINAL -->|"No"| BACKOFF["Adaptive backoff<br/>(5s → 60s)"]
    BACKOFF --> START

Why the fallback? Some HPC clusters don't have slurmdbd (the SLURM accounting daemon) configured, making sacct unavailable. squeue always works but provides less information (no exit codes, no memory stats, no elapsed time for completed jobs).

The 2-second retry handles a race condition: a job can vanish from squeue (it finished) before sacct has ingested the accounting record.

Docker Setup: Containerized SLURM Cluster

OxyMake ships a Docker Compose setup for local testing and CI:

graph TB
    subgraph "docker-compose.yml"
        MYSQL["mysql<br/><i>MariaDB 10.11</i><br/>Port 3306"]
        DBD["slurmdbd<br/><i>Accounting daemon</i><br/>Port 6819"]
        CTL["slurmctld<br/><i>Controller</i><br/>Port 6817"]
        REST["slurmrestd<br/><i>REST API gateway</i><br/>Port 6820"]
        C1["c1<br/><i>Compute node</i>"]
        C2["c2<br/><i>Compute node</i>"]
    end

    SHARED[("/work<br/><i>Shared volume</i>")]
    DATA[("/data/lab<br/><i>Host bind mount</i>")]
    JWT[("shared-slurm<br/><i>JWT key volume</i>")]

    MYSQL --> DBD
    DBD --> CTL
    CTL --> REST
    CTL --> C1
    CTL --> C2

    JWT --- CTL
    JWT --- REST
    JWT --- DBD
    SHARED --- CTL
    SHARED --- C1
    SHARED --- C2
    DATA --- CTL
    DATA --- C1
    DATA --- C2

    style SHARED fill:#cfc,stroke:#333
    style DATA fill:#cfc,stroke:#333
    style JWT fill:#ff9,stroke:#333
    style REST fill:#9ff,stroke:#333

Start the Cluster

cd tests/slurm-docker
docker compose up -d

# Wait ~20 seconds for all services to initialize
docker compose exec slurmctld sinfo -N -h
# Output:
#   c1  normal  idle
#   c2  normal  idle

Cluster Configuration

The slurm.conf defines a minimal 2-node cluster:

ClusterName=oxymake-demo
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
AccountingStorageType=accounting_storage/slurmdbd
NodeName=c[1-2] CPUs=2 RealMemory=2048 State=UNKNOWN
PartitionName=normal Nodes=c[1-2] Default=YES MaxTime=INFINITE State=UP

Key settings:

• select/cons_tres with CR_Core: Consumable resources at the core level — each job gets exactly the cores it requests.
• sched/backfill: Allows smaller jobs to start while larger jobs wait for resources, improving utilization.
• slurmdbd with MariaDB: Full accounting so sacct works.

JWT Authentication Setup

The Docker cluster configures JWT authentication automatically:

1. slurmctld generates a random 256-bit key at startup (/etc/slurm/jwt_hs256.key)
2. The key is shared via the shared-slurm Docker volume
3. slurmdbd and slurmrestd pick up the key and add AuthAltTypes=auth/jwt to their configuration
4. Clients authenticate with X-SLURM-USER-TOKEN (JWT) and X-SLURM-USER-NAME headers

Generate a token for local testing:

# Generate a JWT token for user "root" (valid 1 hour)
docker compose exec slurmctld scontrol token lifespan=3600
# Output: SLURM_JWT=eyJhbGciOi...

export SLURM_JWT=eyJhbGciOi...

Port Mapping

Submit a Test Job

docker compose exec slurmctld bash -c '
  echo "#!/bin/bash
hostname
date
sleep 5
echo done" > /work/test.sh && sbatch /work/test.sh'
# Output: Submitted batch job 1

# Check status:
docker compose exec slurmctld sacct --parsable2 --noheader -o JobID,State,ExitCode
# Output: 1|COMPLETED|0:0

Teardown

docker compose down -v   # Remove containers and volumes

Two Modes: CLI vs REST API

Mode 1: CLI (sbatch / sacct)

The default and most common mode. OxyMake shells out to SLURM CLI commands. This works on any cluster where the user has SLURM in their $PATH:

# OxyMake internally runs:
sbatch --parsable job.sh                    # Submit → returns job ID
sacct -j 12345 --parsable2 -o JobID,State   # Poll status
scancel 12345                                # Cancel if needed

Pros: Works everywhere, no extra setup, respects Munge auth. Cons: One process spawn per command, rate limiting required at scale.

Mode 2: REST API (slurmrestd)

For programmatic access, SLURM provides slurmrestd — an HTTP/JSON gateway to the same operations:

# Start slurmrestd (typically done by the cluster admin)
slurmrestd -a rest_auth/local 0.0.0.0:6820

# Submit a job via HTTP
curl -X POST http://slurmctld:6820/slurm/v0.0.44/job/submit \
  -H "Content-Type: application/json" \
  -H "X-SLURM-USER-NAME: $USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  -d '{
    "script": "#!/bin/bash\nhostname\ndate",
    "job": {
      "name": "ox_test",
      "partition": "normal",
      "cpus_per_task": 4,
      "memory_per_node": { "number": 8, "set": true, "infinite": false },
      "tasks": 1
    }
  }'

# Poll status
curl http://slurmctld:6820/slurm/v0.0.44/job/12345 \
  -H "X-SLURM-USER-NAME: $USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT"

# Cancel
curl -X DELETE http://slurmctld:6820/slurm/v0.0.44/job/12345 \
  -H "X-SLURM-USER-NAME: $USER" \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT"

Pros: No process spawning, structured JSON responses, lower latency at scale. Cons: Requires slurmrestd to be running, JWT authentication setup, not universally available.

Both modes are supported. CLI mode is the default. To use REST mode, pass --slurm-api http://host:6820 (or set SlurmConfig::api_url). Authentication uses X-SLURM-USER-NAME (from $USER) and X-SLURM-USER-TOKEN (from $SLURM_JWT, optional).

REST API Flow

The full lifecycle of a job submitted via REST mode:

sequenceDiagram
    participant OX as OxyMake<br/>(Rust)
    participant REST as slurmrestd<br/>:6820
    participant CTL as slurmctld
    participant C as Compute Nodes<br/>(c1, c2)

    Note over OX: Generate job script<br/>+ write to staging_dir

    OX->>REST: POST /slurm/v0.0.44/job/submit<br/>Headers: X-SLURM-USER-NAME, X-SLURM-USER-TOKEN<br/>Body: {script, job: {name, partition, cpus, mem}}
    REST->>CTL: Internal SLURM protocol
    CTL-->>REST: job_id: 12345
    REST-->>OX: {"job_id": 12345}

    loop Adaptive Polling (5s–60s)
        OX->>REST: GET /slurm/v0.0.44/job/12345
        REST->>CTL: Query job state
        CTL-->>REST: Job state + metadata
        REST-->>OX: {"job_state": "RUNNING", ...}
    end

    CTL->>C: Dispatch job to node
    C->>C: Execute sbatch script
    C-->>CTL: Exit code 0

    OX->>REST: GET /slurm/v0.0.44/job/12345
    REST-->>OX: {"job_state": "COMPLETED", "exit_code": 0}

    alt Cancel needed
        OX->>REST: DELETE /slurm/v0.0.44/job/12345
        REST->>CTL: scancel 12345
    end

Environment requirement: Unlike CLI sbatch (which inherits the submitter's shell environment), the REST API starts with an empty environment. OxyMake injects default PATH and HOME variables to ensure scripts can find basic utilities.

The Bridge: OxyMake DAG → sbatch Dependency Chain

The core translation from OxyMake's DAG to SLURM's execution model:

flowchart LR
    subgraph "OxyMake DAG"
        direction TB
        A["generate-s01"]
        B["generate-s02"]
        C["process-s01"]
        D["process-s02"]
        E["merge"]
        F["report"]
        A --> C
        B --> D
        C --> E
        D --> E
        E --> F
    end

    subgraph "SLURM Submission"
        direction TB
        SA["sbatch generate-s01.sh<br/>→ SLURM 100"]
        SB["sbatch generate-s02.sh<br/>→ SLURM 101"]
        SC["sbatch --dependency=afterok:100<br/>process-s01.sh → SLURM 102"]
        SD["sbatch --dependency=afterok:101<br/>process-s02.sh → SLURM 103"]
        SE["sbatch --dependency=afterok:102:103<br/>merge.sh → SLURM 104"]
        SF["sbatch --dependency=afterok:104<br/>report.sh → SLURM 105"]
    end

    A -.-> SA
    B -.-> SB
    C -.-> SC
    D -.-> SD
    E -.-> SE
    F -.-> SF

Topological submission: Jobs are submitted in topological order. When OxyMake submits process-s01, it already knows that generate-s01 was assigned SLURM ID 100, so it can add --dependency=afterok:100.

Cached jobs are transparent: If generate-s01 is cached (outputs exist and are up-to-date), it is never submitted to SLURM. When process-s01 is submitted, its dependency list omits the cached job entirely — the outputs are already on the shared filesystem.

Environment Support on HPC

SLURM clusters have unique environment constraints:

Conda / Module System

HPC clusters use module load for software management. OxyMake generates the appropriate setup:

[rule.train.environment]
conda = "torch-env"

Generates:

module load conda 2>/dev/null || true
eval "$(conda shell.bash hook)"
conda activate torch-env

Containers are not applied by the SLURM backend

Known limitation. docker, apptainer and nix environments have no effect on the SLURM executor, and uv has only a partial one. The local executor wraps the command (docker run, apptainer exec, nix develop -c, uv run); the SLURM backend does not.

Most HPC clusters prohibit Docker (requires root), so a docker environment is rewritten to an Apptainer invocation in the generated sbatch script:

[rule.inference.environment]
docker = "nvcr.io/nvidia/pytorch:24.01-py3"

Generates:

# WARNING: Docker not supported on most HPC clusters.
# Consider using Apptainer (environment = { apptainer = "..." }).
OXYMAKE_CONTAINER_CMD="apptainer exec nvcr.io/nvidia/pytorch:24.01-py3"

OXYMAKE_CONTAINER_CMD is assigned but never expanded: the rule command is appended to the script verbatim, so the job runs outside the container. The same holds for an explicit apptainer environment, and for nix, which emits no setup at all. A uv environment emits uv sync -r <requirements> but does not run the command under uv run, so the interpreter on PATH is unchanged. Only conda takes effect, because conda activate mutates the script's own shell.

This matters beyond the missing isolation: the cache key hashes the declared environment regardless of backend, so a SLURM job that ran uncontained is stored under a key that records the container, and a later local run of the same rule — which does apply the container — reads it as a hit.

Until this is fixed, put the container invocation in the rule command itself on SLURM:

[rule.inference]
shell = "apptainer exec /shared/images/pytorch-24.01.sif python infer.py"

Shared Filesystem Constraint

Job scripts, inputs and outputs must live on a filesystem visible to both the scheduling node and the compute nodes — while state.db must not:

flowchart LR
    subgraph "Login / Submit Node"
        OX["ox run<br/>--executor slurm"]
        WS["workspace/<br/><i>Local disk only</i><br/>.oxymake/state.db<br/>+ inputs + outputs"]
    end

    subgraph "Shared Filesystem<br/>(NFS / Lustre / GPFS)"
        STAGE["staging_dir/<br/><i>sbatch scripts</i>"]
    end

    subgraph "Compute Nodes"
        C1["c1: slurmd"]
        C2["c2: slurmd"]
    end

    OX --> WS
    OX -->|"write scripts"| STAGE
    STAGE -->|"read scripts"| C1
    STAGE -->|"read scripts"| C2
    C1 -->|"read/write"| WS
    C2 -->|"read/write"| WS

    style WS fill:#fcc,stroke:#333
    style STAGE fill:#cfc,stroke:#333

Critical constraint: state.db uses SQLite WAL mode, which does not work on network filesystems (NFS, Lustre, GPFS). The ox run process must execute on a node with local disk. Compute nodes never access state.db — they only read sbatch scripts and read/write data files.

This constrains the whole workspace, not just the database. .oxymake/ is created in the working directory and cannot be relocated — there is no flag, config key, or environment variable for it — and a rule's inputs and outputs are resolved against that same directory. So the database cannot be kept on local disk while the project tree sits on shared storage: the entire workspace has to be on local disk, and the compute nodes must be able to reach it. On a cluster where $HOME is the only path visible to compute nodes and is itself NFS, there is no supported configuration; that is a known limitation, not a tuning question.

Only staging_dir is independently configurable, and it must be on the shared filesystem so compute nodes can read the generated scripts.

Configuration

Configure the SLURM executor in .oxymake/config.toml or Oxymakefile.toml:

[executor.slurm]
partition = "gpu"
account = "my-lab"
qos = "high"
staging_dir = "/scratch/oxymake"
max_submit = 100
poll_interval_min = "5s"
poll_interval_max = "60s"
extra_flags = ["--mail-type=FAIL", "--mail-user=user@lab.edu"]

[executor.slurm.job_array]
enabled = true
max_array_size = 1000
max_concurrent = 50

Switching to a Real Cluster

Moving from the Docker test cluster to a production HPC environment:

Grid'5000

[executor.slurm]
partition = "default"
staging_dir = "/home/$USER/oxymake-staging"
extra_flags = ["--reservation=my-reservation"]

# On a Grid'5000 frontend:
oarsub -I -t deploy -l nodes=4,walltime=2:00:00
# Then inside the reservation:
ox run --executor slurm -j 16

IDRIS (Jean Zay)

[executor.slurm]
partition = "gpu_p13"
account = "abc@v100"
qos = "qos_gpu-t3"
staging_dir = "$WORK/oxymake-staging"
extra_flags = ["--hint=nomultithread"]

# On Jean Zay:
module load python/3.11 cuda/12.1
ox run --executor slurm

GCP + Slurm-GCP

[profile.gcloud]
executor = "slurm"
partition = "batch"
account = "default"
jobs = 100

[profile.gcloud-gpu]
executor = "slurm"
partition = "gpu"
account = "default"
jobs = 20

ox run --profile gcloud

Google Cloud's HPC Toolkit deploys a SLURM cluster with autoscaling — nodes spin up on demand when jobs enter the queue and spin down when idle. The Filestore NFS mount provides the shared filesystem required by OxyMake's SLURM executor.

For a full setup guide including cluster provisioning, SSH tunneling, and cost control, see the Cloud HPC cookbook, which works a Google Cloud cluster as one concrete example.

Common Pitfalls

Philosophy: Complementary, Not Overlapping

OxyMake and SLURM solve orthogonal problems:

SLURM vs Ray: When to Use Which

Rule of thumb: Use SLURM when you have a shared HPC cluster with existing SLURM infrastructure. Use Ray when you need elastic scaling, fast job turnaround, or in-memory data passing between tasks.

Why Not Snakemake + SLURM?

Snakemake also integrates with SLURM, but with important differences:

Snakemake: Manages the DAG from a long-running process. Submits jobs one at a time as dependencies complete. Uses file timestamps for caching. Cannot do task fusion or materialization elimination.

OxyMake: Submits the entire dependency chain up front via --dependency=afterok. SLURM sees the full picture and can backfill more aggressively. Uses content hashes (not timestamps) for caching. Optimization passes (fusion, materialization elimination) reduce the number of jobs before submission.

Quick Start

1. Configure the executor

# Oxymakefile.toml
[executor.slurm]
partition = "normal"
staging_dir = "/scratch/$USER/oxymake"

2. Run your workflow on SLURM

ox run --executor slurm

OxyMake handles caching, DAG optimization, and sbatch generation. SLURM handles task placement, resource allocation, and scheduling. Your workflow file does not change.

3. Monitor execution

ox status                   # OxyMake's view (aggregated)
squeue -u $USER             # SLURM's view (per-job)
sacct -j <id> --format=...  # Detailed job accounting

4. Run the demo (Docker)

# Build OxyMake
cargo build --bin ox

# Start the test cluster and run the full demo
just demo-slurm
# Or manually:
bash tests/slurm-docker/run-demo.sh

Further Reading

• Executors -- all available executors and configuration
• Execution Modes -- shell, run, script, call
• The Three Graphs -- RuleGraph, JobGraph, ExecGraph
• OxyMake × Ray Deep Dive -- the Ray executor
• Content-Addressable Cache -- how cache keys work

Peak memory in run history

With ox run --executor slurm --follow, OxyMake records allocation status and elapsed time together with the maximum sacct MaxRSS across the job's steps. It does not sum step peaks. A zero or missing MaxRSS means accounting supplied no usable observation; ox history shows - (null in JSON). A successful finishing attempt with a positive measurement appears in whole MiB rounded up. Failed jobs and cancellations do not record memory. Submission without --follow, squeue fallback, and the REST backend do not supply memory.

See history for attempt semantics and its rounding difference from benchmark TSV. Persistence is covered by mock SLURM commands; verification on a live cluster remains with a cluster operator.

Stopping remote follow

With --follow, the first Ctrl+C or SIGTERM stops the client promptly with exit code 130. It does not cancel submitted SLURM allocations. Use ox status to check them and ox cancel to cancel them. Three consecutive poll rounds with errors end follow with exit code 1, naming the REST endpoint (or local sacct/squeue scheduler access) and last error. An error-free round resets the counter. Both stopped outcomes report last known counts without claiming the run completed. See remote follow.

Idempotent Execution

If you have used Terraform, you already understand OxyMake's execution model. ox run does not mean "launch these jobs." It means "ensure these outputs exist."

This is a fundamental design choice that affects everything from how you think about running workflows to how multiple people can work on the same pipeline simultaneously.

The Convergent Model

When you run ox run, OxyMake looks at each job in the requested subgraph and makes a decision:

The result: running the same command twice does nothing extra. Running it while another instance is already working cooperates instead of conflicting.

ox run --rule '/human/'         # Launches the human-cohort jobs
ox run --rule '/human/'         # all skipped (cached), nothing re-runs
ox run --rule '/human/'         # (while first is running) attaches to running jobs
ox run                          # Launches yeast+mouse, attaches to human

The Terraform Analogy

If you are familiar with infrastructure-as-code tools, the mapping is direct:

Just as terraform apply creates only the resources that don't already exist, ox run executes only the jobs whose outputs are missing or stale.

Cooperative Sessions

The most powerful consequence of idempotent execution is that multiple ox run processes can work on the same project simultaneously, without conflicts.

How It Works

OxyMake uses SQLite (WAL mode) as a coordination layer. When a session wants to execute a job, it claims it atomically:

UPDATE jobs SET status = 'running', session_id = ?, locked_by = ?
WHERE id = ? AND status = 'pending';

If another session already claimed the job (0 rows affected), the current session either waits for it (if it needs the output) or moves on to other work.

Example: Two Terminals

# Terminal 1: start the human pipeline
ox run --rule '/human/'
#  Session 1: 2,100 jobs to run

# Terminal 2 (while T1 is running): start the mouse pipeline
ox run --rule '/mouse/'
#  Session 2: 3,423 jobs to run. 0 conflicts with session 1.

# Terminal 3: run everything
ox run
#  Session 3: 10,247 total jobs
#    2,100 running (human, session 1) — attaching
#    3,423 running (mouse, session 2) — attaching
#    1,312 cached (completed by sessions 1+2) — skipping
#    3,412 to run (yeast + remaining) — executing

Session 3 does not duplicate work. It attaches to what sessions 1 and 2 are already doing, skips what they have finished, and picks up the rest.

Stale Session Recovery

If a session crashes (power failure, OOM kill), its jobs are not stuck forever. Each session sends a heartbeat every third of its lease (90 s by default; OX_SESSION_LEASE_SECS overrides). If the heartbeat is older than the lease, the session is considered dead: the first session waiting on one of its jobs resets that session's running jobs to pending and claims the one it needs.

No manual cleanup required.

The Lifecycle Commands

The convergent model needs symmetric operations. OxyMake provides five commands that form a complete algebra of workflow control:

Cancel

ox cancel --where cohort=human    # Stop human jobs
ox cancel --rule call             # Stop all variant calls
ox cancel --session 2             # Stop everything session 2 is doing
ox cancel                         # Stop everything

Canceled jobs have their partial outputs deleted and their status reset to pending. The next ox run will re-execute them.

Invalidate

ox invalidate --rule call                  # Delete variant-call outputs + cache entries
ox invalidate --rule call --cascade        # + all downstream outputs
ox invalidate --since "2026-03-22"         # Everything computed after this date
ox invalidate --run 3                      # Everything from run #3

The --cascade flag is important: invalidating a feature rule without cascade leaves stale calls that depend on the old feature values. With --cascade, OxyMake traverses the DAG forward and invalidates everything downstream.

Why This Matters

The idempotent execution model means:

1. No accidental double-execution. Two people running the same command cooperate instead of conflicting.
2. Fearless re-running. You can always run ox run again. If everything is up to date, it finishes instantly.
3. Incremental by nature. Add new rules, change parameters, re-run. Only the affected subgraph recomputes.
4. Crash-resilient. Completed work survives process death. Just re-run.
5. Observable. ox status shows exactly what is happening across all sessions.

Crate Graph — How OxyMake Fits Together

A first-time contributor clones two dozen ox-* crates and needs a mental model before reading any code. This page is that model: which crate does what, which depends on which, and the one rule that keeps the whole thing legible.

If you only remember one sentence, remember this:

ox-core takes no ox-* dependency. Every other crate points inward, toward ox-core. Nothing points back out.

That is the textbook hexagonal (ports-and-adapters) shape. ox-core is the domain. The crates around it are either supporting domain libraries, driven adapters (things the engine calls — executors, storage, reports), driving adapters (things that call the engine — the CLI, the MCP server), or the composition layer that wires them together (ox-api).

This is not the same picture as the three-graph data pipeline (RuleGraph → JobGraph → ExecGraph) in The Three Graphs. That describes how a workflow is resolved at runtime. This page describes how the code is layered. Newcomers routinely conflate the two — they are orthogonal.

The shape

graph TB
    subgraph driving["Driving adapters — entrypoints (call the engine)"]
        cli["ox-cli<br/>the ox binary"]
        mcp["ox-mcp<br/>MCP server for agents"]
    end

    subgraph app["Composition layer (wires the engine together)"]
        api["ox-api<br/>embeddable Rust facade"]
    end

    subgraph support["Supporting domain libraries (depend only on ox-core)"]
        format["ox-format"]
        state["ox-state"]
        cache["ox-cache"]
        plan["ox-plan"]
        codec["ox-codec-core"]
        lock["ox-lock"]
    end

    core(["ox-core<br/>domain core — ZERO ox-* deps"])

    subgraph driven["Driven adapters (the engine calls them)"]
        execlocal["ox-exec-local"]
        execray["ox-exec-ray"]
        execslurm["ox-exec-slurm"]
        envsys["ox-env-system"]
        envuv["ox-env-uv"]
        storage["ox-storage-local"]
        repjson["ox-report-json"]
        repterm["ox-report-term"]
        render["ox-render"]
        translate["ox-translate"]
        dashboard["ox-dashboard"]
        tui["ox-monitor-tui"]
    end

    cli --> api
    cli -->|"+ every driven adapter (see table)"| driven
    mcp --> core
    mcp --> format
    mcp --> state
    mcp --> cache
    mcp --> plan
    api --> core
    api --> format
    api --> state
    api --> cache
    api --> plan

    format --> core
    state --> core
    cache --> core
    plan --> core
    codec --> core
    lock --> core

    execlocal --> codec
    execlocal --> core
    execray --> codec
    execray --> core
    execslurm --> core
    envsys --> core
    envuv --> core
    storage --> core
    repjson --> core
    repterm --> render
    repterm --> core
    translate --> format
    translate --> core
    dashboard --> state
    tui --> state

    classDef hub fill:#1f6feb,color:#fff,stroke:#0b3a8c,stroke-width:2px;
    class core hub;

Every arrow A --> B means "crate A depends on crate B". They all flow inward. ox-core has no outgoing ox-* arrow — that is the load-bearing invariant. (ox-render also has no ox-* dependency: it is a pure terminal- styling leaf that ox-report-term builds on, not a second hub.)

Roles, one line each

ox-core is the hub; the rest are grouped by their architectural role.

The hub

Supporting domain libraries (depend only on ox-core)

Composition layer

Driving adapters (entrypoints — they call the engine)

Driven adapters (the engine calls them — each implements an ox-core trait)

Not yet wired into the ox binary

These crates compile and depend only inward, but no entrypoint consumes them yet. They are staged for a future release, not dead code.

Outside the engine graph

The exact edges (verified against cargo tree)

The table below is the authoritative ox-* → ox-* edge list. It is generated from each crate's [dependencies] and matches cargo tree -e no-dev --workspace. The diagram above shows the shape; this table is the ground truth. If you change an inter-crate dependency, update this table (and re-confirm the inward-pointing rule).

To regenerate this view locally:

cargo tree -e no-dev --workspace      # full dependency tree
cargo tree -e no-dev -i ox-core       # invert: who depends on ox-core (≈ everyone)

Why this matters

The inward-pointing rule is what lets you add a new executor, a new storage backend, or a new report format without touching ox-core — you implement the relevant ox-core trait in a new ox-exec-* / ox-storage-* / ox-report-* crate and register it in ox-cli (or ox-api). The domain never learns about its adapters. That is the whole point of the hexagon, and it is the project's single best legibility asset.

For the formal boundary between what OxyMake proves and what it assumes of the substrate, see the Boundary — Substrate Axioms note in the repository.

Bioinformatics Pipeline

This cookbook walks through a multi-sample FASTQ-to-BAM-to-VCF variant calling pipeline in OxyMake. The workflow uses sort, grep, and wc as stand-ins for real bioinformatics tools (BWA, samtools, GATK), so you can run it on any machine without installing anything.

The concepts transfer directly to a production pipeline: just swap the shell commands for real tool invocations.

What You Will Learn

• Wildcard-driven sample processing across multiple samples
• Named inputs for rules with multiple input files
• Tags for organizing pipeline stages
• Target-based filtering to run a subset of samples
• --rule filtering to run a subset of stages

The Complete Oxymakefile

Create a directory and save this as Oxymakefile.toml:

ox_version = "0.1"

[config]
# Wildcard value lists. The key must match the wildcard name, either exactly
# or as its plural: `{sample}` resolves from `samples`, `{chrom}` from
# `chroms`. A key like `chromosomes` would NOT resolve `{chrom}`.
samples = ["NA12878", "NA12891", "NA12892"]
chroms  = ["chr1", "chr2", "chr3"]

# ── Default target ──────────────────────────────────────────────
[rule.all]
input = ["results/cohort_report.txt"]

# ── Stage 1: Generate mock FASTQ reads ─────────────────────────
[rule.simulate_reads]
output = ["fastq/{sample}_R1.fastq", "fastq/{sample}_R2.fastq"]
tags = { stage = "simulate", speed = "fast" }
shell = """
mkdir -p fastq
for i in $(seq 1 50); do
  echo "@{sample}_read${i}/1 chr$((i % 3 + 1)):$((i * 100))" >> {output[0]}
  echo "ACGTACGTACGTACGT" >> {output[0]}
  echo "+" >> {output[0]}
  echo "IIIIIIIIIIIIIIII" >> {output[0]}
  echo "@{sample}_read${i}/2 chr$((i % 3 + 1)):$((i * 100))" >> {output[1]}
  echo "TGCATGCATGCATGCA" >> {output[1]}
  echo "+" >> {output[1]}
  echo "IIIIIIIIIIIIIIII" >> {output[1]}
done
"""

# ── Stage 2: Align reads → sorted BAM ──────────────────────────
# Stand-in: sort the FASTQ by read name to simulate alignment + sorting.
[rule.align]
input = { r1 = "fastq/{sample}_R1.fastq", r2 = "fastq/{sample}_R2.fastq" }
output = ["aligned/{sample}.bam"]
tags = { stage = "align", cost = "heavy" }
resources = { cpu = 4, mem = "8G" }
shell = """
mkdir -p aligned
echo "## BAM for {sample}" > {output}
echo "## Aligned from {input.r1} and {input.r2}" >> {output}
cat {input.r1} {input.r2} | grep "^@" | sort >> {output}
echo "## EOF" >> {output}
"""

# ── Stage 3: Call variants per chromosome ───────────────────────
# Stand-in: grep reads matching the chromosome, count them as "variants."
[rule.call_variants]
input = { bam = "aligned/{sample}.bam" }
output = ["vcf/{sample}_{chrom}.vcf"]
# Without this, `vcf/{sample}_merged.vcf` would also match this pattern
# (chrom="merged") and collide with `merge_vcf`.
wildcard_constraints = { chrom = "chr[0-9]+" }
tags = { stage = "call", cost = "heavy" }
resources = { cpu = 2, mem = "4G" }
shell = """
mkdir -p vcf
echo "##fileformat=VCFv4.2" > {output}
echo "##source=oxymake-cookbook" >> {output}
echo "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO" >> {output}
grep "{chrom}" {input.bam} | awk '{{
  split($2, a, ":");
  printf "%s\\t%s\\t.\\tA\\tG\\t30\\tPASS\\tDP=20\\n", a[1], a[2]
}}' >> {output}
"""

# ── Stage 4: Merge per-chromosome VCFs into one per sample ─────
[rule.merge_vcf]
input = ["vcf/{sample}_{chrom}.vcf"]
output = ["vcf/{sample}_merged.vcf"]
# `{chrom}` appears in the input but not the output: this rule aggregates
# over every chrom value, so it needs an explicit expand mode.
expand = "product"
tags = { stage = "merge" }
shell = """
echo "##fileformat=VCFv4.2" > {output}
echo "##source=oxymake-merge" >> {output}
echo "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO" >> {output}
for f in {input}; do
  grep -v "^#" "$f" >> {output}
done
sort -k1,1 -k2,2n -o {output} {output}
"""

# ── Stage 5: Per-sample QC report ──────────────────────────────
[rule.qc]
input = { bam = "aligned/{sample}.bam", vcf = "vcf/{sample}_merged.vcf" }
output = ["qc/{sample}_report.txt"]
tags = { stage = "qc", speed = "fast" }
shell = """
mkdir -p qc
echo "=== QC Report: {sample} ===" > {output}
echo "Total reads: $(grep -c "^@" {input.bam})" >> {output}
echo "Variants called: $(grep -vc "^#" {input.vcf})" >> {output}
echo "Chromosomes: $(grep -v "^#" {input.vcf} | cut -f1 | sort -u | tr '\n' ' ')" >> {output}
"""

# ── Stage 6: Cohort report ─────────────────────────────────────
[rule.cohort_report]
input = ["qc/{sample}_report.txt"]
output = ["results/cohort_report.txt"]
# Aggregates over every sample.
expand = "product"
tags = { stage = "report" }
shell = """
mkdir -p results
echo "=============================" > {output}
echo " Variant Calling Cohort Report" >> {output}
echo "=============================" >> {output}
echo "" >> {output}
for f in {input}; do
  cat "$f" >> {output}
  echo "" >> {output}
done
echo "--- Summary ---" >> {output}
echo "Samples processed: $(echo {input} | wc -w | tr -d ' ')" >> {output}
"""

Create the Project

mkdir bioinfo-pipeline && cd bioinfo-pipeline
# Save the Oxymakefile.toml above
ox init   # if you want the .oxymake directory pre-created

No input data files are needed -- the simulate_reads rule generates everything from scratch.

Run the Full Pipeline

ox plan

Plan: 7 rules, 22 jobs, 0 source files
Targets: results/cohort_report.txt
  1. [simulate_reads-NA12892] rule=simulate_reads -> [fastq/NA12892_R1.fastq, fastq/NA12892_R2.fastq]
  2. [align-NA12892] rule=align -> [aligned/NA12892.bam]
  3. [call_variants-chr3-NA12892] rule=call_variants -> [vcf/NA12892_chr3.vcf]
  4. [call_variants-chr2-NA12892] rule=call_variants -> [vcf/NA12892_chr2.vcf]
  5. [call_variants-chr1-NA12892] rule=call_variants -> [vcf/NA12892_chr1.vcf]
  6. [merge_vcf-NA12892] rule=merge_vcf -> [vcf/NA12892_merged.vcf]
  ...
  22. [cohort_report] rule=cohort_report -> [results/cohort_report.txt]

ox run -j 4

OxyMake runs up to 4 jobs in parallel. The simulate_reads jobs run first (no dependencies), then align, then call_variants fans out across samples and chromosomes, and finally everything converges into the cohort report.

Filter by Sample

Run only one sample during development by requesting its leaf target (wildcards in the target select the matching jobs):

ox run "qc/NA12878_report.txt"

This builds the pipeline for NA12878 only, skipping NA12891 and NA12892. Combined with caching, this lets you iterate on pipeline logic without waiting for all samples.

Later, run the full cohort:

ox run

NA12878 is cached. Only NA12891 and NA12892 are computed.

Filter by Rule

Run only the QC stage with --rule (exact name or /regex/; assumes upstream outputs exist):

ox run --rule qc

Inspect the plan with the tags attached to each job (ox dag has no tag-grouping mode; the tags live in the plan JSON):

ox plan --json | jq '.jobs[] | {job_id, tags}'

Named Inputs

Several rules use named inputs for clarity. Compare:

# Positional (works but cryptic with multiple inputs)
input = ["aligned/{sample}.bam", "vcf/{sample}_merged.vcf"]
shell = "check {input[0]} {input[1]}"

# Named (self-documenting)
input = { bam = "aligned/{sample}.bam", vcf = "vcf/{sample}_merged.vcf" }
shell = "check {input.bam} {input.vcf}"

Named inputs make your workflow readable as it grows.

Adding a New Sample

Edit Oxymakefile.toml:

[config]
samples = ["NA12878", "NA12891", "NA12892", "NA12893"]

Run again:

ox run -j 4

Only the NA12893 jobs run. Everything else is cached.

Adapting to Real Tools

Replace the stand-in commands with real bioinformatics tools:

[rule.align]
input = { r1 = "fastq/{sample}_R1.fastq", r2 = "fastq/{sample}_R2.fastq" }
output = ["aligned/{sample}.bam"]
tags = { stage = "align", cost = "heavy" }
resources = { cpu = 8, mem = "32G" }
shell = """
bwa mem -t {resources.cpu} reference.fa {input.r1} {input.r2} \
  | samtools sort -@ 4 -o {output}
samtools index {output}
"""

The workflow structure stays the same. Only the shell commands change.

Next Steps

• Rules and Wildcards -- wildcard expansion and constraints
• Tags and Filtering -- organizing large workflows
• Idempotent Execution -- cooperative multi-session runs

Climate Time-Series Pipeline

This cookbook builds a multi-station climate analysis pipeline in OxyMake. It covers feature engineering, index generation, and regional aggregation across a network of weather stations -- all driven by wildcards, snapshots, and execution history. Mock data (random readings) keeps the example self-contained.

What You Will Learn

• Config-driven station network and parameter sweeps
• Wildcard expansion across stations, features, and rolling windows
• Named inputs for multi-file rules
• Snapshots to compare analysis milestones
• Execution history as a lightweight lab notebook
• Tag-based filtering for fast iteration

The Complete Oxymakefile

Create a directory and save this as Oxymakefile.toml:

ox_version = "0.1"

[config]
stations = ["BOS", "DEN", "SEA", "AUS", "PDX"]
windows  = [5, 10, 20, 60]
metric   = ["trend", "anomaly"]

# ── Default target ──────────────────────────────────────────────
[rule.all]
input = ["reports/network_summary.txt"]

# ── Stage 1: Generate mock temperature readings ────────────────
[rule.mock_readings]
output = ["data/readings/{station}.csv"]
tags   = { stage = "data", speed = "fast" }
shell  = """
mkdir -p data/readings
echo "date,temp" > {output}
temp=15
for day in $(seq 1 252); do
  # Random daily temperature delta between -3 and +3 degrees
  d=$(awk "BEGIN {{srand($day * 17 + $(echo {station} | cksum | cut -d' ' -f1)); printf \"%.4f\", (rand() - 0.5) * 6}}")
  temp=$(awk "BEGIN {{printf \"%.2f\", $temp + $d}}")
  printf "2025-%03d,%s\\n" "$day" "$temp" >> {output}
done
"""

# ── Stage 2: Compute features ─────────────────────────────────
[rule.features]
input  = { readings = "data/readings/{station}.csv" }
output = ["data/features/{station}_{window}d.csv"]
tags   = { stage = "features", speed = "fast" }
shell  = """
mkdir -p data/features
echo "date,{station}_trend_{window}d,{station}_var_{window}d" > {output}
tail -n +2 {input.readings} | awk -F, -v lb={window} '
  BEGIN {{ OFS="," }}
  {{
    temps[NR] = $2
    if (NR >= lb) {{
      trend = (temps[NR] - temps[NR - lb + 1]) / lb
      sum = 0; sq = 0
      for (i = NR - lb + 1; i <= NR; i++) {{
        r = temps[i] - temps[i-1]
        sum += r; sq += r * r
      }}
      var = (sq - sum*sum/lb) / (lb - 1)
      printf "%s,%.6f,%.6f\\n", $1, trend, var
    }}
  }}
' >> {output}
"""

# ── Stage 3: Generate indices ─────────────────────────────────
[rule.indices]
input  = ["data/features/{station}_{window}d.csv"]
output = ["data/indices/{station}_{metric}.csv"]
tags   = { stage = "indices" }
shell  = """
mkdir -p data/indices
echo "date,{station}_{metric}" > {output}

if [ "{metric}" = "trend" ]; then
  # Average trend across windows → warming vs cooling stations
  paste -d, data/features/{station}_*d.csv \
    | tail -n +2 \
    | awk -F, '{{ sum=0; n=0; for(i=2;i<=NF;i+=2){{ sum+=$i; n++ }}; if(n>0) printf "%s,%.6f\\n",$1,sum/n }}' \
    >> {output}
else
  # Anomaly: deviation from the mean trend
  paste -d, data/features/{station}_*d.csv \
    | tail -n +2 \
    | awk -F, '{{ sum=0; n=0; for(i=2;i<=NF;i+=2){{ sum+=$i; n++ }}; if(n>0) printf "%s,%.6f\\n",$1,-sum/n }}' \
    >> {output}
fi
"""

# ── Stage 4: Cross-station composite index ────────────────────
[rule.composite]
input  = ["data/indices/{station}_{metric}.csv"]
output = ["data/composite/{metric}_index.csv"]
tags   = { stage = "composite" }
shell  = """
mkdir -p data/composite
echo "date,station,weight" > {output}
# Rank-based regional index: center station values cross-sectionally to zero
paste -d, data/indices/*_{metric}.csv \
  | tail -n +2 \
  | awk -F, '
    BEGIN {{ split("{station}", stations, " ") }}
    {{
      n = 0; sum = 0
      for (i = 2; i <= NF; i += 2) {{ vals[++n] = $i; sum += $i }}
      mean = sum / n
      wsum = 0
      for (i = 1; i <= n; i++) {{ w[i] = vals[i] - mean; wsum += (w[i]>0?w[i]:-w[i]) }}
      if (wsum > 0) for (i = 1; i <= n; i++) w[i] /= wsum
      for (i = 1; i <= n; i++) printf "%s,%s,%.6f\\n", $1, stations[i], w[i]
    }}
  ' >> {output}
"""

# ── Stage 5: Cumulative index score ──────────────────────────
[rule.score]
input  = { weights = "data/composite/{metric}_index.csv", readings = "data/readings/{station}.csv" }
output = ["data/score/{metric}_score.csv"]
tags   = { stage = "score" }
shell  = """
mkdir -p data/score
echo "date,daily_index,cumulative_index" > {output}
# Simple: weight * daily reading, summed across stations
awk -F, '
  NR == FNR && FNR > 1 {{ weights[$1,$2] = $3; next }}
  FNR > 1 {{ readings[$1] = $2 }}
' {input.weights} data/readings/*.csv

# Simplified: accumulate a weighted daily index.
# Sorting is done with `sort`, not awk's `asorti` — that function is a GNU awk
# extension and is missing from the BSD awk shipped on macOS.
tail -n +2 {input.weights} | awk -F, '
  {{ idx[$1] += $3 * (rand() - 0.48) * 0.02 }}
  END {{ for (d in idx) printf "%s,%.6f\\n", d, idx[d] }}
' | sort -t, -k1,1 | awk -F, '
  {{ cum += $2; printf "%s,%.6f,%.6f\\n", $1, $2, cum }}
' >> {output}
"""

# ── Stage 6: Summary report ──────────────────────────────────
[rule.report]
input  = ["data/score/{metric}_score.csv"]
output = ["reports/network_summary.txt"]
tags   = { stage = "report", speed = "fast" }
shell  = """
mkdir -p reports
echo "======================================" > {output}
echo "  Climate Network Pipeline — Summary"    >> {output}
echo "======================================" >> {output}
echo ""                                        >> {output}
echo "Network: {station}"                      >> {output}
echo "Windows: {window}"                        >> {output}
echo "Metrics: {metric}"                        >> {output}
echo ""                                         >> {output}
for f in {input}; do
  index=$(basename "$f" _score.csv)
  lines=$(tail -n +2 "$f" | wc -l | tr -d ' ')
  final=$(tail -1 "$f" | cut -d, -f3)
  echo "Index: $index"                          >> {output}
  echo "  Observation days: $lines"             >> {output}
  echo "  Final cumulative index: $final"       >> {output}
  echo ""                                       >> {output}
done
echo "--- Pipeline complete ---"               >> {output}
"""

Create the Project

mkdir climate-pipeline && cd climate-pipeline
# Save the Oxymakefile.toml above

No input data files are needed -- mock_readings generates synthetic data.

Explore the DAG

ox plan

Plan: 7 rules, 40 jobs, 0 source files
Targets: reports/network_summary.txt
  1. [mock_readings-PDX] rule=mock_readings -> [data/readings/PDX.csv]
  2. [features-PDX-60] rule=features -> [data/features/PDX_60d.csv]
  3. [features-PDX-20] rule=features -> [data/features/PDX_20d.csv]
  ...
  38. [composite-trend] rule=composite -> [data/composite/trend_index.csv]
  39. [score-trend] rule=score -> [data/score/trend_score.csv]
  40. [report] rule=report -> [reports/network_summary.txt]

The DAG fans out across stations and windows, then converges through indices and regional aggregation into a single report.

Run the Full Pipeline

ox run -j 4

OxyMake runs up to 4 jobs in parallel. The mock_readings jobs run first (no dependencies), then features fans out across stations x windows, and everything converges into the network report.

Iterate on a Single Station

During development, focus on one station by requesting its leaf target (wildcards in the target select the matching jobs):

ox run "data/indices/BOS_*.csv"

This builds the pipeline for BOS only. Later, run the full network:

ox run

BOS is cached. Only the remaining stations are computed.

Filter by Rule

Run only the feature computation stage with --rule (exact name or /regex/):

ox run --rule features

Snapshots: Compare Analysis Milestones

After a successful run, save a snapshot:

ox snapshot create baseline --message "5-station trend + anomaly"

Now add a new window (120 days) and a new metric. Edit the config:

[config]
windows = [5, 10, 20, 60, 120]
metric  = ["trend", "anomaly", "seasonal"]

Run again and save another snapshot:

ox run -j 4
ox snapshot create v2 --message "Added 120d window + seasonal metric"

Compare the two milestones:

ox snapshot diff baseline v2

Workflow hash changed (config modified)
Added:    15 jobs (features/*_120d, indices/*_seasonal, ...)
Changed:  2 jobs (composite, report — new inputs)
Unchanged: 40 jobs

This tells you exactly what changed between analysis iterations without manually tracking file modifications.

Execution History as a Lab Notebook

Each ox run is recorded with timing, job counts, and optional notes:

ox run -j 4 --note "Baseline: 5 stations, 4 windows"
# ... iterate ...
ox run -j 4 --note "Added seasonal metric, 120d window"

Review your analysis timeline:

ox history

RUN          STARTED              DURATION   OK  FAIL  SKIP  NOTE
run-a1b2c3   2025-01-15 09:12     12.3s     42    0     0   Baseline: 5 stations, 4 windows
run-d4e5f6   2025-01-15 09:45      4.1s     15    0    40   Added seasonal metric, 120d window

Drill into a specific run:

ox history --run-id run-a1b2c3

This shows per-job wall time and exit codes, plus peak RSS in MiB for local cold launches on Linux/macOS and SLURM CLI runs with --follow. Local RSS covers the child and already-reaped descendants; background or daemonised work can escape it. Warm-worker dispatches remain unmeasured. See the benchmark reference for CPU time and measurement scope.

Scaling the Network

Add more stations by editing [config]:

[config]
stations = ["BOS", "DEN", "SEA", "AUS", "PDX", "ORD", "ATL", "LAX", "JFK", "MIA"]

Run again:

ox run -j 8

Only the new stations are computed. Everything else is cached. As the network grows from 5 to 50 to 500 stations, the same Oxymakefile works -- OxyMake expands the wildcards and parallelizes automatically.

Next Steps

• Growing a Workflow Organically -- evolving from 3 rules to 300+ over weeks of analysis
• Agent-Driven Workflows -- automating the pipeline with LLM agents and NDJSON event streams
• Rules and Wildcards -- wildcard expansion and constraints
• Snapshots -- saving and comparing workflow state

ML Training Pipeline

Coming soon.

This page will show a machine learning training pipeline in OxyMake, covering:

• Data preparation: feature extraction, train/test splitting, and normalization
• Hyperparameter sweeps: wildcard-driven grid search across learning rates, architectures, and regularization parameters
• GPU resource management: declaring GPU requirements per rule for SLURM/Kubernetes scheduling
• Model evaluation: automated metric collection and comparison
• In-memory passing: using call mode with Arrow IPC to pass DataFrames between feature computation and training without disk I/O

Growing a Workflow Organically

Coming soon.

This page will illustrate how real research workflows evolve over time, covering:

• Starting small: a 3-rule exploratory workflow
• Adding complexity incrementally: new rules never invalidate existing cached results thanks to content-addressable caching
• Workflow composition: splitting large workflows across files with include directives
• Snapshot milestones: saving and comparing workflow states as research progresses
• Run annotations: using ox run --note to create a lightweight research lab notebook from the execution history

Agent-Driven Workflows

Coming soon.

This page will demonstrate how AI agents can drive OxyMake pipelines programmatically, covering:

• Structured NDJSON events: parsing --json output for typed event streams
• Programmatic gate approval: agents evaluating metrics and approving quality checkpoints via ox gate approve
• Automated error recovery: detecting failures from JSON events, adjusting parameters, and retrying
• Multi-agent coordination: multiple agents driving different stages of a pipeline
• End-to-end example: a complete pipeline driven by an LLM agent without human intervention

Cloud HPC with SLURM

OxyMake's SLURM executor targets any SLURM cluster — on-prem, academic (Grid'5000, Jean Zay), or cloud. This guide works one concrete cloud example end-to-end: a Google Cloud cluster provisioned with the HPC Toolkit. The same shape applies to AWS ParallelCluster, Azure CycleCloud, or any managed SLURM-on-cloud offering — only the provisioning commands change; the OxyMake profile and run loop are identical. It covers cluster provisioning, profile configuration, SSH tunneling for remote access, and running pipelines end-to-end.

Prerequisites

• A GCP project with billing enabled
• gcloud CLI installed and authenticated (gcloud auth login)
• Terraform >= 1.3
• The Cloud HPC Toolkit (ghpc CLI)

Cluster Architecture

The HPC Toolkit deploys a standard SLURM cluster on GCP:

graph TD
    subgraph VPC["GCP VPC"]
        Login["Login Node<br/>(SSH entry)"] --> Controller["Controller<br/>(slurmctld)"]
        Controller --> C0["c2-0 node"]
        Controller --> C1["c2-1 node"]
        Controller --> CN["c2-N node"]
        NFS["Filestore (NFS) — /mnt/shared"]
    end

Key points:

• Controller node runs slurmctld and schedules jobs
• Compute nodes auto-scale — spin up when jobs are queued, shut down when idle
• Filestore provides the shared NFS filesystem required by OxyMake's SLURM executor
• Login node is your SSH entry point for running ox run

Step 1: Provision the Cluster

Create the blueprint

Create a file oxymake-cluster.yaml:

# oxymake-cluster.yaml — HPC Toolkit blueprint
blueprint_name: oxymake-slurm

vars:
  project_id: YOUR_PROJECT_ID
  deployment_name: oxymake-slurm
  region: us-central1
  zone: us-central1-a

deployment_groups:
  - group: primary
    modules:

      # Shared filesystem (required by OxyMake SLURM executor)
      - id: homefs
        source: modules/file-system/filestore
        settings:
          local_mount: /mnt/shared
          size_gb: 1024

      # Network
      - id: network
        source: modules/network/vpc

      # SLURM partition — general-purpose compute
      - id: compute_partition
        source: community/modules/compute/schedmd-slurm-gcp-v6-partition
        use: [network, homefs]
        settings:
          partition_name: batch
          machine_type: c2-standard-8    # 8 vCPU, 32 GB
          max_count: 10                  # Auto-scales 0 → 10 nodes
          enable_placement: false

      # GPU partition (optional)
      - id: gpu_partition
        source: community/modules/compute/schedmd-slurm-gcp-v6-partition
        use: [network, homefs]
        settings:
          partition_name: gpu
          machine_type: a2-highgpu-1g    # 1× A100
          max_count: 4
          enable_placement: false

      # SLURM controller + login node
      - id: slurm_controller
        source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
        use: [network, compute_partition, gpu_partition]
        settings:
          login_node_count: 1

      # Login node
      - id: slurm_login
        source: community/modules/scheduler/schedmd-slurm-gcp-v6-login
        use: [network, slurm_controller]
        settings:
          machine_type: e2-standard-4

Deploy

# Generate Terraform from the blueprint
ghpc create oxymake-cluster.yaml

# Deploy
ghpc deploy oxymake-slurm

# Wait for the cluster to be ready (~5 minutes)
gcloud compute ssh oxymake-slurm-login0 --zone us-central1-a -- sinfo

You should see the batch and gpu partitions in the output.

Step 2: Configure the OxyMake Profile

Add a [profile.gcloud] section to your Oxymakefile.toml:

[profile.gcloud]
executor = "slurm"
partition = "batch"
account = "default"
jobs = 100                      # SLURM handles scheduling; allow many concurrent
keep_going = true               # Don't abort the full DAG on a single failure

[profile.gcloud-gpu]
executor = "slurm"
partition = "gpu"
account = "default"
jobs = 20

Run with the profile:

ox run --profile gcloud
ox run --profile gcloud-gpu    # For GPU workloads

Profile fields map to SLURM flags:

CLI flags always override profile values: ox run --profile gcloud --partition gpu overrides the partition from batch to gpu.

Step 3: Prepare the Cluster

SSH into the login node and set up OxyMake:

gcloud compute ssh oxymake-slurm-login0 --zone us-central1-a

On the login node:

# Install OxyMake (from prebuilt binary or cargo)
curl -fsSL https://oxymake.noogram.dev/install.sh | sh
# or: cargo install oxymake

# Clone your workflow into the shared filesystem
cd /mnt/shared
git clone https://github.com/your-org/your-pipeline.git
cd your-pipeline

# Verify SLURM is accessible
sinfo                      # Should show partitions
ox run --executor slurm --dry-run   # Should show the DAG without submitting

Important: Run ox run from the login node (or controller), not from a compute node. The state.db must be on a local filesystem — /mnt/shared is NFS, so OxyMake stores state.db in a local directory by default.

Step 4: Run a Pipeline

# Dry run — see what would be submitted
ox run --profile gcloud --dry-run

# Submit the pipeline
ox run --profile gcloud

# Monitor jobs
squeue -u $USER              # SLURM's view
ox run --profile gcloud --status  # OxyMake's view (if supported)

On GCP with auto-scaling, compute nodes spin up on demand. The first run may take a few extra minutes while nodes boot. Subsequent runs are faster as nodes remain warm for the configured idle timeout (default: 5 minutes).

SSH Tunnel for Remote Access

When running OxyMake from your local machine (not SSH'd into the cluster), you can either tunnel to the SLURM CLI tools (Option A/B) or use REST mode via slurmrestd (Option C).

Option A: SSH ProxyCommand (recommended)

Add to your ~/.ssh/config:

Host oxymake-slurm
    HostName <login-node-external-ip>
    User your-username
    IdentityFile ~/.ssh/google_compute_engine
    # Or use gcloud's IAP tunnel:
    # ProxyCommand gcloud compute ssh oxymake-slurm-login0 --zone us-central1-a --tunnel-through-iap --plain -- -W %h:%p

Then SSH in and run:

ssh oxymake-slurm "cd /mnt/shared/your-pipeline && ox run --profile gcloud"

Option B: IAP Tunnel (no public IP required)

If your login node has no external IP (common for secure setups), use Identity-Aware Proxy:

# Direct SSH via IAP
gcloud compute ssh oxymake-slurm-login0 \
    --zone us-central1-a \
    --tunnel-through-iap

# Or set up a SOCKS proxy for port forwarding
gcloud compute ssh oxymake-slurm-login0 \
    --zone us-central1-a \
    --tunnel-through-iap \
    -- -D 1080 -N -f

# Forward the OxyMake dashboard port (if using ox dashboard)
gcloud compute ssh oxymake-slurm-login0 \
    --zone us-central1-a \
    --tunnel-through-iap \
    -- -L 8080:localhost:8080 -N -f

Option C: SSH Tunnel for slurmrestd

Forward the slurmrestd port to your workstation and use REST mode:

# Forward slurmrestd (port 6820) to localhost
gcloud compute ssh oxymake-slurm-login0 \
    --zone us-central1-a \
    --tunnel-through-iap \
    -- -L 6820:slurmctld:6820 -N -f

# Run OxyMake in REST mode via the tunnel
ox run --executor slurm --slurm-api http://localhost:6820

Note: REST mode requires slurmrestd to be running on the cluster. Set SLURM_JWT for JWT authentication if required by your cluster.

Cluster Lifecycle

Scale down

GCP auto-scaling shuts down idle nodes. To force-stop:

# Drain all compute nodes
scontrol update partition=batch state=DRAIN

# Or destroy the cluster entirely
ghpc destroy oxymake-slurm

Cost control

For intermittent workloads, consider stopping the controller and login node when not running pipelines:

gcloud compute instances stop oxymake-slurm-controller --zone us-central1-a
gcloud compute instances stop oxymake-slurm-login0 --zone us-central1-a
# Restart when needed:
gcloud compute instances start oxymake-slurm-controller --zone us-central1-a
gcloud compute instances start oxymake-slurm-login0 --zone us-central1-a

Troubleshooting

Oxymakefile Format

OxyMake workflows are defined in Oxymakefile.toml, a declarative TOML file. This page is the complete format reference.

Top-Level Fields

format_version = "2"
ox_version = ">=0.7.0"       # Minimum capable binary, not a reproducibility pin.

Absent format_version selects legacy schema 1. Only "1" and "2" are supported. Schema 1 leaves ox_version informational and warns for each file that no binary requirement is enforced; its resource identity is unchanged.

Schema 2 requires exactly >=MAJOR.MINOR.PATCH with canonical decimal version components, for example >=0.7.0. Missing requirements, shorthand (0.1), other comparators, ranges, prerelease/build suffixes, and an unsatisfied minimum are errors. Diagnostics name the file, requirement and running binary version. Every included file is checked before any external config source is read or a workflow is planned or run. Included files declare their own schema and minimum; a legacy include still warns and has no enforced binary requirement.

Schema 2 rejects unknown structural keys, including nested rule, profile, gate, executor, log, input/output descriptor and guard fields. Dynamic maps (config, tags, params, resource names, wildcard constraints, profile overrides and named input/output paths) keep their user-defined names. Named resource classes, variables tables and resource exports are not implemented.

Use ox migrate --to-format 2 to review an upgrade.

Config Section

The [config] section defines workflow-level variables used for wildcard expansion:

[config]
samples = ["A", "B", "C"]
chromosomes = ["chr1", "chr2", "chr3"]
models = ["linear", "ridge", "lasso"]

Config values are arrays of strings. They drive wildcard expansion in rules.

Rule Definitions

Each rule is a [rule.<name>] table:

[rule.process]
input = ["data/{sample}.csv"]
output = ["results/{sample}.txt"]
shell = "python process.py {input} {output}"

Rule Fields

environment selects a software backend; env declares environment variables. The names are intentionally distinct and have no third alias.

materialize belongs to an output descriptor, not directly to the rule. Its values are always, auto, never, and final:

format_version = "2"
ox_version = ">=0.7.0"

[rule.a]
output = [{ path = "a", materialize = "always" }]
shell = "touch a"

Cross-platform cache reuse

cache_platform controls whether the platform (OS/architecture) participates in a rule's cache key. The default, "exact", restricts reuse to the same platform. Set it to "any" only when the rule's outputs are suitable for reuse across platforms:

[rule.merge_counts]
input = ["data/*.parquet"]
output = ["build/counts.parquet"]
shell = "duckdb -c '...'"
cache_platform = "any"

The engine cannot verify this claim. A rule that emits platform-specific artifacts, such as machine code, must use "exact". The parser rejects cache_platform = "any" together with reproducibility = "non_reproducible"; "approximate" and "seed_deterministic" are allowed.

Executor override

A rule may declare executor = "local" to run on the host running ox. It is a no-op in a local run. In a SLURM run, selecting an uncached job with this override uses the local scheduler for the selected graph: local rules execute on this host, other rules use SLURM, and ox run waits for completion even without --follow. Dependencies, cache checks, retries, logs, cancellation and the --jobs limit still go through the scheduler. This limit also bounds cluster submissions and defaults to 1; ox must stay alive. Both hosts must already see the input and output files through a shared filesystem.

Ray's whole-DAG driver cannot execute a task on the submitting host. An active job declaring executor = "local" is therefore rejected before Ray submission, including with --follow. Use a separate local run for those targets. Cached jobs and jobs excluded by target selection, --until, or --omit-from neither block Ray nor switch SLURM dispatch.

Only the exact value "local" is accepted. Other values are parse errors naming the rule; choose the run's backend with ox run --executor. Per-rule site placement, several remote backends in one run, and artifact transfer inside a run are outside this field's scope. Job-start events and history record the executor selected for each executed job.

Output cleanup

clean_outputs is an optional per-rule string (currently local-executor only):

Three values express three distinct lifecycles; a boolean cannot represent all of them. The default preserves the existing guarantee that stale outputs from a failed run cannot masquerade as valid results. OxyMake always clears its own .oxytmp staging files before execution. Output verification and hashing after execution are unchanged. A failed job is still recorded as a failure and never writes a successful cache entry. Changing the policy invalidates the job's cache key.

Warning: "never" hands the staleness guarantee to the script. The script must validate existing files, replace stale data, and exit successfully only when every declared output is complete. Preserved files alone do not prove that a failed run succeeded. "on-failure" also requires the script to validate any existing outputs it reuses on a successful run.

Slurm and Ray carry this field but keep their existing output behavior; this policy controls automatic cleanup by the local executor. Explicit ox clean and output lifecycle policies such as temp are separate mechanisms.

Incremental cache of an external source

For a dataset of 509 parquet files (about 4 GB over S3), declare the actual files as outputs and let an idempotent extraction script reuse complete files:

[rule.fetch_dataset]
input = ["scripts/extract.py", "dataset-manifest.json"]
output = ["cache/2025-01.parquet", "cache/2025-02.parquet"] # List all 509 files.
clean_outputs = "never"
shell = "python scripts/extract.py dataset-manifest.json"

The script checks each completed file against the manifest, downloads missing or stale files to its own temporary paths, then renames each completed file into place. Editing the script invalidates the rule, but complete downloads remain available to reuse. A transient network failure at file 400 preserves the earlier downloads for the next attempt. Avoid OxyMake's reserved .oxytmp suffix for the script's temporary files. Unlike a stamp-only rule, OxyMake tracks and hashes the real dataset outputs.

Execution Modes

Four modes form a spectrum from flexibility to optimizability:

shell -- Opaque shell command. Maximum flexibility, no optimization.

[rule.align]
shell = "bwa mem ref.fa {input} > {output}"

run -- Inline script with language specification.

[rule.stats]
lang = "python"
run = """
import pandas as pd
df = pd.read_csv("{input}")
df.describe().to_csv("{output}")
"""

script -- External script file.

[rule.analyze]
lang = "python"
script = "scripts/analyze.py"

call -- Pure function reference. Supports in-memory Arrow IPC passing.

[rule.features]
input = [{ path = "data/{sample}.parquet", format = "parquet" }]
output = [{ path = "features/{sample}.parquet", format = "parquet", materialize = "auto" }]
call = "pipeline.features:compute_features"

Wildcards

Wildcards in {braces} are resolved from [config] arrays or from a requested target path that matches a rule output pattern. Existing files are recognized as source inputs; their names do not themselves enumerate wildcard values.

[config]
samples = ["A", "B"]

[rule.process]
input = ["data/{sample}.csv"]     # {sample} expanded from config.samples
output = ["results/{sample}.txt"]

Resources

[rule.heavy_job]
output = ["results/big.txt"]
shell = "compute_heavy"
resources = { cpus = 4, mem_gb = 16, gpu = 1 }

Named resource classes (schema 2)

Schema 2 can collect repeated declarations into a root-level named class:

format_version = "2"
ox_version = ">=0.7.0"

[resource_classes.standard]
cpu = 2
mem = "4G"

[rule.first]
output = ["first.txt"]
shell = "compute --threads {threads} > {output}"
resource_class = "standard"

[rule.larger]
output = ["larger.txt"]
shell = "compute --threads {threads} > {output}"
resource_class = "standard"
resources = { cpus = 6 }

A class contains integer, float, or string resource values only, as do schema-2 rule resources. Custom resource names are allowed. The class guard rejects parent, extends, resource_class, variables, environment, executor, when, shell, run, script, call, input, output, and resources. Other names are treated as custom resources. It cannot inherit another class or declare variables, execution, conditions, or environment settings. Definitions may follow their references or live in schema-2 included files, which share one class namespace. Schema-1 files ignore class declarations and references and warn with the file name and format version; migrate those files to schema 2 before using classes. Duplicate class definitions and unknown class references are errors. A rule's inline resources replace individual canonical dimensions from its class, so cpus may override class cpu, but aliases may not be duplicated within either table. An override keeps the class spelling: class cpu = 2 plus rule cpus = 6 yields cpu = 6, so {resources.cpu} and {threads} both expand to 6; {resources.cpus} stays literal. A new dimension keeps the rule spelling.

Classes resolve before command interpolation. Their effective values therefore expand {resources.NAME} using the effective spelling. As with inline resources, {threads} reads the canonical CPU value from either cpu or cpus. Cache identity also follows those effective values: class names, definition order, and definition file paths do not enter the key. As before, a resource affects a key only when interpolation places its value in the execution text.

What a declared resource actually does, per route

A declared resource means something different on every execution route. Read this table before relying on resources for anything beyond scheduling hints. Definitions: Requested — the declaration is parsed and attached to the job. Admitted — something checks the declaration before letting the job start, and can refuse or delay it. Enforced — something stops the running job from exceeding the declaration. Measured — the actual usage is observed and reported back, independent of what was declared.

The portable resource vocabulary shared by Ray and local scheduler admission is:

All memory suffixes above use binary scaling, so "1GB" and "1GiB" both mean 1,073,741,824 bytes. The explicit custom: prefix always names a custom token: custom:mem_mb is distinct from the memory key mem_mb.

Aliases may not be combined for the same resource (cpu with cpus, for example), and a bare custom name may not be combined with its prefixed form (metal with custom:metal). Invalid units, negative or non-finite values, fractional byte results, overflow, and token counts finer than 0.0001 are errors. These checks normalize a declaration without rewriting the original resources table. Executor-specific keys, including SLURM directives such as time, retain the vocabulary documented by that executor. time = 60 is a SLURM time limit; on Ray it would request a custom token named time and the default feasibility check rejects it unless a node advertises 60 units. Do not copy SLURM-only keys into a Ray workflow.

Ray rejects custom names with leading or trailing whitespace, and names colliding (case-insensitively) with cpu, cpus, gpu, gpus, memory, object_store_memory, or the reserved prefixes node: and accelerator_type:. This also applies to explicit custom: names. Other names, including quotes, backslashes and non-ASCII characters, are forwarded as data.

Native Ray DAG tasks reserve CPU, GPU, memory and custom resources. Memory is forwarded as memory=<bytes> and included in the live-node feasibility check. This is logical scheduling admission, not a hard RSS limit. The per-job Jobs API and array paths do not reserve memory; there, a declared memory value is instead forwarded to the job as the OXYMAKE_MEMORY_LIMIT_BYTES environment variable (crates/ox-exec-ray/src/runtime_env.rs:91-98). That variable is a hint for user code to read, not a limit OxyMake or Ray enforces — nothing in OxyMake sets an rlimit or cgroup from it, and nothing kills a job for exceeding it.

Local scheduler admission is opt-in with ox run --resource-budget KEY=VALUE. The flag is repeatable and comma-separated, for example --resource-budget cpu=6,mem_gb=32 --resource-budget metal=1; it uses exactly the aliases, binary memory units and duplicate checks above. Capacity counts are whole tokens; job demands may be fractional. Memory capacity is normalized to whole bytes, so mem_gb=1, mem_mb=1024, and memory=1GiB are identical. Two spellings of one canonical resource across flags are an error.

Only budgeted canonical resources are interpreted and enforced. For example, a cpu = 6 budget also constrains a cpus = 4 demand. A job with no demand consumes no resource tokens and still uses one -j / --jobs slot. Zero demand fits zero capacity; a positive demand against zero capacity, or any demand exceeding total capacity, is a configuration error naming the job and resource before execution of the selected DAG starts, even when every job is cached. Rules excluded by target selection, --until or --omit-from do not participate in validation. In mixed SLURM runs, only locally routed jobs consume this budget; remote jobs retain their SLURM resource requests. Pure remote runs reject --resource-budget.

-j N caps concurrent jobs, while --resource-budget caps resources held by those jobs. This is distinct from --memory-budget, which caps in-memory materialized outputs and never admits or rejects a job. The resource budget is per ox run: two concurrent runs each get their full configured budget. OxyMake does not detect host capacity or coordinate budgets across processes.

Resource interpolation in commands

A declared resource can reach the command text itself, through two placeholders resolved while the job graph is built (crates/ox-core/src/resolver.rs:1350-1357):

• {threads} substitutes the canonical CPU resource value (the Snakemake convention), declared as either cpu or cpus.
• {resources.NAME} substitutes any declared resource under its own key, e.g. {resources.mem_gb} for resources = { mem_gb = 16 }.

Validation rejects the placeholder shapes whose declarations can be checked statically: {resources.NAME} and {params.NAME} require that exact key on the rule, {threads} requires either cpu or cpus, and {log} requires a configured stdout log path. The error names the rule, placeholder, and available keys. This check applies before planning, dry runs, and execution.

{config.NAME} is checked one step later, while the job graph is built, because --set NAME=value can supply a key that no [config] section declares and overrides are applied after validation. An unknown config key is still an error, and it still stops the run before any job starts — it is simply reported by the resolver rather than by ox lint.

A bare {name} is deliberately not checked. OxyMake cannot distinguish a mistyped wildcard such as {samp} from legitimate brace text in shell and inline code, including awk '{print $1}', ${HOME}, and JSON literals. Such a bare typo can therefore remain literal; namespaced placeholders are the fail-closed forms where static checking is possible.

Command interpolation covers inline commands only — shell and run blocks, where the interpolated text is the command OxyMake executes (crates/ox-core/src/resolver.rs:1136-1204). For script, only the path string is interpolated, not the file's contents; for call, only the function-reference string is interpolated, not the function body. A {threads}/{resources.NAME} reference inside a script file or a called function is not substituted — it reaches the script/function unexpanded. To expose a value to a script or called function, put the placeholder in an env value instead; environment values use the same interpolation path.

The interpolated execution block — the actual command text after substitution — is what enters the cache key as rule_source (crates/ox-cache/src/key.rs, crates/ox-cli/src/commands/run.rs). The resources table itself is not a separate cache key input. The practical consequence: a rule that writes resources = { cpu = 4 } and a shell command that never mentions {threads} or {resources.cpu} can have its cpu value changed without invalidating the cache, because the command text — the only place resources reach the key — did not change. A rule that interpolates {threads} or {resources.NAME} into its command does get cache invalidation when that value changes, as a side effect of the command text changing, not because resources is tracked directly.

A shared multi-user machine without root

This is the pattern for the case that motivated the guarantee table above: several people running many rules on one Linux box, no root, no cgroups.

OxyMake bounds how many jobs start, not what each one takes. On a shared machine, get all three of these, not just the first:

1. Declare resources on every rule — resources = { cpu = N, mem_gb = M } — even though local admission ignores them by default. This is what makes the next two steps possible and keeps the numbers visible in one place.
2. Run with a matching --resource-budget — e.g. ox run --resource-budget cpu=<cores you're allowed>,mem_gb=<memory you're allowed>. This caps how many jobs your own ox run starts concurrently. It does not cap what a single started job does once running, and it does not know about — or coordinate with — anyone else's ox run, your own other concurrent runs, or any process outside OxyMake.
3. Cap library thread pools yourself, inside the command, using {threads}/{resources.NAME} interpolation (above) so the number is declared once: e.g. shell = "OMP_NUM_THREADS={threads} POLARS_MAX_THREADS={threads} {command}". OxyMake has no mechanism that does this for you.

A per-run --resource-budget coordinates nothing across runs or users. Two people each running ox run --resource-budget cpu=8 on the same 8-core machine will each believe they have the whole machine; OxyMake will start up to 8 CPU-tokens' worth of work per run, not per machine.

Memory limits: reservation, not enforcement

Nothing in OxyMake today enforces a declared memory value against a running process — see the guarantee table above. If a future version adds local memory enforcement, the mechanism matters: ulimit -v (RLIMIT_AS) caps a process's address space, not its resident set (RSS). Address space is routinely far larger than resident memory for a modern runtime — a process can reserve gigabytes of virtual address space it never touches — so an address-space limit rejects far more processes than a resident-memory limit would, or none at all if set loosely enough to be harmless. It is not a drop-in for "limit this job's memory." One team has reported, unconfirmed and still under investigation on their side, that wrapping a job in a ulimit -v cap appeared to break an S3 client's TLS setup on its first call; whether the cap, a related environment change, or something else in their environment caused it has not been established. It is recorded here only as a caution, not as a finding about ulimit -v in general: an enforcement feature should name its actual mechanism (RSS via cgroup, address space via rlimit, or none) rather than promise "memory".

Benchmark output

Set benchmark on a rule to write a two-line TSV after a successful job:

[rule.measured]
output = ["results/output.txt"]
shell = "produce results/output.txt"
benchmark = "benchmarks/output.tsv"

The columns are s, h:m:s, max_rss, and cpu_time, in that order. max_rss is expressed in MiB with two decimal places when the executor supplies a per-job measurement, and cpu_time is expressed in seconds. History's peak_mem_mb instead rounds up to whole MiB: 1500 KiB renders as 1.46 in benchmark TSV and 2 in history. The round-up keeps a positive sub-MiB history observation distinguishable from zero. An unmeasured value is written as -; a measured zero stays zero.

On Linux and macOS, local cold launches (buffered and streaming output) use wait4 on the job's own child process. RSS is that child's high-water mark, including already-reaped descendants; it is not a simultaneous process-tree memory total. Background or daemonised work can escape the observation. The collector converts Linux KiB and macOS bytes to bytes before formatting. CPU time is the same child's user plus system time, in seconds; no CPU/wall ratio is inferred. It is recorded in benchmark TSV, not in job_history. Failed and signalled children retain any measurements returned by wait4 in their executor JobResult; scheduler publication rules still apply. A timed-out child is measured in the lower-level ProcessResult, but the executor returns a timeout error and drops that result, so no measurement reaches JobResult.

With --warm-workers fork, the retained Python template owns and reaps one child per dispatch using os.wait4. That child's RSS and CPU time use the same conversion and output paths as a cold child, so successive dispatches through one template do not inherit one another's dispatch-added peaks. The RSS figure is nevertheless the child's high-water mark and starts with the template's resident footprint inherited at fork. Only memory above that warm baseline was added by the dispatch; the reported total is not the memory the dispatch would need by itself. A NumPy- or PyTorch-heavy template can therefore dominate the reported peak for a small dispatch. CPU time is local to the child. The optional reply fields are backward compatible: an older template omits them and a newer binary records them as absent; an older binary ignores fields added by a newer template. Malformed or missing usage is absent, never zero.

--warm-workers persistent remains unmeasured. One process executes every dispatch in-process, so its RSS high-water mark contains prior dispatches and its CPU cannot be separated without a quiescence contract. OxyMake does not publish a delta or sample as a per-dispatch figure. Unsupported platforms also leave RSS and CPU absent.

Conditional Guards

[rule.expensive]
output = ["results/{seed}.txt"]
shell = "compute {seed}"
when = "seed in @selected_seeds"

Guards are evaluated at DAG resolution time. Jobs whose guard is false are never created.

Include Directives

Split large workflows across files:

include = ["rules/alignment.toml", "rules/qc.toml"]

Environment Specification

A rule declares its environment with an environment inline table whose key is the backend and whose value is that backend's argument:

[rule.analyze]
output = ["results/summary.txt"]
shell = "python analyze.py"
environment = { uv = "requirements.txt" }

Supported keys: uv, conda, docker, apptainer, nix. Omitting environment runs the command on the host as-is.

The table is validated: an environment table that names none of those keys — environment = { type = "uv", requirements = "…" }, for instance — or that carries an extra key beside a recognised backend is a parse error naming the accepted keys. It is never silently dropped.

For uv, a value ending in .toml is treated as a project file: uv discovers it on its own, so it is not passed on the command line, but its bytes (and those of an adjacent uv.lock) enter the cache key. Any other value is a requirements file, passed as uv run --with-requirements <file>.

A top-level environment table sets the default for every rule that does not declare its own:

environment = { uv = "pyproject.toml" }

Declared Environment Variables

A root-level [env] table applies to every rule in that file. A rule-level inline env table overrides individual keys; other workflow-level keys survive the merge:

[env]
OMP_NUM_THREADS = "{threads}"
POLARS_MAX_THREADS = "{threads}"

[rule.analyze]
output = ["results/summary.txt"]
shell = "python analyze.py"
resources = { cpus = 4 }
env = { POLARS_MAX_THREADS = "2" }

Values are interpolated exactly like command text, including {threads}, {resources.NAME}, {params.NAME}, {config.NAME}, and rule wildcards. The same validation applies, so an undeclared {resources.typo} is rejected before execution. Variable names must match [A-Za-z_][A-Za-z0-9_]* so the same declaration is valid on local, Ray, and SLURM executors.

The resolved merged table is part of the cache key. Changing a value therefore invalidates affected jobs. There is no per-variable or workflow-level opt-out: OxyMake cannot know whether a variable changes output, so hashing every declared variable is the only safe default. An empty table contributes no bytes, and a workflow with no env declarations keeps its existing cache keys.

OxyMake does not define OX_CPUS, derive OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, or another library-specific list, or add a canonical memory placeholder. Use {threads} or the declared-text form of {resources.mem} in a variable the workflow names. Per-variable “does not affect output” annotations are likewise not supported.

See docs/format/env-vars.md for composition with inherited and executor-owned variables, and Environments for software backends.

Next Steps

• CLI Commands -- how to run workflows
• Expression Language -- guard and expression syntax
• Configuration -- project-level settings

Existing wildcard outputs

Output-pattern matches take precedence over disk existence when their inputs resolve. An existing hand-maintained file may remain a source when an input of its own producer is missing. Fixed inputs of wildcard producers (such as shared configuration) and missing sources deeper in the graph remain errors. Known cached outputs require resolvable inputs or verified adoption. Use wildcard_constraints to exclude hand-maintained names from rule ownership. See Existing files and rule ownership.

Rust callers can use ox_core::resolver::resolve_with_source_fallback to supply an &dyn Fn(&std::path::Path) -> bool policy for that fallback. Returning false keeps a known generated output from becoming a source after missing-input resolution. resolve allows the fallback by default. Explicit ResolveRequest::existing_files are always leaves; other errors are propagated.

resolve_with_source_fallback_at also accepts a filesystem base directory, keeping relative path checks independent of process cwd. CLI, MCP and SessionBuilder use ox_api::resolution::discover_source_files and ox_api::resolution::resolve to apply the same output filtering and cache-provenance policy relative to the Oxymakefile directory (workflow_directory). An absent cache is an empty store and permits unrecorded manual sources; a cache that cannot be opened denies fallback because provenance cannot be checked.

CLI Commands

OxyMake provides the ox command-line tool. Every command supports --json for structured NDJSON output.

Core Commands

ox init

Initialize a new OxyMake project in the current directory.

ox init

Creates a starter Oxymakefile.toml and .oxymake/ directory.

ox run

Execute the workflow, ensuring requested outputs exist.

ox run                          # Build default targets
ox run results/report.html      # Build a specific target
ox run -j 8                     # Parallel execution (8 jobs)
ox run -j 8 --resource-budget cpu=6,mem_gb=32
ox run --rule stats             # Only run jobs from a rule (exact or /regex/)
ox run --json                   # Structured NDJSON output
ox run --note "experiment v2"   # Annotate the run
ox run --no-cache               # Ignore the cache, re-run everything

Options:

• -j N, --jobs N -- Maximum concurrent jobs (default: 1)
• --resource-budget KEY=VALUE -- Per-run local admission capacity; repeat the flag or separate pairs with commas, for example cpu=6,mem_gb=32. Uses the portable resource aliases and units, and is available with --executor local or for locally routed jobs in mixed SLURM runs. Pure remote runs reject it.
• --rule RULE -- Only run jobs from this rule (exact name or /regex/)
• -k, --keep-going -- Continue independent jobs after a failure
• -n, --dry-run -- Show jobs and expanded execution blocks without executing
• --json -- Emit NDJSON events on stdout
• --report-json PATH -- Write the NDJSON event stream to a file
• --note TEXT -- Attach a note to this run
• --no-cache -- Ignore cached outputs and re-execute
• --cache-remote DIR -- Share output blobs through a directory blob store (forces hash validation; see below)
• --executor EXEC -- Choose executor: local (default), slurm, ray. A SLURM graph with an active rule declaring executor = "local" uses scheduler dispatch and waits for completion, including without --follow. --jobs bounds cluster submissions too (default: 1); ox must stay alive. Ray DAG runs reject active local overrides. Pruned or cached local jobs do not change dispatch. See Executor override.
• --ray-allow-pending -- Ray only; skip the default live-node feasibility check and allow waiting for a future node. Resource values and names are still validated. Using this flag with another executor is a usage error. It can also be enabled with ray_allow_pending = true in a selected profile.

-j N and --resource-budget control different limits. -j N limits the number of concurrently running jobs. --resource-budget limits the declared local resources those admitted jobs may hold; it does not change the job count, so a rule with no resource declaration still uses one -j slot but reserves no resource tokens. --memory-budget is separate again: it controls how much completed output OxyMake keeps in memory, not whether a job may start.

A resource budget belongs to one ox run process. Two concurrent runs each receive their configured capacity; OxyMake makes no host-wide capacity or cross-process resource-arbitration promise.

Dry-run execution blocks

ox run --dry-run keeps its summary and job lines unchanged, then adds the resolved execution block beneath each job. Shell blocks show the exact command after {input}, {output}, {threads}, {resources.*}, {params.*}, {config.*}, and wildcard interpolation. Inline run blocks show the resolved source text and language. Script blocks show the interpreter followed by the resolved script path (using sh when no language is declared). Call blocks show the language followed by the resolved function reference; this is a description of the invocation, not the generated adapter code.

Human output prefixes every physical line of execution text with |. This includes blank lines and a final blank line, so indentation and line breaks remain visible without reformatting the command. With --json, each dry_run_job event instead contains an execution object using the workflow mode's fields: type plus command for shell, code and lang for run, path and optional lang for script, or function and lang for call. JSON string escaping preserves multi-line text exactly.

Resolved execution text can contain values supplied through configuration or parameters. Treat dry-run output as potentially sensitive, just like job environments and logs; OxyMake does not redact those values.

Ray checks each active task's CPU (default 1), GPU, memory and custom request against one live node's total capacity before submitting the DAG. Busy capable nodes queue normally. The State API inspection has a 10-second end-to-end deadline; malformed, unknown, partial or truncated snapshots fail closed. An empty live-node list produces one warning and submits because a cluster may be scaling from zero; visible nodes that cannot satisfy a request still cause an error. The check is a snapshot and does not monitor topology changes. See the Ray chapter.

With -f ../Oxymakefile.toml, run and plan use the Oxymakefile directory for workflow paths, .oxymake, cache checks and execution. Explicit --report-json and --cache-remote paths remain relative to the invoking directory.

Concurrent sessions. Several ox run may share a workspace: each job is claimed in .oxymake/state.db before it is dispatched, so a job is executed by one session and the others wait for its result (see ox gate for the behaviour and OX_SESSION_LEASE_SECS for the lease after which a dead session's jobs are taken over).

--cache-remote <dir> stores each job's output blobs in the given directory (content-addressed, BLAKE3-verified on restore) and restores missing outputs from it. Validation is always promoted to hash when the flag is set. The directory is a blob transport, not a complete portable cache: the local SQLite index under .oxymake/cache/ that maps computation keys to output paths and hashes does not travel with the blobs, so a fresh checkout pointing at the same directory re-executes unless that local index is transferred too. A remote computation-key manifest that removes this requirement is future work. See Caching.

Local interruption. The first Ctrl+C (or SIGTERM, which is what ox cancel sends) starts a graceful shutdown: the session is recorded interrupted, every in-flight job is marked cancelled in the ledger, and its process group gets SIGTERM. The shutdown is bounded — a job that has not exited after the grace period (default 10 s, OX_SHUTDOWN_GRACE_SECS overrides; 0 escalates immediately) is killed with SIGKILL, so a child that traps or ignores SIGTERM cannot hold the run open. A second signal force-exits with 130, cancelling this session's still-running rows on the way out. Either way no row of the interrupted session is left running, and rows owned by a concurrent session are never touched (ADR-012).

A job failure without --keep-going is not an interruption: the run stops dispatching new jobs and lets the ones already running finish, then exits 1. A Ctrl+C during that wait cancels them with the same bounded shutdown; the exit code stays 1, since a job did fail.

Remote follow (--executor ray|slurm --follow). Polling stops with exit code 1 after three consecutive rounds containing a poll error. An entirely successful round resets the counter, so one or two transient failures do not abort a long run. A successful poll of one job does not hide another job's persistent error. Ray waits 3 seconds between rounds and SLURM waits 5 seconds; request latency is additional, so this is a retry bound, not a wall-clock deadline. The summary names the endpoint and last error and reports last known job counts.

The first Ctrl+C or SIGTERM stops following promptly, including during a poll request, and exits 130. It reports Follow interrupted and last known counts. For Ray, interruption also stops the driver and records its unfinished jobs as cancelled after the stop succeeds. The message confirms the driver stopped or reports the cancellation error without claiming an outcome. SLURM interruption and endpoint loss do not cancel remote jobs; unfinished work may still be running. Use ox status to inspect it and ox cancel to stop it. Stopped follows record the session as interrupted, preserve requested --timings, and do not print Completed. Endpoint loss and failed cancellation leave unfinished job rows unchanged. With --json, these outcomes emit run_follow_stopped with reason interrupted or endpoint_unreachable, the exit code, message, and last known counts.

Exit codes:

• 0 -- Success (all jobs succeeded or were cached)
• 1 -- Runtime error or one or more jobs failed
• 2 -- Command-line usage error
• 130 -- Interrupted by Ctrl+C / SIGTERM

ox plan

Show the execution plan without running anything.

Planning resolves the complete dependency graph, applies the same cache and staleness analysis as ox run, and displays the jobs that may execute. Existing generated outputs are not treated as source files, so a missing intermediate includes its producer and downstream rebuilds; a warm cache reports zero jobs. Job lines (and the JSON reason field) explain selections such as output missing and upstream rebuilt.

With the same filesystem and cache settings, plan reports an upper bound of what run executes. Planning cannot know the bytes a rebuilt job will produce, so it conservatively selects downstream jobs with upstream rebuilt. At runtime, if all rebuilt outputs match content hashes recorded under the same cache key, consumers receive their normal cache check and may be skipped. A producer whose cache key changed still forces its consumers, even when its bytes are identical. Changed outputs still invalidate consumers transitively. Missing recorded hashes, non-cacheable rules, --no-cache, and cache.validation = "mtime" retain conservative rebuilding.

ox plan                     # Show what would run (optimized)
ox plan --json              # Structured plan output
ox plan --no-optimize       # Show the raw plan (skip optimization passes)
ox plan --level rules       # Show the RuleGraph instead of the JobGraph

ox lint

Validate the Oxymakefile without executing.

ox lint                     # Check for errors
ox lint --json              # Structured diagnostics

Checks for: syntax errors, missing inputs, cycles, ambiguous rules, undefined wildcards.

Inspection Commands

ox dag

Visualize the dependency graph.

ox dag                      # Graphviz DOT output (default)
ox dag --format mermaid     # Mermaid graph syntax
ox dag --group-by rule      # Collapse nodes by field
ox dag --json               # Structured JSON

ox status

Show current execution status.

ox status                   # Summary of current state
ox status --json            # Structured status

Running vs. orphaned

ox status does not report jobs.status verbatim. A job counts as running only while the session that claimed it is active and its heartbeat is younger than the claim lease (90 s by default, OX_SESSION_LEASE_SECS overrides). A row left at running by a session that was interrupted, completed, or stopped heartbeating is reported as orphaned, with its age and the reason:

Sessions: 0 active
Jobs: 14 total (1 completed, 0 running, 1 orphaned, 2 failed, 10 pending, 0 cached, 0 cancelled)
Orphaned: 1 jobs abandoned by a dead session
  substrate_tests                orphaned  12m17s  (owning session was interrupted)
  the next `ox run` re-evaluates them; `ox status` never reclaims
Pending: 10 jobs waiting
  repro_spheres                  waiting for: substrate_tests (orphaned)

ox status is a reader: it never rewrites a row it reports as orphaned. Reclaiming happens at the start of the next ox run. The same derivation backs ox top and the web dashboard — see ADR-012.

--json carries the distinction too: jobs.orphaned alongside jobs.running, an orphaned_jobs array (each entry with reason, reason_description, declared_status, session_id, elapsed_secs), and waiting_for_orphaned on each pending job.

ox logs

View job logs.

ox logs stats-alice         # Logs for a specific job
ox logs --failed            # Logs for all failed jobs

ox history

List past runs.

Per-job detail includes the executor selected for each executed job, including local overrides in SLURM runs. The text table has an EXECUTOR column; JSON uses executor.

ox history                     # Recent runs
ox history --json              # Structured history
ox history --run-id run-1234   # Per-job detail for one run

With --run-id … --json, each job is emitted as one JSON object per line. Besides timing and exit code, the object carries the provenance of the cache decision — input_hashes, output_hashes, params_hash, env_hash, reproducibility_class and artifact_provenance — so a cache report can be reconstructed from recorded state instead of being caught live. A job the cache layer never keyed (--no-cache, or cache_validation = "mtime") reports null for these. peak_mem_mb is the executor's per-job peak-memory observation, stored as whole MiB rounded up (the text column is MEM MiB; the JSON key remains peak_mem_mb). Unlike history's whole-MiB round-up, benchmark TSV max_rss uses two decimal places: 1500 KiB appears as 2 in history and 1.46 in the TSV; a positive sub-MiB value rounds up to 1 in history.

ox run --executor slurm --follow records the maximum sacct MaxRSS across that job's steps (.batch, .0, etc.), retaining allocation status. Zero, missing accounting, and squeue fallback leave memory null. SLURM REST currently reports no peak-memory observation. Submission without --follow does not collect this accounting.

Local cold launches on Linux and macOS record the child's wait4 RSS in history, rounded up to MiB. This includes already-reaped descendants but is not a simultaneous process-tree total; background or daemonised work can escape it. Fork-mode warm workers also record the child each dispatch forks and reaps, so the same retained template can publish distinct peaks for successive jobs. Persistent-mode warm workers remain null: one process serves every dispatch, so a per-dispatch RSS peak does not exist. CPU time is measured for cold and warm-fork children too, but appears only in benchmark TSV, not history.

The figure belongs to the attempt that finished the job; earlier failed attempts are not accumulated. Failures and cancellations leave memory null. The scheduler's ErrorStrategy::Ignore is an exception: it treats a failed attempt as completed and its job_completed event can carry that attempt's memory. Mock commands verify persistence; a live SLURM cluster has not verified this path.

Management Commands

ox gate

Manage gates (human-in-the-loop checkpoints).

A gate is declared in the Oxymakefile and names the rules it guards:

[gate.qc_check]
after = ["align"]            # informational: the rules whose results you review
before = ["publish"]         # every job of these rules waits for the gate
message = "Check alignment QC before publishing."

When ox run reaches a job of a rule listed in before, it registers the gate as pending in .oxymake/state.db, prints the message and waits, polling the gate until a decision is recorded:

ox gate list                              # All gates: id, name, status, run
ox gate approve qc_check                  # Approve by name — the run resumes
ox gate approve qc_check --reason "ok" --approver alice
ox gate reject qc_check --reason "bad"    # Reject — the guarded jobs are cancelled
ox gate approve 3                         # By id (shown by `ox gate list`)

Rules of the gate ledger:

• One record per gate and run. A decision belongs to the ox run that asked for it; the next run that reaches the gate registers a fresh pending record and waits again. Runs whose guarded outputs are already up to date never reach the gate.
• Names first, ids as a fallback. approve/reject match the gate name against pending records; with a single run waiting there is exactly one. If two runs wait on the same gate, the name is ambiguous and the command lists the ids to use instead.
• Two approved runs of the same job execute it once. Before it dispatches a job, ox run claims it in state.db (the cooperative claim protocol of ADR-012). Approving both ids therefore makes one session execute the job and the other wait: it prints <job> is being executed by <session>; waiting for its result instead of running it here, then takes over the owner's result — a completion counts as done in both sessions (both exit 0, exactly one execution happened), a failure or cancellation by a live owner is mirrored. Sessions heartbeat every third of a lease (default 90 s, OX_SESSION_LEASE_SECS overrides); if the owner is killed mid-job, the waiting session reclaims the job once the lease has expired and executes it itself. A waiting run stops on the first Ctrl+C / SIGTERM without touching the peer's job. Defence in depth: the local executor also takes an exclusive flock(2) on each output path under .oxymake/locks/ for the duration of execution and commit, and a job whose paths are locked by another live process fails closed (job '<id>' is already being executed by another session (pid N): output '<path>' is locked) — for instance a reclaimed job whose killed owner's orphaned shell is still writing. That lock holds on local filesystems with a working flock(2); the claim protocol holds wherever SQLite does, hosts included.
• after adds no dependency edge. A gate is evaluated once a guarded job's own inputs are ready, so list in after rules that are upstream of the before rules through the DAG.
• Local executor only. Gates are enforced by the scheduler; --executor slurm and --executor ray submit the DAG without it, so a gated workflow is refused on those executors.
• The ledger is the enforcement. If .oxymake/state.db cannot be opened, a gated run fails instead of running the guarded rules unapproved. If a gate record cannot be written (the run reports that the gate "could not be registered"), the guarded jobs stay blocked and the registration is retried on every poll; an absent record never opens a gate.

ox snapshot

Manage workflow snapshots for comparison.

ox snapshot save baseline-v1        # Save current state
ox snapshot diff baseline-v1        # Compare with snapshot
ox snapshot list                    # List snapshots

ox invalidate

Invalidate cached outputs to force re-execution.

ox invalidate stats                 # Invalidate a rule
ox invalidate results/alice.txt     # Invalidate a specific output

ox clean

Remove outputs and cache.

ox clean                    # Remove all outputs
ox clean --cache            # Also remove cache
ox clean --state            # Delete a corrupt state.db (it is a regenerable cache)

ox cancel

Cancel running jobs.

ox cancel                   # Cancel all running jobs
ox cancel stats-alice       # Cancel a specific job

ox cancel marks the jobs cancelled in .oxymake/state.db and sends SIGTERM to the owning ox run, which handles it on the same graceful path as Ctrl+C — including the bounded SIGTERM → SIGKILL escalation described under ox run.

ox top

Live TUI dashboard for monitoring execution.

ox top                      # Interactive dashboard

Shows real-time job status, resource utilization, and DAG progress.

It uses the same running-vs-orphaned derivation as ox status: the Running Jobs panel lists only jobs a live session is executing, and the panel title carries an N orphaned count when rows were abandoned. ox top never reclaims them.

Global Options

Every command accepts:

Most subcommands additionally accept --json (structured NDJSON output) and -v/-vv (increase verbosity).

Next Steps

• Oxymakefile Format -- workflow definition reference
• Configuration -- project settings

ox migrate

Review a schema migration with ox migrate --to-format 2; add --write to apply it to the whole include graph. See the command reference.

ox migrate

ox migrate --to-format 2 [-f Oxymakefile.toml] [--write]

The default is a report: no workflow files are changed. The report names every file and old/new version declaration. --write applies the complete checked include graph. Only destination format 2 is supported.

Absent format_version means legacy schema 1. The known generated ox_version markers 0.1 and 0.3 become >=0.7.0, the release introducing schema 2. All other legacy declarations, including a missing declaration, require manual resolution: the tool cannot infer what binary the author intended. Valid schema-2 files can coexist with legacy files and remain unchanged.

Migration preserves command bytes, comments and software environments, but may normalize document formatting, including line endings. It does not extract shell assignments. Unknown keys (including future features), unsupported schemas, include cycles, missing files, invalid merged workflows and unsatisfied requirements cause refusal before writing any original.

All replacements and recovery copies are staged before writing. A detected concurrent edit or staging error leaves originals intact; a replacement error rolls back files already replaced. This is not a crash-atomic filesystem transaction. If the filesystem prevents rollback, the error names the retained recovery copies. Keep normal version-control backups and avoid editing the graph concurrently with migration.

Migration preserves cache identity. Any future change to execution semantics will state its own cache consequences in that release's notes. A minimum binary requirement is a capability floor, not a reproducible build guarantee.

ox lock

Generate or verify a reproducibility lockfile.

The ox lock command captures a cryptographic snapshot of the entire workflow — rule definitions, config values, input hashes — into an ox.lock file. Use it to detect unintended changes between runs or across machines.

Subcommands

ox lock generate

Generate an ox.lock file from the current workflow state.

ox lock generate                        # Write ox.lock next to Oxymakefile.toml
ox lock generate -o locks/my.lock       # Write to a custom path
ox lock generate -f path/Oxymakefile.toml

Options:

ox lock verify

Verify the current state against an existing ox.lock.

ox lock verify                          # Verify against ox.lock
ox lock verify -l locks/my.lock         # Verify against a custom lockfile

Options:

Exit codes:

• 0 — Lock matches current state
• 1 — Mismatch detected (details printed to stderr)

Examples

# Pin the workflow before a release
ox lock generate
git add ox.lock && git commit -m "lock: pin workflow v2.1"

# CI: verify nothing drifted
ox lock verify || { echo "Workflow changed since lock!"; exit 1; }

See Also

• CLI Commands — full command index
• Content-Addressable Cache — how OxyMake tracks state

ox test

Test and validate a workflow without executing it.

The ox test command resolves the DAG, checks for structural errors, and optionally simulates execution order — all without running any shell commands. Use it to catch misconfigurations before committing to a full run.

Usage

ox test                             # Validate entire workflow
ox test results/report.html         # Validate a specific target
ox test --dry-run                   # Simulate execution order
ox test --json                      # Output NDJSON diagnostics

Arguments

Options

What It Checks

• Oxymakefile parses without errors
• All wildcards resolve against [config] values
• Dependency graph is acyclic
• Every input is either a source file or produced by a rule
• Wildcard constraints are satisfied

Examples

# Quick validation in CI
ox test || exit 1

# Check a single target's dependency chain
ox test results/{sample}_stats.tsv

# Dry-run to see execution order
ox test --dry-run

See Also

• ox lint — lighter-weight syntax check
• ox plan — show full execution plan

ox dashboard

Web dashboard for monitoring and DAG visualization.

The ox dashboard command starts a local HTTP server that serves an interactive web UI. The dashboard reads from the OxyMake state database and provides real-time job status, DAG visualization, and run history.

Usage

ox dashboard                        # Start on http://127.0.0.1:9876
ox dashboard --port 8080            # Custom port
ox dashboard --bind 0.0.0.0         # Listen on all interfaces
ox dashboard --db path/to/state.db  # Custom state database

Options

Features

• Status cards — at-a-glance counts of running, succeeded, and failed jobs
• DAG visualization — interactive dependency graph
• Job table — sortable list of all jobs with status and timing
• Run history — browse past runs and their outcomes

Running vs. orphaned

Like ox status, the dashboard reports an effective status rather than jobs.status verbatim: a job is running only while the session that claimed it is active with a heartbeat younger than the claim lease (90 s, OX_SESSION_LEASE_SECS overrides). A row left behind by a session that was interrupted, completed, or that stopped heartbeating shows as orphaned, in its own card and with its own badge — never the RUNNING badge, and never counted in the ETA or the jobs/sec strip.

This shows up in the API as:

orphan_reason is one of session_interrupted, heartbeat_stale, session_completed, session_missing.

The dashboard is read-only: it reports orphaned rows, it never reclaims them. Reclaiming happens at the start of the next ox run. See ADR-012.

Examples

# Start dashboard alongside a long-running workflow
ox run -j 8 &
ox dashboard
# Open http://127.0.0.1:9876 in a browser

# Expose to the local network (e.g. for a shared workstation)
ox dashboard --bind 0.0.0.0 --port 8080

See Also

• ox top — terminal TUI dashboard
• ox status — CLI status summary

ox translate

Translate a Snakefile into OxyMake TOML.

The ox translate command parses a Snakemake Snakefile and emits an equivalent Oxymakefile.toml. Use it to migrate existing Snakemake workflows to OxyMake without rewriting rules by hand.

Usage

ox translate Snakefile                       # Writes Snakefile.translated.toml
ox translate Snakefile -o Oxymakefile.toml   # Writes a custom path

When -o is omitted, the translator writes two files next to the input:

• <INPUT>.translated.toml — the generated Oxymakefile
• <INPUT>.translated.toml.escalations.toml — written only when the IR contains escalations

Every run emits a one-line summary to stderr:

translated: N rules (X mechanical, Y with escalations); dropped: Z unsupported top-level constructs; includes: K files NOT followed

ox translate exits with status 2 when escalations were recorded so CI or shell scripts can gate on a clean translation. The files are still written; only the exit code changes.

Arguments

Options

Translation Notes

The translator handles the most common Snakemake patterns:

• rule blocks → [[rule]] sections
• input / output → inputs / outputs
• expand() calls → OxyMake wildcard {sample} syntax
• params → [rule.params]
• shell → command

Complex Python logic inside Snakefiles (e.g., run: blocks, conditional inputs, lambda wildcards) may require manual adjustment after translation. Review the generated TOML and run ox lint to verify.

Examples

# Quick migration — produces Snakefile.translated.toml
ox translate Snakefile
ox lint -f Snakefile.translated.toml   # Verify the result
ox plan -f Snakefile.translated.toml   # Check execution plan

# Custom output path
ox translate Snakefile -o Oxymakefile.toml

# CI gate: fail the job when escalations were emitted
ox translate Snakefile || echo "needs manual review"

See Also

• Oxymakefile Format — full TOML reference
• ox lint — validate the generated file

ox query

Query the dependency graph using Bazel-style expressions.

Usage

ox query <EXPRESSION> [OPTIONS]

Expressions

Options

Examples

# What does annotate depend on?
ox query 'deps(annotate)'

# What depends on the data rule? (reverse dependencies)
ox query 'rdeps(data)'

# All paths from data to annotate
ox query 'allpaths(data, annotate)'

# JSON output for programmatic use
ox query 'deps(annotate)' --json

See Also

• ox dag -- visualize the DAG
• ox plan -- show the execution plan

ox export

Export an Oxymakefile to another workflow format.

Usage

ox export <FORMAT> [OPTIONS]

Formats

Options

Examples

# Export to stdout
ox export snakemake

# Export to file
ox export snakemake -o Snakefile

# Export a specific Oxymakefile
ox export snakemake -f pipelines/Oxymakefile.toml -o Snakefile

Bidirectional Translation

OxyMake supports bidirectional Snakemake translation:

• Import: ox translate Snakefile converts Snakemake to OxyMake TOML
• Export: ox export snakemake converts OxyMake TOML back to Snakemake

This enables zero-friction migration in both directions.

See Also

• ox translate -- import from Snakemake
• ox cache-export -- describe cached outputs for another machine
• Oxymakefile Format -- the OxyMake workflow format

ox cache-export

Describe cached outputs so another machine can adopt them.

Usage

ox cache-export <TARGET>... [-o <PATH>]

The command writes a versioned JSON manifest containing every output path and BLAKE3 hash, input hashes, job-spec hash, reproducibility class, platform scope, producing platform, manifest kind, and producing ox version. Output files are not copied; transport them separately and preserve their workflow-relative paths.

ox run build/counts.parquet
ox cache-export build/counts.parquet -o counts.ox-cache.json

Without -o, the manifest is written to stdout. There is no default manifest filename. Only completed targets with current cache provenance can be exported; rebuild an older target first if necessary.

See also

• ox cache-import -- adopt the described outputs
• ox export -- translate an Oxymakefile to Snakemake or WDL

ox cache-import

Adopt outputs produced in another workflow tree as verified local cache entries.

Usage

ox cache-import <MANIFEST> [-f <OXYMAKEFILE>]

Place every output named by the manifest at the same workflow-relative path, then run ox cache-import. OxyMake re-hashes every output before changing .oxymake/cache/cache.db; one mismatch rejects the entire manifest. It also checks that the local rule's job specification, platform scope, output set, and reproducibility policy agree with the manifest.

# After copying counts.ox-cache.json and build/counts.parquet:
ox cache-import counts.ox-cache.json
ox run build/model.rds

Imported outputs whose producer inputs are absent become trusted resolver leaves. Downstream jobs can therefore run without copying the raw inputs or maintaining a second, pruned Oxymakefile.

An exact entry can only be imported on its producing OS/architecture. Rules with cache_platform = "any" may cross that boundary. Non-reproducible rules are always refused.

The manifest has a fixed kind and a versioned format. Newer unsupported versions are rejected before their body is decoded, with both producer and consumer ox versions and an upgrade remedy in the error.

See also

• ox cache-export -- create an adoption manifest
• Content-addressable cache

Configuration

OxyMake uses a layered configuration system. Workflow-level settings live in Oxymakefile.toml, and project-level settings live in .oxymake/config.toml.

Workflow Configuration

The [config] section in Oxymakefile.toml defines variables for wildcard expansion:

[config]
samples = ["A", "B", "C"]
models = ["linear", "ridge"]

These values drive wildcard resolution in rules.

Named Profiles

Per-run defaults live in the Oxymakefile, as [profile.NAME] sections selected with ox run --profile NAME:

[profile.cluster]
jobs = 32
executor = "slurm"
partition = "gpu"

[profile.ray]
executor = "ray"
ray_allow_pending = true

ray_allow_pending is the profile equivalent of --ray-allow-pending. Explicit CLI flags take precedence over profile values, except that ray_allow_pending = true cannot be disabled from the CLI: there is no negative flag. Set it to false in the profile or select a different profile to restore the feasibility check.

There is no project-level settings file. ox init creates Oxymakefile.toml and an empty .oxymake/ directory; it does not write a config.toml, and no .oxymake/config.toml is read if you create one by hand.

User Global Config

$XDG_CONFIG_HOME/oxymake/config.toml (or ~/.config/oxymake/config.toml) is read for exactly two top-level keys:

cache_validation = "hash"   # default cache validation strategy
open_dashboard = true       # open the web dashboard on `ox run`

Environment Variables

OxyMake reads the following environment variables:

Everything else is set with a command-line flag or in the Oxymakefile.

Configuration Precedence

cache_validation — the one setting with a full resolution chain — is resolved in order (later overrides earlier):

1. Built-in default (mtime+hash)
2. User global config (~/.config/oxymake/config.toml)
3. cache_validation under [config] in the Oxymakefile
4. OX_CACHE_VALIDATION
5. --cache-validation on the command line

State Directory

The .oxymake/ directory contains:

.oxymake/
  state.db          # SQLite execution state + audit log
  cache/            # Content-addressable output cache
  logs/             # Per-rule stdout/stderr
  events/           # NDJSON run event streams

.oxymake/ is created in the directory ox is invoked from. Its location is not configurable: no flag, config key, or environment variable relocates it.

The state database (state.db) uses SQLite WAL mode for concurrent access, and must reside on local disk (not NFS/Lustre/GPFS). Because a rule's inputs and outputs are also resolved relative to the working directory, satisfying that requirement means running the whole workspace from local disk — the database cannot be kept local while the project tree lives on shared storage.

Next Steps

• Oxymakefile Format -- workflow definition reference
• CLI Commands -- command reference
• Expression Language -- expression syntax

Expression Language

OxyMake includes a minimal expression language for conditional guards and dynamic values in workflow definitions. The language is deliberately limited: pure functions, no loops, no side effects.

Guard Expressions

The when field on a rule accepts a boolean expression:

[rule.expensive_model]
output = ["results/{seed}_{model}.txt"]
shell = "train --seed {seed} --model {model}"
when = "seed in @selected_seeds"

If the guard evaluates to false, the job is not created in the DAG.

Supported Operators

Membership

when = "sample in @high_priority_samples"   # Check if wildcard is in a config list
when = "model in ['linear', 'ridge']"       # Check against inline list

Comparison

when = "wildcards.threshold >= 0.5"
when = "wildcards.replicate != 'control'"

Logical

when = "sample in @fast_samples and model == 'linear'"
when = "not (sample in @excluded)"

Variable References

Wildcards

Access wildcard values with bare names or the wildcards. prefix:

shell = "process {sample}"                   # Bare wildcard in commands
when = "wildcards.sample in @selected"       # Explicit prefix in guards

Config References

Reference config arrays with @:

when = "sample in @priority_samples"         # @name refers to config.name

Built-in Variables

String Interpolation

In shell, run, and script fields, {braces} perform string interpolation:

shell = "python process.py --input {input} --output {output} --sample {wildcards.sample}"

Shell safety. In a shell field, interpolation is raw text inserted into the command passed to the configured shell with -c; it is not shell escaped and {input}/{output} lists are space-joined. Never interpolate third-party file names, wildcard values, or configuration values into shell without validating them for the shell context. See SECURITY.md.

Double braces {{ and }} produce literal braces (useful in Python code):

run = """
result = {{"key": "value"}}
"""

Design Philosophy

The expression language is intentionally not Turing-complete. Complex configuration logic should happen outside the Oxymakefile:

python gen_config.py > config.toml     # Generate config externally
ox run --config config.toml            # Use generated config

This preserves static parseability: any tool can read an Oxymakefile without executing code.

Next Steps

• Oxymakefile Format -- complete format reference
• Rules and Wildcards -- wildcard patterns
• Tags and Filtering -- tag-based job selection