reference · oto bench --help at v0.6.1

oto bench

measure answer quality against a gold question set

oto bench [-h] [--from SOURCE] [--type TYPE] [--project PROJECT] [--gold GOLD]
                 [--systems SYSTEMS] [--ablations] [--k K] [--out OUT] [--force]
                 [{run,validate,start,add}]

Verbs

run scores the gold set
validate checks it
start writes a starter file
add appends questions from a file

Options

Option	What it does
`--from SOURCE`	for add: a JSON file {"source_doc": ..., "questions": [...]} (repeatable)
`--type TYPE`	for add: default question type
`--project PROJECT`	project root (default: current directory)
`--gold GOLD`	gold set path (default: <project>/gold/questions.jsonl)
`--systems SYSTEMS`	comma-separated systems to compare. `dense` is the embedding baseline and needs the bench extra; it is reported as unmeasured if absent
`--ablations`	also run Oto with the graph off and with time off, to show each pillar defends the metric it claims to
`--k K`	k for coverage
`--out OUT`	write machine-readable results here
`--force`	run despite gold-set problems

Where to use it

Start measuring.

oto bench start --project claims

a starter gold set; fill in who wrote the questions before anything else

Questions came out of a document's review.

oto bench add --project claims --from claims/proposals/total-loss-procedure.questions.json

appends them to the gold set

Is the set worth running?

oto bench validate --project claims

refuses a set with no recorded provenance or a question with nothing to grade against

Run it, and show each pillar defends its metric.

oto bench run --project claims --ablations --out bench.json

scores the set, with the graph off and time off as ablations, and the embedding baseline or an honest unmeasured

As the terminal prints it

usage: oto bench [-h] [--from SOURCE] [--type TYPE] [--project PROJECT] [--gold GOLD]
                 [--systems SYSTEMS] [--ablations] [--k K] [--out OUT] [--force]
                 [{run,validate,start,add}]

positional arguments:
  {run,validate,start,add}
                        run scores the gold set; validate checks it; start writes a starter file;
                        add appends questions from a file

options:
  -h, --help            show this help message and exit
  --from SOURCE         for add: a JSON file {"source_doc": ..., "questions": [...]} (repeatable)
  --type TYPE           for add: default question type
  --project PROJECT     project root (default: current directory)
  --gold GOLD           gold set path (default: <project>/gold/questions.jsonl)
  --systems SYSTEMS     comma-separated systems to compare. `dense` is the embedding baseline and
                        needs the bench extra; it is reported as unmeasured if absent
  --ablations           also run Oto with the graph off and with time off, to show each pillar
                        defends the metric it claims to
  --k K                 k for coverage
  --out OUT             write machine-readable results here
  --force               run despite gold-set problems