Standard / bench

OGS Benchmark Specification

Open Gastronomy Standard — Benchmark v0.1

1. Purpose

The OGS Benchmark (OGS-Bench) defines a reproducible, open methodology for measuring the gastronomy intelligence of large language models and other AI systems against the Open Gastronomy Standard. It is intended to become the official global benchmark for evaluating gastronomy-capable AI across:

• Taste, aroma, and texture reasoning
• Dish composition and cooking-method analysis
• Food-and-beverage pairing judgment and explainability
• Knowledge of classical cuisines, ingredients, and wine regions
• Structured, schema-conformant JSON generation against OGS itself

OGS-Bench is model-agnostic. Any system that can consume text prompts and produce text (ideally JSON) responses can be scored — including "gpt-5.4", "gpt-4.1", open-source LLMs, rule-based systems, and human panels.

2. Design Principles

3. Capability Categories

OGS-Bench v0.1 defines eight capability categories. Each has its own ID prefix, scoring methods, and weight in the aggregate score.

Implementers MAY use alternate weights for their own leaderboards, but the default weights above MUST be reported for any result that claims to be an "OGS-Bench v0.1 aggregate score".

4. Task Identifiers

Benchmark task IDs follow the OGS ID convention with a new entity type benchmark-task:

ogs:core:benchmark-task:<category-code>-<NNN>

Example: ogs:core:benchmark-task:pair-004.

The core namespace is reserved for the official task set shipped with this specification. Third-party task packs MUST use a distinct namespace (e.g., ogs:acme:benchmark-task:pair-001).

5. Task Document Structure

Every task is a standalone JSON document matching schema/benchmark-task.schema.json.

5.1 Required fields

5.2 Optional fields

5.3 Prompt object

"prompt": {
  "instruction": "Select the OGS aromatic family for 'lemon zest'.",
  "input": {
    "item": "lemon zest"
  },
  "answer_format": "one_of:fruit,floral,herbal,spice,earth,wood,dairy,savory,confection,vegetal,marine",
  "response_schema": null
}

• instruction: always required. This is the core ask to the model.
• input: required, but MAY be an empty object. Contains structured payload the model should reason about.
• answer_format: an advisory string describing how the response should be formatted (e.g., json, one_of:a,b,c, integer_0_10, set_of_codes).
• response_schema: optional JSON Schema the response will be validated against when scoring.type is schema_validation or json_match.

5.4 Expected object

The shape of expected depends on the scoring type:

6. Scoring

Each task declares exactly one scoring.type. All scorers produce a real-valued score in [0, 1]. The benchmark runner aggregates these at four levels:

1. Task score — raw scorer output for a single task.
2. Capability score — unweighted mean of task scores in the same category.
3. Category score — same as capability score for v0.1 (one capability per category for now).
4. Aggregate score — weighted sum over categories using the weights in §3, expressed on a 0–100 scale.

6.1 Scorer specifications

6.2 Aggregate formula

Let C = {VOCAB, SENSE, COMP, PAIR, XPLAIN, RISK, STRUCT, REGION} and w_c the weight of category c. Let s_c be the mean of all task scores in category c. Then:

aggregate_score = 100 * Σ_c w_c * s_c

If a category has zero attempted tasks, the aggregate is reported as partial and the missing category weight is redistributed proportionally.

7. Running the Benchmark

The reference runner lives in bench/ and exposes a small CLI:

# List available tasks
python -m bench list

# Run a specific model on all tasks
python -m bench run --model gpt-5.4 --out results/gpt-5.4.json

# Run only a subset of categories
python -m bench run --model gpt-4.1 --categories pairing,risk --out results/gpt-4.1.pair.json

# Compare two result files
python -m bench report results/gpt-5.4.json results/gpt-4.1.json

Model adapters currently supported out of the box:

Adapters are pluggable: new ones may be registered via bench.models.register.

Every run produces a result document matching schema/benchmark-result.schema.json, which records the model ID, per-task scores, per-category aggregates, prompts, raw responses, and the benchmark version used.

8. Conformance

A benchmark submission is OGS-Bench v0.1 conformant if it:

1. Uses the official task set at the declared benchmark_version.
2. Reports both per-category and aggregate scores using the weights in §3.
3. Publishes its result document, which MUST include raw model responses and the exact prompts (the reference runner does this by default).
4. Discloses the adapter and any non-default prompting options (temperature, system message, tool use, etc.).

Submissions MAY additionally report results on third-party task packs, but those do not count toward the OGS-Bench v0.1 aggregate.

9. Versioning

OGS-Bench follows the same Semantic Versioning rules as OGS core:

• MAJOR — breaking changes to task/result schemas, scoring definitions, or category weights.
• MINOR — adding tasks, new categories, new scorers (backwards-compatible).
• PATCH — clarifications, bug fixes, added rationale text.

Every task carries an explicit benchmark_version. Runners MUST refuse to score tasks whose MAJOR version they do not support.

10. References

• ogs-core.md — Core specification
• ogs-sense.md — Sensory model
• ogs-comp.md — Composition model
• ogs-match.md — Matching model
• schema/benchmark-task.schema.json — Task document schema
• schema/benchmark-result.schema.json — Result document schema
• bench/ — Reference runner and tasks