Skip to content

Calibration Arm D — Frozen Joined CSV Schema

Status: Frozen at JOINED_SCHEMA_VERSION = "joined.v1"
Canonical source: swarm/calibration/joined.py
Pre-registration: docs/research/calibration-prereg.md#arm-d-freeze-joined-csv-schema
Contract: Adaptive agents arms 1–3 and all downstream studies join calibration results against this schema by run_id.

Schema Overview

The joined CSV contains one row per accepted interaction from a calibration run. Each row merges: - Proxy-side data (from arm D's inline ProxyComputer pass): the proxy's v_hat and p_hat, the interaction's ground-truth label p_true, and metadata. - Judge-side data (from arm B's judge ensemble): per-judge scores and rationales.

The schema deliberately separates proxy and judge signals so downstream consumers can ask: "where does the proxy track ground-truth but judges diverge?" — i.e., where the proxy may be fooled.

Frozen Base Columns (9 columns)

These columns are immutable as part of the joined.v1 contract. Any rename or reorder must bump JOINED_SCHEMA_VERSION and will break downstream consumers loudly.

Column Type Nullability Format Provenance Description
interaction_id string not null UUID (string repr) Arm D (inline proxy) Unique identifier for the interaction. Generated by the fixture when the scenario is created; matches by construction because proxy and judges run in the same process.
scenario string not null "mixed", "obfuscation", or "self_optimizer" Arm D config Scenario type; defines which interaction generator was used. Fixed at run time via --scenario flag.
seed integer not null non-negative int Arm D config Random seed for scenario generation; ensures reproducibility of the interaction pool. Fixed at run time via --seed flag.
interaction_type string not null interaction type name (e.g., "REPLY", "TRADE") Arm D (interaction object) Type of interaction. Extracted from interaction.interaction_type.name (or stringified if not an enum).
accepted integer not null 0 or 1 Arm D (interaction object) Whether the interaction passed the acceptance filter (1 = accepted, 0 = rejected). Only accepted interactions are included by default (accepted_only=True in build_proxy_rows). Formatted as integer in CSV.
p_true float not null Decimal, 6 places (e.g., 0.753421) Arm D (interaction object) Ground-truth probability that the interaction is beneficial, P(v = +1). The latent label the entire calibration study anchors to. Extracted from interaction.p; missing p raises AttributeError (no silent 0.5 fallback).
v_hat float not null Decimal, 6 places (e.g., -0.234567) Arm D (ProxyComputer) Raw proxy score before sigmoid normalization, in [-1.0, +1.0]. Computed from observables (task_progress_delta, rework_count, verifier_rejections, tool_misuse_flags, counterparty_engagement_delta) via ProxyComputer.compute_labels().
p_hat float not null Decimal, 6 places (e.g., 0.417890) Arm D (ProxyComputer) Normalized proxy probability estimate, sigmoid(k * v_hat), in [0.0, 1.0]. The proxy's assessed likelihood the interaction is beneficial. k (sigmoid temperature) is configurable; default is 1.0.
ground_truth integer nullable -1, +1, or empty string Arm D (interaction object) Realized outcome label, sampled from Bernoulli(p_true). Extracted from interaction.ground_truth; empty string if absent. Represents the latent v ∈ {-1, +1} that p_true was the probability of.

Dynamic Judge Columns (paired per judge)

For each judge in --judges <judge1> <judge2> ..., two columns are appended in the order judges appear:

Column Pattern Type Nullability Format Provenance Description
judge_{judge_name}_score float nullable Decimal, 6 places, or empty string Arm B (LLM judge) Judge's quality score for the interaction, in [0.0, 1.0]. Empty string if the judge did not rate this interaction (e.g., sampling strategy or error). Judged against rubric rubric.v1 (pre-registered) unless overridden.
judge_{judge_name}_rationale string nullable Free text, CSV-escaped Arm B (LLM judge) Judge's brief written justification for the score. Empty string if absent or if the judge did not rate this interaction.

Judge names are drawn from --judges argument at run time. Known backends: - "mock" — deterministic mock judge (no API calls) - "claude" — Anthropic Claude (via ANTHROPIC_API_KEY) - "gpt4o_mini" — OpenAI GPT-4o-mini (via OPENAI_API_KEY) - "llama" — Ollama Llama 3.x (running locally) - "qwen", "mistral", "glm" — Other Ollama models

Join Key and Reproducibility

  • Join key: run_id (implicit from the run directory name; not a column).
  • Downstream studies load the CSV from runs/<timestamp>_calibration_join_seed<seed>/joined.csv.
  • Join on (scenario, seed, interaction_id) to correlate with other tables.

  • No direct run_id column, but seed is stable across multiple runs of the same scenario.

  • Reproducibility guarantees:

  • Same scenario + seed reproduces identical interaction_ids and p_true values.
  • Same scenario + seed + judge backend reproduces identical judge scores (deterministic on mock; LLM judges may vary).
  • The config.json in the run directory records schema_version, judge models, rubric versions, and git rev for full auditability.

Example Row

For a hypothetical run with judges ["mock", "claude"]:

interaction_id,scenario,seed,interaction_type,accepted,p_true,v_hat,p_hat,ground_truth,judge_mock_score,judge_mock_rationale,judge_claude_score,judge_claude_rationale
550e8400-e29b-41d4-a716-446655440000,obfuscation,42,REPLY,1,0.753421,0.234567,0.558962,1,0.750000,agent_type=honest,0.720000,"Low effort visible in verbiage."

Nullability and Empty Cell Handling

  • p_true, v_hat, p_hat, scenario, seed: Never empty; these are core contract fields.
  • ground_truth: Empty string ("") when interaction.ground_truth is None. This is valid and expected for synthetic scenarios.
  • judge_*_score: Empty string when a judge did not rate the interaction (e.g., stratified sampling, or cross-scenario evaluation of a pre-existing arm B run). Consumers must handle gracefully.
  • judge_*_rationale: Empty string when the rationale is absent (standard for mock judge; some LLM judges may omit this field).

Adaptive studies must treat empty judge cells as missing data, not as zeros.

Data Types and Format Precision

  • Floating-point fields (p_true, v_hat, p_hat, judge_*_score): All rendered with exactly 6 decimal places (e.g., 0.750000). When re-loading via csv.DictReader, parse with float() to recover full precision.
  • Integer fields (scenario, seed, interaction_type, accepted, ground_truth): Rendered as their natural string representation.
  • accepted is an integer (0 or 1), not a boolean string.
  • ground_truth is an integer (-1 or +1) or empty string, never "null" or "None".

Provenance Summary

Data Arm Computed Where Reference
interaction_id, p_true, ground_truth D (fixture) Interaction generator tests/fixtures/interactions.py
scenario, seed, interaction_type, accepted D (config + fixture) calibration_join.py main Passed via --scenario / --seed; read from interaction object
v_hat, p_hat D (inline proxy) ProxyComputer.compute_labels() swarm/core/proxy.py
judge_*_score, judge_*_rationale B (external anchor) Judge backend (MockJudge or LLMJudge) swarm/judges/judge.py; scores read from judge.score(view) verdict

Spec / Code / Prereg Deltas

Observed Discrepancies

  1. Pre-reg mentions agreement_metrics in the CSV; code does not include them.
  2. Pre-reg text: "The deliverable is a CSV of accepted interactions with {v_hat, p, ground_truth, judge_score_claude, judge_score_gpt4o_mini, judge_score_llama, agreement_metrics} that downstream studies can join against by run-id."
  3. Code reality: agreement_metrics (Krippendorff's α, ICC, Spearman ρ, disagreement-by-bin) are not columns in the joined CSV. They are computed by arm C (inter-rater agreement) and written to separate outputs: runs/<ts>_calibration_agreement/summary.csv and runs/<ts>_calibration_agreement/bins.csv.
  4. Rationale: Agreement metrics are aggregate properties of the judge ensemble, not per-interaction properties. Including them as repeated columns in the joined CSV would wastefully replicate the same values in every row.

  5. Impact: Downstream consumers must join the joined CSV with arm C's summary if they need agreement statistics. Separate runs required.

  6. No "run_id" or "rubric_version" column in the joined CSV.

  7. Pre-reg implied: The joined CSV is the "frozen schema downstream studies join against by run_id," suggesting run_id might be a column.
  8. Code reality: run_id is implicit (read from the directory name); not a CSV column. rubric_version is not included because all rows in a run use the same rubric.
  9. Rationale: Immutable columns (scenario, seed, interaction_type, accepted) plus run-directory metadata are sufficient for joins. Adding redundant columns violates normalization and would force the schema to change if rubric policies evolve.

  10. Impact: Downstream consumers must track which run directory (and thus which rubric version) each CSV came from via their own index. The run's config.json records the rubric version for auditability.

  11. Column order is significant.

  12. Code enforcement: BASE_COLUMNS is a frozen tuple in swarm/calibration/joined.py. Tests assert it cannot be reordered without bumping JOINED_SCHEMA_VERSION.
  13. No pre-reg specification: The pre-reg lists column names but does not mandate order. However, the code intentionally freezes order to prevent accidental silent breaking changes.
  14. Impact: Consumers relying on positional indices must verify they match the schema version.

Breaking Risks (Adaptive Study Integration)

The adaptive studies (arms 1–3) join the joined CSV by (scenario, seed, interaction_id) to augment their own runs with the calibration anchor. Potential breaking scenarios:

  1. Removing interaction_id: Adaptive studies cannot join without it. Risk: CRITICAL.
  2. Removing or renaming p_true: The entire study's ground-truth signal is lost. Risk: CRITICAL.
  3. Removing judge columns: Studies lose the external quality anchor. Risk: MAJOR. Partial removal (fewer judges) is acceptable as long as downstream code handles missing columns gracefully.
  4. Reordering BASE_COLUMNS: Positional-index consumers will fail silently. Mitigated by: Test assertion on schema version.
  5. Empty rows on stratified sampling: Arm B samples interactions (≥50 per bin), not all. Some rows in the joined CSV will have empty judge cells. Mitigation: Consumed by downstream as "unrated by this judge," not as an error. Essential for cost control on LLM judge runs.
  6. Different rubric versions: Arm B is pre-registered to rubric.v1; runs with rubric.v2 or later are recorded as pre-reg deviations. Downstream consumers must not assume all rows use the same rubric unless they verify via config.json. Mitigated by: config.json records rubric version and deviation status.

Freeze Readiness

Schema is frozen and code is production-ready. The joined CSV is emitted by experiments/calibration_join.py (arm D) with version-locked structure. Tests in tests/test_calibration_joined.py verify: - ✅ Schema version assertion (joined.v1) - ✅ BASE_COLUMNS immutability (tuple, frozen order) - ✅ CSV round-trip (write → read via DictReader → consistency) - ✅ Missing judge handling (empty cells, not nulls) - ✅ Missing ground_truth handling (empty string, not null)

Ready for adaptive study integration when: - ✅ Arm A (proxy fidelity) is run and optimal k is selected. - ✅ Arm B (judge scoring) produces judge_scores.csv at sufficient scale (≥50 per bin, stratified across scenarios). - ✅ Arm C (inter-rater agreement) verifies judges converge (α ≥ 0.5). - ⏳ Arm D joins and validates the CSV schema (this document).

Remaining before declaring final schema frozen: - Confirm adaptive study code can parse the joined CSV without errors (test load from a real arm B/D run). - Verify downstream pre-reg joiners (adaptive arms 1–3) match column expectations (search for hard-coded column names or assumptions in adaptive runner code). - Document any adaptive-study-specific post-processing (e.g., per-bin aggregation of judge scores for toxicity metrics).

References

  • Code: swarm/calibration/joined.py (BASE_COLUMNS, JoinedRow, to_row)
  • Runner: experiments/calibration_join.py (writes joined.csv)
  • Tests: tests/test_calibration_joined.py (schema assertions, CSV round-trip)
  • Pre-registration: docs/research/calibration-prereg.md#arm-d-freeze-joined-csv-schema
  • Arm A (fidelity): experiments/calibration_fidelity.py, swarm/calibration/fidelity.py
  • Arm B (judges): experiments/calibration_judge.py, swarm/judges/judge.py
  • Arm C (agreement): experiments/calibration_agreement.py, swarm/judges/__init__.py (run_agreement)
  • Adaptive pre-reg: docs/research/adaptive-agents-prereg.md (describes join strategy and confounds)