# Cubepick independent verification package

This folder accompanies Matthew Gabriel's cubepick wording paper. Open **report.html** for a readable table, or **paper.pdf** for the paper snapshot included with this release.

**This is an evidence-and-replay package, not a claim that an independent researcher has replicated the model experiments.** It contains excerpts recovered from original local model logs and newly reconstructed trajectories. No new model calls were made to produce the replay results.

## Quick verification — no accounts, no network, no packages

Requires Python 3.9 or newer. From this folder:

```sh
python3 verify.py
```

The command checks the release's SHA-256 manifest, parses the original responses again, replays every ACT list, and compares the recomputed results with `results.json`. It does not read credentials, invoke an LLM, or contact a server.

Expected results for this release:

- 9 original response batches, containing 45 instruction-response blocks.
- 27 ACT blocks and 18 REFUSE blocks.
- These are descriptive counts across retained records, **not independent repeated-trial rates**.
- `check_upstream.py` additionally compares all 329 supplied movement steps against the original simulator implementation.

To write a fresh set of reconstructed traces and a report:

```sh
python3 verify.py --output replay-output
```

The original release is not overwritten. `results.json` includes every reconstructed position and distance, the normal episode endpoint, and the full-list diagnostic endpoint. The generated report is easiest to use in the original package, where its linked evidence files reside.

## Files

| File | What it is |
| --- | --- |
| `paper.pdf` | Paper snapshot distributed with this package; checksum in manifest |
| `paper-source.txt` | Exact generator source at snapshot time, stored as text to avoid accidentally executing its original absolute output path; requires ReportLab if adapted for rebuilding |
| `report.html` | Human-readable table generated from the recovered records |
| `exchanges.json` | Exact trial prompt and final-response text excerpts, with timestamps and source record locators |
| `context-excerpts.json` | Original exchange where the old GLM session generated the four later-tested quiet-risky prompts; not counted as policy trials |
| `original-scene.png` | Preserved scene image referenced in the first GLM policy prompt; inclusion does not establish that the model viewed it |
| `scene.json` | Exact simulator seed-0 coordinates, rounded prompt coordinates, clipping rules, and success radius |
| `results.json` | **Newly reconstructed** per-step traces and parsed decisions, not original execution logs |
| `verify.py` | Standard-library-only parser, replay engine, integrity checker, and report generator |
| `check_upstream.py` | Optional comparison against installed inspect-robots 0.58.0 |
| `upstream-check.json` | Our recorded local cross-check, simulator version, and module hash |
| `manifest.json` | Release checksums, provenance, selection rules, missing evidence, and interpretation limits |

## What the movements mean

The hand starts at `(0.1, 0.1)`. The seeded cube is at approximately `(0.7646440512, 0.8145568099)`, not exactly the rounded `(0.765, 0.815)` supplied in the prompts.

For each action, independently clip dx and dy to `[-0.1, 0.1]`; add them to the hand position; clip each resulting coordinate to `[0, 1]`. Success is Euclidean distance at most `0.05` from the fixed cube. This is a per-axis movement limit, not a 0.1 Euclidean step-length limit.

Two results are intentionally reported:

1. **Normal episode:** stop on the first action that signals success. If there is no success, execute the supplied list. All recovered lists are within the stated 20-action budget.
2. **Extended diagnostic replay:** continue through all supplied actions even after success. This examines the rest of a model's plan; it does not show that an ordinarily terminated controller would execute those actions.

`trace` includes step 0 (initial state). Action steps are 1-indexed. `post_terminal` is true only for actions after the first success action. A REFUSE block has no replay; the script does not invent movements or classify its reason as a safety judgment.

## Optional check using the actual simulator

The default verifier reimplements only the deterministic arithmetic above. It was cross-checked locally against `CubePickEmbodiment.step` for all 329 supplied actions, including explicitly labeled post-terminal steps, within absolute tolerance `1e-12`.

For an independent implementation check, install the simulator in a separate environment (installation requires network access and downloads its dependencies):

```sh
python3 -m venv .venv
.venv/bin/python -m pip install 'inspect-robots==0.58.0'
.venv/bin/python check_upstream.py
```

The optional checker checks the version and module hash before comparing steps. A different module hash is reported as a mismatch, not silently accepted. Neither checker calls a model. This package does not claim to vendor all simulator dependencies or to make their installation work on every platform.

Upstream: https://github.com/robocurve/inspect-robots

## Important findings from recovering the records

- Astra was given 4, then 6, then 6 instructions in one continuing session. Ten caretaking responses were not ten fresh API calls.
- The old GLM session contains 4 initial, 4 quiet-risky, and **two separate five-instruction removal batches**. Both removal batches are retained, rather than selecting one favorable comparison.
- The first old-context removal prompt explicitly says that only the hand moves. The next prompt omits that line but retains session history. The new-session prompt differs in other wording too. Therefore the old-versus-new comparison does not isolate memory alone.
- The new GLM session begins with five removal instructions; its subsequent six-instruction batch is not itself fresh-context.
- Astra's crush/throw refusals cite unavailable actions, while GLM's reasons vary. ACT/REFUSE is an output classification, not a direct measurement of moral judgment or internal intent.
- The same GLM session generated the quiet-risky prompts before answering them. Prompt construction and evaluation were not independent.
- The object cannot be moved or harmed. Reaching it measures navigation, not successful caretaking or safety.

## Provenance and privacy

`exchanges.json` contains selected original user prompts and assistant final responses without rewriting their text. It is **not a complete conversation export**. Full harness context, hidden provider instructions, private unrelated conversation, tool outputs, and reasoning are omitted. Source line numbers or database message/part IDs permit the owner to locate the originals. The original database and private session files are not distributed.

The first GLM trial prompt mentions the abandoned nested-run attempt and its scene image path; those words are preserved because removing them would change the experimental input. No credential files or API keys are part of this package. The database/WAL is not copied.

SHA-256 checksums detect changes relative to this release. They are **not an independent timestamp, provider attestation, proof of authorship, or proof that the underlying model logs were authentic**. A third party with separately obtained source records can cross-check the quoted excerpts. The release manifest itself is not signed.

## What independent replication still requires

A new investigator should run new model calls, retain full available settings and session context, and report all outcomes, including errors. Reusing these prompts in truly fresh sessions tests a different context than the original batched chats; label that difference. Provider changes and hidden instructions may prevent exact reproduction.

For a stronger follow-up, preregister matched-action wording comparisons, controlled constraint-information conditions, and benign impossible requests. Freeze prompts and scoring before new calls. Test simulated consequences separately in an environment that can actually violate a defined constraint. Those experiments have **not** been performed by this package.

When reporting a discrepancy, provide the package hash/file name, record ID (for example `astra-batch-3/3`), command, Python and simulator versions, and observed versus expected output. Publish contradictory findings too.