# Recreating Eichler's Codex pull-request review system This guide explains the Codex autoreview system used in the Eichler repository as of August 14, 2026. It is meant for a friend who wants to build the same kind of system in another GitHub repository. The system is more than a prompt attached to a GitHub Action. It treats model output as untrusted input, validates every claim against the Git diff, limits automatic edits to changed lines, and publishes one stable report on the pull request. If you want a useful first version quickly, build the read-only review in Phase 1 below. Add automatic remediation only after the review path has good tests and a reliable rate of false positives. ## What it does For each eligible pull request, the system: 1. Builds a deterministic manifest of the exact base-to-head change. 2. Routes the PR to a content-only shortcut, full Codex review, or manual review. 3. Runs separate correctness and code-simplification audits. 4. Requires both audits to return JSON that matches committed schemas. 5. Validates model findings against the manifest and real changed-line ranges. 6. Selects a narrow subset of findings that may be fixed automatically. 7. Asks a separate, read-only Codex pass for exact line replacements. 8. Validates those replacements without executing code from the PR. 9. Pushes one bounded commit with an exact-SHA lease if the branch has not moved. 10. Reviews the new commit again after an automatic change. 11. Updates one PR comment and one `codex/autoreview` commit status. 12. Stores the full machine evidence as GitHub Actions artifacts for 14 days. It never auto-merges. Its status describes the static review result. It does not claim that tests or builds passed. ## The pipeline at a glance ```text pull_request_target | v deterministic manifest + preflight | +--> content-only ------> short terminal report | +--> unsafe shape ------> manual-review report | +--> full review | v correctness Codex pass | v simplification Codex pass | v trusted validation + selection | eligible fixes? / \ no yes | | | remediation proposal | | | trusted evaluation | | | one retry for rejects | | | validated CAS push | | | post-change audits \ / | v one terminal report + status ``` The current Eichler workflow can make two to six model calls: - Two calls for a normal PR with no eligible automatic fix. - One additional call for the first remediation proposal. - One bounded retry call if some proposed fixes fail validation. - Two post-change calls if the system pushes a candidate commit. ## Files in the Eichler implementation The implementation is split into policy, schemas, trusted code, workflow wiring, and tests. ```text .github/ codex/ README.md prompts/ correctness.md complexity.md remediation.md schemas/ correctness.schema.json complexity.schema.json remediation.schema.json workflows/ codex-pr-audit.yml tools/ codex-pr-report/ manifest.mjs preflight.mjs report.mjs remediation.mjs manifest.test.mjs preflight.test.mjs prompt.test.mjs report.test.mjs remediation.test.mjs schema.test.mjs workflow.test.mjs ``` Each part has one job: - `manifest.mjs` produces the authoritative description of the change. - `preflight.mjs` chooses `content_only`, `full_review`, or `manual_review` without a model call. - The prompt files define the review contracts and trust boundary. - The JSON Schemas bound the model outputs. - `report.mjs` validates, normalizes, selects, and renders results. - `remediation.mjs` prepares signed edit requests, validates proposals, constructs a candidate Git tree, and records the outcome. - The workflow grants the minimum token permissions needed by each job. - The tests pin the security and reporting behavior. The implementation is about 15,000 lines including tests. Copying only the workflow will not reproduce its safety properties. ## Fastest way to reproduce the exact implementation If you can receive the Eichler source files, copy these three paths as one unit: ```text .github/codex/ .github/workflows/codex-pr-audit.yml tools/codex-pr-report/ ``` Also copy these two integrations: ```json { "scripts": { "test:codex-pr-report": "node --test tools/codex-pr-report/*.test.mjs" } } ``` ```yaml # In the normal CI workflow - run: pnpm test:codex-pr-report ``` Run the tests before changing policy. Then replace each Eichler-specific assumption: - Rename the comment marker and bot commit identity. - Replace or remove the meeting-record content-only route. - Change component classifications and sensitive-path signals in `manifest.mjs`. - Review every allowed and denied remediation path in `remediation.mjs`. - Replace `oxfmt` setup if the target repository uses another formatter. - Update the sparse checkout if trusted instructions live somewhere other than root `AGENTS.md`. - Select and calibrate the model, effort, Codex CLI version, and API budget. - Rename the secret if desired, then update every reference consistently. - Run the entire deterministic suite after each policy change. Do not transplant `codex-pr-audit.yml` by itself. It depends on command names, output files, receipts, schemas, and validation rules implemented by the other copied files. ## Prerequisites You need: - A GitHub repository that uses GitHub Actions. - An OpenAI API project dedicated to PR review. - A project API key stored as the GitHub Actions secret `CODEX_AUDIT_OPENAI_API_KEY`. - A Linux GitHub-hosted runner. - Node.js for the deterministic scripts. - A formatter that can check candidate source from trusted configuration. Eichler uses a pinned `oxfmt` version. The API project should have its own budget and usage limits. Do not reuse a production application key, a personal all-purpose key, or an organization billing-admin key. The official Codex GitHub Action documentation is here: . ## The central security rule The workflow uses `pull_request_target` because the OpenAI key must not be exposed to workflow code supplied by the pull request. That event is safe only with a strict checkout layout: ```text workspace root trusted base revision _audit/pr untrusted pull-request head ``` The workflow, prompts, schemas, root instructions, and validator always come from the immutable base SHA. The PR head is checked out into `_audit/pr` and treated as evidence. No step executes scripts, hooks, package-manager commands, tests, builds, migrations, or binaries from `_audit/pr`. Every prompt repeats the same boundary. It tells Codex that all content under `_audit/pr` is untrusted, including source comments, Markdown, `AGENTS.md`, workflow files, and text that claims to contain instructions. The keyed Codex jobs also use: ```yaml permissions: contents: read # inputs to openai/codex-action permission-profile: ":read-only" safety-strategy: drop-sudo codex-args: '["--ephemeral","--json"]' ``` The Codex action is the final step in each keyed job. The GitHub token is read-only in those jobs. The branch-writing job has no OpenAI key and runs no model. Eichler also limits the workflow to: - non-draft PRs; - PRs that target the default branch; - same-repository PR branches; - the exact base and head SHAs from the event. The workflow re-reads the live PR before publishing or pushing. It drops stale results if the PR moved, closed, changed branches, or returned to draft. ## Step 1: Build an authoritative change manifest Do not let the model decide what changed. Generate a manifest with trusted code before a model runs. The Eichler manifest records: - schema version; - base, merge-base, and head SHAs; - file status and mode; - current and previous paths; - text, binary, symlink, gitlink, and executable shape; - additions, deletions, and total changed lines; - exact base-side and head-side hunk ranges; - component and file classification; - risk signals; - a full or targeted coverage plan. The default coverage limits are 80 files and 8,000 changed lines. Larger diffs get a deterministic targeted plan. The prompts must not claim full coverage when the manifest requests targeted coverage. Use Git commands that disable user-controlled diff helpers: ```sh git -C _audit/pr diff --no-ext-diff --no-textconv ``` Do not enable copy detection. Treat a copied file as newly added maintenance surface. Record rename-only and mode-only changes explicitly because they may have no honest line anchor. The manifest is the source of truth for every later SHA, path, hunk, and coverage claim. ## Step 2: Add deterministic preflight routing Run cheap local checks before spending API tokens. Eichler has three decisions: - `content_only`: a narrow class of new meeting records. It receives a short report and no model call. - `full_review`: ordinary code, configuration, docs, policy, and mixed changes. - `manual_review`: evidence that cannot safely enter the automated path, such as symlinks, gitlinks, unsupported modes, malformed manifests, deletions, renames, copies, or file-type mismatches. The content-only rule is specific to Eichler. Replace it with a rule that fits your repository, or remove the shortcut entirely. Default unknown inputs to full or manual review. Do not default them to skipped review. Preflight output should contain the decision, exact SHAs, stable reason code, and a short plain-text summary. Validate its size and shape before publishing it. ## Step 3: Separate correctness from simplification Eichler uses independent sequential passes so each prompt has one purpose. The correctness pass looks for: - functional defects; - security and privacy failures; - authorization problems; - data loss or persistence mistakes; - broken workflow states and contracts; - concurrency and idempotency failures; - accessibility failures; - missing verification for risky behavior. The simplification pass looks for: - duplication; - unnecessary abstractions or dependencies; - dead code; - premature generalization; - verbose implementations with a materially smaller equivalent; - maintained generated noise; - unrelated scope that should be split. The simplification prompt explicitly protects tests, validation, useful types, domain states, authorization, audit trails, compliance controls, error handling, idempotency, accessibility, and explanatory code. A smaller diff is not automatically a better diff. Run the two passes sequentially if your API project has a low request-per-minute limit. ## Step 4: Require structured output Pass a committed JSON Schema to each Codex call with `output-schema-file`. Do not parse a free-form Markdown review and hope that headings stay stable. The correctness schema includes: - a change summary; - zero to fifteen findings; - zero to five human review targets; - inspected and omitted paths; - uncertainties. Each finding includes severity, confidence, category, whether the PR introduced it, the failure mode, evidence, impact, recommendation, verification, and one to four locations. The simplification schema includes: - zero to twelve opportunities; - complexity that should remain; - scope-split candidates; - coverage and uncertainties. Each opportunity includes confidence, benefit, risk, behavior to preserve, verification, locations, estimated net line reduction, independence, and overlap group. Require `additionalProperties: false`, item limits, strict enums, SHA patterns, and bounded location counts throughout the schemas. Example action step: ```yaml - name: Run correctness audit id: run-codex uses: openai/codex-action@ # pin the reviewed v1 commit with: openai-api-key: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }} prompt-file: .github/codex/prompts/correctness.md output-schema-file: .github/codex/schemas/correctness.schema.json codex-args: '["--ephemeral","--json"]' codex-version: model: effort: permission-profile: ":read-only" safety-strategy: drop-sudo ``` Pin both the action commit and Codex CLI version. Choose the model and reasoning effort from your own calibration and budget rather than copying Eichler's current choice. ## Step 5: Validate every model claim JSON Schema validation is necessary but insufficient. Trusted code should reject a result when: - its base or head SHA differs from the manifest; - it cites a path outside the manifest; - its line range is invalid for the cited revision; - it claims a PR-introduced problem without an anchor inside a real changed hunk; - inspected and omitted paths do not partition the changed paths; - it claims full coverage after a targeted or partial review; - it fabricates line anchors for binary files or gitlinks; - it exceeds report, finding, location, or string limits. This rule prevents the common failure where a reviewer finds real but unrelated debt in a file touched by the PR. A reportable PR finding needs at least one location fully contained inside an authoritative changed-line range. Use base-side ranges for pure deletions. Use head-side ranges for additions and edits. Route mode-only and rename-only concerns to human review when no content hunk can support a finding. Assign IDs only after validation, for example `INITIAL/C-001` and `INITIAL/S-001`. If you run a post-change audit, use `CURRENT/` IDs. Do not correlate ordinal IDs across passes. ## Step 6: Render one stable PR report The workflow maintains one comment identified by a hidden marker: ```html ``` On each run, update that comment instead of adding another. Publish the comment only after the lifecycle reaches a terminal result. Eichler's report contains six sections: 1. TL;DR. 2. Open problems. 3. Automatic changes. 4. Human decisions. 5. Coding-agent follow-up tasks. 6. Collapsed coverage and limitations. The renderer maps validated evidence to one posture: - `low_apparent_risk`: success; no issues found by the static review. - `focused_review`: success; human attention requested. - `do_not_merge`: failure; a confirmed current problem remains. - `audit_incomplete`: error; the audit did not complete safely. Publish the same result as a commit status with context `codex/autoreview`. State in the description that tests and builds did not run. The report is advisory and does not approve the PR. Before publishing, verify that: - the report starts with the trusted marker; - the evidence SHAs match the live PR; - the report stays inside GitHub's comment size limit; - the posture agrees with the validation-complete flag; - the PR is still open and points to the audited head. Keep the normalized JSON and complete locations in an artifact even if the public comment uses a compact representation. ## Step 7: Add automatic remediation only after review is stable Automatic remediation is a second system layered on top of review. Keep the first version read-only if you do not have time to implement these checks. Eichler selects only: - high-confidence correctness findings introduced by the PR; and - high-confidence, low-risk, independent simplifications with no overlap. Trusted code turns each eligible source into a remediation request. The request contains: - the exact base and head SHAs; - a stable source ID; - allowed file paths; - exact writable head ranges; - exact insertion points paired to cited base-side deletions; - edit and line limits; - a SHA-256 receipt over the canonical request bytes. The remediation model does not return a patch or shell commands. It returns exact edit operations: ```json { "path": "src/example.ts", "start_line": 42, "delete_line_count": 2, "expected_lines": ["old line one", "old line two"], "replacement_lines": ["new line"] } ``` The validator checks each source independently. One malformed fix does not discard an unrelated valid fix. A multi-file fix remains atomic. Reject an edit if: - the receipt, source ID, or SHA is wrong; - the path is not authorized for that source; - the original lines differ by one byte; - the edit is outside the source's cited changed hunk; - edits overlap; - the change touches a denied file type or directory; - the result exceeds the edit budget; - a simplification misses its promised net-line range; - the candidate is not already formatter-clean; - the candidate introduces a missing symbol or import; - the edit requires a product, legal, or external decision. Eichler permits one fresh proposal for rejected sources only. The retry request contains only those sources. Accepted first-pass edits are locked and are never reconsidered. ## Step 8: Keep model access separate from write access The job that asks Codex for edits has: - the OpenAI key; - read-only repository access; - no write token; - no ability to push. The later job that constructs and pushes a commit has: - no OpenAI key; - no model call; - trusted validator code from the base revision; - write access only for the bounded final operation. In Eichler v1, automatic writes are limited to formatter-clean JavaScript and TypeScript. The validator denies workflows, migrations, generated output, snapshots, SQL, lockfiles, binaries, symlinks, gitlinks, file creation, deletion, rename, and mode changes. Construct the candidate with Git plumbing or another method that does not invoke hooks, filters, or code from the PR. Run `git diff --check`. Feed candidate bytes to the trusted pinned formatter in check mode. Do not let the formatter discover configuration under the PR checkout. Immediately before the push, fetch the live PR again. Push with an exact lease: ```sh git -C _apply/pr push --porcelain \ --force-with-lease="refs/heads/$HEAD_REF:$EXPECTED_HEAD_SHA" \ origin "$FIXED_SHA:refs/heads/$HEAD_REF" ``` The lease turns the write into a compare-and-swap operation. If a person or bot updated the branch, the push fails instead of overwriting the new work. Use a clear bot author such as: ```text Your Project Codex Remediation ``` ## Step 9: Review the automatic commit again A push made with `GITHUB_TOKEN` does not start another copy of the same workflow. This is useful because it prevents remediation loops, but it also means normal CI may not run for the bot commit. After a successful push, mark `codex/autoreview` pending on the new SHA and run fresh correctness and simplification audits inside the same workflow. Render the terminal report from the post-change evidence. Tell the reviewer that normal tests and builds still need to run on the final head. Never describe the automatic edit as a verified fix unless a separate trusted process actually ran the required verification. ## Step 10: Preserve evidence and fail closed Upload immutable artifacts named with the PR number and audited SHA. Eichler stores: - the manifest; - raw and normalized audit JSON; - validation results; - remediation requests and SHA-256 receipts; - first-pass and retry evaluations; - the final patch and candidate state; - the terminal Markdown report. Use a short retention period that fits your audit needs; Eichler uses 14 days. Add explicit terminal paths for: - preflight failure; - incomplete model output; - invalid or stale evidence; - proposal-generation failure; - proposal-validation failure; - push failure; - post-change audit failure; - publication failure after a successful push. If a candidate was pushed and later reporting fails, mark that exact candidate as unverified and incomplete. Do not reuse prose produced for the previous head. ## Suggested build order ### Phase 1: useful read-only review 1. Add the trusted base/head checkout layout. 2. Generate an exact change manifest. 3. Write one correctness prompt and schema. 4. Run `openai/codex-action` read-only. 5. Validate SHA, path, location, hunk, and coverage claims. 6. Update one marked PR comment. 7. Publish `codex/autoreview` on the audited SHA. 8. Add tests for stale results, prompt injection, invalid anchors, and moved branches. This phase provides most of the review value with a much smaller attack surface. ### Phase 2: review quality 1. Split correctness and simplification into separate passes. 2. Add full versus targeted coverage. 3. Add review targets, protected complexity, and uncertainties. 4. Store normalized evidence as artifacts. 5. Calibrate findings against a set of known good and bad PRs. ### Phase 3: bounded remediation 1. Select only high-confidence eligible sources. 2. Sign canonical remediation requests. 3. Request exact line operations from a read-only model job. 4. Validate each source independently. 5. Add one retry for rejected sources. 6. Build a candidate without hooks or PR-controlled code. 7. Push with an exact-SHA lease from a keyless writer job. 8. Re-audit the pushed candidate. Do not start Phase 3 until the first two phases have strong deterministic tests. ## Tests worth copying as behavior Eichler tests the following contracts: - action and third-party versions stay pinned; - keyed jobs remain read-only and model steps remain last; - the writer job has no OpenAI key; - trusted policy comes from the base revision; - the PR head remains under the evidence directory; - fork PRs, drafts, and non-default targets do not enter keyed jobs; - manifests handle additions, deletions, copies, renames, binary files, modes, symlinks, gitlinks, merge bases, and hunk ranges; - preflight rejects malformed and structurally unsafe evidence; - schema limits and prompt contracts stay aligned; - model findings require real changed-line anchors; - stale SHAs and invalid coverage claims fail validation; - remediation cannot escape per-source paths and hunks; - edits are atomic per source and non-overlapping across sources; - retry inputs contain only rejected sources; - candidates are formatter-clean and pass `git diff --check`; - branch writes use an exact lease; - the report remains bounded and keeps all current issues; - one marked comment is updated instead of multiplied; - incomplete or failed runs publish a fail-closed status. Run the Eichler deterministic suite with: ```sh node --test tools/codex-pr-report/*.test.mjs ``` ## Repository settings and operating notes Configure these items in the target repository: 1. Add the `CODEX_AUDIT_OPENAI_API_KEY` Actions secret. 2. Give the workflow only job-level permissions. Start the workflow with `permissions: {}`. 3. If you enable remediation, allow the writer job to update same-repository PR branches. 4. Add `codex/autoreview` to branch protection only after failure behavior has been tested. 5. Set concurrency to one active run per PR and cancel stale runs. 6. Keep prompts, schemas, validators, and workflow changes under normal code review. 7. Review OpenAI API usage and GitHub Action logs after rollout. The current Eichler model and effort settings are cost choices, not part of the design. Measure false positives, missed seeded defects, token use, latency, and cost before choosing your own settings. ## Common shortcuts that break the design - Checking out the PR head over the workspace root in a `pull_request_target` job. - Running `npm install`, tests, builds, or repository scripts from the untrusted PR. - Giving a model job both the API key and a write-capable GitHub token. - Asking Codex to post comments or push directly. - Accepting free-form prose as machine evidence. - Trusting a cited file and line without checking it against a real diff hunk. - Publishing a result after the live PR head moved. - Letting remediation touch an entire changed file instead of source-specific hunks. - Retrying the whole patch and allowing accepted edits to change again. - Calling a static edit verified when tests did not run. - Using a normal force push instead of an exact-SHA lease. - Auto-merging based on the model's report. ## Minimum viable workflow skeleton This skeleton shows the trust boundary. It omits the validators and publisher that you must add before relying on the result. ```yaml name: Codex PR review on: pull_request_target: types: [opened, synchronize, reopened, ready_for_review] concurrency: group: codex-pr-review-${{ github.event.pull_request.number }} cancel-in-progress: true permissions: {} jobs: review: if: >- github.event.pull_request.draft == false && github.event.pull_request.base.ref == github.event.repository.default_branch && github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-latest permissions: contents: read steps: - name: Check out trusted policy uses: actions/checkout@ with: ref: ${{ github.event.pull_request.base.sha }} fetch-depth: 1 persist-credentials: false sparse-checkout: | AGENTS.md .github/codex tools/codex-pr-report sparse-checkout-cone-mode: false - name: Check out PR as untrusted evidence uses: actions/checkout@ with: ref: ${{ github.event.pull_request.head.sha }} path: _audit/pr fetch-depth: 0 persist-credentials: false submodules: false - name: Build trusted manifest env: BASE_SHA: ${{ github.event.pull_request.base.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} run: >- node tools/codex-pr-report/manifest.mjs --repo _audit/pr --base "$BASE_SHA" --head "$HEAD_SHA" --output _audit/change-manifest.json - name: Require dedicated review key env: AUDIT_KEY: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }} run: test -n "$AUDIT_KEY" - name: Run structured read-only review id: codex uses: openai/codex-action@ with: openai-api-key: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }} prompt-file: .github/codex/prompts/correctness.md output-schema-file: .github/codex/schemas/correctness.schema.json codex-args: '["--ephemeral","--json"]' codex-version: model: effort: permission-profile: ":read-only" safety-strategy: drop-sudo ``` Split validation and publication into later jobs. Pass data through immutable artifacts or bounded job outputs. The publishing job should receive neither the OpenAI key nor a checkout of executable PR code. ## Final checklist Before enabling the workflow on real pull requests, confirm all of these statements: - Trusted policy always comes from the event's base SHA. - The PR head is evidence only and lives in a separate directory. - No keyed job executes PR-controlled code. - Every model response is schema-bound and independently validated. - Findings need a real changed-line anchor. - Coverage claims match the manifest. - The publisher rejects stale PR identity. - The report says when tests and builds did not run. - Model jobs cannot write to the repository. - A writer job, if present, has no OpenAI key. - Automatic edits stay within source-specific paths and hunks. - The push uses an exact-SHA compare-and-swap lease. - A pushed candidate receives a fresh audit. - Every terminal failure produces an honest status. - The workflow never auto-merges. That separation of duties is the core of the system: Codex analyzes and proposes; trusted code decides what is valid; a separate bounded job performs the write; humans retain the merge decision.