Download the Markdown

Recreating Eichler’s Codex pull-request review system

Recreating Eichler's Codex pull-request review system

This guide explains the Codex autoreview system used in the Eichler repository as of August 14, 2026. It is meant for a friend who wants to build the same kind of system in another GitHub repository.

The system is more than a prompt attached to a GitHub Action. It treats model output as untrusted input, validates every claim against the Git diff, limits automatic edits to changed lines, and publishes one stable report on the pull request.

If you want a useful first version quickly, build the read-only review in Phase 1 below. Add automatic remediation only after the review path has good tests and a reliable rate of false positives.

What it does

For each eligible pull request, the system:

  1. Builds a deterministic manifest of the exact base-to-head change.
  2. Routes the PR to a content-only shortcut, full Codex review, or manual review.
  3. Runs separate correctness and code-simplification audits.
  4. Requires both audits to return JSON that matches committed schemas.
  5. Validates model findings against the manifest and real changed-line ranges.
  6. Selects a narrow subset of findings that may be fixed automatically.
  7. Asks a separate, read-only Codex pass for exact line replacements.
  8. Validates those replacements without executing code from the PR.
  9. Pushes one bounded commit with an exact-SHA lease if the branch has not moved.
  10. Reviews the new commit again after an automatic change.
  11. Updates one PR comment and one codex/autoreview commit status.
  12. Stores the full machine evidence as GitHub Actions artifacts for 14 days.

It never auto-merges. Its status describes the static review result. It does not claim that tests or builds passed.

The pipeline at a glance

pull_request_target
        |
        v
deterministic manifest + preflight
        |
        +--> content-only ------> short terminal report
        |
        +--> unsafe shape ------> manual-review report
        |
        +--> full review
                 |
                 v
       correctness Codex pass
                 |
                 v
       simplification Codex pass
                 |
                 v
      trusted validation + selection
                 |
           eligible fixes?
             /       \
           no         yes
           |           |
           |    remediation proposal
           |           |
           |    trusted evaluation
           |           |
           |    one retry for rejects
           |           |
           |    validated CAS push
           |           |
           |    post-change audits
             \       /
                 |
                 v
       one terminal report + status

The current Eichler workflow can make two to six model calls:

Files in the Eichler implementation

The implementation is split into policy, schemas, trusted code, workflow wiring, and tests.

.github/
  codex/
    README.md
    prompts/
      correctness.md
      complexity.md
      remediation.md
    schemas/
      correctness.schema.json
      complexity.schema.json
      remediation.schema.json
  workflows/
    codex-pr-audit.yml

tools/
  codex-pr-report/
    manifest.mjs
    preflight.mjs
    report.mjs
    remediation.mjs
    manifest.test.mjs
    preflight.test.mjs
    prompt.test.mjs
    report.test.mjs
    remediation.test.mjs
    schema.test.mjs
    workflow.test.mjs

Each part has one job:

The implementation is about 15,000 lines including tests. Copying only the workflow will not reproduce its safety properties.

Fastest way to reproduce the exact implementation

If you can receive the Eichler source files, copy these three paths as one unit:

.github/codex/
.github/workflows/codex-pr-audit.yml
tools/codex-pr-report/

Also copy these two integrations:

{
  "scripts": {
    "test:codex-pr-report": "node --test tools/codex-pr-report/*.test.mjs"
  }
}
# In the normal CI workflow
- run: pnpm test:codex-pr-report

Run the tests before changing policy. Then replace each Eichler-specific assumption:

Do not transplant codex-pr-audit.yml by itself. It depends on command names, output files, receipts, schemas, and validation rules implemented by the other copied files.

Prerequisites

You need:

The API project should have its own budget and usage limits. Do not reuse a production application key, a personal all-purpose key, or an organization billing-admin key.

The official Codex GitHub Action documentation is here: https://learn.chatgpt.com/docs/github-action.

The central security rule

The workflow uses pull_request_target because the OpenAI key must not be exposed to workflow code supplied by the pull request.

That event is safe only with a strict checkout layout:

workspace root       trusted base revision
_audit/pr             untrusted pull-request head

The workflow, prompts, schemas, root instructions, and validator always come from the immutable base SHA. The PR head is checked out into _audit/pr and treated as evidence. No step executes scripts, hooks, package-manager commands, tests, builds, migrations, or binaries from _audit/pr.

Every prompt repeats the same boundary. It tells Codex that all content under _audit/pr is untrusted, including source comments, Markdown, AGENTS.md, workflow files, and text that claims to contain instructions.

The keyed Codex jobs also use:

permissions:
  contents: read

# inputs to openai/codex-action
permission-profile: ":read-only"
safety-strategy: drop-sudo
codex-args: '["--ephemeral","--json"]'

The Codex action is the final step in each keyed job. The GitHub token is read-only in those jobs. The branch-writing job has no OpenAI key and runs no model.

Eichler also limits the workflow to:

The workflow re-reads the live PR before publishing or pushing. It drops stale results if the PR moved, closed, changed branches, or returned to draft.

Step 1: Build an authoritative change manifest

Do not let the model decide what changed. Generate a manifest with trusted code before a model runs.

The Eichler manifest records:

The default coverage limits are 80 files and 8,000 changed lines. Larger diffs get a deterministic targeted plan. The prompts must not claim full coverage when the manifest requests targeted coverage.

Use Git commands that disable user-controlled diff helpers:

git -C _audit/pr diff --no-ext-diff --no-textconv <merge-base> <head>

Do not enable copy detection. Treat a copied file as newly added maintenance surface. Record rename-only and mode-only changes explicitly because they may have no honest line anchor.

The manifest is the source of truth for every later SHA, path, hunk, and coverage claim.

Step 2: Add deterministic preflight routing

Run cheap local checks before spending API tokens.

Eichler has three decisions:

The content-only rule is specific to Eichler. Replace it with a rule that fits your repository, or remove the shortcut entirely. Default unknown inputs to full or manual review. Do not default them to skipped review.

Preflight output should contain the decision, exact SHAs, stable reason code, and a short plain-text summary. Validate its size and shape before publishing it.

Step 3: Separate correctness from simplification

Eichler uses independent sequential passes so each prompt has one purpose.

The correctness pass looks for:

The simplification pass looks for:

The simplification prompt explicitly protects tests, validation, useful types, domain states, authorization, audit trails, compliance controls, error handling, idempotency, accessibility, and explanatory code. A smaller diff is not automatically a better diff.

Run the two passes sequentially if your API project has a low request-per-minute limit.

Step 4: Require structured output

Pass a committed JSON Schema to each Codex call with output-schema-file. Do not parse a free-form Markdown review and hope that headings stay stable.

The correctness schema includes:

Each finding includes severity, confidence, category, whether the PR introduced it, the failure mode, evidence, impact, recommendation, verification, and one to four locations.

The simplification schema includes:

Each opportunity includes confidence, benefit, risk, behavior to preserve, verification, locations, estimated net line reduction, independence, and overlap group.

Require additionalProperties: false, item limits, strict enums, SHA patterns, and bounded location counts throughout the schemas.

Example action step:

- name: Run correctness audit
  id: run-codex
  uses: openai/codex-action@<pinned-commit> # pin the reviewed v1 commit
  with:
    openai-api-key: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }}
    prompt-file: .github/codex/prompts/correctness.md
    output-schema-file: .github/codex/schemas/correctness.schema.json
    codex-args: '["--ephemeral","--json"]'
    codex-version: <tested-version>
    model: <model-you-have-tested-and-budgeted>
    effort: <tested-effort>
    permission-profile: ":read-only"
    safety-strategy: drop-sudo

Pin both the action commit and Codex CLI version. Choose the model and reasoning effort from your own calibration and budget rather than copying Eichler's current choice.

Step 5: Validate every model claim

JSON Schema validation is necessary but insufficient. Trusted code should reject a result when:

This rule prevents the common failure where a reviewer finds real but unrelated debt in a file touched by the PR. A reportable PR finding needs at least one location fully contained inside an authoritative changed-line range.

Use base-side ranges for pure deletions. Use head-side ranges for additions and edits. Route mode-only and rename-only concerns to human review when no content hunk can support a finding.

Assign IDs only after validation, for example INITIAL/C-001 and INITIAL/S-001. If you run a post-change audit, use CURRENT/ IDs. Do not correlate ordinal IDs across passes.

Step 6: Render one stable PR report

The workflow maintains one comment identified by a hidden marker:

<!-- your-project-codex-pr-audit -->

On each run, update that comment instead of adding another. Publish the comment only after the lifecycle reaches a terminal result.

Eichler's report contains six sections:

  1. TL;DR.
  2. Open problems.
  3. Automatic changes.
  4. Human decisions.
  5. Coding-agent follow-up tasks.
  6. Collapsed coverage and limitations.

The renderer maps validated evidence to one posture:

Publish the same result as a commit status with context codex/autoreview. State in the description that tests and builds did not run. The report is advisory and does not approve the PR.

Before publishing, verify that:

Keep the normalized JSON and complete locations in an artifact even if the public comment uses a compact representation.

Step 7: Add automatic remediation only after review is stable

Automatic remediation is a second system layered on top of review. Keep the first version read-only if you do not have time to implement these checks.

Eichler selects only:

Trusted code turns each eligible source into a remediation request. The request contains:

The remediation model does not return a patch or shell commands. It returns exact edit operations:

{
  "path": "src/example.ts",
  "start_line": 42,
  "delete_line_count": 2,
  "expected_lines": ["old line one", "old line two"],
  "replacement_lines": ["new line"]
}

The validator checks each source independently. One malformed fix does not discard an unrelated valid fix. A multi-file fix remains atomic.

Reject an edit if:

Eichler permits one fresh proposal for rejected sources only. The retry request contains only those sources. Accepted first-pass edits are locked and are never reconsidered.

Step 8: Keep model access separate from write access

The job that asks Codex for edits has:

The later job that constructs and pushes a commit has:

In Eichler v1, automatic writes are limited to formatter-clean JavaScript and TypeScript. The validator denies workflows, migrations, generated output, snapshots, SQL, lockfiles, binaries, symlinks, gitlinks, file creation, deletion, rename, and mode changes.

Construct the candidate with Git plumbing or another method that does not invoke hooks, filters, or code from the PR. Run git diff --check. Feed candidate bytes to the trusted pinned formatter in check mode. Do not let the formatter discover configuration under the PR checkout.

Immediately before the push, fetch the live PR again. Push with an exact lease:

git -C _apply/pr push --porcelain \
  --force-with-lease="refs/heads/$HEAD_REF:$EXPECTED_HEAD_SHA" \
  origin "$FIXED_SHA:refs/heads/$HEAD_REF"

The lease turns the write into a compare-and-swap operation. If a person or bot updated the branch, the push fails instead of overwriting the new work.

Use a clear bot author such as:

Your Project Codex Remediation <codex-remediation@users.noreply.github.com>

Step 9: Review the automatic commit again

A push made with GITHUB_TOKEN does not start another copy of the same workflow. This is useful because it prevents remediation loops, but it also means normal CI may not run for the bot commit.

After a successful push, mark codex/autoreview pending on the new SHA and run fresh correctness and simplification audits inside the same workflow. Render the terminal report from the post-change evidence.

Tell the reviewer that normal tests and builds still need to run on the final head. Never describe the automatic edit as a verified fix unless a separate trusted process actually ran the required verification.

Step 10: Preserve evidence and fail closed

Upload immutable artifacts named with the PR number and audited SHA. Eichler stores:

Use a short retention period that fits your audit needs; Eichler uses 14 days.

Add explicit terminal paths for:

If a candidate was pushed and later reporting fails, mark that exact candidate as unverified and incomplete. Do not reuse prose produced for the previous head.

Suggested build order

Phase 1: useful read-only review

  1. Add the trusted base/head checkout layout.
  2. Generate an exact change manifest.
  3. Write one correctness prompt and schema.
  4. Run openai/codex-action read-only.
  5. Validate SHA, path, location, hunk, and coverage claims.
  6. Update one marked PR comment.
  7. Publish codex/autoreview on the audited SHA.
  8. Add tests for stale results, prompt injection, invalid anchors, and moved branches.

This phase provides most of the review value with a much smaller attack surface.

Phase 2: review quality

  1. Split correctness and simplification into separate passes.
  2. Add full versus targeted coverage.
  3. Add review targets, protected complexity, and uncertainties.
  4. Store normalized evidence as artifacts.
  5. Calibrate findings against a set of known good and bad PRs.

Phase 3: bounded remediation

  1. Select only high-confidence eligible sources.
  2. Sign canonical remediation requests.
  3. Request exact line operations from a read-only model job.
  4. Validate each source independently.
  5. Add one retry for rejected sources.
  6. Build a candidate without hooks or PR-controlled code.
  7. Push with an exact-SHA lease from a keyless writer job.
  8. Re-audit the pushed candidate.

Do not start Phase 3 until the first two phases have strong deterministic tests.

Tests worth copying as behavior

Eichler tests the following contracts:

Run the Eichler deterministic suite with:

node --test tools/codex-pr-report/*.test.mjs

Repository settings and operating notes

Configure these items in the target repository:

  1. Add the CODEX_AUDIT_OPENAI_API_KEY Actions secret.
  2. Give the workflow only job-level permissions. Start the workflow with permissions: {}.
  3. If you enable remediation, allow the writer job to update same-repository PR branches.
  4. Add codex/autoreview to branch protection only after failure behavior has been tested.
  5. Set concurrency to one active run per PR and cancel stale runs.
  6. Keep prompts, schemas, validators, and workflow changes under normal code review.
  7. Review OpenAI API usage and GitHub Action logs after rollout.

The current Eichler model and effort settings are cost choices, not part of the design. Measure false positives, missed seeded defects, token use, latency, and cost before choosing your own settings.

Common shortcuts that break the design

Minimum viable workflow skeleton

This skeleton shows the trust boundary. It omits the validators and publisher that you must add before relying on the result.

name: Codex PR review

on:
  pull_request_target:
    types: [opened, synchronize, reopened, ready_for_review]

concurrency:
  group: codex-pr-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions: {}

jobs:
  review:
    if: >-
      github.event.pull_request.draft == false &&
      github.event.pull_request.base.ref == github.event.repository.default_branch &&
      github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Check out trusted policy
        uses: actions/checkout@<pinned-commit>
        with:
          ref: ${{ github.event.pull_request.base.sha }}
          fetch-depth: 1
          persist-credentials: false
          sparse-checkout: |
            AGENTS.md
            .github/codex
            tools/codex-pr-report
          sparse-checkout-cone-mode: false

      - name: Check out PR as untrusted evidence
        uses: actions/checkout@<pinned-commit>
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          path: _audit/pr
          fetch-depth: 0
          persist-credentials: false
          submodules: false

      - name: Build trusted manifest
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
        run: >-
          node tools/codex-pr-report/manifest.mjs
          --repo _audit/pr
          --base "$BASE_SHA"
          --head "$HEAD_SHA"
          --output _audit/change-manifest.json

      - name: Require dedicated review key
        env:
          AUDIT_KEY: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }}
        run: test -n "$AUDIT_KEY"

      - name: Run structured read-only review
        id: codex
        uses: openai/codex-action@<pinned-commit>
        with:
          openai-api-key: ${{ secrets.CODEX_AUDIT_OPENAI_API_KEY }}
          prompt-file: .github/codex/prompts/correctness.md
          output-schema-file: .github/codex/schemas/correctness.schema.json
          codex-args: '["--ephemeral","--json"]'
          codex-version: <tested-version>
          model: <tested-model>
          effort: <tested-effort>
          permission-profile: ":read-only"
          safety-strategy: drop-sudo

Split validation and publication into later jobs. Pass data through immutable artifacts or bounded job outputs. The publishing job should receive neither the OpenAI key nor a checkout of executable PR code.

Final checklist

Before enabling the workflow on real pull requests, confirm all of these statements:

That separation of duties is the core of the system: Codex analyzes and proposes; trusted code decides what is valid; a separate bounded job performs the write; humans retain the merge decision.