CI policy enforcement

Who it is for

Developers and repository maintainers who need pull requests to satisfy shared CI policy as well as application tests.

The problem and constraints

A repository can run its own tests successfully while its workflow weakens a security boundary. Triggers, permissions, action references, local paths, and container inputs all affect which code runs and under what authority.

The policy checker has to interpret workflow structure rather than look for a few reassuring strings. At the same time, broad support for every execution format can turn a small checker into a second parser platform.

How it is used

Run the pinned external checker alongside the application’s own tests. The shared checker evaluates policy; repository-specific tests continue to evaluate the application. Keep the checker’s version explicit so the proposed change does not silently replace the rules it must satisfy.

The pinned policy checker and the repository’s own tests answer different questions, so a passing application suite cannot stand in for a policy result.

Open diagram at full size

Read the workflow as text

• A pull request goes to two independent checks.
• Pinned shared policy checker: workflow events and permissions; action, local path and container references; bounded YAML parsing.
• Dockerfile actions: forbidden, instead of parsed.
• Repository tests: the application tests run on their own.
• The checker gives a policy result, and the tests give an application result.
• Each result needs an identity: the candidate, the checker version, the test version.

What the implementation checks

Implementation surfaces of the custom Ruby checker. Execution evidence and its limits are summarized below.

The checker reads files that the pull request controls, so it also has to protect its own runtime. It calls fixed system binary paths and clears known interpreter, loader and shell-startup hooks, to stop a caller from replacing or preloading the checker. It bounds file size, file count and total size before parsing YAML, and node count and depth after parsing.

A decision that reduced the surface

The checker forbids Dockerfile actions instead of attempting to build and maintain a custom Dockerfile parser. That trades compatibility for a smaller policy surface.

A second decision kept AI review out of the merge path. The policy runs no language-model reviewer. A person can still ask for an advisory AI review, but it is not a required, paid, or flaky merge gate.

The architecture also separates a pinned shared checker from each repository’s application tests. The two answer different questions, so a passing application suite cannot substitute for a policy decision.

Evidence and limits

The checker has 16 test methods. That is a count of tests, not a current passing run.

A historical research report describes six passing repositories, three blocking canaries, and rejection of a hostile pull request. These are reported historical CI outcomes, not fresh benchmark results.

No measured incident reduction or false-positive rate is established.

A count of tests does not show what the checker does with a bad workflow, so I ran both. On October 2, 2026, I took the checker and its tests as they stood at an earlier commit of the policy repository and ran them in a Ruby 3.2 container, with the repository mounted read-only. The suite ran 8 tests with 36 assertions and no failures; that earlier suite is smaller than the 16 test methods counted above. I then gave the checker two throwaway repositories. The bad one used a forbidden trigger, write-all permissions, an action referenced by a tag instead of a full commit SHA, a Docker image without a sha256 digest and a tracked environment file. The checker reported all 5 violations, each with its file, and exited with code 1. The good one passed with exit code 0. The run is evidence about that earlier version only.

$ docker run --rm -i -v ~/project:/src:ro -w /src ruby:3.2 bash -s < run-ci-demo.sh; echo "docker_exit=$?"
$ ruby -v; git --version
ruby 3.2.11 (2026-03-27 revision 5483bfc1ae) [aarch64-linux]
git version 2.47.3
$ git -C /src archive 34b709f scripts tests | tar -x -C /work/policy

$ ruby tests/check_repository_policy_test.rb
Run options: --seed 16365

# Running:

........

Finished in 0.233003s, 34.3343 runs/s, 154.5044 assertions/s.

8 runs, 36 assertions, 0 failures, 0 errors, 0 skips

$ cat bad/.github/workflows/ci.yml; git -C bad ls-files
name: CI
on: [pull_request_target]
permissions: write-all
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker://alpine:latest
.env
.github/workflows/ci.yml
$ (cd bad && ruby ../policy/scripts/check_repository_policy.rb); echo "exit=$?"
Repository policy violations (5):
...
  - .github/workflows/ci.yml:2: pull_request_target is forbidden
  - .github/workflows/ci.yml:3: workflow permissions must not use write-all
  - .github/workflows/ci.yml:8: remote action or reusable workflow must use a full commit SHA: actions/checkout@v4
  - .github/workflows/ci.yml:9: Docker action must use a full sha256 digest: docker://alpine:latest
exit=1

$ cat good/.github/workflows/ci.yml; git -C good ls-files
name: CI
on: [pull_request]
permissions:
  contents: read
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
.env.example
.github/workflows/ci.yml
$ (cd good && ruby ../policy/scripts/check_repository_policy.rb); echo "exit=$?"
Repository policy passed: 1 workflow/action YAML files scanned.
exit=0
docker_exit=0

Learnings

What I learned: a policy layer needs to reason about execution semantics, not simply whether an application test is green.

What changed: the implementation keeps shared policy pinned, checks concrete workflow surfaces, and excludes Dockerfile actions rather than broadening into a custom parser.

What to test next: rerun the 16 test methods and the reported repository/canary cases against the current pinned checker, preserving exact versions and outputs.