Making CI Review Your Release Notes: An Agentic CI/CD Pipeline Built on claude -p

In the previous post we saw how to keep codebase documentation refreshed on a schedule with claude -p. There, claude -p was the hand that rewrote documents. This time we place the same claude -p closer to day-to-day operations: a pipeline where AI performs the first-pass review of release notes and only the problematic ones reach a human. The write-up takes the form of a proposal that puts two implementations of the same review side by side, a direct Claude API call and claude -p.

Background

In many teams the release process requires a human to review the release notes and an engineering manager to approve before the merge. As release frequency grows, the manager becomes an approval bottleneck. Cross-checking notes against actual commits line by line on every release is tedious, and tedious review tends to turn perfunctory.

The core of this proposal is to flip the approval structure: the moment a release tag is created, AI reviews the release notes first, and only flagged releases escalate to a human. On GitLab the flow looks like this. Pushing a release tag like v1.2.3 triggers the pipeline,¹ which reviews the tag message (the release notes) against the changes since the previous tag and returns a verdict of APPROVE or NEEDS_REVIEW. On APPROVE the pipeline passes; on NEEDS_REVIEW it fails, and the manager looks only at that release.

Review criteria

The pipeline gives Claude four review criteria.

• Commit consistency. Do the release notes match the actual commit history?

• Missing breaking changes. Are user-impacting changes absent from the notes?

• Clarity. Are the sentences clear enough for users to understand?

• Security labeling. Are security-related changes labeled appropriately?

One piece of shared groundwork: register ANTHROPIC_API_KEY as a Masked and Protected variable under Settings > CI/CD > Variables in GitLab.

Approach A: calling the Claude API directly

A shell script gathers the release notes and the commit log, then makes a single curl call to the Claude API for a verdict.² The structure is simple and the per-call cost is predictable.

# =====================================================================
# Automated AI release note review (Approach A: direct Claude API call)
# ---------------------------------------------------------------------
# Flow:
#   1) A developer pushes a release tag (e.g. v1.2.3)
#   2) The tag pipeline runs automatically
#   3) Collect the tag message + the commit log since the previous tag
#   4) Send them to the Claude API for review
#   5) Pass or fail the pipeline based on the verdict (APPROVE / NEEDS_REVIEW)
#      On NEEDS_REVIEW the pipeline fails and the manager only looks at failures
# =====================================================================

# workflow: top-level rules controlling when the whole pipeline runs
workflow:
  rules:                                        # evaluated top to bottom
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/' # only on semantic version tags like v1.2.3
      when: always
    - when: never                                # otherwise do not create a pipeline at all

stages:
  - ai-review

release-note-ai-review:
  stage: ai-review
  image: alpine:3.20                             # a light image is enough: curl and jq

  variables:
    CLAUDE_MODEL: "claude-sonnet-5"              # Sonnet balances review quality and cost
    MAX_TOKENS: "2000"                           # plenty for the review-result JSON
    GIT_DEPTH: "0"                               # full history, to compare with the previous tag

  before_script:
    - apk add --no-cache curl jq git

  script:
    # Step 1: collect the text under review
    - RELEASE_NOTES=$(git tag -l --format='%(contents)' "${CI_COMMIT_TAG}")   # treat the annotated tag body as the notes
    - PREV_TAG=$(git describe --tags --abbrev=0 "${CI_COMMIT_TAG}^" 2>/dev/null || echo "")
    - |
      if [ -n "$PREV_TAG" ]; then
        COMMIT_LOG=$(git log --oneline "${PREV_TAG}..${CI_COMMIT_TAG}")
      else
        COMMIT_LOG=$(git log --oneline "${CI_COMMIT_TAG}")
      fi
    - |
      if [ -z "$RELEASE_NOTES" ]; then
        echo "Release notes (tag message) are empty. Use an annotated tag."
        exit 1
      fi

    # Step 2: build the request body (JSON) safely with jq (string concatenation breaks on special characters)
    - |
      REQUEST_BODY=$(jq -n \
        --arg model "$CLAUDE_MODEL" \
        --arg notes "$RELEASE_NOTES" \
        --arg commits "$COMMIT_LOG" \
        --arg tag "$CI_COMMIT_TAG" \
        --argjson max_tokens "$MAX_TOKENS" \
        '{
          model: $model,
          max_tokens: $max_tokens,
          system: "You are a release note reviewer. Output JSON only. Format: {\"verdict\": \"APPROVE\" or \"NEEDS_REVIEW\", \"score\": 0-100, \"issues\": [array of problems], \"summary\": \"one-line summary\"}. Criteria: (1) do the notes match the commit history (2) are user-impacting changes missing (3) are the sentences clear (4) are security-related changes labeled appropriately.",
          messages: [
            { role: "user",
              content: ("Release tag: " + $tag + "\n\n## Release notes\n" + $notes + "\n\n## Actual commit list\n" + $commits) }
          ]
        }')

    # Step 3: call the Claude API
    - |
      RESPONSE=$(curl -sS https://api.anthropic.com/v1/messages \
        -H "x-api-key: ${ANTHROPIC_API_KEY}" \
        -H "anthropic-version: 2023-06-01" \
        -H "content-type: application/json" \
        -d "$REQUEST_BODY")
    - |
      if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
        echo "Claude API error:"; echo "$RESPONSE" | jq '.error'; exit 1
      fi

    # Step 4: parse the response and extract the verdict. content[0].text holds the JSON string the model produced
    - REVIEW_JSON=$(echo "$RESPONSE" | jq -r '.content[0].text')
    - REVIEW_JSON=$(echo "$REVIEW_JSON" | sed 's/```json//g; s/```//g')   # in case the model wrapped it in a code block
    - VERDICT=$(echo "$REVIEW_JSON" | jq -r '.verdict')
    - SCORE=$(echo "$REVIEW_JSON" | jq -r '.score')
    - echo "$REVIEW_JSON" | jq .
    - echo "$REVIEW_JSON" > ai-review-result.json

    # Step 5: final verdict. APPROVE passes; anything else fails the pipeline and acts as a gate
    - |
      if [ "$VERDICT" = "APPROVE" ]; then
        echo "AI review passed (score ${SCORE}). Auto-approved."
      else
        echo "AI review requests human attention (score ${SCORE}). Share this pipeline link with the manager."
        exit 1
      fi

  artifacts:
    paths:
      - ai-review-result.json                    # preserve the review-result JSON
    expire_in: 30 days                           # audit retention period
    when: always                                 # failures matter most, so always store

  timeout: 10m                                   # cap the wait in case of API latency

The defining trait, and limitation, of Approach A is that Claude only sees the commit-title list we spoon-feed it. Whether a feature described in the notes actually landed in the code changes is beyond its reach, because we never handed it the diff.

Approach B: Claude Code CLI (claude -p)

This approach runs Claude Code in headless mode (claude -p).³ No commit-collection code is needed. Claude acts as an agent, running the allowed git log, git diff, and git show commands itself to cross-check the release notes against the actual changes. It can verify at greater depth, down to whether a feature mentioned in the notes exists in the actual diff.

# =====================================================================
# Automated AI release note review (Approach B: Claude Code CLI, claude -p)
# ---------------------------------------------------------------------
# Differences from Approach A:
#   - Instead of curl-ing the API directly, run Claude Code in headless mode (-p)
#   - Claude acts as an agent, running git log/diff itself to
#     cross-check the release notes against the actual changes (no collection code)
#   - --allowedTools permits read-only tools, keeping the run safe
# =====================================================================

workflow:
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
      when: always
    - when: never

stages:
  - ai-review

release-note-ai-review:
  stage: ai-review
  image: node:20-alpine                          # Claude Code is an npm package, so a Node.js image is required

  variables:
    GIT_DEPTH: "0"                               # full-history clone so Claude can compare with the previous tag
    CLAUDE_MODEL: "claude-sonnet-5"              # model to use (passed via the --model flag)

  before_script:
    - apk add --no-cache git jq
    - npm install -g @anthropic-ai/claude-code
    - claude --version                           # print the version to confirm installation

  script:
    # Step 1: extract the release notes (tag message)
    - RELEASE_NOTES=$(git tag -l --format='%(contents)' "${CI_COMMIT_TAG}")
    - |
      if [ -z "$RELEASE_NOTES" ]; then
        echo "Release notes (tag message) are empty. Use an annotated tag."
        exit 1
      fi

    # Step 2: run Claude Code headless
    # Key point: we do not hand over the commit log. Claude runs the allowed git commands itself.
    - |
      RESULT=$(claude -p "Review the release notes for release tag ${CI_COMMIT_TAG}.

      ## Release notes
      ${RELEASE_NOTES}

      ## How to review
      Use git log and git diff to inspect the actual changes since the previous tag and cross-check them against the release notes.

      ## Criteria
      1. Do the notes match the actual commits/changes
      2. Are user-impacting (breaking) changes missing
      3. Are the sentences clear
      4. Are security-related changes labeled appropriately

      ## Output format (your final answer must be this JSON only)
      {\"verdict\": \"APPROVE\" or \"NEEDS_REVIEW\", \"score\": 0-100, \"issues\": [\"problem\"], \"summary\": \"one-line summary\"}" \
        --bare \
        --model "${CLAUDE_MODEL}" \
        --output-format json \
        --allowedTools "Read,Grep,Glob,Bash(git log *),Bash(git diff *),Bash(git show *),Bash(git tag *),Bash(git describe *)" \
        --max-turns 15)
      # Flag notes:
      # -p                : headless (print) mode. Run the prompt, print the result, exit
      # --bare            : skip local hooks/MCP/CLAUDE.md discovery so CI behaves identically every run
      #                     authentication is handled via the ANTHROPIC_API_KEY environment variable
      # --model           : which model to use
      # --output-format   : with json, output has the shape {result, is_error, total_cost_usd, ...}
      # --allowedTools    : tools allowed to run without approval prompts
      #                     read tools and read-only git commands only, so no writes or edits
      #                     "Bash(git log *)" whitelists specific commands via the space and *
      # --max-turns 15    : cap on agent turns, preventing runaway cost from endless exploration

    # Step 3: parse the run result. Filter Claude Code's own errors (auth failures, etc.) first
    - |
      if [ "$(echo "$RESULT" | jq -r '.is_error')" = "true" ]; then
        echo "Claude Code run error:"; echo "$RESULT" | jq -r '.result'; exit 1
      fi
    - REVIEW_JSON=$(echo "$RESULT" | jq -r '.result')                     # .result holds the final answer (JSON)
    - REVIEW_JSON=$(echo "$REVIEW_JSON" | sed 's/```json//g; s/```//g')
    - VERDICT=$(echo "$REVIEW_JSON" | jq -r '.verdict')
    - SCORE=$(echo "$REVIEW_JSON" | jq -r '.score')
    - echo "Review cost \$$(echo "$RESULT" | jq -r '.total_cost_usd')"     # log the per-run cost
    - echo "$REVIEW_JSON" | jq .
    - echo "$REVIEW_JSON" > ai-review-result.json

    # Step 4: final verdict (gate)
    - |
      if [ "$VERDICT" = "APPROVE" ]; then
        echo "AI review passed (score ${SCORE}). Auto-approved."
      else
        echo "AI review requests human attention (score ${SCORE}). Share this pipeline link with the manager."
        exit 1
      fi

  artifacts:
    paths:
      - ai-review-result.json
    expire_in: 30 days
    when: always

  timeout: 15m                                   # more headroom than Approach A to allow for agent exploration

Comparing the two approaches

Both share the same verdict gate (APPROVE/NEEDS_REVIEW) and the same artifacts structure, which makes them easy to run side by side in a pilot.

For light cross-checks where commit titles suffice, Approach A is cheap and fast. For deep review that must catch divergence between the notes and the actual code, Approach B earns its cost.

What to tune in production

Moving from pilot to production, a few spots typically get adjusted to fit organizational policy.

• Gate strength. Instead of failing the pipeline on NEEDS_REVIEW, you can post an MR comment only. Conversely, even on APPROVE you can keep the deploy job at when: manual so a human touches it once. Agreeing first that AI approval is strictly an assist matters more than the mechanics.

• Notes source. This example treats the annotated tag message as the notes. If you use the GitLab Release feature or CHANGELOG.md, switch the source to the Releases API.

• Two-tier models. Run ordinary releases on a cheaper model and escalate only critical ones, like security releases, to a higher-tier model to hold both cost and quality.

• Notifications. Wire a Slack or email webhook on failure and only the filtered exceptions get pushed to the manager.

Other uses of the same pattern

The skeleton of "have claude -p deliver a verdict and wire the result into pipeline status" applies well beyond release notes. Feed it an MR's git diff to scan for security vulnerabilities, or check whether an MR's changes contradict the documents built with acquire-codebase-knowledge and leave the result as a comment. Change the prompt and the same pipeline skeleton extends to a different review.

Wrapping up

The difference between Approach A and Approach B is the best explanation of the word "agentic." Approach A is a function call: we gather the material needed for the judgment and hand it over once. Approach B is agentic: Claude decides for itself what to look at and investigates the repository. Approach B assists humans more deeply than Approach A at the price of more cost and time, which is why fencing it in with --max-turns and read-only --allowedTools is necessary.

Either way the core benefit is the same. Instead of the manager reading every release, they confirm only the exceptions AI filtered out. Keep in mind that verdicts can wobble, design the gate safely, and an agentic pipeline takes over release note review while reducing the approval bottleneck and leaving an audit trail.

1. GitLab, CI_COMMIT_TAG predefined variable, GitLab Docs.

   ↩

2. Anthropic, Messages API, Claude API Docs.

   ↩

3. Anthropic, Run Claude Code programmatically (headless mode), Claude Code Docs.

   ↩