QA in Your CI/CD Pipeline: Scoped Tokens, Real Exit Codes, and Reports You Can Trust

There's a particular kind of confidence that comes from knowing your regression suite ran against the code that is actually serving traffic right now. Not "I tested it locally." Not "the tests will run later, after someone remembers." The real thing: a pipeline that refuses to go green until QA has run against the revision you deployed, and that says so explicitly when it did not.

That's what CI/CD integration is supposed to be. Getting there takes one extra step that most guides skip: the QA run must be pointed at an environment that is serving the revision you just tested, and the pipeline has to prove that before it reads the result as a verdict. A test run against an older deploy is not wrong, it is just answering a different question — and it will happily tell you the wrong thing about the commit you care about.

Here's how that works with Qalizer, and why the pieces — scoped tokens, job IDs, artifacts and exit codes — are each load-bearing.

Why manual regression fails

Manual regression has two structural problems. First, it's stale by the time it matters — you test what you have, then keep merging, and by release time the app has moved past what the suite verified. Second, it's distant from the change — when a regression surfaces, tracing it to a specific commit is hard because so many changes landed between. You get a pile of suspects instead of a single culprit.

CI-driven regression addresses both, with one caveat worth being precise about. It runs on every push, so the suite stops being stale; and when it runs against the revision you just deployed, a red result points at a change that is still close enough to investigate. What CI does not do is prove causation. A red pipeline says the run failed against whatever that environment was serving at that moment — it can be a regression, a flake, a dependency outage, a stale deploy, or a test that was already broken. Reading it as "the latest commit broke it" is the most common way to misuse a gate, and it is how teams learn to ignore one.

What the CI API actually runs

This is the part where most "wire up your tests" posts quietly overpromise, so let me be exact about Qalizer's CI surface.

The CI API starts browser scenarios: one scenario by id, a scenario set by name, or a list of scenario ids. Everything else — API tests, liveness checks, visual regression, load tests, Gremlins, data injection — is not startable from the CI API. An API test can be promoted to a browser scenario (its steps become navigate + an API assertion), and then it runs in CI like any other scenario. That's the intended path, not a workaround.

Runs cost 1 credit per scenario execution, charged when the run is created. Charges are compensated in the cases where the run could not be admitted at all — for example a monitor that is already busy, or an execution location that turns out to be unusable before dispatch — and the refund is keyed to the run so a retry cannot double-charge. That is narrower than "refunded whenever a run does not start", and much narrower than "refunded when the tests fail": a run that correctly finds a bug in your application is a successful, delivered result, and it is charged like one. See the Terms for the full credit-restoration policy.

API tokens: the access you grant to a machine

For a pipeline to trigger tests, it needs to authenticate to Qalizer the same way you would — but as a machine, with an identity you can revoke.

That's what CI tokens are for. You create one in your account, give it a name (I name mine after the pipeline, like gh-actions-web), and it is the only credential the pipeline holds. A few properties worth knowing:

• Three fixed scopes, all CI-only: ci:runs:create (start a run, cancel a queued run), ci:runs:read (poll status), ci:reports:read (download reports). CI tokens cannot touch monitors, billing, admin or profile endpoints — not because of a checkbox you must remember, but because those scopes are never issued to a CI token.
• Per-pipeline and revocable. Lose a runner or decommission a repo, and you revoke or rotate that one token. Rotating returns the new secret once and invalidates the old one.
• Shown once. The full token is returned only when it is created or rotated, and never again in any list, log or admin response. Store it immediately.
• Secret by convention. Never hardcode it in pipeline YAML. GitHub: secrets.QALIZER_TOKEN. GitLab: a masked CI/CD variable. Jenkins: a credentials binding.

Treat the token like a deploy key: a machine identity with the least privilege it needs, rotated when its owner changes.

The CI loop, step by step

Here is the real flow, with the real endpoint paths:

1. POST /api/v1/ci/runs with Authorization: Bearer <token> and a JSON body naming the monitor and exactly one selection (scenario_id, scenario_set_name, or scenario_ids).
2. The API answers 202 with a job_id — not a "run id". That job_id is the handle for everything else. (A run_id may also be present once the run is linked to a test-run row; poll and fetch reports by job_id.)
3. Poll GET /api/v1/ci/runs/{job_id} (requires ci:runs:read). The response's status is one of queued, running, passed, failed, error, cancelled or timed_out.
4. When the job is terminal, download the artifacts from GET /api/v1/ci/runs/{job_id}/report/download?format=json|html|pdf|junit (requires ci:reports:read). There is nothing to upload before you download it — the HTML report lives on Qalizer until your step fetches it.
5. Map the outcome to an exit code. 0 only for a proven pass; non-zero for everything else.

Two details that matter more than they look:

• passed is the only green status. failed, error, cancelled and timed_out are all red, and error is specifically "infrastructure or execution problem — no verdict was reached". Treating anything other than passed as success is how a pipeline starts lying.
• passed is not the same as "zero failures". The status response includes passed_is_proven, and a run that executed nothing, lost its runner, or never confirmed cleanup is reported as unproven — which the status vocabulary expresses as error, never as passed. The JUnit artifact carries the same rule.

Anyone can curl that loop by hand. To save you writing the plumbing, Qalizer also publishes a bootstrap script, qalizer-run.sh, that does the start → poll → download → verify sequence against a Docker runner and uses a stable exit-code contract:

Because the exit code is the contract, wiring Qalizer into any CI system is a matter of "call this API, then this stage depends on the exit status." No CI-specific plugin needed — just the API and the exit code.

Before you trigger: prove what the target is serving

The single most useful thing you can add to a QA gate is a step that fails unless the environment you are about to test is serving the revision you just built. Without it, a green or red result tells you about the last deploy, and every conclusion you draw about the current commit is unfounded.

There are two halves, and both are yours to build — Qalizer tests the URL you give it, and cannot know which commit that URL is running:

1. Deploy the tested revision in the same pipeline, before the QA stage — not "on merge" from another workflow, and not a manual deploy someone may have skipped.
2. Verify deployment readiness and identity before triggering QA: wait for the environment to answer healthy, then read the identity it reports (a version endpoint, a BUILD_SHA/GIT_SHA-style value, or an image tag) and compare it with ${{ github.sha }}. If it does not match, fail the job with "environment not serving this commit" — do not run QA and do not report a verdict.

      # 1) Deploy the SAME revision that triggered this workflow.
      - name: Deploy the tested revision
        run: ./deploy.sh --revision "${{ github.sha }}"

      # 2) Wait for readiness, then prove the running revision is the one we
      #    deployed. Both failures are pipeline failures, not test failures.
      - name: Verify the environment serves the tested revision
        run: |
          set -euo pipefail
          for i in $(seq 1 60); do
            curl -fsS "${{ vars.QALIZER_TARGET_URL }}/healthz" >/dev/null 2>&1 && break
            sleep 5
          done
          LIVE_SHA=$(curl -fsS "${{ vars.QALIZER_TARGET_URL }}/version" | jq -r '.revision // empty')
          if [ "$LIVE_SHA" != "${{ github.sha }}" ]; then
            echo "::error::Target is serving '${LIVE_SHA:-unknown}', not ${{ github.sha }}."
            echo "::error::Refusing to run QA against a revision we did not deploy."
            exit 1
          fi

Use whatever your platform exposes for the identity check — a health endpoint plus a build-info endpoint is enough. The point is not the specific endpoint; it is that the gate has a positive statement ("the target is running commit X") rather than an assumption. When the check fails, the right outcome is a failed deployment stage, not a failed QA run: you have learned nothing about the commit yet.

If your workflow can only test a long-lived staging environment that tracks main, be explicit about that in the job name. "Staging smoke, tracks main" is honest; "regression for this commit" is not, and it is the claim that gets misread during an incident.

A GitHub Actions example

The workflow below mirrors the API contract above and is deliberately plain curl + jq, so you can read every step and adapt it if your runner differs. It is not a plugin: no action, no wrapper, just HTTP. It assumes the deploy-and-verify steps above already succeeded — that is the ordering that makes its verdict mean something.

name: Qalizer Regression
on:
  pull_request:
  push:
    branches: [main]

jobs:
  qalizer:
    runs-on: ubuntu-latest
    steps:
      - name: Start Qalizer run
        id: qalizer
        env:
          QALIZER_TOKEN: ${{ secrets.QALIZER_TOKEN }}
          QALIZER_API: ${{ vars.QALIZER_API_URL }}
        run: |
          set -euo pipefail
          # Exactly ONE selection: scenario_id | scenario_set_name | scenario_ids
          BODY=$(jq -n --arg m "${{ vars.QALIZER_MONITOR_ID }}" \
                       --arg set "${{ vars.QALIZER_SET_NAME }}" \
                       '{monitor_id: $m, scenario_set_name: $set,
                         source: "ci", environment: "ci",
                         execution_location: "server_only",
                         idempotency_key: "gha-${{ github.run_id }}-${{ github.run_attempt }}"}')
          RESP=$(curl -fsS -X POST "$QALIZER_API/api/v1/ci/runs" \
                   -H "Authorization: Bearer $QALIZER_TOKEN" \
                   -H "Content-Type: application/json" \
                   --data-binary "$BODY")
          JOB_ID=$(printf '%s' "$RESP" | jq -r '.job_id')
          echo "job_id=$JOB_ID" >> "$GITHUB_OUTPUT"
          echo "Started Qalizer job $JOB_ID"

      - name: Poll Qalizer job
        id: poll
        env:
          QALIZER_TOKEN: ${{ secrets.QALIZER_TOKEN }}
          QALIZER_API: ${{ vars.QALIZER_API_URL }}
          JOB_ID: ${{ steps.qalizer.outputs.job_id }}
        run: |
          set -euo pipefail
          for i in $(seq 1 90); do
            RESP=$(curl -fsS "$QALIZER_API/api/v1/ci/runs/$JOB_ID" \
                     -H "Authorization: Bearer $QALIZER_TOKEN")
            STATUS=$(printf '%s' "$RESP" | jq -r '.status')
            case "$STATUS" in
              queued|running) sleep 10 ;;
              passed)
                [ "$(printf '%s' "$RESP" | jq -r '.passed_is_proven')" = "true" ] \
                  || { echo "::error::Run reported passed but is not proven"; exit 2; }
                echo "Qalizer job $JOB_ID passed"; exit 0 ;;
              failed) echo "::error::Qalizer job $JOB_ID failed"; exit 1 ;;
              cancelled|timed_out) echo "::error::Qalizer job $JOB_ID $STATUS"; exit 4 ;;
              *) echo "::error::Qalizer job $JOB_ID ended as $STATUS (unproven)"; exit 2 ;;
            esac
          done
          echo "::error::Timed out waiting for Qalizer job $JOB_ID"; exit 4

      # ALWAYS download BEFORE uploading: the artifact is fetched from Qalizer,
      # and this step must run even when the poll step failed.
      - name: Download Qalizer reports
        if: always() && steps.qalizer.outputs.job_id != ''
        env:
          QALIZER_TOKEN: ${{ secrets.QALIZER_TOKEN }}
          QALIZER_API: ${{ vars.QALIZER_API_URL }}
          JOB_ID: ${{ steps.qalizer.outputs.job_id }}
        run: |
          set -euo pipefail
          mkdir -p qalizer-reports
          for fmt in junit html json; do
            curl -fsS "$QALIZER_API/api/v1/ci/runs/$JOB_ID/report/download?format=$fmt" \
                 -H "Authorization: Bearer $QALIZER_TOKEN" \
                 -o "qalizer-reports/qalizer-$JOB_ID.$fmt" \
              || echo "::warning::No $fmt report available for $JOB_ID"
          done
          ls -l qalizer-reports

      - name: Upload Qalizer reports as artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: qalizer-reports
          path: qalizer-reports/
          if-no-files-found: warn

A few notes on what's happening:

• One selection, explicitly. scenario_set_name is the name of the set inside that monitor, not a global id. Providing two of the three selection fields is rejected with 422.
• Poll on job_id. The status field is what you branch on; report_urls in the same response tells you which artifacts exist.
• Download before upload. The report step runs if: always() so a red gate still produces evidence, and it fetches the files first — there is nothing local until it does.
• Exit codes are the contract. The mapping distinguishes a test failure from "we never got a verdict". That difference is what stops a broken runner from looking like a passing test.
• jq ships on the standard GitHub runner, so there is no extra install step.

The same three steps port to GitLab CI (a script: block with the same curl/jq logic, allow_failure: false, and $CI_PIPELINE_ID as the idempotency key) and Jenkins (the same sh steps inside a withCredentials block).

Where the run executes

By default a CI run asks for execution_location: "auto", which uses your own local runner if one is online, fresh and has a free slot, and otherwise falls back to Qalizer's shared server pool. If you need to be certain where it runs, set server_only or local_only. With local_only, an unavailable runner is an error naming the location — the run is never silently moved to the pool.

The status response reports the location that was actually used (execution_location). A surprising number of "our CI ran somewhere else" conversations are really "auto picked the other option".

Blocking merges the right way

The exit code alone does not block a merge. A failing job status does — and only if the check is required on the protected branch. The setup that actually gates:

1. Push the workflow to the default branch once so GitHub registers the check.
2. Repository Settings → Branches → Branch protection rules (or a ruleset) for main.
3. Enable Require status checks to pass before merging and select this job's check — qalizer in the example above (shown as Qalizer Regression / qalizer in the picker).
4. Optionally, Require branches to be up to date before merging, so the QA result belongs to the commit that will land.

Then verify it the only way that counts: open a pull request that deliberately breaks a tested page and confirm the merge button is disabled with the Qalizer check red. If the button is still green, the check isn't required — a green run alone never proves a gate exists.

Two practical cautions:

• Monitor busy is a retry, not a failure. Starting a run while another operation holds that monitor returns 409 with the code monitor_busy. Retry, or run the QA stage after the other operation.
• Cancellation is limited. A queued run can be cancelled; a scenario already executing in a browser runs to completion, and the cancel endpoint says so instead of pretending.

Honest caveats

Let me be straight about what CI integration does not do. It doesn't make flaky tests less flaky — it makes them visible more consistently, which is a feature. It will occasionally block a merge for a false positive, so keep a "re-run the QA stage" path for genuine flakes, as long as the flake gets triaged rather than papered over. And it needs a steady, deterministic test set to be useful as a gate: gate on your stable core first, then expand what blocks the merge as confidence grows.

Above all, remember what a red result proves and what it doesn't. It proves the run failed, against the environment it was pointed at, at that moment. It does not prove that the newest commit caused the failure, that the target was serving that commit, or that the failing behaviour reproduces. Treat the pipeline output as a pointer to where to look next, not as a verdict you can attach to a commit message.

One more: CI credits come out of the same balance as everything else, one per scenario execution. A 40-scenario set on every push is a budgeting question, not a free one.

Zero-click regression is the goal

The end state I'm chasing with all of this is simple: every push gets regression-tested against the revision that was actually deployed, and the merge only goes through when the tests say so. No one remembers to run the suite. No one argues about whether QA "should" run. The pipeline does it, and the job status, artifacts and exit codes make the result unambiguous.

That's the most reliable QA gate you can install — not because it's clever, but because it's automatic. It never forgets, it never skips, and it brings the evidence to the exact place the decision is made: your merge queue.

If you're still treating regression as a manual ritual, the upgrade path is short: create a CI token in your account, add the deploy-and-verify steps, then the Qalizer stage, and mark the check required. Qalizer is in a paid public beta and runs cost credits — a new account starts with 100 signup credits, so you can prove the gate works on a small set before buying more.