Robot Framework

TraceCov ships a Robot Framework listener. It records every request your suites send through requests-based libraries such as RequestsLibrary, and writes a coverage report into the output directory, linked from report.html.

Requires Robot Framework 7.0 or later.

$ uv pip install tracecov

Setup

Add the listener to your robot command, with the path of your OpenAPI specification after a ;. A relative path starts from the directory you run robot in:

$ robot --listener "tracecov.robotframework.TraceCov;openapi.yaml" --outputdir results tests

A URL works too:

$ robot --listener "tracecov.robotframework.TraceCov;https://api.example.com/openapi.json" --outputdir results tests

Use ; as the separator. Robot Framework also accepts :, but splits a URL at every colon. The listener fetches a URL without authentication; download a protected specification first and pass its path.

You keep your test files as they are. The listener records every request a suite sends through RequestsLibrary, with or without Create Session, or through your own keywords that call requests. In this example, replace https://api.example.com with your API's URL:

*** Settings ***
Library             RequestsLibrary
Suite Setup         Create Session    petstore    https://api.example.com


*** Test Cases ***
List Pets
    GET On Session    petstore    /pets

Create Pet
    POST On Session    petstore    /pets    json=${{ {"name": "Tom"} }}    expected_status=201

The listener records requests from failing tests as well as passing ones.

Reports

The listener writes tracecov-report.html and tracecov-report.json into the output directory, and prints the HTML report path after Robot Framework's own:

Output:  /work/results/output.xml
Log:     /work/results/log.html
Report:  /work/results/report.html
TraceCov: /work/results/tracecov-report.html

report.html links to it as API Coverage in its Summary table. The link stays correct when --report puts report.html in another directory.

The reports hold coverage counts and the method, host and path of each request that matched no operation. They do not store header values, query values or bodies.

See Coverage Concepts for what each metric measures and JSON Report Format for the JSON layout.

Coverage per suite and test

Professional Edition

Labels require TraceCov Professional. The Community edition records coverage and writes the reports without labels, and logs a warning in log.html. The warning does not change the exit code.

The JSON report has a labels section with one entry per test and one per suite, named by their full names. Each entry holds that test's or suite's totals, in the same shape as the report's summary, with no breakdown by operation:

$ jq '.labels | keys' results/tracecov-report.json
[
  "Tests",
  "Tests.Pets",
  "Tests.Pets.Crud",
  "Tests.Pets.Crud.Create Pet",
  "Tests.Pets.Crud.List Pets"
]

The HTML report does not show labels.

A suite's entry covers every test in it, and every suite inside it. Requests sent from a suite's setup or teardown count for that suite and the suites that contain it, never for a test.

SchemathesisLibrary, Schemathesis's Robot Framework library, adds a random suffix to its test names, so its per-test labels change on every run. Compare runs by their suite labels.

Query each test's coverage after the run

The JSON report holds each test's totals. To list which operations, parameters, responses and schema keywords each test reached, for example what only failing tests reached, save the run's coverage data with data:

$ robot --listener "tracecov.robotframework.TraceCov;openapi.yaml;data=tracecov.data" --outputdir results tests

A relative data path starts from the output directory, so the listener writes results/tracecov.data next to the reports and creates missing directories; an absolute path stays as given. Under pabot each process rewrites the file as it finishes, and the last one writes every process's coverage: read it after pabot exits.

Load it into a CoverageMap of the same specification and ask by test name. output.xml has each test's status and tags:

from pathlib import Path

from robot.api import ExecutionResult
from tracecov import CoverageMap

coverage = CoverageMap.from_path("openapi.yaml")
coverage.merge_data(Path("results/tracecov.data").read_bytes())

tests = list(ExecutionResult("results/output.xml").suite.all_tests)
for test in tests:
    for row in coverage.label_entries([test.full_name]):
        print(test.full_name, test.status, list(test.tags), row)

failing = [test.full_name for test in tests if test.status == "FAIL"]
passing = [test.full_name for test in tests if test.status == "PASS"]
for row in coverage.label_entries(failing, excluding=passing):
    print("Only failing tests reached", row)

Each row names one element and whether the test sent valid values, invalid values or both, for example {"method": "GET", "path": "/pets/{petId}", "location": "path", "name": "petId", "reached": "invalid"}. See label_entries() for the row keys. Labels are test and suite full names, so pick tests by tag the same way the script picks them by status: [test.full_name for test in tests if "smoke" in test.tags].

The file holds what the reports hold plus which elements each test and suite reached; it stores no header values, query values or bodies. Load it with the TraceCov version and the specification that wrote it, for example in the same CI job. merge_data() raises ValueError for a file that does not fit the specification, but cannot tell apart two specifications of the same shape.

Parallel runs with pabot

Replace robot with pabot:

$ pabot --listener "tracecov.robotframework.TraceCov;openapi.yaml" --outputdir results tests

Each pabot process saves its coverage under results/pabot_results/. The processes merge it into one tracecov-report.html and tracecov-report.json in results. The last process to finish writes the complete report. You need no extra CI step. The API Coverage link assumes report.html stays in results, where pabot writes it by default.

When the listener fails to start

If the listener cannot load the specification, Robot Framework prints an error and runs the tests without it:

[ ERROR ] Taking listener 'tracecov.robotframework.TraceCov;openapi.yaml' into use failed: Importing listener 'tracecov.robotframework.TraceCov' failed: Creating instance failed: FileNotFoundError: [Errno 2] No such file or directory: 'openapi.yaml'

The run still passes, with no coverage report. Add --exitonerror to make robot exit with a non-zero code instead, so CI fails:

$ robot --exitonerror --listener "tracecov.robotframework.TraceCov;openapi.yaml" --outputdir results tests

--exitonerror also stops the run on any other Robot Framework error, such as a library that fails to import.

When requests do not count

Requests to paths your specification does not describe appear under Unmatched requests in the HTML report, and in unmatched in the JSON report. Requests to other hosts your suite calls, such as a token endpoint, appear there too.

A common cause is a path prefix: the suite calls /api/v1/pets while the specification documents /pets. Put the prefix in the specification's server URL, for example servers: [{url: https://api.example.com/api/v1}] or servers: [{url: /api/v1}]. TraceCov uses only the path part of the server URL, so its host may differ from the one your suite calls. Swagger 2.0 specifications use basePath the same way.

If the report lists no requests at all, check that your suite's HTTP client is built on requests. See Not captured.

Send results to Elasticsearch

Send each run's JSON report to Elasticsearch to chart coverage across runs, and keep the report as a CI artifact. The jq filter below turns the report into one document for the whole run (label is null) and one per label. Each document holds the summary metrics; for example, chart operations.percent over generated_at. A document per label keeps the index's field count fixed however many tests you add.

GitHub ActionsGitLab CI

- name: Run API tests
  run: robot --exitonerror --listener "tracecov.robotframework.TraceCov;openapi.yaml" --outputdir results tests

- name: Upload coverage report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: tracecov-report
    path: results/tracecov-report.*

- name: Send coverage to Elasticsearch
  if: always()
  env:
    ELASTIC_URL: ${{ secrets.ELASTIC_URL }}
    ELASTIC_API_KEY: ${{ secrets.ELASTIC_API_KEY }}
  run: |
    jq -c --arg run "$GITHUB_RUN_ID" '.generated_at as $at | ({label: null} + .summary), ((.labels // {}) | to_entries[] | {label: .key} + .value) | {index: {}}, {run: $run, generated_at: $at} + .' results/tracecov-report.json \
      | curl -sf -X POST "$ELASTIC_URL/tracecov/_bulk" -H 'Content-Type: application/x-ndjson' \
          -H "Authorization: ApiKey $ELASTIC_API_KEY" --data-binary @- \
      | jq -e '.errors == false'

api-tests:
  script:
    - robot --exitonerror --listener "tracecov.robotframework.TraceCov;openapi.yaml" --outputdir results tests || status=$?
    - |
      jq -c --arg run "$CI_PIPELINE_ID" '.generated_at as $at | ({label: null} + .summary), ((.labels // {}) | to_entries[] | {label: .key} + .value) | {index: {}}, {run: $run, generated_at: $at} + .' results/tracecov-report.json \
        | curl -sf -X POST "$ELASTIC_URL/tracecov/_bulk" -H 'Content-Type: application/x-ndjson' \
            -H "Authorization: ApiKey $ELASTIC_API_KEY" --data-binary @- \
        | jq -e '.errors == false'
    - exit ${status:-0}
  artifacts:
    when: always
    paths:
      - results/tracecov-report.*

ELASTIC_URL and ELASTIC_API_KEY are CI secrets you define. Elasticsearch answers 200 even when it rejects documents, so the last jq fails the step when the response reports errors. On GitLab, the job keeps the test run's exit code and also fails when the upload fails.

Not captured

• Clients other than requests. The listener misses Browser library traffic and other HTTP clients. Record them with tracecov proxy.
• Sessions created before the first suite starts, for example in another listener's __init__.

Related

• Which Tests Cover What: labels in the pytest plugin
• Coverage Concepts: what each coverage state means
• JSON Report Format