Skip to content

100% endpoint coverage, 56% of parameters: what TraceCov found in Schemathesis

Schemathesis called every operation in GitLab's API schema. TraceCov showed that 562 of the schema's form parameters were never sent as form fields.

GitLab v3 schema, Schemathesis coverage phase 4.18.0 4.18.5
Operations reached 358 / 358 (100%) 358 / 358 (100%)
Parameters exercised 729 / 1,291 (56.47%) 1,291 / 1,291 (100%)
Schema keywords exercised 1,130 / 1,790 (63.13%) 1,789 / 1,790 (99.94%)

"Exercised" means at least one request hit the parameter or keyword with a valid or an invalid value. One fix accounts for the whole difference.

The symptom

Schemathesis generates API tests from OpenAPI schemas. Its coverage phase builds a deterministic set of requests per operation: boundary values, enum members, missing required fields, wrong types. On May 8, 2026, we started recording those requests with TraceCov across the Schemathesis schema corpus.

On gitlab.com/v3.json, a 358-operation Swagger 2.0 schema, every operation received requests, yet TraceCov listed 562 uncovered parameters, all with the same location:

{
  "kind": "parameter_uncovered",
  "method": "PUT",
  "path": "/api/v3/application/settings",
  "location": "form_data",
  "parameter": "admin_notification_email"
}

No query, path, or header parameter was missing. The gap list held nothing else apart from six keywords that lacked only a valid or only an invalid value. TraceCov does not list keyword gaps under a parameter that was never hit, so all 660 missed keywords belong to these 562 form fields.

Were the values generated?

Schemathesis did generate the fields. Every case contained them:

{
    'admin_notification_email': '',
    'after_sign_out_path': '',
    ...
}

Schemathesis 4.18.0 sent them as JSON:

PUT /api/v3/application/settings
Content-Type: application/json

{"admin_notification_email": "", ...}

The schema declares these fields as form fields, and TraceCov matches formData parameters against form-encoded and multipart bodies only. The requests reached the endpoint and counted toward endpoint coverage, but Schemathesis sent none of the 562 fields as form fields.

The cause: a schema that contradicts itself

GitLab's schema declares formData parameters on 117 operations. On 113 of them, the operation also says it accepts only JSON:

put:
  consumes: ["application/json"]
  parameters:
    - name: admin_notification_email
      in: formData
      type: string

The other four operations declare no consumes or list multipart/form-data. Schemathesis sent their four fields as form fields, and TraceCov counted them.

The Swagger 2.0 specification allows formData only when consumes lists application/x-www-form-urlencoded or multipart/form-data, so the two declarations conflict. Schemathesis followed consumes, encoded the fields as JSON, without a warning, and every endpoint showed as covered.

The contradiction is rare. Of the 2,163 Swagger 2.0 schemas in the Schemathesis corpus, two contain it: GitLab with 562 affected parameters, and one other schema with a single parameter. That is why this page uses GitLab.

Which declaration wins, and what it costs

Following consumes was a defensible choice, and on the real GitLab it probably did no harm. The schema looks generated by grape-swagger, and Grape parses JSON, form-encoded, and multipart bodies into the same params by default. We did not send these requests to a GitLab server, so this is unverified.

Schemathesis still picked the parameter location over consumes, for two reasons:

  • in: formData is the more specific declaration. It is attached to each field, while consumes applies to the whole operation. In this schema it looks like a generator default: all 148 operations that declare consumes start with application/json.
  • Only form encoding tests the fields as declared. A server that parses form bodies only would see none of the 562 fields in a JSON body, and the test run would still pass.

The cost falls on servers that accept JSON only. They now receive form bodies on these operations and may reject them with 400 or 415. Such a response still exercises the endpoint, but the declared field constraints are no longer tested against a body the server parses. The right fix in that case is the schema: change in: formData to a body parameter.

A schema linter catches this contradiction before any test runs. It misses the other ways requests lose data, listed below, because those live in the test code.

The fix

Schemathesis 4.18.5, released May 13, 2026, sends formData parameters as a form payload when consumes declares only non-form media types (aad6f61d). It uses multipart/form-data when a field has type: file, and application/x-www-form-urlencoded otherwise:

PUT /api/v3/application/settings
Content-Type: application/x-www-form-urlencoded

admin_notification_email=&after_sign_out_path=&...

Measured revision by revision:

Schemathesis revision Parameters Keywords
d1b7b2b61 (4.18.0) 729 / 1,291 1,130 / 1,790
4.18.1 729 / 1,291 1,130 / 1,790
93dc5d747 (parent of the fix) 729 / 1,291 1,130 / 1,790
aad6f61d2 (the fix) 1,291 / 1,291 1,789 / 1,790
4.18.5 1,291 / 1,291 1,789 / 1,790

The same audit produced the coverage-phase fixes listed in the 4.18.1 changelog. On this schema they changed nothing.

What this means for your test suite

Endpoint coverage counts requests that reach an operation. It cannot tell whether the request carried the data the operation declares. The encoding here was wrong, but requests lose data in other ways that also leave endpoint coverage at 100%:

  • A client helper drops null or empty fields, so optional parameters never reach the server. Parameter coverage shows them as uncovered.
  • A fixture always sends the same value for an enum or a bounded number. Keyword coverage shows the untested enum members and bounds.
  • A body goes out as JSON where the API expects multipart. Parameter coverage shows the fields as uncovered, as it did here.

TraceCov measures at the level the schema describes: each parameter, and each keyword on it. To see what your own tests send, start with the Quick Start. If you already run Schemathesis, its Docker images ship with TraceCov installed.

Methodology

We ran the Schemathesis coverage phase (positive and negative modes) on gitlab.com/v3.json from the Schemathesis corpus at each of the five revisions above. Every generated request went to TraceCov 0.24.2 without being sent to a server, so response coverage is not measured. All runs used the same Python environment, and the coverage phase is deterministic, so no seed is involved. No run produced errors.

At 4.18.5, one keyword is never exercised, three lack a valid value, and four lack an invalid one.