Skip to main content

Coverage

A source code audit report lists the security checks the audit performed, as well as the vulnerabilities it found.

While the agents analyze your code, they also work through a methodology checklist: a list of security checks a reviewer would perform on an application like yours. The report lists the checks on that list with the answer each one got, and a finding is only one of the answers a check can get.

Why this exists​

A report with no findings for a class of vulnerability could mean three different things: the class was checked and your code is clean, the class does not apply to your application, or the class was never examined. Coverage lets you tell these apart, because the report shows the absence of a finding and the absence of a test as different lines.

The four answers​

Your report has a Testing Methodology section. It has one row per check, and each row shows the answer that check got. A row can show one of these four answers.

AnswerWhat it means
TestedAt least one audit task examined your code for this check. Any findings appear in the findings section.
AnalyzedThe audit's code reading and architecture mapping answered this check. That work runs on every assessment.
Not applicableYour codebase does not contain the surface this check targets.
Requires runtimeThe check needs a running application, which a source code audit does not start.

In other words, each check in the section was examined directly, answered by the code reading the audit always does, out of scope because your application has nothing for it to test, or out of reach because source code alone cannot settle it.

The section starts with the totals for the four answers, so you can see the overall result before you read individual rows.

The report lists only the checks the audit settled. The API's methodology rows can carry a fifth status, not_completed, for a check whose audit work did not finish. The report leaves those checks out of its rows and its totals, so a check is never shown as tested when it was not.

Coverage of each entry point​

Besides the methodology checklist, the audit asks a question of every entry point it documents, once for each attack class that can apply to it: injection, authorization, session management, cross-site scripting and the other classes listed on the entry point. An auditor reads the entry point's code and answers each question with one of these:

  • The protection the class asks about is in place. The class counts as checked, and the entry point's details show the auditor's reason and the source lines that show the protection.
  • The protection is missing or wrong. The auditor raises a suspected issue, which becomes a finding once it is confirmed.
  • The class does not apply, because the entry point has nothing it is about, for example no session credential for session management. The entry point's details show the reason, with the cited lines when there are any.
  • The source code alone cannot answer the question. The audit then examines that entry point and class more deeply.

Where the audit asked a class as several named checks, such as an insecure direct object reference check under authorization, each check has its own answer. Open a class on an entry point to read them. On the Methodology tab, each such check shows how many entry points it was asked of, how many hold, how many do not apply and how many the source could not settle (undecided), and the findings where it fails. The question of each attack class is listed there too, with the answers it got.

Open an entry point under Documentation to read these answers. An audit that finished before auditors answered each question this way shows the answers it recorded.

Five methodology packs run on every audit​

Five methodology packs run on every code audit. Four cover web applications, mobile applications, infrastructure as code, and compiled binaries. The fifth holds checks AISafe wrote for defects that real audits found and the published guides do not ask about plainly, such as a regular expression that takes seconds on a crafted string.

AISafe does not ask you to choose between them, because real projects mix these kinds of code. A mobile client talks to a web backend, the backend deploys through Terraform, and somewhere in the tree there is a compiled component. If you had to pick one pack up front, you would be deciding what your application is before anyone had read it, and a wrong choice would leave a whole part of your application unexamined.

So every check runs, and the checks that do not fit your application are marked not applicable. They still appear in the report with that answer.

When a whole pack has nothing to test, the report shows its rows as one summary line with the counts. This keeps a backend report from filling up with a hundred inapplicable mobile rows, while the counts still account for every check in the pack.

What "not applicable" means​

It means the audit looked for the surface a check needs and found that your application does not have it. For an endpoint, the audit looks at everything its request runs, including the middleware and shared code it calls, not only the handler. For example, the repository contains no native code for a binary check to examine.

It is an answer about your application: AISafe gives it only after the audit has looked for the surface and found none.

Checks that code cannot answer​

Some checks depend on how a deployed system behaves rather than on what the source code says. No amount of careful reading can settle them.

The report marks such a check Requires runtime instead of leaving it out, so you can see where a code audit's reach ends. If those checks matter to you, follow the audit with a service that tests a live target: a Black-Box Pentest or a White-Box Pentest.

Regulated regimes are opt-in​

When you start an audit you can tick one or more regulated regimes: the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), or the Payment Card Industry Data Security Standard (PCI DSS).

Ticking a regime adds two things together: that regime's checks, and the ability to recognize the data those checks are about. For example, a check that asks where personal data travels comes together with the ability to recognize personal data in your code. You cannot select one without the other, and you would have no reason to, because neither half can answer anything on its own.

If you leave all three unticked, the audit still runs the five packs that are always on.

What a regulatory pack adds​

Ticking GDPR does not turn the run into "a GDPR audit". It adds checks tied to the technical articles of the regulation that a code audit can address:

  • security of processing
  • erasure of personal data
  • integrity and confidentiality
  • data protection by design and by default

It deliberately leaves out obligations that are met by documents rather than by code. Staff training, processor contracts, and breach-notification timelines are real duties under the regulation, but a code audit cannot read any of them. Your report does not mention them, and you should not read any result in it as covering them.

HIPAA and PCI DSS work the same way. Their checks address the technical controls that exist in your source code. Organizational and procedural controls are outside what reading code can reach, and the audit says nothing about them.

Attacks the audit refuted​

Artifacts has an Assurance tab. It lists the attacks the audit's agents tried against your code and found that the code stops, each with the reason it failed and the lines that show it. Opening an attack shows that reason in the agent's own words, the cited lines (which open in the Code explorer), and the endpoints, components and threats it was aimed at.

The tab opens on a heatmap of refuted attacks by component and vulnerability class; on a large codebase it opens on one row per scope, and a scope opens into its components. Selecting a cell, a row or a class lists those attacks, and you can search the attacks and their reasons.

Each attack is counted in the vulnerability class the agent gave it. A cell with a coloured ring is one where a finding of that class still stands, in the colour of its severity. Some audits show refuted attacks per component instead of by class. A refuted attack is evidence that one specific attempt failed. It is not a guarantee that no other attempt of that class could succeed.

What the scan read​

Before any check runs, the audit reads the source you attached. Images, documents and build output are not source code, so the audit leaves them out on purpose.

If the attached source contains no readable code at all, AISafe refuses the audit before it starts and before anything is charged. The message says what AISafe found instead, so you can attach the right branch or archive.

A repository of Terraform or other configuration is still a real audit target. AISafe analyzes configuration for infrastructure, dependency, and secret issues even though it contains no application code.

What this is not​

An audit is not an assessment of compliance, and AISafe is not an assessor.

What you get is narrower than a compliance assessment: findings mapped to control frameworks (see Compliance mapping), plus a record of which methodology checks the audit worked through and what answer each one got. You or your auditor can read that record, question it, and check it against the code.

The record is not a judgment about your application's standing under any standard, and nothing in it supports such a judgment. That decision belongs to your auditor, and the coverage record is one of the things they can read while they make it.

Standing rules​

Every audit also runs against a set of standing rules. These are guidance that our security team writes about how each class of weakness actually appears in source code. Each run records the rules assigned to its stages and the rules cited by the work it produced. These records help you review the audit's coverage.