Security Analysis | Sahithyan's Notes

Verification compares a system against its requirements and design. Analysis tools support it, and analysis can be manual, automated, or both. Relying only on manual techniques is usually too expensive.

Security evaluation applies the same assurance goal to a complete system and its evidence.

Analysis Fundamentals

Types of Analysis

Static analysis
Verifies software without running it. Covers source code scanning, code inspection, and type checking.
Dynamic analysis
Runs the software on specific inputs and checks results against an oracle. Covers functional testing, web app scanners, and fuzz testing.
Hybrid analysis
Combines static and dynamic analysis.

Measurement

Measurement Terminology

Treat a defect-detection tool as a classifier. For each code location it either raises a report or stays silent (positive or negative), and that call is either right or wrong (true or false). This gives 4 outcomes.

True positive
A report where a real defect exists. This is the intended result.
False positive
A report where no defect exists. It costs developer time and erodes trust in the tool.
False negative
Silence about a location with a real defect. This is the dangerous outcome, since the vulnerability ships unnoticed.
True negative
Silence about defect-free code. It is correct but not directly observable, since you do not know which locations the tool cleared.

Developers care most about false positives, which waste effort. Auditors care most about false negatives, which are missed defects.

Precision

The fraction of real defects out of all locations the tool reported. Probability a given report is worth acting on. Developers want this high.

\text{Precision} = \frac{TP}{TP + FP}

Recall

Also known as sensitivity, true positive rate. The fraction of real defects out of all real defects present. Probability a defect will not be missed. Auditors want this high.

\text{Recall} = \frac{TP}{TP + FN}

F1 Score

The harmonic mean of precision and recall. It is low unless both inputs are high, so a tool cannot score well by sacrificing one for the other.

\frac{1}{F_1} = \frac{1}{2} \times \left(\frac{1}{\text{Precision}} + \frac{1}{\text{Recall}}\right)

F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}

ROC Curve

Plots true positive rate against false positive rate as the reporting threshold varies. Catching more real defects also means more false alarms.

Formal methods aside, automated tools do not understand system architecture, mission, or environment. Most have significant false positive or false negative rates. They work best as part of a process, not as the only mechanism.

Tool Adoption

Adopting Tools

A tool is useless without understanding of the vulnerability types it reports and how to fix them. RealPlayer developers marked genuine Flawfinder findings as “ignore” instead of fixing them, and each became a CVE.

Rollout steps:

Define Objectives
Choose the vulnerability types to find, the codebases to scan, and a measure of success.
Start With a Soft Gate
Show reports to developers without blocking a merge or release.
Train Developers
Developers must understand each report, confirm whether it is genuine, and know how to fix it.
Run a Small Pilot
Use easily understood checks. Measure false positives and tune the tool’s rules.
Appoint a Champion
This person supports developers, triages disputed reports, and maintains the tool configuration.
Expand the Rollout
Make the gate mandatory only after the team trusts the reports and can resolve them consistently.