The Hitchhiker's Guide to Program Analysis, Part III: Mostly Harmless LLMs

Figure 1 shows a warning that is easy to dismiss when viewed locally. The loop bound is checked against the size of items, and the write occurs immediately after the check. An LLM-based filter, or a human reviewer reading only the local snippet, can therefore give a convincing no-bug rationale: the guard at Line 17 appears to bound the loop, so the out-of-bounds warning looks like a false alarm.

The local rationale misses the type of the checked value. The value is not an int, but a 64-bit long field in a control object defined by ALSA (the Advanced Linux Sound Architecture), the Linux kernel's audio subsystem. A negative long can pass the guard and then be narrowed to a positive 32-bit int. For example, becomes after the narrowing conversion. Thus, the guard does not establish the condition needed to make the loop safe, namely n <= 16, and the write to items[i] can go out of bounds.

This example shows why a plausible local explanation is not enough to discharge a warning. The visible check and the reported write are adjacent, but the value's type and admissible range are determined by a framework object outside the local snippet. A no-bug decision must account for those facts over the attacker-controlled input range; otherwise it can miss executions that reach the reported error state.

The bug above is not beyond program analysis. A sufficiently precise analysis, e.g., a path-sensitive value analysis such as Frama-C/Eva (Cuoq et al., 2012), or symbolic execution such as KLEE (Cadar et al., 2008), can detect the reported error state once the relevant code is placed in an analyzable context. Locally, the required analysis is straightforward: keep the user-controlled value unconstrained within its program type, follow the guard, model the narrowing conversion, and check the resulting array bound.

The difficulty is bringing this reasoning to bear in a real Linux kernel driver. Running the analyzer directly on the whole driver module is not realistic: starting from driver registration or module initialization would force the analyzer to explore a large kernel-driven state space before reaching the warned sink. This is why symbolic-execution systems often rely on selected entry points, environment models, under-constrained execution, or selective exploration to make analysis tractable (Ramos and Engler, 2015; Chipounov et al., 2011). Nor does the warning itself provide the information needed to make the warned behavior analyzable. In our setting, a warning typically identifies only the bug location and a coarse source-to-sink taint flow.

A static analysis harness fills this gap by supplying the warning-specific context that the backend analyzer needs. It initializes the driver and framework state needed to reach the warned sink and then invokes a selected entry point, keeping warning-dependent values unconstrained within their program types. Such context could in principle be supplied by manually modeling each subsystem or by writing deterministic harness generators. Evident uses an LLM as a practical way to recover this context from driver and framework conventions that are costly to model manually across subsystems. The LLM therefore only helps construct the harness.

This creates a validation problem. As Figure 1 indicates, an LLM-generated harness may look plausible because it reconstructs a benign execution of the driver. By concretizing under-specified inputs or state according to normal use, it can overconstrain the analysis and remove executions that reach the reported error state. Evident therefore validates the generated harness against the reported warning before allowing it to enter backend analysis.

Even if the LLM is not asked to decide whether the warning is real, its harness can still make the warning disappear by changing the analyzed context. In the motivating example, a plausible harness might replace the attacker-controlled value with the range suggested by the local guard. Such a harness, as shown in Figure 1, could contain the following line, where Frama_C_interval(a,b) introduces an abstract integer value constrained to the interval for Frama-C/Eva:

This example shows that harness construction can fail even when the final decision is delegated to a formal backend. If the LLM fixes the warning-dependent input to [0,16], the backend is essentially analyzing a different program context in which the negative long values have already been removed. A no-bug result on such a harness would only show that the reported error state is unreachable after the input domain has been narrowed.

The needed check is concrete in this example: the tainted input reported by the detector must be initialized with an abstract value over its program type, not replaced by a benign concrete range chosen by the LLM. Evident therefore validates the harness before backend analysis and rejects harnesses that concretize or over-constrain the warning-dependent input.

Evident starts from the information reported by the front-end detector: where the tainted input comes from, where it reaches a warned operation, and what error state should be checked there. For harness construction, Evident splits this reported evidence into two parts: a value-flow witness that captures the warning-dependent computation to exercise, and a call witness that identifies candidate harness entries.

Value-flow Witness. A value-flow witness refines the detector-reported taint flow into the program points that harness construction should exercise. It records how a warning-dependent value moves through assignments, conversions, parameter passing, and uses on its way to the warned operation. The witness may be interprocedural; in the motivating warning, the tainted control object passes through the ALSA framework before a value read from it becomes the driver callback's loop bound. For harness construction, Evident views the witness as fragments aligned with the call witness. We write for the fragment inside the sink-side function , as illustrated in Table 2.

For the case in Figure 2, the local value-flow witness is:

Call witness. As shown in Table 2, a call witness records how that sink-side function is reached from an external or framework entry point. We write the call witness as , where is the outermost function reported on the path and is the sink-side function. In Figure 2, the is snd_ctl_ioctl, is snd_ctl_elem_write, and is msm_routing_put_.... is omitted for clarity; it is an ALSA framework routine.

The call witness also induces a set of candidate harness entry points. Rather than forcing the harness to start at , Evident considers suffixes of the call witness: . Each candidate chooses a different entry depth. Starting near preserves more framework behavior but requires more kernel state to be initialized. Starting closer to reduces the amount of framework state, but requires the harness to reconstruct the objects and assumptions that the skipped framework calls would have supplied.

The call witness is also not just a sequence of direct calls. In the real warning, snd_ctl_elem_write is an ALSA framework routine that receives the control value and dispatches to the driver callback through the control's put operation, effectively calling kctl->put(...). This indirect call is common in kernel frameworks. Therefore, the call witness tells Evident not only which functions are on the path, but also which framework object must be reconstructed and which operation binding must be made explicit in the harness.

Task. The harness constructor is the only component in Evident that uses an LLM. Given the warning input and entry candidates from the previous step, it produces a candidate C harness for one suffix . The harness constructs the arguments and state needed to invoke , keeps warning-dependent values unconstrained within their C types, and calls into so that the remaining call witness can reach the sink-side function .

For the motivating warning, this task is not just to call the driver callback. If the harness starts from the ALSA framework routine snd_ctl_elem_write, it must construct the control objects needed for the framework dispatch and bind the driver's put operation so that the indirect call reaches msm_routing_put_.... If the harness starts closer to the driver callback, it must instead reconstruct the objects that the framework would have supplied, including the kcontrol and ucontrol arguments. The entry candidates expose this tradeoff to the constructor.

Figure 3 shows a simplified harness sketch for the motivating warning using a suffix that starts at the sink-side callback. The real generated harness contains additional declarations, stubs, and initialization code, but the excerpt shows the key obligations: it constructs the framework-supplied arguments (e.g., shift at Line 22 in Figure 2), keeps the warning-dependent value unconstrained within its C type, and invokes the selected entry.

Type-preserving abstract-value initialization. For warning-dependent values, Evident uses type-preserving abstract-value initialization: the harness initializes the value with an abstract value whose domain is derived from the C type of the lvalue, rather than from a concrete range chosen by the LLM. In the paper presentation, this is written as a single assignment x = abstract_value(x), where is the lvalue being initialized and the abstract_value function is a harness primitive that introduces an Eva abstract value over the C type of . The LLM chooses where the abstract value should be introduced, but it does not choose the value's type or range.

This design is important for kernel code, where warning-dependent values often appear through typedefs, nested structures, and unions. In the motivating warning, the loop bound is read through a framework-defined union field whose underlying type is long, even though the value is later stored in an int. Asking the LLM to write backend-specific interval expressions would require it to choose both a type and a range, which is exactly where an over-constrained harness can remove attacker-controlled values. With abstract_value, the LLM only places the abstract-value initialization at the relevant program location; the range remains determined by the program type and backend semantics.

Code navigation. Harness construction follows the pattern of tool-augmented coding agents: the LLM is not expected to infer the relevant driver and framework structure from a single prompt. Evident provides the warning input, entry candidates, relevant snippets, and abstract-input requirements, and lets the LLM request source facts through code-navigation tools. In our implementation, a lightweight tool layer routes these requests to clangd, which provides language-server queries such as go-to-definition, references, callers, callees, and type information (8; 18).

Repair loop. Harness construction is iterative. After each proposal, Evident tries to build the harness and run the available checks. Failures are returned to the LLM as structured diagnostics, and the LLM is asked to repair the harness under the same warning input. Some failures are ordinary compile errors, such as missing declarations, incompatible pointer types, or invalid field accesses. Others are validation failures, such as failing to keep a warning-dependent value unconstrained within its program type or generating a harness whose detector-reported intermediate checkpoints are not reachable. We describe validation in more detail in §4.3.

If no candidate harness passes within the attempt budget, Evident marks the case unresolved.

Validation checks a generated harness against the warning input summarized in Table 2. It asks four questions: does the harness reach the reported sink ; does it keep the warning-dependent value from source unconstrained within its program type; does it execute the relevant fragment of the value-flow witness ; and does it avoid fixing other state that can affect the reported error state to unjustified constants?

Abstract-input audit. The first validation check is tied to the warning-dependent source and the values derived from it. Evident audits assignments in the generated harness to catch concrete or overly narrow initialization of warning-dependent state. Values identified by the warning input must be introduced through the type-preserving form abstract_value. The audit also checks for aggregate initialization that silently fixes warning-relevant state, such as zeroing an object that contains a tainted value or a state field that controls whether the tainted value reaches .

The audit distinguishes structural wiring from value restriction. A harness may need concrete assignments to allocate storage, connect pointer fields, or install callback tables. For example, the motivating harness may bind an ALSA control's put operation so that the framework dispatch reaches the driver callback. By contrast, numeric or state values that influence must remain unconstrained within their program types unless the warning input or trusted setup code justifies a narrower constraint.

Target reachability and dead-sink. The second validation check is tied to the reported sink . Evident inserts Frama-C logging probes at and at selected earlier hook points on the chosen entry path, then runs a lightweight Eva pass. A probe is observed only when Eva propagates a non-bottom abstract state to the instrumented statement.

Reachability of is the normal acceptance condition. It shows that the harness has constructed enough entry context for analysis to arrive at the reported location. Missing reachability is treated as a harness failure by default, because it often reflects incomplete framework wiring rather than a genuine dead sink. In the motivating warning (Line 22 in Figure 2), the sink-side function reads through kcontrol->private_value ; unless the harness constructs kcontrol and initializes that field (Line 7-8 in Figure 3) , Eva can lose the abstract state before the reported sink.

Evident accepts an unreachable reported sink only under a stricter dead-sink condition. First, an earlier hook in the sink-side function or on the selected entry path must be reached, so the harness is not simply dead. Second, the reported sink itself must remain unreachable. Third, the lightweight run must report no alarms before the sink. A dead-sink result is accepted only when the sink-side context is reached, the target branch is unreachable, and no earlier alarm explains the loss of reachability.

Checkpoint reachability. The third check is tied to the value-flow witness . Reaching the reported sink is not enough: a harness may reach the same program location without exercising the warning-dependent value reported by the detector. Evident therefore instruments selected checkpoints from the portion of relevant to the chosen entry candidate and checks whether they are reachable with non-bottom abstract state.

These checkpoints may include the load of the warning-dependent value, a narrowing conversion, or an assignment to a loop-bound variable. The check does not prove the detector's taint relation or the full source-to-sink flow. It only serves as an admission check: an accepted harness must exercise the warning-relevant fragment of the reported witness, rather than merely reach through an unrelated context.

Dependency audit. The final check considers state beyond the reported source . Such state may determine whether the reported sink is reached, whether the warning-dependent value reaches , or what range it has when the reported error state is checked. For example, a guard such as if (g) x &= 0xff can sanitize a tainted value depending on the value of g. If g is a global variable and Eva starts the analysis with default zero-initialized globals, the backend may miss behaviors that remain possible in the original program.

Evident therefore runs a dependency analysis by Frama-C/From on the harnessed program and audits state that can influence reachability of or the check of . When such state is not fixed by trusted setup code, the harness must expose it through type-preserving abstract-value initialization. This check is especially important for dead-sink results: a proof that is unreachable must not depend on model-supplied concrete values for guards or global state. It must follow from the code and build configuration under unconstrained warning-relevant state.

After a harness passes validation, Evident analyzes the accepted harness with Frama-C/Eva. The backend checks the reported error state under the program context encoded by . Evident reports no bug only when the backend shows unreachable on an accepted harness. Harness construction, LLM explanations, validation success, alarms, unknown results, timeouts, and analysis failures are not evidence for no bug.

We do not use Frama-C/Eva's -lib-entry mode as the main way to analyze driver entries. Although -lib-entry can abstract globals at analysis entry, drivers often contain large static object graphs and recursive objects that prevent analysis from reaching the warning-specific code. RQ4 evaluates a rule-guided variant that relies more heavily on this style of direct entry analysis; it leaves many cases unresolved.

Evident may analyze an accepted harness under multiple Frama-C/Eva precision settings. In our implementation, the lightweight run uses Eva precision level 1, while the full run uses level 11. The lightweight run is cheaper and can quickly discharge warnings whose reported error state is already unreachable; the full run spends more analysis effort to reduce alarms caused by imprecision. These settings change precision and cost, not the decision rule: a no-bug result requires some accepted backend run to show unreachable. If accepted runs only alarm, return unknown, time out, or fail, the warning remains still alarmed or unresolved as defined in §3.

Dataset. We evaluate on 200 warnings drawn from the BugLens dataset (Li et al., 2025a), which includes Android Linux kernel driver warnings reported by two existing static detectors: SUTURE (Zhang et al., 2021) and CodeQL-OOB, a CodeQL-based out-of-bounds detector (Backhouse, 2018). The dataset contains 126 SUTURE warnings and 74 CodeQL-OOB warnings. RQ1 uses all 200 warnings. RQ2–RQ4 use the 126 SUTURE warnings, which are the subset used for model-sensitivity, comparison, and design-variant experiments.

For the SUTURE subset, we use the two main driver modules in the dataset, sound and i2c, because prior BugLens results show that these two modules cover all confirmed true bugs in that subset. The SUTURE benchmark cases, msm-sound and msm-i2c, are evaluated on the Android 10 Qualcomm Pixel kernel android-msm-coral-4.14-android10 (android-10.0.0_r0.56), Linux 4.14.150. The CodeQL-OOB cases use a separate Qualcomm MSM kernel tree with Linux 4.4.21-perf+.

Ground truth. Ground-truth labels were established from the labels and case analyses reported in the SUTURE and BugLens papers, followed by additional manual validation by the authors. In total, the ground-truth set contains 40 real bugs and 160 non-bugs. We treat these labels as the best available reference for this dataset, while noting that they are derived from prior reports and manual review rather than from exhaustive formal specifications.

Metrics. We report correctness against these labels. Correct results (Cor.) are the sum of true positives and true negatives: real bugs that remain alarmed, and non-bugs that are discharged as no-bug. False positives are non-bug warnings that Evident leaves alarmed, and false negatives are real bugs that Evident incorrectly discharges as no-bug. Accuracy is , where includes unresolved cases. Unresolved cases are reported separately and do not count as correct, even though they do not discharge the warning.

Baselines. The first baseline directly asks Codex, instantiated with GPT-5.5 with the xhigh reasoning setting, to classify each warning given its report. Codex operates as a fully agentic baseline: it is given read-only access to the entire target kernel tree and may freely explore additional code context using its built-in shell and code-search capabilities (e.g., rg, cat). We disable web search, browser access, and all external tools, so its analysis relies solely on the model and the source code; it has no access to ground-truth labels or any intermediate results of Evident. The second family of baselines uses the BugLens workflow, which performs its own repository-level knowledge retrieval to gather code context. We evaluate both the original BugLens setting with o3-mini, using results reported in the BugLens paper, and a GPT-5.5 variant to keep the model consistent with the rest of our experiments.

Implementation. In the main experiment, Evident uses GPT-5.5 (gpt-5_5-2026-04-23) for harness scoping and Frama-C 32.0 (Germanium) for formal analysis, through the Eva and From plug-ins. The experiments were conducted on a machine running Ubuntu 22.04.5 LTS (Jammy), equipped with an Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz and 64 GB of RAM. The OCaml version was 4.14.1. Each case is analyzed under a fixed per-stage timeout, with Eva runs capped at 15 minutes.

Cost and runtime. Cost is not a primary evaluation target, but Evident's LLM use is modest. With GPT-5.5, harness generation consumes about 27K uncached input tokens, 42K cached input tokens, and 11K output tokens per case attempt, costing about $0.50. This is well below the agentic Codex baseline in our runs ($1.20/case). BugLens consumes a similar token usage to Evident; its cost is thus roughly equal under GPT-5.5, and lower in its original o3-mini configuration due to cheaper model pricing. Runtime is dominated by Frama-C/Eva: quick analysis times out at 300s and full analysis at 900s. Serial execution averages about 10 minutes per case, but cases are independent and parallelize naturally across workers.

Evident reaches a verdict on 179 of 200 warnings. Across all warnings, 151 results are correct (true positives or true negatives), for an accuracy of 75.5%. The 28 incorrect verdicts are false positives: non-bug warnings that Evident conservatively leaves undischarged. Evident produces no false negatives in the main experiment.

Residual cases. Table 4 breaks down the 21 unresolved cases and 28 false positives. Unresolved cases expose support boundaries in the current implementation. They arise from unsupported C or analysis patterns, missing models for kernel framework objects or helper APIs, validation failures after the repair budget is exhausted, or final-analysis timeout. Missing models occur when the harness cannot reconstruct required kernel state, such as data structures in the kernel whose effects are needed to reach or analyze the reported sink.

The 28 false positives are residual proof failures rather than missed bugs. In 16 cases, the backend lacks non-local relations needed to prove the warning infeasible, such as relations between a pointer and its length field, a payload cursor and its grammar, an enum index and table metadata, or a driver-specific numeric value and its protocol bound. In the remaining 12 cases, the harness or environment model leaves framework objects, global state, helper effects, or initialization-dependent fields too abstract. Eva therefore observes behaviors that are feasible in the abstract state but infeasible under the intended kernel or driver execution.

Overall, the residual cases show the empirical error profile in this dataset: all incorrect verdicts are false positives, and no false negatives are observed. For no-bug decisions, this means that Evident's failures are conservative in this experiment: unsupported or imprecise cases remain alarmed or unresolved rather than being discharged. The remaining errors point to missing kernel-framework models and missing non-local semantic relations, rather than to LLM's failures.

Table 5 evaluates Evident with LLMs of varying capability on the same 126-warning subset. It shows that model choice affects both correctness and coverage. GPT-5.5 obtains the highest accuracy, 0.75, and leaves the fewest cases unresolved. Lower-capability models produce fewer correct verdicts and leave more cases unresolved, including some real bugs. Across all models, however, these failures do not become incorrect no-bug results: every configuration has zero false negatives on this dataset.

The performance of GLM-5.1 (37), a leading open model, is also notable: despite using only 40B active parameters, it remains close to the closed models in this comparison. It achieves 0.66 accuracy and produces no false negatives, although it leaves more cases unresolved than GPT-5.5. This suggests that Evident is not tied to a single frontier proprietary model. A capable open model can construct enough accepted harnesses to recover many backend-supported verdicts.

The unresolved cases from weaker models mostly fail before final bug classification in two ways. Some fail during harness construction. In our implementation, the LLM first gathers code facts and then emits a single C harness; weaker models sometimes discover late that required definitions or object relationships are still missing, continue requesting lookup tools, and exhaust the attempt budget before producing a usable harness. Other cases produce a harness but fail admission validation. These failures come from invented local models, ill-typed object layouts, or incomplete reconstruction of the environment state needed to reach the reported bug location. Evident classifies both kinds of failures as unresolved rather than no-bug results.

These results highlight a robustness property of Evident's decision principle. A no-bug result is reported only when the harness is accepted by validation and the backend shows the reported error state unreachable. As a result, lower-capability models mainly reduce coverage by producing more failed or unresolved harnesses. They do not relax the backend-decision rule used to discharge warnings.

Table 6 compares Evident with direct LLM-based filtering on the same 126-warning subset. For GPT-5.5 and o3-mini, we use the BugLens scaffold rather than a bare prompt (Li et al., 2025a). These numbers are not meant to reproduce the BugLens paper, because our task uses a different ground truth: we evaluate bug reachability, not security impact. A reachable signed overflow, out-of-bounds access, or invalid pointer dereference counts as a bug even if an LLM judges it unlikely to be exploitable. To make the comparison aligned with our task, we remove BugLens's security-impact accessor and compare only the warning classification under our reachability-based ground truth.

The results show that LLM-only filtering can miss real bugs even with strong models and structured scaffolds. GPT-5.5 with the BugLens scaffold and the Codex agent both identify the motivating narrowing bug from Section 2, but still produce 14 and 18 FNs respectively; o3-mini produces 17 FNs. Many missed bugs follow a similar pattern: the model accepts a plausible framework or driver check as sufficient sanitization, while the actual code makes the check conditional, version-dependent, incomplete, or state-dependent. Prior LLM-filtering experiments also report sanitizer-overtrust failures (Li et al., 2025a).

The comparison highlights the consequence of where the final verdict is made. The model-decided baselines produce 14–18 false negatives on this subset: real bugs are sometimes dismissed as no-bug. We do not attribute these failures to a single cause, since they can arise from missing context, incorrect reasoning about guards, or version-specific framework assumptions. The important difference is procedural. In Evident, an LLM-generated rationale is never enough to discharge a warning. A no-bug result requires an accepted harness and a backend result showing the reported error state unreachable. On the same subset, Evident produces zero false negatives; its residual failures are 25 false positives and 7 unresolved cases.

Table 7 evaluates design variants on the 126-warning subset. The variants test three parts of Evident's division of responsibility: how the harness is constructed, how warning-dependent values are represented, and whether generated harnesses are validated before backend analysis.

Rule-guided harnesses. This variant replaces Evident's LLM-guided context construction with a more rigid setup. The analysis provides the entry pattern and wiring rules, and the LLM only emits harness code consistent with them. Warning-dependent state is not introduced through Evident's abstract_value(x) interface, but is instead delegated to the backend's global abstraction at analysis entry. The variant is conservative but incomplete: it introduces no false negatives, but leaves 80 of 126 cases unresolved and finds only 18 of the 39 real bugs. Many failures arise before a useful backend result is obtained, because the rule-guided setup does not reconstruct enough kernel- and framework-managed state to reach the relevant callbacks or reported locations. This is the same obstacle that makes direct target analysis difficult in the first place.

LLM-decided input. This variant removes both the type-preserving abstract-value interface and the validation gate. It uses a simpler harness-construction prompt in which the LLM must choose how to initialize warning-dependent inputs, without the abstract_value(x) primitive used by Evident. The generated harness may therefore use concrete values, intervals, or backend-specific abstract expressions chosen by the model. This variant produces 7 false negatives and 14 unresolved cases. The false negatives occur when the generated harness restricts warning-dependent inputs too narrowly, allowing the backend to show absence under a model-chosen input space rather than under the C types of the warning-dependent lvalues.

Abstract values without validation. This variant isolates the effect of validation more directly. It keeps the same harness-construction scaffold as Evident, including the type-preserving abstract_value(x) interface, but disables the validation checks before backend analysis. It obtains the highest aggregate accuracy in Table 7, 0.77, but still introduces 6 false negatives. These false negatives arise because an accepted-looking backend result can be produced on a harness that fails to exercise the reported warning. The full system rejects such harnesses before backend analysis; without validation, the backend may show absence only within an inadmissible generated context.

For reference, Table 7 also includes the best model-decided baseline from RQ3. It produces a label for every case, but incurs 17 false negatives. Together, the variants show that aggregate accuracy is not the only relevant measure for warning discharge. The simpler LLM-decided setup can narrow the checked input space; removing validation from the full scaffold can admit harnesses that do not exercise the reported warning; and rule-guided harness construction is too incomplete. The full configuration gives up a small amount of aggregate accuracy relative to the no-validation variant, but it eliminates the false negatives observed in the design variants while retaining much higher coverage than rule-guided harnessing.

To understand how Evident differs from model-decided triage in practice, we examine four representative cases from our evaluation. These cases are not intended as capability limits for LLM-based agents. A more elaborate agent may avoid some of the same mistakes by retrieving more code, adding preprocessing, using stronger prompts, or majority voting.

The cases instead illustrate a structural difficulty of using the model for bug triage. Some program fragments cannot be judged from the local snippet alone; the answer may turn on a specific fact outside the snippet, such as how an object is constructed or how a function is called. Contemporary agents can typically retrieve any code snippets they want, but it has no mechanism that enforces any specific code to be inspected. Furthermore, even if the right context is retrieved, it may still not be what drives the answer. Thus a no-bug decision can look the same whether it is supported by the program or produced by a plausible shortcut.

Evident avoids this difficulty by moving the decisive part of triage out of the model’s internal code reasoning. The final evidence no longer depends on whether an unobservable reasoning step actually happened, whether the right context was used, or whether a plausible explanation merely masked a guess.

Nested types. The motivating example shows that the exact value type is important, but that type may not be visible at the local expression. More generally: resolving types like snd_ctl_elem_value::id::index requires two dependent lookups, first into snd_ctl_elem_value and then into the embedded snd_ctl_elem_id. A field access through nested structures requires such lookups in such a chain.

In practice, we often observe that models did not strictly follow such lookup chains. They would stop after one or two levels and form a conclusion from partial type information. The conclusion is sometimes correct, but it does not reveal whether the model actually resolved the relevant type or relied on a shortcut that happened to work.

Evident avoids making this nested type resolution part of the model's internal reasoning. Warning-dependent values are introduced as abstract values through their actual lvalue types, so the compiler resolves the full definition chain against the analyzed tree. The decisive fact is therefore checked outside the model rather than inferred from a partial inspection of the code.

Code-as-seen is not code-as-run. The code visible to the model is not always the code that executes. In one case, the LLM-only baseline reasoned about a target guarded by a complex compound condition, ~100 lines below the guard in a ~400-line function, apparently requiring nontrivial reasoning about framework versions and hardware support. In the analyzed build, however, one conjunct was compiled to a constant:

With CONFIG_USE_Q6_32CH_SUPPORT disabled, the branch cannot execute regardless of the remaining conjuncts. The deciding fact is not in the function the model reads, so the baseline treats the dead branch as live code and reaches the wrong conclusion. Although preprocessing would remove this particular mismatch, it only resolves configuration choices expressed at the preprocessor level. It cannot eliminate branches that are dead if they are in the ordinary C logic, such as helper function or driver state, where the deciding fact is in program semantics rather than in conditional compilation. Evident sidesteps the gap rather than asking the model to bridge it: the harness is compiled against the analyzed build configuration, so the code the backend analyzes is the code that actually executes under the given harness. The deciding fact never passes through the model.

Memory contamination from nearby software versions. Prior work raises the concern that models may have seen target code or patches during training (Ramos et al., 2025; Wu et al., 2024), which gives the models an unfair advantage. We observe the opposite, failure mode in model-decided triage: the model remembers plausible facts from a different Linux kernel version. Later Linux kernels added ALSA-core input validation for control API values (Iwai, 2022); in some LLM-only runs on older Android kernels, the model reasoned as if such framework-level sanitization was already present, concluding that user-controlled indices had been canonicalized before reaching the driver callback. We do not claim a causal link to training exposure; the point is that parametric memory can bias the model toward version-mismatched framework behavior, and prompting for source evidence cannot guarantee that remembered facts are suppressed or distinguished from the analyzed version. Evident forces such assumptions to materialize: a remembered sanitizer must appear either as code, which must exist in the analyzed tree to compile and to reach the validation checkpoints, or as a constraint on warning-dependent inputs, which validation rejects outright. A version-mismatched belief therefore cannot remain an implicit assumption in the analysis.

Inherent nondeterminism. Feeding the model the same warning does not yield the same triage decision. In our inspection, the same code was plausibly analyzed in opposite directions across runs: one response discarded the warning, another declined to after emphasizing a different part of the same context. This instability is known in LLM-assisted triage: BugLens stabilizes judgments by majority voting, while LLift reports that repeated runs can produce different outcomes because of model randomness (Li et al., 2025a, 2024). Evident does not remove this nondeterminism but limits its effect: a wrong or incomplete harness leaves the case unresolved instead of an unsound discard, and an accepted no-bug decision must be backed by the full evidence chain. However in model-decided triage, the same randomness acts directly on the verdict.