How Far Is an AI Claim From the Evidence? — Evidence Distance

Question

Two AI answers can both cite sources and still differ greatly in how directly those sources support the claims being made. How can we describe that difference?

This article uses the term Evidence Distance for the distance between a claim and the directly inspectable evidence that supports it.

Evidence Distance is the number and strength of transformations, assumptions and inferential steps separating a claim from the evidence that can directly verify it.— Working definition — a reasoning concept introduced here, not a standardized metric.

What This Really Means

A citation proves that a source was referenced. It does not automatically prove that the source directly supports the sentence beside the citation.

Between an original observation and a final AI answer there may be extraction, parsing, chunking, retrieval, summarization, interpretation, aggregation and inference.

Primary evidence
   ↓
Extracted data
   ↓
Retrieved passage
   ↓
Summary
   ↓
Interpretation
   ↓
Inference
   ↓
Final claim

Every step can be legitimate. But every step can also introduce distortion.

Simplest Example

Suppose a temperature sensor reports 38.2 °C.

The original sensor value directly supports the first claim. It does not by itself prove the third.

Evidence exists
        ≠
Evidence directly supports every derived claim

Where the Example Stops Working

Evidence Distance is not simply a count of sentences or processing steps.

One mathematically valid transformation may preserve evidence almost perfectly, while a single speculative interpretation may create a much larger epistemic jump.

The concept therefore includes both distance by transformation and distance by inference.

Direct Answer

Evidence Distance increases when a claim depends on more transformations, more assumptions, weaker provenance or stronger inference than the underlying evidence directly warrants.

Short distance:
source → claim

Longer distance:
source → extraction → summary → assumption → inference → claim

A longer distance does not automatically make a claim false. It means the reasoning chain requires more validation.

Why This Matters for AI

LLMs are particularly good at producing fluent bridges between pieces of information. That capability is useful, but fluency can hide how far the conclusion has moved beyond the directly retrieved evidence.

An answer can therefore look well sourced while containing claims whose support exists only through several unstated inferential steps.

Four Components of Evidence Distance

1. Transformation Distance

How many times has the original evidence been converted, summarized, normalized or reformatted before reaching the model?

2. Inferential Distance

How many reasoning steps are required before the evidence supports the final conclusion?

3. Provenance Distance

Can the retrieved statement still be traced to an identifiable original source, or has it passed through summaries and copies whose origin is unclear?

4. Applicability Distance

How much reasoning is required to establish that the evidence applies to this exact version, user, jurisdiction, time period or situation?

Evidence Distance in RAG

RAG can shorten one important distance: the distance between a model and external knowledge.

It does not automatically shorten the distance between a retrieved passage and the final claim.

RAG solves:
model ↔ external information

RAG does not automatically solve:
evidence ↔ justified conclusion

Connection to the Retrieval Trigger

The Retrieval Trigger asks whether internal model knowledge is sufficient.

Evidence Distance asks a later question: after evidence has been retrieved, how directly does it support what the system wants to claim?

Need evidence?
   ↓
Retrieval Trigger
   ↓
Retrieve
   ↓
How directly does evidence support the claim?
   ↓
Evidence Distance

A Practical Diagnostic

For an important AI-generated claim, reconstruct the support chain backwards.

CLAIM
  ↑
Which inference produced it?
  ↑
Which retrieved evidence supports that inference?
  ↑
Which source produced that evidence?
  ↑
Can the original source still be inspected?

If the chain breaks, the system has encountered an evidence-traceability problem.

If the chain remains intact but requires several strong assumptions, the Evidence Distance is still high.

Example: Product Data

Suppose a manufacturer specification says a laptop weighs 1.48 kg.

Manufacturer specification
→ 1.48 kg
→ "The laptop weighs 1.48 kg"

Low Evidence Distance

Now consider:

Manufacturer specification
→ 1.48 kg
→ comparison with three competitors
→ assumption about travel behavior
→ "This is the best laptop for frequent travelers"

Much higher Evidence Distance

The specification is real. The recommendation is still several reasoning steps away from it.

Example: Scientific Evidence

A paper may report an observed association. An AI answer can incorrectly compress that into a causal statement.

Observed association
        ↓
Interpretation
        ↓
Unstated causal assumption
        ↓
"X causes Y"

The problem is not that no evidence exists. The problem is that the final statement crosses a larger evidential distance than the study directly supports.

Connection to Agent-Ready Context

Structured information objects can reduce avoidable Evidence Distance by preserving source identity, units, timestamps, conflicts and validity instead of forcing the model to reconstruct them from flattened text.

A Simple Working Scale

This scale is illustrative, not a scientific standard. Its purpose is to force an explicit question: How much did we have to add between the evidence and the claim?

Misconceptions

A citation means Evidence Distance is low. No. A citation can sit beside a claim that substantially exceeds what the cited source establishes.

Long Evidence Distance means the claim is wrong. No. Complex conclusions can be well supported, but they require a stronger reasoning chain.

Primary sources eliminate inference. No. Primary evidence can still require interpretation.

More sources automatically reduce distance. Ten weak repetitions of the same unsupported claim do not create direct evidence.

Edge Case: Derived Data

Some derived values are more reliable than raw human descriptions. A correctly calculated total from verified transactions can be highly dependable even though it is one transformation away from the original records.

Evidence Distance therefore measures epistemic separation, not a simplistic preference for untouched data.

Limitations

Evidence Distance is a conceptual diagnostic, not a universal quantitative score.

Different domains require different treatment of inference. Mathematical proof, clinical evidence, software logs, market analysis and historical interpretation cannot share one universal numerical threshold.

What Would Change This Answer?

Future AI systems may generate explicit provenance graphs, machine-checkable claim-evidence links and typed reasoning traces. That could make parts of Evidence Distance measurable automatically.

The underlying problem remains: a claim can move farther from its evidence even while every intermediate sentence sounds plausible.

Conclusion

The important distinction is not merely between sourced and unsourced answers.

It is between claims that remain close to directly inspectable evidence and claims that depend on increasingly long chains of interpretation and assumption.

Evidence
   ↓
Evidence Distance
   ↓
Reasoning
   ↓
Answer Validity Boundary

Evidence Distance gives us a language for describing that space.

Primary Sources

Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020).

Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023).

W3C, PROV Overview. A standardized model for representing provenance relationships.