> open source · PyPI package

σ-RAG: Significance-Threshold Retrieval

Standard RAG always returns the top-k chunks — even when none of them are relevant. σ-RAG borrows a technique from particle physics: estimate the background noise distribution, set a significance threshold, and refuse to answer when the evidence isn't there.

• Python
• NumPy
• sentence-transformers
• Statistics
• RAG
• Open Source

The Numbers

Why Top-k Retrieval Causes Hallucinations

Standard RAG has a fundamental flaw: it always returns the top-k chunks regardless of whether any of them are actually relevant. When you ask a question about pasta carbonara and your corpus contains only physics papers, the retriever faithfully returns the three least-irrelevant physics papers — and the LLM has no way to know they're background noise.

The result: the LLM receives irrelevant context, treats it as evidence, and generates a confident, detailed, completely fabricated answer. This is the hallucination problem that σ-RAG is designed to fix.

What Particle Physics Taught Me About RAG

During my PhD in particle physics, one of the first things you learn is that you don't declare a new particle discovery just because you see “the largest excess we've found today.” The ATLAS and CMS experiments at the LHC require a local significance of 5σ — meaning the probability that background processes alone could produce an excess at least as large is below 2.87 × 10⁻⁷.

The procedure has two steps that are always kept separate:

Standard RAG has neither step. σ-RAG applies the same logic to embedding space: estimate the background distribution of cosine similarities between unrelated document pairs, then gate retrieval on whether a chunk clears the significance threshold.

How the Noise Floor Works

The key insight: cosine similarities between random, unrelated document pairs form a distribution — the background. Sample enough cross-document pairs from your corpus and you get a Gaussian with mean μ and standard deviation σ. This is the baseline — what a background-level (irrelevant) match looks like in your particular embedding space.

threshold = μ_background + n·σ_background   # default n = 2

# At n=2:  false-alarm rate ≈ 2.3%
# At n=3:  false-alarm rate ≈ 0.13%
# At n=5:  false-alarm rate ≈ 2.9 × 10⁻⁷  (LHC discovery bar)

A chunk clears the bar only if its cosine similarity to the query exceeds the threshold. If zero chunks clear the bar, the pipeline returns a calibrated “no evidence” response and never calls the LLM. No background context → no hallucination opportunity.

The background is estimated from cross-document pairs only — pairs from the same document are naturally more similar and would inflate the noise floor. This is analogous to measuring background in a sideband rather than in the signal region itself.

Getting Started

pip install sigma-rag           # core (numpy only)
pip install "sigma-rag[local]"  # + sentence-transformers (recommended)

from sigma_rag import SigmaIndex, SigmaRAGPipeline

# 1. Build index
index = SigmaIndex()
index.add_documents(corpus_docs)
index.calibrate()   # fits the noise floor from cross-document pairs

# 2. Query — only statistically significant chunks reach the LLM
pipeline = SigmaRAGPipeline(index, n_sigma=2.0)

# Answerable query
response = pipeline.query("What significance level was required for the Higgs discovery?")
print(response.has_evidence)          # True
print(len(response.retrieval.significant))  # e.g. 2

# Unanswerable query — suppressed before the LLM is ever called
response = pipeline.query("What is the best carbonara recipe?")
print(response.has_evidence)          # False  ← hallucination prevented

Suppression vs. Coverage Tradeoff

The key question: as you raise the σ threshold, how quickly do you suppress out-of-domain queries, and does in-domain coverage hold up? The sweet spot is where both curves sit at 100% simultaneously.

How Much Context Reaches the LLM?

Top-k grows your LLM context linearly with k regardless of relevance. σ-RAG adapts — passing fewer chunks on in-domain queries (less noise) and zero on unanswerable ones.

Precision–Recall on In-Domain Queries

Varying σ traces a smooth precision-recall curve. Top-k collapses to a few discrete points. σ-RAG gives you a continuous operating point dial — tighten σ for higher precision, relax it for higher recall.

Package Design

Designed to be dependency-free at the core — only NumPy required. Heavier dependencies are optional extras that unlock progressively better embeddings.

Interesting Engineering Decisions

Choosing Your σ Level

The σ threshold directly controls the false-alarm rate — the probability a genuinely irrelevant chunk clears the bar. Lower σ = more recall, higher false-alarm rate. Higher σ = more precision, lower false-alarm rate.

Open Questions