Medical RAG Evaluation

Does retrieval help a language model answer biomedical research questions, and does the model admit it when the evidence is not there? A retrieval-augmented system is usually judged on whether it finds the right document. This asks a narrower question: whether it finds the paragraph that holds the answer, and what the model does when that paragraph is missing.

PubMedQA: 1,000 research questions written from PubMed abstracts, each labelled yes, no or maybe by experts, with the abstract minus its conclusion as the evidence. Every number comes from the official 500-question test split, with a 95% bootstrap confidence interval. The other 500 questions were the development set, used while building the pipeline; nothing was tuned on test.

Each abstract is split into its labelled sections (background, objective, methods, results and so on), and each section is one retrievable passage, the usual way RAG systems chunk text. Retrievers: BM25, TF-IDF, LSA with 256 dimensions, BM25 and LSA fused by reciprocal rank, and two sentence-transformer models, all-MiniLM-L6-v2 and PubMedBERT embeddings. A harder index adds 60,000 unlabelled PubMedQA abstracts as distractors.

Answers come from one model, gemini-3.8-flash, at temperature 0, on the same 500 questions under five conditions.

Condition	Evidence given	Tests
closed	none	what the model knows on its own
passages	top 3 retrieved passages (BM25)	standard RAG
parent	every section of the top-ranked abstract	retrieve by passage, read the whole document
oracle	every section of the question's own abstract	upper bound
removed	top 3 passages after deleting the question's own abstract from the index	the evidence does not exist: the right answer is "insufficient"

The model must answer from the evidence only, cite the evidence it used, and say "insufficient" when the evidence is about a different study. Output must be JSON with an answer from a fixed set; anything else is counted as invalid and wrong, never coerced into an answer. The same instruction is used for every evidence condition, so differences come from the evidence, not the prompt.

Finding the paper is easy. Finding the evidence in it is not. BM25 puts the question's own abstract in the top 5 passages for 97.0% [95.4, 98.4] of test questions, and in the top 1 for 92.4%. Its results section, where the answer actually is, makes the top 5 only 51.8% [47.3, 56.4] of the time and the top 1 for 7.6%. The question is phrased like the study's objective, so retrieval ranks the background and objective sections first. A system that hands the model its top 3 passages often hands it the question restated, not the answer.

The latent-semantic baseline did worse than plain BM25, and fusing it in did not help. Dense retrievers narrow the gap but do not close it: PubMedBERT embeddings find the right paper in the top 5 for 99.0% of questions and its results section for 78.3% [74.6, 81.8]; all-MiniLM-L6-v2 reaches 99.0% and 69.7%. With every retriever the results section is the top passage for under 10% of questions. With 60,000 distractor abstracts in the index, BM25 finds the paper in the top 5 for 86.8% of questions and its results section for 25.0%.

Model gemini-3.8-flash, run on 2026-10-17, 500 test questions, 0 failed calls. The majority-class baseline, always answering yes, scores 55.2%.

Condition	Accuracy	Abstained	Accuracy when answering	Citation precision
closed	55.8% [51.4, 60.4]	0.4%	56.0%
passages	32.4% [28.4, 36.6]	57.2%	75.7%	99.5%
parent	67.8% [63.6, 72.0]	12.2%	77.2%	99.5%
oracle	72.2% [68.0, 76.2]	6.2%	77.0%	100%
removed	0.2% [0.0, 0.6]	99.6%	50.0%

Closed-book, the model is no better than always answering yes. Standard RAG with the top 3 passages scores lower than closed-book, 32.4%, because the model says "insufficient" on 57.2% of questions when the passages do not answer them; when it does answer, it is right 75.7% of the time. Handed every section of the top-ranked abstract, it reaches 67.8%, a paired gain over passages of 35.4 points [30.8, 39.8]. The oracle abstract reaches 72.2%.

The two findings are the same finding. Within the passages condition, accuracy is 73.7% [67.8, 79.5] on the 205 questions where the results section was among the retrieved passages and 3.7% [1.7, 6.1] on the 295 where it was not. The retrieval gap is the answer gap.

With the question's own abstract removed from the index, the model answered yes or no anyway on 0.2% [0.0, 0.6] of questions, 1 of 500, and said "insufficient" on 99.6%. That is the easy kind of missing evidence: with the abstract gone, the retrieved passages come from other papers and nothing in them looks like an answer. The passages condition is the harder case, the right paper retrieved without its results section. When it cites, it cites the right study: citation precision is 99.5% for passages and parent and 100% for oracle.

Missing evidence is the normal case in a clinical setting, and a system that fills the gap with a confident answer is the failure that matters. Under this instruction the model filled it once in 500 questions, which is the behaviour to require, with two caveats: the same instruction makes it refuse more than half of ordinary RAG questions, and it was measured for one model under one instruction.

PubMedQA questions are written from article titles, which makes finding the paper easier than in real search, and the 1,000-abstract index makes it easier still; the distractor run is the harder setting. Yes, no or maybe is a simplification of what a study shows, and "maybe" is the rarest label at 11% and the hardest. One model family: the results are tied to the model name recorded in each results file. This is a research-question benchmark, not clinical decision support.