{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "Evaluating Long-Context Question & Answer Systems",
  "keywords": "长上下文,问答,评估",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/qa-evals/",
  "publish_date": "2025-05",
  "corpus_stage": "推理部署",
  "notes": "长上下文问答系统评估",
  "id": "54a4018ffe09",
  "collected_at": "2026-10-02T14:17:13",
  "title": "Evaluating Long-Context Question & Answer Systems",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nEvaluating Long-Context Question & Answer Systems\n[\nllm\neval\nsurvey\n]\n· 28 min read\nWhile evaluating Q&A systems is straightforward with short paragraphs, complexity increases as documents grow larger. For example, technical documentation, novels and movies, as well as multi-document scenarios. Although some of these evaluation challenges also appear in shorter contexts, long-context evaluation amplifies issues such as:\nInformation overload:\nIrrelevant details in large documents obscure relevant facts, making it harder for retrievers and models to locate the right evidence for the answer.\nPositional variance:\nEvidence may appear at the beginning, middle, or end of documents, making it a challenge for models with limited effective context or those susceptible to the “lost in the middle” problem.\nMulti-hop reasoning:\nThe correct answer depends on synthesizing several distinct pieces of evidence scattered throughout the text(s), challenging the model’s ability to retain and integrate information that is far apart.\nHallucinations at scale:\nLarger contexts increase the risk of models returning plausible yet incorrect responses due to poor retrieval or limited effective context.\nOpen-ended questions:\nQueries on broad themes or interpretative topics rarely have a single definitive answer, especially for large documents or corpora.\nIn this write-up, we’ll explore\nkey evaluation metrics\n, how to\nbuild evaluation datasets\n, and\nmethods to assess Q&A performance\nthrough human annotations and LLM-evaluators. We’ll also\nreview several benchmarks\nacross narrative stories, technical and academic texts, and very long-context, multi-document situations. Finally, we’ll wrap up with advice for evaluating long-context Q&A on our specific use cases.\nAn overview of what we'll cover in this writeup\nBy the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their\nfinal\ncohort of “AI Evals for Engineers and PMs” in July. Here’s a\n35% discount code\n.\nKey Evaluation Metrics\nEvaluating Q&A systems goes beyond just checking for factual accuracy. Specifically, we might want answers to be based solely on the provided text, not the model’s knowledge. But even technically correct answers aren’t necessarily helpful. Thus, to evaluate Q&A systems effectively, we should consider two orthogonal dimensions:\nFaithfulness:\nHow strictly the answer relies on only the source document.\nHelpfulness:\nHow relevant, comprehensive, and useful the response is for the user.\nFaithfulness measures whether an answer strictly relies\nonly\non the source document.\nThis means the model shouldn’t add external information or make things up (aka hallucinate). Faithfulness is especially important for legal agreements, financial contracts, or medical and insurance forms, where answers must be based solely on the given text. Faithfulness is synonymous with groundedness, where answers must be anchored on the original document.\nFaithfulness also includes the Q&A system knowing when to say, “I don’t know.” If the source document doesn’t contain the answer, the ideal response is something like, “I don’t have that information in the provided text.” Related to this challenge are two errors by Q&A systems:\nFalse positives:\nWhen the system makes up an answer that doesn’t exist in the source document (hallucinations).\nFalse negatives:\nWhen the system incorrectly states that the source document doesn’t contain information that actually is present, either due to poor retrieval or attention limitations over large contexts.\nWe also want to distinguish faithfulness from correctness. An answer might be correct based on general knowledge but still be unfaithful if it contradicts the document. Examples include patient-specific medical instructions that differ from the usual guidelines, definitions in financial or legal agreements that depart from the standard, and historical fiction with alternate timelines. Users depend on Q&A systems to return responses that are faithful to their specific documents, rather than general truths.\nFor systems that provide citations, we can also assess citation accuracy. This evaluates if the cited text supports the answer. Benchmarks like\nQASPER\nexplicitly evaluate whether models reference the right supporting evidence for the answer. This combined assessment—checking both faithfulness and citation accuracy—provides finer-grained metrics on overall faithfulness and evidence retrieval.\nHowever, a faithful answer isn’t always a helpful answer.\nThis is where we also want to evaluate the helpfulness of responses.\nHelpfulness measures whether an answer is relevant, sufficiently detailed, yet concise.\nRelevance means the answer directly addresses the user’s question without straying off-topic. Comprehensiveness ensures the answer contains the necessary details. Conciseness balances comprehensiveness by ensuring the answer is succinct, without unnecessary details or fluff.\nWhile a brief, one-sentence response to a complex question might be faithful, it falls short of being helpful if the answer needs more details. Conversely, overly long responses filled with extraneous details can overwhelm users, making it hard for users to find the core answer they need. An ideal response should contain most, if not all, of the relevant information from the source document, in a concise way that meets the user’s needs.\nA study by\nXu et al. (2023)\nfound that domain experts in fields like biology or economics preferred answers that were both comprehensive and faithful, particularly for long-form questions. In contrast, crowd-workers often emphasized surface aspects such as conciseness or detail. Thus, if we’re building our Q&A system for power users and experts, the system should focus on returning faithful and comprehensive answers.\nThere’s a tension between faithfulness and helpfulness. An answer can be perfectly faithful yet totally unhelpful. For example, if we ask about a legal contract: “What happens if the tenant misses a payment?” A faithful yet unhelpful answer could be, “Clause 4.2 of the lease agreement addresses missed payments.” Although technically accurate, it’s not helpful as it doesn’t tell us what actually happens if a payment is missed. The same goes for Q&A systems that simply copy-paste large sections from documents. A useful system should synthesize the information and return a direct answer that meaningfully addresses the user’s question.\nAll in all, the best answers achieve both faithfulness and helpfulness by:\nStaying grounded in the source text (faithful)\nDirectly addressing the user’s question (relevant)\nProviding sufficient detail and context (comprehensive)\nPresenting information clearly and succinctly (concise)\nBuilding an Evaluation Dataset\nEvaluating long-context Q&A begins with creating a robust evaluation dataset. This involves testing how well a Q&A system can navigate book-length documents to answer questions.\nFirst, we’ll start with creating a variety of realistic, context-specific questions. While human annotators excel at crafting great questions, this is time-consuming and impractical at scale, especially for lengthy documents. A more efficient approach is to use language models to draft questions that annotators can then accept or edit—this augments human judgment with machine speed and scale.\nHowever, just scaling with a language model isn’t enough. We also need to guide the model toward generating natural, useful questions. Thus, instead of vague prompts like “Generate questions about this chapter,” we can be more specific, such as: “Summarize the main characters in this chapter. Then, generate one question about each character’s backstory based on what we’ve read so far.” More precise prompting helps steer models toward producing useful questions for our evaluation dataset.\nThis approach builds on the methodology of existing benchmarks.\nNarrativeQA\nintentionally generates questions based on summaries rather than full texts. This encourages questions that test narrative comprehension rather than shallow fact recall. For the same reason,\nQASPER\ncreates questions based on abstracts from academic papers that models then answer based on the full paper. By learning from these benchmarks, we can construct evaluation datasets that effectively measure meaningful comprehension of long-context documents.\nWe’ll want to ensure question diversity when creating questions.\nHaving a range of question types helps us evaluate the Q&A system’s capabilities without overfitting to any single type of question. Depending on our use case, an evaluation dataset could include a mix of:\nFact recall:\nThese evaluate basic fact retrieval, like “Who is the protagonist?”, “When was the treaty signed?”, or “What is the legal clause mentioned in Section 2.1?” While simple, they confirm whether our Q&A system can reliably extract information.\nDefinitions:\nThese assess a model’s ability to explain domain-specific content based on the document. Examples include “What does this acronym mean in the paper?”, “Explain the magic system introduced in Chapter 7,” or “Define the economic theory discussed on page 203.” This is important for technical documents to ensure the system can handle specialized terminology in context.\nSummarization:\nThese measure whether the system can identify the core ideas and coherently summarize them. For example, “Summarize the main findings of the paper”, “Recap what has happened in the book so far”, or “What are the key themes discussed in Part 2?”\nInference and reasoning:\nThese evaluate the ability to reason beyond explicitly stated facts by integrating information from different parts of the document to form a coherent answer. For example, “Why did the character make this choice?” or “What can we infer about the society from these laws?”\n“No-Info”:\nUnlike previous categories, these questions cannot be answered from the document. For example, “What did Gandalf do in the final battle at Hogwarts?” or “What is the penalty for trademark infringement in this residential lease agreement?” A faithful Q&A system should recognize that the required information isn’t present and respond accordingly instead of making up an answer.\nOur Q&A evals should also be robust to the position of evidence within the document.\nWe ensure this by having questions with evidence that appear at the beginning, middle, or end, as well as creating multi-hop questions that require details from several sections or documents. Benchmarks like\nHELMET\nevaluate how model accuracy changes based on the location of supporting information, evaluating the model’s ability to pay attention to and combine information from the entire document instead of relying solely on nearby context.\nMethods to Assess Q&A Performance\nHuman annotators are crucial for building a high-quality, ground-truth dataset\n. This is useful for calibrating automated evaluators, and with enough annotated examples, we can also train evaluation classifiers or reward models. Here’s how this might look for the metrics of faithfulness and helpfulness:\nFaithfulness annotation\ninvolves evaluating whether an answer accurately reflects the source text. Ideally, we’d like simple binary labels—faithful or unfaithful—but reality is rarely that straightforward. Answers typically exist on a spectrum. As a result, a mostly correct answer that misses a critical detail should be graded differently from one that incorrectly represents minor or peripheral information.\nRelated to faithfulness is the\n“no-info” annotation\n. This checks whether the model correctly identifies when the provided context doesn’t contain the information to answer the question. The goal here is to identify hallucinations, where the model invents answers instead of acknowledging the gap. As part of this exercise, we could have the following labels:\nIncorrect answer / hallucination:\nThe model tries to answer despite missing information, even if the response sounds plausible.\nIncorrect refusal:\nThe model mistakenly claims the information isn’t present, perhaps due to retrieval errors or inadequate attention to the long context.\nCorrect refusal:\nThe model accurately recognizes the absence of necessary details and appropriately declines to answer.\nHelpfulness comparisons\ninvolve annotators judging which of two faithful answers better meets the user’s needs. Rather than asking for absolute ratings, annotators make relative judgments, answering a straightforward question: “Which answer is more helpful?” People find comparing two answers easier than assigning absolute ratings, resulting in greater consistency across annotators. When comparing helpfulness, annotators should consider:\nRelevance\n: Does one answer more directly and precisely address the question?\nComprehensiveness\n: Does one answer include key information that the other misses?\nConciseness\n: Is one answer more succinct and easier to understand?\nHere are some practical tips for setting up a reliable annotation process:\nStart with clear guidelines: Include examples for each category and clarify how to handle edge cases. Also, be concise—it makes it easier to read the entire guide.\nIterate on the guidelines: Our initial draft won’t be perfect. Collect annotator feedback on unclear or challenging cases to improve our guidelines.\nUse qualification tasks: Before assigning actual tasks, provide annotators with practice examples with known correct answers. This ensures they understand the guidelines and can apply them consistently.\nMeasure inter-annotator agreement: Check for consistency among annotators using metrics like Cohen’s Kappa. Low agreement can indicate unclear guidelines or ambiguous scenarios needing further clarification.\nConsider expert annotators for specialized domains: General annotation tasks can usually be handled by crowd-workers, but domains like medicine or law often require subject-matter experts for accurate and meaningful evaluations.\nThat said, while human annotation is traditionally considered the gold standard, it’s not always practical or scalable, especially for large documents.\nThis is where LLM-evaluators (also called “LLM-as-Judge”) can help.\nVia this approach, we provide clear criteria—or our annotation guidelines—to a model, and have it evaluate the quality of Q&A responses.\nBut first, it’s important to recognize why older automated metrics fall short. Historically, the language modeling community relied on n-gram-based metrics like BLEU and ROUGE, which measure word overlap between generated responses and reference answers. Although these metrics work somewhat for tasks like machine translation, they correlate poorly with human judgment on open-ended tasks such as Q&A.\nFor example, the\nL-Eval\nbenchmark highlighted the poor correlation between token-overlap metrics and human judgment for Q&A responses. A correct answer using words that differ from the reference answer can get unfairly penalized by a low ROUGE score, leading to a misleading negative signal. This is especially noticeable when model responses and reference answers vary in length. Without length normalization, token-overlap metrics can mistakenly reward verbose yet mediocre answers over concise, accurate ones.\nThis is why model-based evaluation is increasingly popular—it offers more reliable and nuanced evals than traditional metrics. We typically start by calibrating an LLM-evaluator against a high-quality, human-annotated dataset. With ground truth, we can evaluate our LLM-evaluator by measuring its recall and precision on faithfulness annotations, and its correlation with human judgments on the helpfulness comparisons.\nTo evaluate faithfulness, we can treat answers as collections of individual claims, each of which can be verified as true or false.\nThis is similar to approaches used in\nNLI-based\nand\nQ&A-based\nsummarization metrics, and\nclaim generation and verification\n. Breaking answers down into atomic claims helps us pinpoint where hallucinations occur. Here’s how it works:\nExtract claims:\nConsider this response about a contract dispute: “The tenant breached the lease because they missed three payments, failed to maintain insurance coverage, and sublet the apartment without permission.” This can be split into:\nClaim 1: The tenant missed three payments.\nClaim 2: The tenant failed to maintain required insurance coverage.\nClaim 3: The tenant sublet the apartment without permission.\nVerify each claim:\nCheck each statement against the source document (in this case, the lease agreement) to confirm its accuracy.\nCalculate faithfulness:\nThe proportion of claims supported by the document provides an overall faithfulness score.\nThis fine-grained approach, as demonstrated by evaluations like\nSummaC\n,\nQAFactEval\n, and\nRefChecker\n, offers more interpretability and nuance. Rather than labeling an entire answer as faithful or not, we gain a nuanced understanding of which claims are incorrect. This also allows assigning partial credit to mostly faithful answers with minor inaccuracies.\nWe can also go a step further by requiring the model to provide citations for each claim. This helps distinguish between two different failure modes: hallucinations (making up answers) and retrieval failures (not retrieving relevant information).\nTo evaluate our evaluator, we can compare its judgments to human annotations on two key metrics: (i) recall (of all unfaithful claims, how many does the evaluator correctly flag?) and (ii) precision (of all claims the evaluator flags as unfaithful, how many are truly unfaithful?)\nEvaluating helpfulness requires a more nuanced approach because often, there isn’t a definitively “helpful” way to answer.\nDifferent situations might call for varying levels of detail or explanation styles. Here are several strategies we can consider:\nReference-based comparison\nworks well when we have high-quality reference answers. The LLM-evaluator compares generated answers against these references to assess relevance, detail, and clarity. However, as models improve, their answers may surpass existing references, making this method less effective over time.\nCriteria-based evaluation\nassesses answers using a clearly defined rubric. This approach allows us to directly reuse our annotation guidelines, focusing on criteria like relevance, comprehensiveness, and conciseness.\nPairwise comparisons\nare particularly useful when iteratively improving Q&A systems. By comparing newly generated answers against previously validated ones, we consistently push quality higher. This method is also ideal for A/B testing different configurations of the Q&A system.\nTo calibrate an LLM-evaluator on helpfulness, pairwise comparisons are especially reliable. By presenting pairs of answers to annotators and LLM-evaluators, we can measure their alignment—how often they agree on the more helpful answer. Correlation metrics, such as Cohen’s Kappa, quantify this alignment effectively. For example, L-Eval found that GPT-4’s pairwise comparisons correlated strongly with human preferences once properly calibrated.\nWhat We Can Learn from Existing Benchmarks\nTo ground our discussion so far, let’s look at some benchmarks for long-context Q&A. Besides providing a common standard, these benchmarks highlight challenges we might encounter in dataset creation and evaluation. Since these datasets are likely already part of model training data, we shouldn’t rely solely on them to evaluate our Q&A system. Instead, we’ll want to create evaluation datasets tailored to our use case.\nWe’ll cover six benchmarks spanning (i) narrative documents, (ii) technical and academic documents, and (iii) very long or multi-document contexts.\nThe NarrativeQA dataset\n, introduced by\nKočiský et al. in 2017\n, is designed to test genuine narrative comprehension rather than surface-level pattern matching. Unlike earlier datasets that allowed models to answer by extracting single sentences, NarrativeQA requires synthesizing information scattered across novels and movie scripts to generate answers.\nFirst, the authors collected over 1,500 stories from Project Gutenberg and movie script websites, along with their corresponding plot summaries from Wikipedia. Annotators then generated question-answer pairs based only on these summaries, without viewing the full texts. (Conversely, models answered questions based on the full text but not the summaries.) This deliberate approach ensured that answers couldn’t be found by simple text matching, focusing the evaluation on understanding the entire text. The resulting dataset contains 46,765 question-answer pairs focused on narrative comprehension.\nStatistics of the NarrativeQA dataset\nNarrativeQA evaluates whether models can integrate information dispersed throughout long narratives, such as entire books or movies, to produce coherent answers. Answers are evaluated on n-gram matching metrics such as BLEU, METEOR, and ROUGE, comparing machine-generated answers against two reference answers for each question.\nNarrativeQA highlights the importance of questions that go beyond simple extraction, requiring models to integrate information across the document. By generating questions from summaries instead of full texts, the authors ensured questions required holistic comprehension of the text, thus reducing superficial, extractive answering strategies.\nNovelQA\n, introduced by\nWang et al. in 2024\n, is a benchmark designed for evaluating reading comprehension on very long texts, often exceeding 200,000 tokens. Similar to NarrativeQA but updated for modern times, NovelQA assesses how well models understand and integrate narratives spanning entire novels. Models were evaluated in two formats: multiple-choice and open-ended generation.\nTwo types of responses in NovelQA\nTo build the dataset, the authors selected a diverse set of 89 English novels and collaborated closely with English literature students familiar with these works. Annotators created 2,305 questions in two phases. First, annotators used a question template and filled in entities from the novel to form valid questions (templates below).\nTemplates used to generate questions in NovelQA\nThen, to enhance question diversity, annotators also freely generated challenging questions. All the questions were then reviewed by the authors, who ultimately accepted 79.4% of the questions. Each question was accompanied by a gold-standard answer and the relevant supporting evidence from the novels to ground evaluations.\nNovelQA evaluates a model’s ability to synthesize, integrate, and recall detailed information across extremely long contexts. Questions fall into these categories:\nDetail-oriented (22.2%): Focus on subtle specifics requiring careful recall.\nSingle-hop (42.8%): Answerable from adjacent sentences or closely related passages.\nMulti-hop (35%): Requires synthesizing information across multiple chapters.\nThe questions cover various narrative aspects, such as characters, plot, setting, and deeper thematic meanings. The benchmark supports both multiple-choice and open-ended generative evaluation methods, with GPT-4 serving as evaluator for generative answers (achieving Cohen’s Kappa of 89.25% against human judgments).\nData distribution by complexity and aspect in NovelQA\nNovelQA’s findings are a shift from the typical “lost in the middle” problem—it showed that model performance declines when evidence appears beyond the 100,000-token mark. The authors also highlighted the importance of rigorous quality control, manually reviewing all crowd-generated questions and accepting only 79.4% of question-answer pairs. Finally, explicitly linking each answer to specific supporting evidence helps with retrieval evals.\nPerformance of models decline when evidence is beyond 100k tokens\nWhile narrative texts present one kind of challenge, comprehending dense, technical documents introduces an entirely different set of difficulties.\nQASPER\n, introduced by\nDasigi et al. (2021)\n, addresses this by testing models on information-seeking questions on academic papers. Specifically, QASPER contains 5,049 questions on 1,585 NLP papers. Similar to NarrativeQA, these questions were crafted by NLP practitioners who had only read paper titles and abstracts. This approach ensures questions often require synthesizing information across the entire paper rather than simple text extraction.\nExample question, answer, and supporting evidence in QASPER\nFirst, 25 NLP practitioners selected papers that interested them and created questions based solely on titles and abstracts. Then, another group of 51 NLP experts answered these questions using the full texts. The latter group’s task included determining if questions were answerable, pinpointing specific supporting evidence (such as text passages, figures, or tables), and providing clear, concise answers. (10% of questions were marked unanswerable and thus excluded.) Separating question generation from answer annotation reduced biases, as question authors had no prior knowledge of the detailed answers.\nQASPER evaluates models on two main aspects: answer accuracy (Answer-F1) and evidence selection (Evidence-F1). Answer-F1 measures the accuracy of model responses, regardless of whether they extract text directly or create new explanations. Evidence-F1 evaluates the model’s ability to identify supporting details. This is particularly challenging, as more than half of the questions require combining evidence from multiple sections or paragraphs.\nThe Evidence-F1 results in QASPER highlight a significant gap between answer generation and evidence retrieval—even when models give accurate answers, they often struggle to identify the exact supporting passages. Additionally, limiting question creators to only titles and abstracts naturally encouraged questions—and answers—that required a deep understanding of the entire paper, moving beyond superficial extraction.\nL-Eval\nby\nAn et al. (2023)\ncovers documents ranging from 3,000 to 200,000 tokens and includes 20 diverse subtasks, 508 extensive documents, and over 2,000 human-annotated question-answer pairs. Unlike previous benchmarks that mainly relied on text-matching metrics, L-Eval also applied LLM-evaluators and measured the difference between both.\nTo build L-Eval, the authors first created four new datasets: Coursera (educational content), SFiction (science fiction stories), CodeU (Python codebases), and LongFQA (financial earnings). They also improved five existing datasets by adding more challenging synthesis-oriented questions, such as augmenting QuALITY to require deeper comprehension of entire documents. Lastly, they reviewed and corrected 12 tasks from prior benchmarks, using Claude-100k to identify and remove inaccuracies or unanswerable questions.\nStatistics of datasets, question types, and domains in L-Eval\nL-Eval evaluates two types of tasks: closed-ended (like multiple-choice, code comprehension, true/false, and math), emphasizing precise reasoning, and open-ended (such as narrative synthesis and summarization), focusing on integrating and summarizing long-form content.\nClosed-ended tasks were evaluated via exact-match accuracy while open-ended tasks had human annotators rating responses from 1 (poor) to 5 (excellent). Additionally, L-Eval used language models like GPT-4 and GPT-3.5 as evaluators through pairwise comparisons for open-ended tasks. These had carefully designed prompts to reduce bias toward overly detailed answers. Traditional n-gram metrics, including ROUGE-L and F1 scores, were also used for efficiency, despite their known sensitivity to response length.\nL-Eval showed that traditional n-gram metrics often fail to reflect true comprehension in long-context scenarios due to mismatched answer lengths. Additionally, the benchmark demonstrated that using LLMs as evaluators in pairwise comparisons provides superior alignment with human assessments compared to traditional metrics, highlighting clear distinctions in model strengths for closed-ended versus open-ended tasks.\nHELMET\n(How to Evaluate Long-context Models Effectively and Thoroughly), introduced by\nYen et al. (2025)\n, addresses issues in earlier benchmarks, such as unrealistic tasks and inconsistent metrics, providing a framework for evaluating long-context language models.\nTo start, the authors identified shortcomings in existing evaluations, including limited context lengths, unreliable methods, and inadequate coverage for non-instruction-tuned models. Then, they created a benchmark with seven task categories: Retrieval-Augmented Generation (RAG), generation with citations, passage re-ranking, many-shot in-context learning, long-document question-answering, summarization, and synthetic recall. Each task contains contexts of up to 128,000 tokens, allowing controlled and consistent assessments with carefully crafted few-shot prompts and model-based metrics.\nTask categories, datasets, and metrics in HELMET\nHELMET specifically evaluates these capabilities in long-context models:\nRetrieval and reasoning: Natural Questions, TriviaQA, and HotpotQA test a model’s ability to find relevant information within extensive contexts containing distractors.\nInstruction following: Generation tasks requiring citations assess whether models can follow precise formatting guidelines while staying accurate.\nComparative reasoning: Passage re-ranking evaluates how well models compare and reason across multiple sections of text.\nIn-context learning: Many-shot tasks measure a model’s ability to quickly adapt and learn from multiple examples provided in-context.\nLong-form comprehension: Long-document question-answering and summarization tasks assess a model’s capability to synthesize and understand extensive texts.\nHELMET showed that synthetic tasks like Needle In a Haystack aren’t as useful, due to their weak correlation with real-world scenarios. Also, by carefully controlling input lengths, HELMET could evaluate model robustness to increasingly long contexts that approached previous models’ limits (≥128K tokens). Similar to previous benchmarks, HELMET replicated the flaws in traditional n-gram metrics such as ROUGE, which can misrepresent quality in longer outputs. Instead, it recommended using model-based evaluations, using models like GPT-4o, for evaluations that align more closely with human judgment.\nComparison of benchmark results across NIAH, Ruler, InfinityBench, and HELMET\nLoong\n, by\nWang et al. (2024)\n, is a benchmark that evaluates long-context comprehension across\nmultiple\ndocuments. While most earlier benchmarks focus on single-document scenarios, Loong presents realistic, multi-document tasks where missing any relevant document results in incorrect answers.\nLoong focuses on multi-document Q&A\nLoong consists of 1,600 evals drawn from financial reports, legal cases, and academic papers in English and Chinese, mainly from 2024. Each task includes evidence spread across multiple documents, mimicking real-world complexity. To generate questions, the authors used two methods: template-based generation, where Q&A pairs were constructed through predefined rules, and free annotation, where GPT-4o was prompted to create additional Q&A pairs.\nLoong evaluates a model’s ability to locate, compare, cluster, and reason on evidence spread across multiple documents, typically ranging from 10,000 to over 250,000 tokens. The benchmark covers four task types:\nSpotlight\n: Finding relevant evidence from one specific document among several.\nComparison\n: Comparing and integrating multiple pieces of information from different documents and returning the right answer.\nClustering\n: Aggregating and grouping relevant information from multiple sources based on specific criteria.\nChain of Reasoning\n: Integrating evidence across documents to return answers.\nThe four evaluation tasks in Loong\nFor evaluation, GPT-4 was used as the LLM-evaluator to score model outputs based on accuracy, hallucinations, and completeness, referencing the golden answer and task requirements. Metrics included (i) average scores (the average evaluation across all questions) and (ii) perfect rate (the percentage of questions receiving a perfect score).\nInterestingly, their analysis of retrieval-augmented generation (RAG) showed that using RAG\nreduced\nperformance on the Loong benchmark. They hypothesized that this is because Loong’s evidence is dispersed across multiple documents. While RAG helped somewhat on spotlight tasks, it performed poorly on tasks demanding deeper synthesis, such as comparison, clustering, and multi-step reasoning.\nThe use of RAG degrades performance compared to the baseline\nHere are some other long-context benchmarks that you may find helpful:\nA Critical Evaluation of Evaluations for Long-form Question Answering\nAda-LEval: Evaluating long-context LLMs with length-adaptable benchmarks\nBABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack\nBAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models\nBizBench: A Quantitative Reasoning Benchmark for Business and Finance\nELI5: Long Form Question Answering\nFrustratingly Hard Evidence Retrieval for QA Over Books\nInfinityBench: Extending Long Context Evaluation Beyond 100K Tokens\nLFED: A Literary Fiction Evaluation Dataset for Large Language Models\nLongBench: A Bilingual, Multitask Benchmark for Long Context Understanding\nLongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks\nLongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion\nMultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents\nQuALITY: Question Answering with Long Input Texts, Yes!\n• • •\nWhew, that was a lot! Here are some key takeaways:\nFaithfulness and helpfulness are orthogonal dimensions.\nAn answer can be faithful yet unhelpful, or helpful yet contain hallucinated information.\nFaithfulness also means knowing when to say “I don’t know”.\nModels should decline to answer when the context lacks information and respond correctly when it does.\nTraditional n-gram metrics struggle on Q&A.\nUse LLM-evaluators instead. They’re better at evaluating semantic quality and align more closely with human judgment.\nThe location of evidence matters.\nAcross the benchmarks discussed, some models struggled with the “lost in the middle” effect while others had poor performance when the evidence was beyond the 100,000 token mark.\nUsing RAG can reduce performance\n, especially for tasks requiring cohesive reasoning across evidence dispersed across a single or multiple documents.\nDid I miss anything important? Any other metrics, methods, or benchmarks you’d suggest I look into? Please\nlet me know\n!\nBy the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their\nfinal\ncohort of “AI Evals for Engineers and PMs” in July. Here’s a\n35% discount code\n.\nReferences\nAn, Chenxin, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. “L-Eval: Instituting Standardized Evaluation for Long Context Language Models.” arXiv. https://doi.org/10.48550/arXiv.2307.11088.\nBai, Yushi, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, et al. 2024. “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.” arXiv. https://doi.org/10.48550/arXiv.2308.14508.\nBai, Yushi, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, et al. 2025. “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks.” arXiv. https://doi.org/10.48550/arXiv.2412.15204.\nDasigi, Pradeep, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. “A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers.” arXiv. https://doi.org/10.48550/arXiv.2105.03011.\nDong, Zican, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. “BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models.” arXiv. https://doi.org/10.48550/arXiv.2309.13345.\nFabbri, Alexander, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. “QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization.” In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.187.\nFan, Angela, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. “ELI5: Long Form Question Answering.” arXiv. https://doi.org/10.48550/arXiv.1907.09190.\nFeng, Song, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. “MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents.” In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6162–76. https://doi.org/10.18653/v1/2021.emnlp-main.498.\nHu, Xiangkun, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. “RefChecker: Reference-Based Fine-Grained Hallucination Checker and Benchmark for Large Language Models.” arXiv. https://doi.org/10.48550/arXiv.2405.14486.\nKočiský, Tomáš, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. “The NarrativeQA Reading Comprehension Challenge.” arXiv. https://doi.org/10.48550/arXiv.1712.07040.\nKoncel-Kedziorski, Rik, Michael Krumdick, Viet Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. 2024. “BizBench: A Quantitative Reasoning Benchmark for Business and Finance.” arXiv. https://doi.org/10.48550/arXiv.2311.06602.\nKuratov, Yuri, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. “BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack.” arXiv. https://doi.org/10.48550/arXiv.2406.10149.\nLaban, Philippe, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2021. “SummaC: Re-Visiting NLI-Based Models for Inconsistency Detection in Summarization.” arXiv. https://doi.org/10.48550/arXiv.2111.09525.\nLing, Zhan, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025. “LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion.” arXiv. https://doi.org/10.48550/arXiv.2501.15089.\nMou, Xiangyang, Mo Yu, Bingsheng Yao, Chenghao Yang, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2020. “Frustratingly Hard Evidence Retrieval for QA Over Books.” arXiv. https://doi.org/10.48550/arXiv.2007.09878.\nPang, Richard Yuanzhe, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, et al. 2022. “QuALITY: Question Answering with Long Input Texts, Yes!” arXiv. https://doi.org/10.48550/arXiv.2112.08608.\nWang, Chonghua, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. “Ada-LEval: Evaluating Long-Context LLMs with Length-Adaptable Benchmarks.” arXiv. https://doi.org/10.48550/arXiv.2404.06480.\nWang, Cunxiang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, et al. 2024. “NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens.” arXiv. https://doi.org/10.48550/arXiv.2403.12766.\nWang, Minzheng, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, et al. 2024. “Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA.” arXiv. https://doi.org/10.48550/arXiv.2406.17419.\nXu, Fangyuan, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. “A Critical Evaluation of Evaluations for Long-Form Question Answering.” arXiv. https://doi.org/10.48550/arXiv.2305.18201.\nYen, Howard, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2025. “HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly.” arXiv. https://doi.org/10.48550/arXiv.2410.02694.\nYu, Linhao, Qun Liu, and Deyi Xiong. 2024. “LFED: A Literary Fiction Evaluation Dataset for Large Language Models.” arXiv. https://doi.org/10.48550/arXiv.2405.10166.\nZhang, Xinrong, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, et al. 2024. “InfinityBench: Extending Long Context Evaluation Beyond 100K Tokens.” arXiv. https://doi.org/10.48550/arXiv.2402.13718.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Jun 2025). Evaluating Long-Context Question & Answer Systems. eugeneyan.com.\n        https://eugeneyan.com/writing/qa-evals/.\nor\n@article{yan2025qa,\n  title   = {Evaluating Long-Context Question & Answer Systems},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2025},\n  month   = {Jun},\n  url     = {https://eugeneyan.com/writing/qa-evals/}\n}\nShare on:\nBrowse related tags:\n[\nllm\neval\nsurvey\n]\nor\nSearch\n« AI Engineer 2025 - Improving RecSys & Search with LLM techniques\nTraining an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "Evaluating Long-Context Question & Answer Systems",
    "description": "Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "Evaluating Long-Context Question & Answer Systems",
    "og:type": "article",
    "og:description": "Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.",
    "og:url": "https://eugeneyan.com/writing/qa-evals/",
    "og:image": "https://eugeneyan.com/assets/og_image/qa-evals-v2.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "Evaluating Long-Context Question & Answer Systems",
    "twitter:description": "Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.",
    "twitter:image": "https://eugeneyan.com/assets/og_image/qa-evals-v2.jpg",
    "article:published_time": "2025-06-22T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "survey"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/54a4018ffe09/code/body-000.txt",
      "chars": 234,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/og_image/qa-evals-v2.jpg",
      "bytes": 217526,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-323b3ed37ce13d986fa7.jpg",
      "src": "https://eugeneyan.com/assets/og_image/qa-evals-v2.jpg",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/narrativeqa.webp",
      "bytes": 61390,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-b6185072b786fd6b51f3.webp",
      "src": "https://eugeneyan.com/assets/narrativeqa.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/novelqa-fig2.webp",
      "bytes": 73480,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-dd1be8cda700dc6c3e8d.webp",
      "src": "https://eugeneyan.com/assets/novelqa-fig2.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/novelqa-table5.webp",
      "bytes": 141574,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-0c907507339a625904a8.webp",
      "src": "https://eugeneyan.com/assets/novelqa-table5.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/novelqa-table7.webp",
      "bytes": 168286,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-18754b9cafa3248c245b.webp",
      "src": "https://eugeneyan.com/assets/novelqa-table7.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/novelqa-fig3.webp",
      "bytes": 59466,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-9c44600e7b4852c1af2b.webp",
      "src": "https://eugeneyan.com/assets/novelqa-fig3.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/qasper-fig1.webp",
      "bytes": 153596,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-4d40c252c70e6df7c1c9.webp",
      "src": "https://eugeneyan.com/assets/qasper-fig1.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/leval-table1.webp",
      "bytes": 139692,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-32a8bc57ed5399cd92e5.webp",
      "src": "https://eugeneyan.com/assets/leval-table1.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/helmet-table3.webp",
      "bytes": 111664,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-24adf44b25f868e10a2d.webp",
      "src": "https://eugeneyan.com/assets/helmet-table3.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/helmet-fig1.webp",
      "bytes": 52968,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-266e361da5fd97ee8518.webp",
      "src": "https://eugeneyan.com/assets/helmet-fig1.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/loong-fig1.webp",
      "bytes": 74382,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-d0f21a47828ebeb533d6.webp",
      "src": "https://eugeneyan.com/assets/loong-fig1.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/loong-fig2.webp",
      "bytes": 114010,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-798c7d0fdc0da15aee98.webp",
      "src": "https://eugeneyan.com/assets/loong-fig2.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/loong-fig3.webp",
      "bytes": 79288,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-cdca0f3a6938a4a16588.webp",
      "src": "https://eugeneyan.com/assets/loong-fig3.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/54a4018ffe09/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/llm/",
      "label": "llm",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/eval/",
      "label": "eval",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/survey/",
      "label": "survey",
      "type": "a"
    },
    {
      "href": "https://maven.com/parlance-labs/evals?promoCode=eugene-is-all-you-need",
      "label": "35% discount code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2105.03011",
      "label": "QASPER",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.18201",
      "label": "Xu et al. (2023)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1712.07040",
      "label": "NarrativeQA",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2410.02694v1",
      "label": "HELMET",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.11088",
      "label": "L-Eval",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2111.09525",
      "label": "NLI-based",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.naacl-main.187/",
      "label": "Q&A-based",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2405.14486",
      "label": "claim generation and verification",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/1712.07040",
      "label": "Kočiský et al. in 2017",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2403.12766",
      "label": "Wang et al. in 2024",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2410.02694",
      "label": "Yen et al. (2025)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2406.17419",
      "label": "Wang et al. (2024)",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2305.18201",
      "label": "A Critical Evaluation of Evaluations for Long-form Question Answering",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2404.06480",
      "label": "Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2406.10149",
      "label": "BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2309.13345",
      "label": "BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2311.06602",
      "label": "BizBench: A Quantitative Reasoning Benchmark for Business and Finance",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/1907.09190",
      "label": "ELI5: Long Form Question Answering",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2007.09878",
      "label": "Frustratingly Hard Evidence Retrieval for QA Over Books",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2402.13718",
      "label": "InfinityBench: Extending Long Context Evaluation Beyond 100K Tokens",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2405.10166",
      "label": "LFED: A Literary Fiction Evaluation Dataset for Large Language Models",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2308.14508",
      "label": "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2412.15204",
      "label": "LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2501.15089",
      "label": "LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2109.12595",
      "label": "MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/abs/2112.08608",
      "label": "QuALITY: Question Answering with Long Input Texts, Yes!",
      "type": "a"
    },
    {
      "href": "https://x.com/eugeneyan",
      "label": "let me know",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/aie-2025/",
      "label": "« AI Engineer 2025 - Improving RecSys & Search with LLM techniques",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/semantic-ids/",
      "label": "Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:25Z",
  "source_url": "https://eugeneyan.com/writing/qa-evals/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2105.03011",
      "bytes": 506609,
      "reused": false,
      "local_path": "papers/repair/52258ec1edf197ac3925.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.18201",
      "bytes": 735968,
      "reused": false,
      "local_path": "papers/repair/73e684d655163223f2eb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1712.07040",
      "bytes": 225236,
      "reused": false,
      "local_path": "papers/repair/7aa4679a253ab94fa0cc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2410.02694v1",
      "bytes": 1976429,
      "reused": false,
      "local_path": "papers/repair/e5652c37ae55c19d97a8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.11088",
      "bytes": 1239004,
      "reused": false,
      "local_path": "papers/repair/465d341a96a4644e69d4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2111.09525",
      "bytes": 387371,
      "reused": false,
      "local_path": "papers/repair/c14dc7bafdd948facdca.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2405.14486",
      "bytes": 2062368,
      "reused": false,
      "local_path": "papers/repair/8ed56531483e9d09d4ee.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1712.07040",
      "bytes": 225236,
      "reused": true,
      "local_path": "papers/repair/7aa4679a253ab94fa0cc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2403.12766",
      "bytes": 4326871,
      "reused": false,
      "local_path": "papers/repair/9688051c75e043cf2749.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2410.02694",
      "bytes": 2164826,
      "reused": false,
      "local_path": "papers/repair/966e3372c8e434d56ff3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.17419",
      "bytes": 889244,
      "reused": false,
      "local_path": "papers/repair/6f7d55589cb42491b63b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.18201",
      "bytes": 735968,
      "reused": true,
      "local_path": "papers/repair/73e684d655163223f2eb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2404.06480",
      "bytes": 654710,
      "reused": false,
      "local_path": "papers/repair/51802ebff1b01cc3c5f0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2406.10149",
      "bytes": 5321327,
      "reused": true,
      "local_path": "papers/repair/92ccea2b3b77a6d9c105.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2309.13345",
      "bytes": 496298,
      "reused": false,
      "local_path": "papers/repair/4adc5eaba953875ad9eb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.06602",
      "bytes": 513781,
      "reused": false,
      "local_path": "papers/repair/55d22aa45acf672f901d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.09190",
      "bytes": 2930782,
      "reused": false,
      "local_path": "papers/repair/60d7a061f41984e06566.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.09878",
      "bytes": 128448,
      "reused": false,
      "local_path": "papers/repair/1845c7a46b7b97a4083e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.13718",
      "bytes": 538632,
      "reused": false,
      "local_path": "papers/repair/5e491fe5771acbb75f58.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2405.10166",
      "bytes": 1091735,
      "reused": false,
      "local_path": "papers/repair/b51207eb79b3cde57667.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2308.14508",
      "bytes": 11630690,
      "reused": false,
      "local_path": "papers/repair/0675a801c53c70b004f7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2412.15204",
      "bytes": 9951437,
      "reused": true,
      "local_path": "papers/repair/c415bedae6d5bdbea511.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2501.15089",
      "bytes": 1402407,
      "reused": false,
      "local_path": "papers/repair/77d6c186556c73f6532c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2109.12595",
      "bytes": 1188691,
      "reused": false,
      "local_path": "papers/repair/290be8f6110b2c582b41.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.08608",
      "bytes": 2773376,
      "reused": false,
      "local_path": "papers/repair/06e38842a50eb6ce419d.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 41154,
  "expected_body_images": 19,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/54a4018ffe09/pages/source.html",
  "local_html": "attachments/54a4018ffe09/pages/article.html",
  "body_base_path": "attachments/54a4018ffe09/pages/",
  "body_markdown_path": "attachments/54a4018ffe09/pages/article.md"
}