CMASS

A benchmark suite for agentic search v1.0 · 25 October 2026

CMASS A ClimbMix Agentic Search Suite for Evaluating Search Agents Across Heterogeneous Tasks

Sahel Sharifymoghaddam*, Lingwei Gu*, Yijun Ge, Jimmy Lin

David R. Cheriton School of Computer Science, University of Waterloo
*Equal contribution

Leaderboard Paper (arXiv) Code (GitHub) Dataset (Hugging Face)
A source question is decomposed into hops; each hop is grounded in ClimbMix documents, and the documents that support a hop are the question's qrels. Source question BCP · TRQA · DSQA · HotpotQA hop 1 hop 2 hop 3 ClimbMix · 553M documents qrel: states a fact a hop needs other documents
verified questions
423
source datasets
4
ClimbMix documents
553M
tokens in the corpus
~400B
qrels (v2026.10.25)
40,736
BM25 Lucene index
559 GB

Abstract

One fixed corpus, four kinds of search task

Evaluating advances in agentic search requires testing whether methods generalize across tasks and corpus scale. Changing web search services impede controlled comparisons, while small corpora assembled around benchmark questions provide a limited test of generalization.

We introduce CMASS, the ClimbMix Agentic Search Suite, which projects three search benchmarks—BrowseComp-Plus, Total Recall QA, and DeepSearchQA—and the multi-hop question-answering dataset HotpotQA onto a shared, fixed corpus. The target, ClimbMix, is a pre-training corpus of 553M documents and approximately 400B tokens, assembled independently of the benchmark questions. Our projection pipeline verifies grounding, answer uniqueness, and set completeness through independent agent audits, controlled revision, and human review. The 423 questions span entity identification, multi-hop reasoning, exhaustive answer sets, and aggregation.

Evaluations of three models expose substantial headroom for open-weight agents and show that high answer accuracy can coexist with incomplete evidence coverage. By enabling measurement of accuracy alongside evidence coverage, search effort, and closed-book performance, CMASS supports developing agents that find sufficient evidence and produce complete answers efficiently.

The suite

Four benchmarks, projected onto ClimbMix

Each source question is decomposed into atomic reasoning hops and kept only if every hop is grounded in ClimbMix documents, its answer is unique and, for set questions, its answer set is complete. Questions that needed it were revised; wording and answers can differ from their upstream versions.

Every question comes with question-level relevance judgments (qrels): the ClimbMix documents that state a fact one of its hops needs, including exact and near duplicates of those documents.

SourceTaskQuestionsQrels
BrowseComp-PlusBCP · Entity identificationEntity identification8011,450
Total Recall QATRQA · AggregationAggregation688,307
DeepSearchQADSQA · Exhaustive answer setsExhaustive answer sets719,889
HotpotQAMulti-hop reasoning Multi-hop reasoning20411,090
Total42340,736
 Qrels are question–document pairs in release v2026.10.25.

Projection pipeline

Every hop grounded, every answer checked

  1. Preparation

    Exact and near-duplicate detection over the whole corpus (Jaccard 0.7 or more; nothing is removed), and one task per source question.

  2. Projection

    Agents decompose each question into hops, search ClimbMix for evidence of each hop, and judge whether the answer can be derived from what they found.

  3. Audit and review

    An auditor checks constraint coverage, uniqueness and set completeness and may revise the question; hunters look for rival answers; the authors review every question.

  4. Qrels

    Judge agents keep the pooled documents that support a hop and add their duplicates; documents the evaluated agents retrieved are then judged the same way, TREC-style.

Flowchart of the projection pipeline for BrowseComp-Plus. Stage 0 prepares the corpus and one task per question. Stage 1 decomposes each of the 830 questions into hops and grounds them in ClimbMix; 326 are labeled projectable, and 65 verified candidates plus 47 near misses, 112 in all, go on. Stage 2 audits them, checks for rival answers, sends them to human review and a final check, and accepts 80. Stage 3 builds 11,450 qrels by candidate pooling, a precision filter and duplicate expansion.
The projection pipeline, with the BrowseComp-Plus counts: 80 of 830 source questions are released, with 11,450 qrels. Open the image for full size.

Stage 2: audit, revision and human review

An auditor agent checks each question's constraints and hops (D2), then its answer's accuracy and completeness (D3). A question that fails goes to a revision agent for a permitted revision (D4); wording changes need human approval, and revised questions repeat the audit and review. Hunter and verifier agents look for rival answers and omissions, and the authors then accept, accept with fixes, or reject each question (D5). A fresh checker agent and validation scripts produce the final records (D6); HotpotQA skips the checker.

The authors' review interface, with an offline demo: review_site/.

Flowchart of Stage 2. D1: scripts validate the inputs and evidence. D2 and D3: an auditor agent audits constraints and hops, then answer accuracy and completeness. On a failure, D4: a revision agent attempts a permitted revision and the revised question repeats the loop; a failed repair is excluded. D5: hunter and verifier agents look for rivals and omissions, then the authors accept, accept with fixes, or reject. D6: a checker agent re-checks the evidence, and scripts validate and finalize the record, which goes to Stage 3.
Stage 2's audit, revision and human review loop; D1 to D6 follow the paper's appendix. Dashed blue arrows route revisions through D4, and red arrows lead to exclusion. Open the image for full size.

The pipeline is dataset- and corpus-agnostic. Code, prompts and offline demos: pipelines/cmass.

Results

What three search agents show

  • Headroom for open-weight agents

    On BCP, Gemma 4 31B and Qwen 3.5 9B answer 27.5% and 16.3% of questions, against 85.0% for GPT-6.1 Sol, while making more tool calls.

  • Accuracy without full evidence

    Sol answers 90.1% of DSQA questions while grounding 75.7% of their hops. On HotpotQA it grounds 90.2% of hops from 17.2% of the relevant documents.

  • Strong closed-book baselines

    With no tools at all, Sol answers 89.7% of HotpotQA questions, against 94.6% with search. Closed-book runs check whether accuracy gains also occur without retrieval.

See the leaderboard

Get the data

Load CMASS from Hugging Face

The current version is CMASS v1.0 (25 October 2026), released on Hugging Face as v2026.10.25. Questions, answers and qrels are obfuscated to discourage accidental training contamination; decode them before evaluation. Pin the release tag and report it with your results: results on v2026.10.20 and v2026.10.25 qrels are not directly comparable. The corpus text is not redistributed; document ids refer to ClimbMix.

Shell
python3 -m pip install datasets
curl --fail --location \
  https://huggingface.co/datasets/castorini/cmass/resolve/v2026.10.25/deobfuscate.py \
  --output deobfuscate.py
Python
from datasets import load_dataset
from deobfuscate import decode_row

repo = "castorini/cmass"
revision = "v2026.10.25"
split = "bcp"  # Also: trqa, hotpotqa, dsqa.
queries = load_dataset(repo, "queries", split=split, revision=revision)
qrels = load_dataset(repo, "qrels", split=split, revision=revision)

query = decode_row(queries[0])
decoded_qrels = [decode_row(row) for row in qrels]
evidence_ids = [row["doc_id"] for row in decoded_qrels
                if row["query_id"] == query["id"]]

print(query["question"])
print(query["answer"])
print(evidence_ids)

Citation

Cite CMASS

BibTeX
@misc{sharifymoghaddam2026projectingbrowsecompplusclimbmixrealistic,
      title={Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search},
      author={Sahel Sharifymoghaddam and Lingwei Gu and Yijun Ge and Jimmy Lin},
      year={2026},
      eprint={2608.20317},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2608.20317},
}