Every few weeks a new model tops a public leaderboard, and someone asks why we still invest so heavily in a search stack when "the model can just read everything." At AlphaSense, we run that experiment continuously so our users don’t have to: every major model release, every major feature update, month after month, against real-world tasks. This article summarizes what the current data says about which model wins and why, how we measure it, the critical role of our search and harness, and what we're building next.

The short version: In finance and business research, today’s bottleneck on answer quality is no longer raw model intelligence. It's context. Frontier models are extraordinary reasoners, but they are mediocre searchers, and the gap between those two skills shows up in both quality and cost.

A frontier model doing its own retrieval over our content with raw vector search (a good proxy for your typical market data/content MCP) costs roughly 3x more per question than the same model running in our harness with AlphaSense Search. And the AlphaSense Search-powered answers are preferred more than 2:1 compared to the vector-RAG baseline, all other things being held equal.

Here are a few model-specific takeaways that are discussed in greater detail later in the article:

• GPT-5.6 Sol tops our benchmark, delivering the best overall performance and quality/cost balance when restricting the system to using a single model.
• GPT-5.6 is more token efficient and thus cheaper overall than a leading open-weights model, Kimi K3.
• Opus 5 over-fetches and delivers worse performance that is 5x more expensive than Opus 4.8, showing that upgrading to the latest models is a non-trivial decision and depends on the task at hand. (Incidentally, Opus 5 is great at slide generation.)
• Gemma 4, an open-weights model from Google DeepMind, delivers the same performance at 40x lower cost than Sonnet 5.

The rest of this post, the first of an ongoing series, covers where these gaps come from, how the benchmark works, what we learned about individual models, and the two engineering programs designed to widen the gap: training our own search agent, and pre-computing the answers our users are about to ask for.

Why the Best Models Are Bad Searchers

Hand a frontier model an MCP on financial documents and a hard research question, and you’ll have to design around three failure modes:

Query generation. Real questions arrive with fiscal-period arithmetic, ticker collisions, speaker specificity, and entity type ambiguity (e.g., is JP Morgan a company or a source). Treating everything like a web search doesn’t work in finance.

Prioritization. Embedding distance is indifferent to source authority. An earnings call transcript, a sell-side preview note, a news article, and a 10-Q filing all score similarly on cosine similarity, but their relevance in the eyes of a human analyst is very different depending on the question being asked. You can trust Twitter for your jokes, but not for the numbers going into your model. Today’s LLMs understand what these documents are, but knowing they’re there, finding them efficiently, and prioritizing them effectively requires a specialized harness.

Stopping. Models are surprisingly bad at knowing when to stop searching and when to go deeper. Models often call it quits too soon (retrieval results bring up 2025 Q4 financials on a question of current financial performance) or don’t know when the document they are looking for doesn't exist (like a future quarter’s financial performance). They are trained to compensate for low-precision retrieval with volume: fetch more, skim, re-fetch, and reason over a context window increasingly full of duplicates and noise. Long, diluted contexts degrade answer quality while inflating cost. This is why AlphaSense is 3x cheaper: With a complete search index trained and tuned on tens of millions of searches, freshness, authority, and relevance scores can be returned for query sessions, letting the model know when to stop.

Some of the most capable models we tested were the worst offenders on token discipline (see Opus 5). AlphaSense Search represents more than a decade of data processing, indexing, and ranking engineering that turns out to be exactly what an agent harness needs.

The net effect is that in an optimized setup, the LLM spends its tokens reasoning over the right context instead of hunting for it. Same model, same corpus: about a third of the cost and significantly better answers.

The Advantage This Benchmark Doesn't Measure: Content

Every configuration, including all models across the vector-RAG baseline and AlphaSense Search alike, runs over the same indexed corpus. That isolates the retrieval, harness, and model variables, which is a useful experimental design.

It also means the results measure none of the work upstream of the index: a large proprietary library of expert interviews and channel checks; sell-side research, global filing coverage, event transcripts, and industry trade journals; and the pipelines that normalize, enrich, link, and tag all of it. The gaps reported here are the floor, measuring just the search agent: The real-world delta, content included, is larger than anything in these charts.

Methodology

The benchmark is based on 245 challenging, multi-step finance questions modeled on real analyst workflows: time-anchored, cross-source, and deliberately hard. We create many benchmarks across features and use cases in AlphaSense, and we update them regularly as tasks evolve and complexity increases.

For each question, we generate an independent scoring rubric before any model produces an answer. Pre-registering rubrics is useful because it prevents graders from drifting toward whatever the models happen to output, and it lets us encode hard requirements up front.

For example, one question on RevPAR is scored out of 62 points: 6 mandatory core checks (26 pts), 14 quality signals (36 pts), and 3 penalty checks (−7 pts).

Retrieval quality and answer quality are graded independently by language-model judges. Results are reported as relative scores (vector-RAG baseline = 1.0) and head-to-head preferences.

Results

Holding the baseline model (Opus 4.8) constant and swapping only the retrieval layer moves preference to 1.7:1, the purest measurement of what the AlphaSense Search stack alone contributes. Choosing the best frontier model (GPT-5.6 Sol) on top of AlphaSense retrieval pushes preference to 1.8:1. Leveraging the optimized AlphaSense harness around that model (i.e., query generation, planning, search score pass-through) gets to 2.1:1, and routing each task to the best model for that task type reaches 2.8:1.

The per-model picture, with everything running on AlphaSense Search:

Quality scores are medians from our July 2026 harness run, relative to the vector-RAG baseline of 1.0. Costs in the chart are median per-question totals including retrieval-loop tokens, not list token prices.

A few datapoints deserve commentary:

Orchestration wins over any single model. No model wins more than a plurality of the benchmark questions. Give the smartest model a lot of great tools, and it can forget to use the simple tool for simple tasks. Different models win on different question types: metric extraction, screening, long-form synthesis. That's why per-task routing produces the 2.8x result. Model routing is not only about cost savings; it’s about leveraging the best model for each use-case, and getting an optimal combination of quality, cost, and speed for each task.

Newer is not automatically better. Opus 5 scores below its predecessor Opus 4.8 (1.46 vs. 1.60) at several times the per-question cost. The main issue is that it over-fetches: more retrieval calls, longer contexts, worse synthesis. On the other hand, Opus 5 has proven to be a great model for Slide Generation. This is a strong argument for continuous model evaluation: In-product performance is a property of the model and use case pair, and it drifts with every release. Without a test suite and rigorous testing, regressions can ship to production silently.

Token price does not equal question cost. Kimi K3 is a lot cheaper per token than GPT-5.6 Sol, yet more expensive per question, because it spends many more tokens assembling context before it can answer. When you buy tokens by the million but deliver answers by the task, tokens-to-completion is the metric that matters.

Open weights will play a critical role. Gemma 4-31B matches Sonnet 5 on answer quality at roughly 40x lower cost. Gemma is a great candidate for sub-agent delegation by a frontier lead model. Open weights mean it’s served on Cerebras hardware where it runs 10–20x faster. The combination of good-enough quality for many tasks and very low latency unlocks high-volume and interactive use cases that would be uneconomical on frontier models.

Why This Matters Right Now

With indiscriminate tokenmaxing mostly behind us, and companies looking for cost control and ROI from their AI investments, the answer lies in getting task-by-task model choice and harness optimization right.

This becomes even more critical as we move to longer-running agentic workflows, where all of these quality and cost benefits (and issues) compound: Even if 95% accuracy was acceptable in a single step with human oversight, this compounds to a coin-toss accuracy in a 10-step agentic process.

What's Next

Two programs aim to move the frontier at AlphaSense even further.

Training our next custom search agent. The evaluation machinery above is also, conveniently, a reinforcement learning environment: pre-registered rubrics, independent judges, and millions of search trajectories. We're using it to fine-tune our next proprietary model specialized in context collection: query planning, retrieval-tool use, coverage tracking, and context evaluation. Our objective is model orchestration-level context quality (~2.8x) at up to 40x lower cost.

Frontier models remain critical. Training a custom search agent does not mean moving off frontier models. Planning, deep multi-step reasoning, coding, data analysis, and Excel/PowerPoint generation stay routed to frontier models, exactly as our routing results argue they should. Context collection is one step in the pipeline — currently the most expensive one — and the step where a specialized model, trained on our corpus and our reward signal, beats a generalist.

Pre-computing the questions users are about to ask. The most efficient token is the one never spent. A growing share of our workload leverages recurring structures: supply-chain relationships, channel check signals, and the key debates around companies, industries, and cross-sector themes.

Today, an agent reconstructs these from hundreds of documents, per user, per query, at run-time. We're building offline pipelines that pre-compute them as first-class, queryable datasets — supply-chain graphs assembled across filings, transcripts, and expert calls; channel-check syntheses coordinated by AI interviewers; key-debate ledgers maintained per ticker with the bull and bear evidence attached, updated as new documents land.

Our SuperAnalyst then starts from the pre-computed structure and spends inference only on the delta, each new question, new document, new angle. For these workflows, that's not an incremental saving; it's orders of magnitude, with better consistency, lower latency, and cleaner citations as side effects. Just like standardized fundamental data helped analysts avoid extracting the same data manually from unstructured company disclosures, we are now structuring the next major layer of analysis that everyone has been doing independently. The result: high quality, speed, and efficiency all at the same time.

The Durable Asset

GPT-5.6 Sol holds the quality bar today with good cost efficiency. But as models keep leapfrogging each other, the best choice today might not be the best tomorrow. Your intelligence layer should be able to adapt at a moment's notice, and that's the business we're in. What compounds on our side is everything around the model: the corpus and the pipelines that feed it; a retrieval stack that makes any model 2x better and 3x cheaper; a routing layer that captures the best of each model release; and, underneath all of it, an evaluation harness rigorous enough to catch a flagship regression within days, and to serve as the reward signal for training the next generation of our own agents.

Finance and business demand frontier models with frontier context, and that’s what we’re building at AlphaSense. We will be releasing regular benchmarking of popular closed- and open-source models and tough datasets that represent the work our customers are doing on our platform.

Appendix

What the question set looks like

Questions are classified along four dimensions before anything is run. The mix in the 245-question set:

There are two main properties in each query that are worth calling out because they drive most of the difficulty:

1. Time Sensitivity: Most questions have implicit or explicit date requirements (e.g.,“latest,” “as of July 2026,” “since mid-May”). The answer and the correct documents change very quickly, so systems can't memorize the correct answer. This makes the dataset quite hard and is also a major gap in existing benchmarks, which are usually frozen in time for years.
2. Source Requirements: Many questions provide explicit instructions on where they expect their data (e.g., “Only Expert Transcripts, official company-provided figures, tier 1 broker report”)

Queries are also deliberately dirty and complex, just like real-world usage. Typos, ticker-only references, mixed languages, and contradictory instructions are part of the workload.

Sample benchmark questions

Here are 14 synthetic questions, representing the types of questions our users ask, which span the list above:

1. Find me commentaries for 2Q26 same store sales trends and FY26 SSS trends (latest) for MCD YUM QSR WEN DPZ CMG and SBUX. one line for 2Q26 one line for FY26, so 14 line total
2. ORAL-SLIM 3 did not meet its co-primary endpoint (announced this morning). Please can you analyse commentary from Eli Lilly, Novo Nordisk, Structure Therapeutics and any other pharmaceutical companies with competing oral GLP-1 programs?
3. Analyze the key trends emerging this year for firms providing HVAC technician certification and skilled-trades training services. Focus on shifts in demand driven by the technician labor shortage, impacts of state licensing changes and EPA Section 608 refrigerant-handling requirements, and how training providers are adapting to new technologies such as VR labs or online instruction. Discuss trends in tuition pricing, employer-sponsored apprenticeship programs, and capacity constraints within the industry.
4. Give me a detailed rundown of all the major acquisitions completed by CWST - make sure all the data is up to date as of July 2026
5. Pandora reports its Q2 CY2026 results on 22 July 2026 (quarter to end-June 2026). Give me the latest sell-side setup: (1) consensus organic revenue growth for Q2 CY26 total AND by region (US, Italy, rest of Europe, China); (2) each recent broker preview note (June-July 2026) with its Q2 organic growth estimate, rating and price target; (3) the key investor debates into the print (US consumer softness, lab-grown diamond ramp, gross margin vs silver and gold input costs, tariff pass-through, buyback pace).
6. Identify the top 7 prominent trends in the global and US paper and packaging industry over the last 8 months as of July 12, 2026. Focus specifically on digital innovation (automation, AI-enabled quality control), smart operations, shifts in the containerboard capacity landscape, restructuring, and asset or investment readjustments, as well as workforce dynamics. For each trend, explain its strategic and operational implications for the packaging sector.
7. Evaluate the market opportunity and competitive landscape for smart electricity meters and NB-IoT connectivity modules for HPL Electric & Power within the Indian market. Benchmark against global peers to estimate the potential size of the AMI opportunity. Analyze historical pricing and realization trends for these products over the past few years. Identify the key drivers of competitive advantage and switching costs for customers. Address the production and consumption dynamics (global vs. local), the current demand-supply balance, and unit economics, including potential for economies or diseconomies of scale. Assess the industry's Porter's Five Forces and utilize economic frameworks such as market structures (oligopoly, monopolistic competition) and game theory to project how competitive dynamics may evolve.
8. Can you find the operating lease commitments for KHC as of q1 2026
9. What is Databricks' latest estimated Revenue, ARR and gross margin as of 2026?
10. looking at Anduril - what are the experts saying about the company, the prospects, the financials, the recent funding round and the valuation
11. Make table of companies with clinical-stage oral GLP-1 and amylin assets. Table should list the company name, market cap, enterprise value, 2025 revenue, 2028 consensus revenue, lead program name, stage, and therapeutic area. If the company is private, list latest post-money valuation or fundraise size including date of such in final column and note n/a in the market cap and EV columns
12. look at US regulated utilities highlight any updates to the dividends declared, any debt issuance, and any quarterly filings. Also include any commentary on market outlook on past performances for the last quarter and what they expect in the future. For formatting, group and organize all of these updates and insights by company
13. Extract details of all list-price increases implemented by the logistics public companies since 2018 from their conference call transcripts. Present the findings in a table categorized by channel DTC, Wholesale, and Franchise including the date the price increases occurred or were announced and the expected timing of the next price increase.
14. What is the current effective US tariff rate on imported wind turbine nacelles and blades from Vietnam as of July 2026?

What is a rubric?

A rubric is a representation that defines what a successful system should produce to meet user needs. The rubric is generated before any systems are run and any outputs are known. Once a rubric is generated, it is frozen. The generator sees only the question, the as-of date, and two derived signals (time-sensitivity and breadth-sensitivity). It never sees an answer, so the rubric can't be anchored to any one system's output.

A rubric is grading the evidence, not the prose of the output. Each criterion in the rubric must be evaluable solely from the retrieved documents. There is a separate rubric that grades the generated answer, which allows us to understand its quality coming from what is found or how it's read. This broadly pushes our evaluation to recall-focused. A set containing the right evidence plus unrelated noise is not penalized for the noise.

How is a rubric evaluated?

A rubric has 7 axes for each criterion (a question can have 1-60 criteria depending on difficulty). Rubric size scales to complexity: a pinpoint lookup gets 3–5 criteria, a bounded comparison 10–16, a multi-hop landscape 18–30. Question 7 above draws 29. Each criterion carries an integer weight from −5 to +5, never 0.

Anything over 4 is a mandatory requirement, and anything negative is a characteristic failure (a missing sub-question, vague claims with no figures, no document about the required entity). Staleness, weak sourcing, and narrowness are never negatives. Roughly 40% of the positive weight is targeted on temporality, authority, and breadth. Topicality is less of a focus because it's implicit table stakes.

Worked rubric — a seven-ticker fact pull

From question 1: "Find me commentaries for 2Q26 same store sales trends and FY26 SSS trends (latest) for MCD YUM QSR WEN DPZ CMG and SBUX. one line for 2Q26 one line for FY26, so 14 line total"

It has 11 criteria, of which 9 are mandatory. Criterion text is condensed for readability.

What passing and failing evidence look like

1. Satisfied (c04 + c07 + c10): "Domino's delivered US same-store sales growth of 3.1% in the second quarter" Domino's Pizza Q2 2026 earnings call transcript, 8 July 2026. A specific figure, a stated basis, a dated primary source inside the window.
2. Partially satisfied (c04): "Domino's cited improving US comps in its latest quarter" broker note, 10 July 2026. Right company, right direction, no figure and a secondary source half credit on c04, no credit on c07.
3. Not satisfied, and triggers c10: "Pizza-category demand was solid through the spring." No company, no figure, no period.