Frontier-level data wrangling at a tenth of the cost

1. Which steps can a cheaper model take over?

An analysis is a chain of steps of very different difficulty. Some are open-ended: choosing the question, the method and the decision a result supports. Others are routine data wrangling: recognising what a column contains, finding the columns that join two tables, or converting a table from JSON to CSV. A frontier model handles both kinds of step, at frontier prices.

An agent harness is the software that turns a language model into an agent: it gives the model its instructions and tools, runs the tools the model calls and feeds the results back to the model until the task is done. The explai agent harness supports data scientists, analysts and business teams from the first look at a dataset to a recommended action. Several specialised agents share this work, and each can run on a different model, so we need to know which steps a cheaper model can take over. The same question decides what a desktop version of explai can run locally, a direction we outlined in Density is all you need.

Public leaderboards can’t answer the question for us as they test the bare model. In our product, the model runs inside the explai agent harness, which affects both accuracy and cost. We therefore measured the models inside it.

2. Methods

2.1 Benchmark

LiveBench is the first of nine benchmarks we have now run on models inside the explai agent harness. We start with it because it tests basic skills.

LiveBench (White et al., 2024) scores answers with code against known solutions, without a language model as judge, so the same answer always receives the same score. Of its seven categories, we used Data Analysis, the one closest to our product. It has 150 questions in three tasks of 50:

• Column type annotation: name a column from a few of its values. Scored right or wrong.
• Table join: say which columns of two tables match. Scored by F1, so a partly correct answer earns partial credit.
• Table reformat: convert a table to another format, such as JSON to CSV. Every cell must match.

A model's score is the share of available points it earned. We used release 2024-06-24, the latest release with public Data Analysis questions.

2.2 Models

We compared Claude Opus 5 with four open-weight models (Table 1). Two of them are mixture-of-experts (MoE) models, which activate only a subset of their parameters for each token (Shazeer et al., 2017). This makes them cheap to run, but all of their weights must still be held in memory.

2.3 The explai agent harness

Every question passed through the same harness; only the model changed (Figure 1). The harness sends the model its instructions and the descriptions of about 67 tools, 28,000 to 42,000 tokens per call, followed by the question and one line specifying the answer format. We did not tune prompts for any model, turned web search off and did not enable extended thinking for Opus 5. The open models reason before answering by default.

2.4 Statistics and cost

Each score has a 95% interval: the range the score would very likely fall in if we could ask many more questions of the same kind. A score of 77.0% (70.8–82.8%) means the model's true score most likely lies between 70.8% and 82.8%. Because every model answered the same questions, we also compare each model with Opus 5 question by question (Miller, 2024). When the interval of that difference includes zero, the data cannot separate the two models.

We report cost per correct answer: the cost of a run, at the providers' prices, divided by the points it earned.

2.5 Reproducibility

The harness pins the benchmark release, runs LiveBench's scoring code unchanged, records the tokens and cost of every call, and stops rather than silently switching to a different model. Every model answered all 150 questions once with the same settings, and empty answers count as wrong. Per-question results are available on request.

3. Results

3.1 Accuracy and cost

DeepSeek-V4-Flash finished 2.5 percentage points behind Opus 5 (95% interval: 7.5 behind to 2.6 ahead), and GLM-5.3-flash 3.2 points behind (9.1 behind to 2.9 ahead). Both ranges include zero, so the gap could be down to chance: on these 150 questions, we cannot say that Opus 5 is better than either model. Qwen3.8-27B (69.9%) and Qwen3.5-9B (57.6%) are measurably lower.

Cost reverses the order (Figure 2). A correct answer from DeepSeek-V4-Flash cost a tenth as much as one from Opus 5, and one from GLM-5.3-flash a fifth as much. Qwen3.5-9B has the lowest price per token, but it earns fewer points, so each of its correct answers costs more than one from DeepSeek-V4-Flash.

3.2 Results by task

The models differ most on table reformat (Figure 3). Opus 5 converted 48 of 50 tables exactly, while the open models converted 34 to 42. On table join, DeepSeek-V4-Flash and GLM-5.3-flash scored about 7 points above Opus 5, and on column type annotation they matched it. With 50 questions per task, gaps under about 10 points are within the margin of error.

3.3 Self-hosting cost

Self-hosting DeepSeek-V4-Flash or GLM-5.3-flash takes two to four H200 GPUs, at an estimated $3,300 to $6,100 a month. With an existing model used through an API instead, a system optimised for caching such as explai costs less than €100 per user a month in our experience. The dense models run on a single card for about $800 to $1,000 a month, and the 27 billion parameters of Qwen3.8-27B at 8-bit also fit in the memory of a strong laptop, but both scored lower (Figure 4).

3.4 Tokens, time and failures

Most tokens are the harness's own instructions (Figure 5). Opus 5 wrote about 460 output tokens per question; the open models, which reason first, wrote 2,600 to 10,400. Prompt caching (Amazon Bedrock documentation) cut Opus 5's cost from about $0.43 to $0.067 per question.

A typical question took about a minute on every model, with medians of 58 to 72 seconds. Opus 5's slowest answer took about two minutes; each open model had answers that took 15 to 35 minutes, and every model's slowest answer was wrong. Fifteen of the 750 answers came back empty, none of them from Opus 5. Ten of these, including all seven from GLM-5.3-flash, exceeded the reply limit on both attempts.

3.5 Sample size

We drew 10,000 random sets of 30 questions from the 150 and scored all five models on the same sets, using their existing answers. On sets this small, the scores of the top four models overlap almost completely (Figure 6), and DeepSeek-V4-Flash scored higher than Opus 5 on about a third of them.

4. Discussion

4.1 Cheap per answer, but too big for one GPU

The two MoE models that matched Opus 5 have 284 billion and 320 billion parameters in total, but each uses only 13 billion or 18 billion of them per token. That makes them cheap to serve through an API, yet hosting them needs good hardware: GLM-5.3-flash needs about 160 GB for its weights even at 4-bit. The dense models that fit on one card trailed. Five models cannot separate the effects of size, architecture and training data, but for local deployment the implication is clear: on these tasks, models that run on modest hardware do not yet match frontier.

4.2 The harness rarely needed its tools

Each LiveBench question contains its own data, so most answers needed no tools. Opus 5 and Qwen3.5-9B made no tool calls at all. The other three models called tools on 11 to 24 hard questions and scored no better on those than Opus 5 did without them. This test therefore measures the models within our instructions more than our tools.

4.3 Some questions cannot be answered

Nine column-type questions were missed by all five models, and at least two cannot be answered from the data given. One shows five prices between 41 and 58 and asks whether they come from the Open, High, Low or Close column. Every model answered Close; the answer key says Low. Such questions lower all scores equally, so the comparison holds, but they reward a lucky guess rather than analysis.

4.4 Thirty questions are not enough

A 30-question test could not have ranked the top four models. We now run every benchmark in full and report an interval with every score.

5. What this means for explai

Column typing and table matching are candidates to route to an open MoE model, with Opus 5 as the fallback. Exact transformations stay on the stronger model or get a deterministic check, such as comparing the row and column counts of input and output. Reasoning models need a larger output budget and a time limit, since their slowest answers were wrong. Finally, instruction size drives cost: caching handles it in the cloud, but on a laptop, processing tens of thousands of instruction tokens before the first word of an answer is the main delay for a desktop version.

6. Limitations and next steps

This study covers one benchmark with three table-centred tasks, one run per model, and Opus 5 without extended thinking. Our next posts report full runs of the other eight benchmarks, covering SQL, function calling, multi-turn tool use, charts, statistics and data-science code, and compare the bare models with the same models inside the explai agent harness. We will also run Opus 5 with extended thinking, repeat runs to measure variation, and test on-device as the desktop version takes shape. LiveBench keeps its newest questions private, so our scores are not comparable with its current leaderboard; because all five models answered the same questions, the comparison between them holds.

Try the explai agent on your own data in the explai Sandbox.