Benchmark results on Brazil's tax reform environments

As of Oct 23, 2026, one model has been run on the sample pack: Claude Opus 5.5, effort high, with a fisco index of 88.6% (the mean of its pass rates on the 8 families in the index). No other model has been run yet; the rows under it are reference anchors, not models. R1, N1, N2 and N3 have not been run by any model yet and are not in the index.

Fisco index

One row per model configuration that was actually run. The reference anchors under it are not models: they bracket the scale, from the answer key to doing nothing, with a simple script that uses no AI in between.

Index = mean of the pass rates on the 8 families in the index (F1, F2, F3, F4, F1T, F4T, S1, I1), each family weighted equally; the mean-score index is the same mean over partial credit. Only a run that ran all of those families gets an index and a rank. R1, N1, N2 and N3 have not been run by any model yet and are not in the index. When a model runs one of them it joins the index, and from then on only runs that include it are ranked. Input tokens include cache reads and writes.

Pass rate by family

Bars are pass rates (every graded field right); whiskers are 95% intervals; the triangle marks what a simple script with no AI gets (for most families, assuming full taxation for everything). With samples this small the intervals are wide, so read differences of a few points as noise.

Families awaiting a model run

These families are in the sample pack, with their graders and reference anchors, but no model has been run on them yet. They are not in the index; when a model runs one of them, it joins the index and the leaderboard above.

What the lookup tools are worth

F1T and F4T leave out of the prompt what the agent has to look up: the annex passages, the NCM descriptions and the cClassTrib table. The same tasks were run with the five offline lookup tools and without them.

How much a rerun moves

Where a family was run more than once on the same tasks, every run is shown and the leaderboard uses the pooled figure, with an interval that treats each task, not each answer, as the unit. A single run of a few dozen tasks can move by several points from one day to the next with nothing changed.

Misses, checked against the law

Every miss of every model run is checked by hand against the law text and the official tables. The model's misses are mostly annex calls: água sanitária coded as sodium hypochlorite instead of the 3808.94.19 that Annex VIII names, refined soybean oil or yogurt placed in the wrong annex, suture needles given Annex III's health-services code instead of Annex IV's medical-devices code, chicken cuts given a neighbouring subheading. In the EFD audits it missed a line count and a code outside a field's list of valid values, and read ICMS CST 200 as a reduced-base CST on three correct lines. On ICMS (I1), CFOPs, rates and abstentions are right almost everywhere; the misses are the DIFAL base in some destination states. For Minas Gerais the answer key follows the state tax authority's own published worked examples. Without the tools, the misses are annex exceptions and conditions it does not recall.

How the numbers are made

Families and the sample pack

Every run here is on the free sample pack, the same tasks a lab gets to evaluate. Tasks are in Portuguese, on Brazilian documents; parties and invoices are fictitious, products, codes and rules are real, and F4 and F4T lines are real product descriptions and declared codes from public federal-purchase NF-e data.

Grading

• Deterministic, no LLM judge. Every answer is graded field by field by code: the TribAPI tax engine computes the expected treatment from the law, the official NCM and cClassTrib tables and the NF-e layout rules, and the grader compares. Rerunning the grader on a saved answer always gives the same score.
• Pass rate is the share of tasks with every graded field right: the binary reward. Mean score is partial credit, a field diff weighted by tax impact (F1, F1T), a fixes-only diff of the IBS/CBS group (F2), half item-level and half field-level F1 over the findings (F3, F4T), and its four fields (consistent or not, the right NCM, both cClassTribs), weighted, each an exact match (F4). EFD audits are graded like F3 over (line, rule) findings, and EFD repairs like F2.
• Format counts. The task asks for one JSON object (or the whole XML in F2); prose around it, a second block or numbers sent as strings score zero, for every model alike.
• Amounts pass within R$ 0,01, the tolerance the NF-e validation rules accept; planted errors are always larger.
• Abstention is graded. A 2027 sale, whose rates the Senate has not set, passes only when the agent abstains; abstaining on an answerable task scores zero.
• Runs. The task prompt is the whole input: no system prompt (tool tasks get only the tools' note stating the call budget), no examples, one answer per task and no retry on a wrong answer. Tool tasks run a tool-use loop under the task's call budget, and every call is logged.

Index and intervals

• Fisco index: the mean of a run's pass rates over the families of the sample pack, each family weighted equally, whatever its task count. A run that skipped a family gets no index and no rank; its family results are still shown.
• 95% intervals: Wilson score intervals on the pass rate when each task was answered once. When a family was run more than once on the same tasks, the interval comes from the per-task pass fractions (normal approximation), so repeated answers to one task are not counted as independent evidence.
• The index carries no interval of its own; the family intervals show how much each part could move.

Ground truth

Every task records the level its answer key rests on. In the sample pack, 267 of 279 tasks rest on our readings of the law (the NCM and NBS lists of the LC 214 annexes, the F4 product rules, each state's ICMS rules for I1, the R1 withholding rules and the EFD validation rules), which are machine-checked (level ai_verified): a reproducible check against the official texts and tables passes with zero errors, but sign-off by a Brazilian accountant is still pending. The other 12 rest on the official tables alone (verified).

Runs and miss analyses

Every figure on this page is generated from the committed run files by the same code that writes the internal baseline report, and a test fails if the page and the run files disagree. Each run folder has a README that goes through every miss against the law; they ship with the sample pack on request. Dates are in Brasília time.

Only models listed above have been run. Other frontier models are next; to have yours evaluated on the sample pack, write to us.