Local meeting notes, benchmarked

This page is generated from the same benchmark results that generate the model catalog shipped inside the app — the numbers you read are the numbers the app acts on.

Abschrift writes meeting notes on your Mac by default: a built-in runtime plus an open model picked for your hardware, downloaded once. That promise is only honest if the local model is actually good. So we measure it — with a benchmark built around the one job that matters: turning a messy meeting transcript into notes you can act on. This edition covers 17 open models from 2B to 122B parameters, from an 8 GB MacBook Air to a 128 GB Mac Studio.

What we measure

Six synthetic meetings (no real people or companies), written to be hard in the ways real meetings are hard: short stand-ups to two-hour strategy workshops, English and German, 4–23k tokens of transcript. Each comes with a scripted ground truth: every fact, decision, and action item that correct notes must carry, with owners and deadlines. Each also contains noise that good notes must drop, like small talk and venting. And each sets attribution traps: heated accusations whose substance belongs in the notes but whose finger-pointing does not.

Every model gets the app's actual notes prompt and identical sampling. Models up to 12B ran on the app's actual bundled runtime (a pinned llama-server build, Q4-class quantization) on a base M1, so speed is measured for real. Models too big for the measurement machine were generated through the OpenRouter API at provider precision and are marked “API” below. Their Apple-Silicon speeds are estimates. The methodology section at the end explains how.

A frontier model, Claude Fable 5, grades each output against the ground truth and the full transcript: fact coverage, false claims weighted by severity, action-item fidelity, section structure, attribution neutrality, noise rejection, and whether the Discussion section alone could power follow-up work. Two grading passes per output, averaged. The same rubric grades notes written by Claude Fable 5 itself. That's the bar, and it scores 97.4.

Does a bigger model write better notes?

Results

Scores are 0–100 composites (weights: facts 30%, correctness 20%, action items 15%, discussion quality 15%, attribution 10%, structure 5%, noise 5%). Long meetings is the composite over the 90–110-minute transcripts only — where small models fall behind first. (A…B) marks mixture-of-experts models: total (active) parameters.

Quality × speed: the map that picks your model

What your Mac gets

The app maps unified memory to a default model. It only asks you to choose when the benchmark shows a genuine quality/speed trade-off on your tier:

Blank cells mean the chip never shipped with that much RAM. Models above ~50 GB download in multiple parts. The app handles that automatically, each part checksum-pinned.

Why isn't Muse Glimmer 30B (composite 91.5) offered in the app? Its on-device runtime path has not been validated on real hardware yet and its mandatory reasoning makes notes take ~2.4x as long as the tier default's — it stays benchmark-only until that changes.

How long does a meeting take?

End-to-end notes generation for a ~45-minute meeting:

Methodology & honesty

• Two measurement paths. Models up to 12B were generated on the exact runtime and quantization the app ships, on a base M1 (16 GB) — quality and speed both measured. Larger models were generated via OpenRouter at the serving provider's precision (typically FP8/BF16) and judged identically. The shipped Q4-class quantization typically costs large models a point or two of quality, so treat their scores as a slight upper bound.
• Speed estimates are anchored, not guessed. Per-chip scaling comes from the community-measured llama.cpp Apple-Silicon benchmark table (a raw memory-bandwidth ratio would overestimate Ultra-class chips almost 2×). For MoE models the effective bytes read per token are calibrated against measured gpt-oss numbers — the calibration reproduces the measured anchors within ~5%, and we still publish ±25% (dense) / ±40% (MoE) error bars.
• RAM floors are computed conservatively: model file + a 32k-token KV cache + compute buffers must fit macOS's GPU working-set limit (~75% of unified memory). Floors for models we could run were validated on real hardware. The rule correctly predicts both the fits and the out-of-memory failures we observed.
• Exact models and modes. Baseline notes: Claude Fable 5, default thinking configuration, the app's exact prompt. Judging runs against the Anthropic API: each output is graded against the meeting's ground-truth fact sheet plus the full transcript. Of the 216 grading passes: 134 by Claude Opus 4.8, 82 by Claude Fable 5 — the judge model has changed over the benchmark's life. Every verdict records its judge, and the two passes per output are averaged, so they can come from different judges. Open models ran with thinking disabled (the app's own runtime setting — hybrid-thinking models like Qwen3.5 are graded in the non-thinking mode they actually ship in), except gpt-oss 20B/120B and Muse Glimmer 30B, whose formats cannot disable reasoning: those ran at default effort with reasoning tokens excluded from the graded output.
• The corpus is synthetic — that's what makes it publishable and exactly gradable. We additionally sanity-check winners on private real meetings before they enter the catalog. Aggregate impressions only: that data never leaves the machine.
• An LLM judge with ground-truth fact sheets is strict but not perfect. Two passes are averaged to reduce variance. The judge and the baseline writer are both Claude models — a judge grading its own kind is a known bias, which is why the baseline is a bar, not a contestant.
• When a new model generation looks promising, the benchmark re-runs, and this page and the in-app catalog regenerate together. A cloud provider with your own API key remains the quality ceiling — one click away in Settings.