Reading a text-to-CAD benchmark before you trust the score

Sat, 07 Nov 2026 · 7 min read · Written with an AI assistant

Chamfer distance, invalidity rate and IoU are now quoted on vendor slides. Here is what each one measures on a real part, where scores collapse, and what no benchmark checks yet: feature trees, constraints, assemblies, sheet metal and SOLID

A vendor slide says its text-to-CAD model has a low Chamfer distance and a single-digit invalidity rate. Neither number tells you whether the part it built is the part you asked for, or whether you can still edit it next week.

Several text-to-CAD benchmarks appeared on arXiv this year, and their scores are starting to travel into sales decks. If you are the engineer asked to judge one of these tools, it helps to know exactly what each metric counts, and which questions none of them ask.

Four numbers, one question

Take a part most of us have drawn a hundred times: a 100 × 60 × 20 mm mounting block with a Ø20 bore through the centre, two Ø9 clearance holes on 70 mm centres, and a 2 mm chamfer on the bore. A text-to-CAD model trained on open data will usually emit something like this CadQuery script:

import cadquery as cq
block = (cq.Workplane("XY").box(100, 60, 20)
         .faces(">Z").workplane().hole(20)
         .faces(">Z").workplane()
         .pushPoints([(-35, 0), (35, 0)]).hole(9))

Note what is missing: the chamfer. Now score it.

Invalidity Rate (IR). Text2CAD-Bench counts a sample as invalid if the code throws a syntax or runtime error, runs past a 60-second timeout, or produces empty or degenerate geometry. Our script runs and makes a solid. It is valid. IR is a test of whether something was built, not of what.
Chamfer Distance (CD). The same benchmark uses CD to compare the generated mesh with the ground-truth mesh; lower is better. A 2 mm chamfer on one edge moves very few surface points. The block will score close to perfect.
IoU. Text2CAD-Bench voxelises both meshes at 256³ and measures volumetric overlap. The block’s volume dwarfs the missing chamfer. Again, close to perfect.
Command and parameter accuracy. DeepCAD proposed Command Accuracy, which checks whether each predicted CAD command is the right type, and Parameter Accuracy, which checks the command’s values. Drawing2CAD notes the catch: parameter accuracy is only evaluated for commands whose type was predicted correctly. If the model writes a cut-extrude where the reference used a revolve, that feature’s dimensions drop out of the parameter score entirely.

The same family of metrics is still standard; RealCAD reports command accuracy, parameter accuracy, median Chamfer distance and invalidity ratio. So when you read a score, ask one question of every number: would this metric have caught the missing chamfer, or a Ø18 bore where Ø20 was asked for? Mostly, the answer is no.

The cliff at sweeps, lofts and shells

Text2CAD-Bench has 600 human-curated examples on four levels. L1 and L2 are basic geometry with standard features. L3 adds complex topology and freeform surfaces. L4 covers real-world domains beyond mechanical parts.

The results are the useful part. From L1 to L3, the authors report that Chamfer distance rose 1.3–2.1× on average and invalidity rose from about 15% to 70–90%. The steepest drop came between L2 and L3, exactly where sweep, loft and shell operations appear (Text2CAD-Bench).

That matches shop-floor experience. A prismatic block is extrudes and holes. A cast housing with a shelled body, a swept cable channel and a lofted transition is where real modelling time goes. A headline score averaged over easy parts tells you little about the parts you would actually hand to a tool.

A second finding is worth pinning above your desk. The domain-specific CAD models tested (Text2CAD, Text2CADQuery and CADFusion) had lower invalidity but much higher Chamfer distance (Text2CAD-Bench). They ran more often and landed further from the target. Code that runs is not the same as the right geometry, so never read IR without CD beside it.

Prompt wording also moved results. Geometric-style prompts did better overall, but on L3 some general models got lower CD from procedural, step-by-step prompts, and the authors suggest matching prompt style to geometric complexity (Text2CAD-Bench). If a vendor demo uses a carefully worded prompt, ask to see it, then try your own.

What no benchmark scores yet

The most honest page of Text2CAD-Bench is its limitations section. It does not cover sheet metal, multi-part assemblies or parametric families with complex constraints, and it names bend allowance and mate constraints as problems needing their own evaluation. Output is CadQuery only, which the authors say limits comparison with proprietary formats, giving SolidWorks macros as their example. Aerospace and automotive parts are underrepresented.

CadQuery is the common target across this research. CAD-Recode turns point clouds into CadQuery Python, and even T2C-Bench, which runs six systems on 900 prompts under one protocol, compares tools on generated geometry. Text2CAD produces parametric sequences with an autoregressive transformer, but a sequence is not a SOLIDWORKS feature tree with fully defined sketches.

Each gap becomes a question for any vendor:

Feature tree. Does the output open in SOLIDWORKS as named, ordered features that rebuild without errors?
Design intent. Are sketches fully defined, with dimensions referencing the faces you would choose, or fixed in space?
Assemblies. Can it add or change a part without breaking mates in the parent assembly?
Sheet metal. Does it use the sheet metal feature with your gauge table and bend allowance, or fake it with extrudes?
Your sector. If you work in aerospace or automotive, what was it tested on?

A benchmark score cannot answer any of these. A demo on your own file can.

Asking beats guessing

The result that matters most for review is from Clarify Before You Draw. Letting the agent ask clarifying questions before generating cut Claude Sonnet 4.5’s mean Chamfer distance by 79.9% and lowered its invalidity ratio from 4.8% to 0.9%. Most prompts an engineer types are underspecified: which face is the datum, is that hole tapped or clearance, does the wall thickness hold after the shell?

So test the behaviour directly. Give the tool a prompt with a deliberate gap, such as “add a mounting hole near the edge” with no size or position, and watch what it does. A tool that silently picks Ø6 at 10 mm from an edge has guessed. A tool that asks has treated you as the person who decides.

This is the behaviour we are designing for in AI-CAD, our engineering copilot for SOLIDWORKS. It is a prototype. Our target is at least 98% precision on bounded tasks, and asking the engineer when the system is not confident. That is a target, not a result, and we have no benchmark scores to quote.

What this means for tooling: build your own acceptance test

Public benchmarks are a reasonable filter, not a purchase decision. The test that matters uses your parts. A practical shape for it:

Pick 10–20 bounded change tasks from real past ECRs: move a hole pattern, thicken a rib, change a bore and its chamfer, add a shelled pocket. Include a few with sweeps or shells, since that is where scores fall away.
Score execution. Did it produce valid geometry, yes or no?
Score deviation. Compare against your own revised model. Check the changed feature, not just the whole-part overlap.
Rebuild the tree. Force a full rebuild (Ctrl+Q) and record errors and warnings.
Edit one dimension. Change a key value by hand. Does the model update as the intent says it should?
Include ambiguous prompts and record whether the tool asks.
Use a human reviewer. Even Text2CAD-Bench, which scored L4 with a vision-language model on a 0–10 rubric, had three annotators with engineering backgrounds re-score a subset to check the automated judge (Text2CAD-Bench). Your senior designer is your annotator.

Keep those test files in-house. Your past change requests are exactly the design IP you would not paste into a public service, so run the evaluation on tools that can run inside your environment.

Chamfer distance, IoU and invalidity rate are honest measurements of narrow things. Read them as answers to “did it build roughly the right lump?” and nothing more. The questions that decide whether a tool belongs in your workflow, such as whether the tree rebuilds, the dimension edits and the tool asks before it guesses, are still yours to test. If you run a test like this and want to compare notes, follow the journal or get in touch.