IBM’s Carbon design system report
Carbon is mature, but more official guidance did not make agent output more reliable.
How well does Carbon Core stay aligned from design through production?
72.5/ 100
Carbon’s 72.5 out of 100 is a good readiness result. Its tokens, components, patterns, and operating guidance are mature, while system-wide alignment checks and published quality results remain incomplete. The separate agent tests add a practical warning: official access by itself did not improve working-software success, so teams should keep tests and review in the loop.
Evidence boundary: This report distinguishes missing public evidence from a missing capability. When a current result was not found in public sources, IBM may still have it internally and can share it for a re-run.
Foundations
75 / 100
Carbon has mature tokens, themes, and delivery. This study did not find a published result verifying every supported output together.
Bindings
68.75 / 100
Carbon has useful design-to-code mappings, but some Tile and TableRow properties remain disabled, incomplete, or unexplained.
Components
68.75 / 100
Carbon’s components and accessibility guidance are mature, but this study did not find published state and responsive results for the full library.
Structure
75 / 100
Carbon documents grid, composition, pattern, and focus behavior well. This study did not find published results from repeated tests of complete product flows.
Governance
75 / 100
Carbon has clear ownership and operating guidance. This study did not find public trend data showing how those practices perform over time.
FoundationsCarbon has a clear token hierarchy, four supported themes, and published foundation packages.Strong shared values help products stay consistent across brands, themes, and platforms.First confirm whether IBM already checks Figma variables against every released token package and theme. If not, add a release check that blocks unexplained differences and identifies the affected output.BindingsMost sampled Figma controls map to code, while Tile and TableRow retain unresolved or commented mappings.Unexplained differences make designers, engineers, and agents guess which surface is authoritative.Restore, replace, or explicitly classify unresolved Tile and TableRow mappings. Then fail a release when Figma properties, Code Connect mappings, or released APIs differ without an approved exception.ComponentsCore components are well documented. The study did not find one current published view of required states, viewports, interactions, and assistive-technology results across the library.Teams need to know that components work in the difficult conditions users actually encounter.Start with one canonical, machine-readable field-error pattern and an automated check that verifies the message is connected to its field. Then define and publish the states, themes, screen sizes, keyboard paths, and assistive-technology checks required across the component library.StructureCarbon documents reusable layout and product patterns. The study did not find published results from repeated flow tests across states and breakpoints.Products fail at the seams between components, even when each component works by itself.Select the product flows Carbon officially supports and test them end to end across focus, validation, loading, recovery, themes, and screen sizes.GovernanceContribution, release, migration, and ownership practices are documented and active.Leaders need to see whether reviews move quickly, migrations finish, and quality problems close.Set targets for reviews, migrations, adoption, exceptions, and regressions. Publish the trends and record the staffing or process decision made when a target is missed.Can an AI agent use Carbon Core successfully?
13 of 15 runs produced working software
Foundations✓✓✓100%Bindings✓✓✓100%Components✓✓✓100%Structure×✓×33.3%Governance✓✓✓100%2 Structure runs failed. The generated validation message was not connected to the field. The implementation displayed an error message after submission, but Carbon TextInput did not reference that message with aria-describedby or aria-errormessage. Assistive technology therefore had no reliable programmatic connection between the Resource name field and its error.
This original five-task benchmark supports agent-assisted work with automated tests and human validation. The failed runs mean it is not evidence for unattended use.
Did direct access to Carbon’s official server help?
Not reliably. Across the expanded task set, the working-software rate was 43% with frozen evidence alone and 41% when the same model could also search Carbon’s official server. The estimated change was two percentage points lower, and the 95% uncertainty range ran from 15 points lower to 10 points higher. This result does not support presenting server access as an overall accuracy improvement.
The comparison used 20 tasks, 5 repeats, and 100 scored runs per condition. The official server was used in 40 enabled runs and completed 71 calls with 0 tool errors.
What to do next: Review the seven tasks that declined with server access, starting with design-to-code bindings and component behavior. For each task, record the query, rank of the first current and applicable result, evidence the agent used, and failed test. Fix search ranking, task-oriented examples, or API guidance according to the observed cause. Before enabling the server as a default accuracy feature, run held-out repeats and require the server condition to match or exceed the frozen-evidence result on every repaired task.
What this result covers: The comparison applies to one model, the 20 registered Carbon tasks, five repeats per condition, the frozen study evidence, Carbon MCP version 1.12.0, and the recorded validation runtime. Area differences are useful investigation leads, not separate proof that the server helps or harms that area.
How Carbon Core can get better
Make official guidance change implementation outcomes.
- Problem
- The official server completed 71 calls without a tool error, but working-software success was 41% with server access and 43% with frozen evidence alone. Binding and component tasks declined most, and the original benchmark also exposed a repeated field-error failure.
- Action
- Trace every declining task from query to returned result to generated code and failed test. Repair result ranking, version cues, task-oriented examples, or API guidance according to the observed cause. At the same time, publish and test one canonical field-error pattern.
- Owner
- Carbon server, component, accessibility, documentation, Code Connect, and developer-experience teams
- Finish line
- Reviewed queries return current, applicable guidance in the first five results; repaired tasks match or exceed their frozen-evidence result in held-out repeats; and every supported form flow automatically verifies the field-to-error relationship.
- Save for the next evaluation
- Queries, ranked results, source versions, generated implementations, failed and passing test output, server identity, and published field-error guidance
- Decision enabled
- Which workflows are ready for default server-assisted use and which guidance or tests still need repair.
Keep Figma and released code aligned before changes ship.
- Problem
- This study did not find a published result from one automated check spanning Figma variables, token packages, component properties, Code Connect mappings, and approved exceptions. IBM may already run parts of this check internally.
- Action
- Generate a versioned manifest of Figma variables and properties, DTCG token files, released packages, Code Connect mappings, typed component APIs, and approved exceptions. Compare it in release CI and block any unexplained difference.
- Owner
- Carbon foundations, Figma library, React, Code Connect, and build-infrastructure teams
- Finish line
- A seeded mismatch fails the release check and identifies the exact surface, property, or token involved.
- Save for the next evaluation
- CI results, compatibility reports, coverage, drift trends, and approved exceptions
- Decision enabled
- Whether a Figma, token, mapping, or component API change is safe to release.
Test complete product experiences across the whole system.
- Problem
- This study did not find one published coverage view spanning component states, screen sizes, assistive technology, layout rules, and end-to-end product flows. IBM may already run some or all of these checks internally.
- Action
- List the states, themes, screen sizes, keyboard paths, assistive-technology checks, and layout rules required for each supported component and flow. Run that checklist in CI and publish the current pass, failure, and exception results.
- Owner
- Carbon component, pattern, accessibility, product, and quality teams
- Finish line
- Required components, states, flows, themes, and breakpoints pass with current results and owned exceptions.
- Save for the next evaluation
- Component quality matrix, flow results, accessibility output, viewport coverage, and exception register
- Decision enabled
- Whether a supported component or product flow is ready across Carbon's promised conditions.
Use operating results to decide where Carbon needs investment.
- Problem
- This study did not find public trend data for adoption, migrations, exceptions, review speed, or quality. IBM may track these internally; sharing current results would allow them to be assessed in a re-run.
- Action
- For each quarterly review, publish the target, current result, trend, accountable owner, and next action for reviews, migrations, adoption, exceptions, and regressions. Record the staffing or process decision made when a target is missed.
- Owner
- Carbon governance, program, release, and design-infrastructure leadership
- Finish line
- The team can show clear trends for adoption, exceptions, migrations, review speed, and quality.
- Save for the next evaluation
- A metrics pack with baselines, targets, trends, owners, and follow-up decisions
- Decision enabled
- Where leadership should invest, delay a migration, or change staffing, ownership, or process.
The bottom line
Carbon is a mature design system with clear product value. The next useful move is not simply to expose more official material. Carbon should identify why server-assisted runs declined on specific binding and component tasks, repair the retrieval or guidance problem, and prove the repair on held-out runs. In parallel, it should publish one dependable field-error pattern, test complete product flows, block unexplained Figma-to-code differences, and use operating results to guide investment.
Access the evidence
Review the scoring method, criterion-level rationales, citations, and downloadable study files.