The new MKA benchmark puts AI-created work files under the microscope

Google DeepMind and Artificial Analysis announced the Multimodal Knowledge Agent benchmark on November 10. Google describes a 200-task evaluation across ten industry verticals, built around visually complex inputs and professional deliverables rather than text answers alone.

Its announcement describes expert-written rubrics weighted toward factual accuracy, with a smaller visual-design component. It also says models run with the same tools, sandbox and turn budget. Those details help define the test conditions; they do not make every real business workflow equivalent to the benchmark.

The practical distinction is between describing an answer and producing usable work. If a task depends on a diagram or a page layout, evaluating only extracted text can miss an important part of the job.

Artificial Analysis explains that the headline metric is Climb Weighted Satisfaction Rate: a weighted share of the harder rubric criteria completed successfully. It distinguishes those criteria from baseline Regression checks. The headline is therefore not simply the percentage of entire projects completed perfectly.

The evaluator also describes an agentic judging process that examines submitted deliverables. For a buyer or an operations lead, the useful question is what was graded and under which conditions—not only which model appears above another on a leaderboard.

A score from one evaluation does not establish a company's error rate, staff-time savings or readiness to automate a consequential decision. Those would require evidence from the actual workflow and its review process. This article makes no claim of measured OrangeHat or customer results.

Google's public showcase repository contains ten examples with task metadata, input files and fine-grained rubric criteria. The repository separates those materials into explicit task folders. This public sample is distinct from the private evaluation set used for reported scores.

That structure suggests a useful pattern for a limited business pilot: define an input bundle, a required output and specific acceptance criteria before running an agent. A hypothetical document-review task might require a reviewer to trace each conclusion back to the relevant source page, then open the delivered file to inspect its content and layout.

For a spreadsheet, a plausible check could examine the formula and its referenced cells rather than accept a formatted total at face value. For a presentation, the check could ask whether an image supports the accompanying claim. These are examples of review questions, not assertions about defects in any named system.

An acceptable file is one part of a working process. A team still needs to decide which inputs it can provide, who reviews the result and how an unsuccessful run is handled. A polished artifact does not answer those operational questions by itself.

MKA's contribution is a more tangible evaluation target: the delivered work and the evidence needed to judge it. Using that idea does not require treating a public leaderboard as a forecast of performance, cost or safety in a different setting.