Chip Li李鸿铭

Gainesville, FL

When can AI be trusted to judge? · Creativity assessment · 2024–2026

Give the AI judge the rubric, and the picture

Can an LLM rate creative work the way expert raters do?

My role
Second author on a study led by Dr. Seyedahmad Rahimi. I built the original rubric-guided GPT-4o scoring tool, first demoed at UF’s AI Days.
Method
421 levels that college students designed in Physics Playground, each rated by two experts on a seven-part rubric. GPT-4o and Gemini scored the same levels with and without the rubric, from the image, the level data, or both.
Key insight
The rubric did most of the work. Adding the picture caught aesthetics and humor that structured data missed, and one model call was as reliable as averaging three.
Outcome
Evidence that rubric-guided LLMs can assess creative work at scale, in Creativity Research Journal and ISLS 2026.

Scoring tool ↗Journal paper ↗ISLS paper

A student-drawn Physics Playground level called Invisible Chef: a chef’s hat, a smiling face, a spoon, and a pot of vegetables around the balloon

Problem

Creativity matters, and it is expensive to measure

The gold standard for judging creative work is a panel of trained raters scoring each artifact. That does not scale to a classroom, let alone a game with thousands of players.

Physics Playground lets students design their own levels: ramps, pendulums, and a ball that has to reach a balloon. Each level is a real creative artifact. Could an LLM rate them the way experts do, inside the game, without stopping play for a test?

Six levels students drew. Rate each one the way the expert raters did, then compare.

  • Student-made level: Swing Arms

    Swing Arms

  • Student-made level: The Birds and the Bees

    The Birds and the Bees

  • Student-made level: RockNRoll

    RockNRoll

  • Student-made level: Trench Run

    Trench Run

  • Student-made level: Get Past the Swing

    Get Past the Swing

  • Student-made level: Returning to Earth

    Returning to Earth

Approach

Hold everything constant except what the judge sees

  • Real artifacts421 levels designed by college students, each saved as an image and as structured level data.
  • Expert ground truthTwo experts rated every level on a seven-part rubric: originality, elaboration of lines and objects, aesthetics, humor and surprise, title creativity, and whether it can be solved.
  • Controlled promptsThe same models scored the same levels with and without the rubric, from the image, the data, or both.
  • Validity and reliabilitySpearman correlations with bootstrapped confidence intervals, one model call against the average of three, and large against small models across the GPT and Gemini families.
Layered prompt: game context, rubric with seven sub-facets and examples, the target level as image and JSON, and the task with output format
The judge’s prompt, layer by layer: context, rubric with examples, the level itself, and the task.

Findings

Structure and sight made the AI judge credible

.81agreement with experts, rubric-guided (ρ)
.61agreement without the rubric
1 callas reliable as averaging three runs
Bigger models judge better: agreement with experts (ρ) in one comparison
GPT-4o.71
GPT-4o-mini.62
GPT-4.1-nano.41
  • The picture mattersWithout the image, the model missed aesthetics and humor. Creativity in a visual artifact cannot be read from data alone.
  • Some criteria resist automationTitle creativity agreed only at about ρ = .35. Wit in a name is still hard for a model to judge.

Impact

A recipe for any AI judge

The result makes it practical to assess creativity unobtrusively inside learning games. It also generalizes: if you use an LLM as a judge, give it the rubric your human raters use, give it the same inputs they see, and measure agreement before you trust it.

The scoring tool is still online at creativity.viablelab.org; the game editor it embedded went dark when the partner server was retired, so new levels can no longer be built inside it.

if I ran it again

The ratings came from two experts and one population of college students. The next question is fairness: does the AI judge score different creative styles and student groups the way human raters do? I would also study whether students accept AI feedback on their creativity, which is a different question from whether it agrees with experts.