Chip Li李鸿铭

Gainesville, FL

UX research portfolio · Human-AI interaction

Hongming (Chip) Li

I research when people trust01, edit02, or hand control03 to AI, and I build the products to find out.

Mixed-methods UX researcher for AI products: field studies, interviews, and usability sessions; surveys, experiments, and behavioral log and prompt-data analysis; LLM evaluation against human judgment. Ph.D. Candidate at the University of Florida, where I design, build, and study AI tools for learning.

Portrait of Hongming (Chip) Li
hi, it’s me, Chip!
  • 151middle-school students in a live deployment of an AI safety layer I built (F1 .928)
  • 10case studies across qualitative, quantitative, and AI-evaluation research
  • 18AI systems I designed, built, and shipped as research instruments
  • 17first-author papers and talks, incl. an AIED 2026 long paper (16.7% acceptance)

Try a finding · 30 seconds

An AI says it is 90% sure. Would you believe it?

We asked three LLMs to code real student messages into one of nine categories, and to say how confident they were. Make your guess, then see what we found.

most people guess high →↓

Can you give me a Java example of a for-loop?

AI says: exampleconf. ≈ 0.9

80

trustAI as the judge

When can AI be trusted to judge?

AI now grades work, polices conversations, flags cheating, and reports its own confidence. I test those judgments against people, and design what happens when they fall short.

A VETTING trace from the lab API: the first draft gives away the answer and is rejected; the rewrite asks a guiding question and passes

VETTING · Mixed methods · AI safety in the field

A safety check students can’t talk their way around

How do you keep an AI writing helper safe for 12-year-olds without making it useless?

  • randomized trial
  • classroom deployment
  • log analysis
  • human audit
  • qualitative coding
  • cost & latency analysis

Moving the rules into a second model that students never talk to caught 91% of unsafe replies for 20% more tokens.

151 middle-school students · 1,250 turns · F1 .928 · Computers & Education: AI (2026) · open-source package

Case studyPaper ↗Code ↗Pilot study ↗

LLM confidence · Quantitative research · AI trust & evaluation

Read AI confidence as a ranking, not a probability

When an AI says it is 90% sure, should people believe it?

  • LLM evaluation
  • calibration metrics (ECE, AUC, ρ)
  • prompt experiments
  • activation steering

Three models were 0.40–0.55 overconfident. Confidence can rank what to review, but cannot set a trust threshold.

633 expert-coded dialogues · 3 models · AIED 2026 long paper (16.7% acceptance) · open-source steering code

Case studyPaperCode ↗Slides

editAI drafts, people decide

Where should people stay in control?

AI can draft study cards, learning objectives, and quizzes in seconds. I study which parts people need to check, change, and own, and build tools around that line.

SmarterFlash dashboard with flashcards, study path, upload, and practice

SmarterFlash · Qualitative research · Human-AI agency

Automate drafts, not ownership

Where should an AI study tool stop helping, so learners still do the learning?

  • think-aloud
  • interviews
  • cognitive walkthrough
  • thematic analysis
  • LLM-seeded hypotheses

Learners welcomed AI drafting, and wanted to edit everything it made. Automation earns adoption; editing earns trust.

4 design principles · ISLS 2026 paper · tested whether an LLM can stand in for early user research

Case studyProduct ↗PaperSlides

ARCHED three-phase workflow: initial generation, collaborative refinement with the analysis engine, and assessment development, with an iterative refinement loop

ARCHED · LOGEN AI · Framework & design research · Human-AI collaboration

One AI proposes, another explains, the educator decides

How do educators stay the decision-maker when AI drafts their learning objectives?

  • framework design
  • expert agreement (weighted κ)
  • blind quality comparison
  • expert feedback

Splitting the AI into a generator and a separate evaluator kept educators in charge. Ten months later, the OECD’s Digital Education Outlook gave ARCHED its own box.

Spotlight paper, iRAISE @ AAAI 2025 · featured in the OECD Digital Education Outlook 2026 · a chapter resource in a Routledge textbook

Case studyLive tool ↗Paper ↗OECD report ↗

hand offAI on your behalf

What do people need before AI acts for them?

Teachers hand AI their class logs, students hand it career choices, researchers hand it their metrics. I design the summaries, explanations, and honest numbers that make that hand-off safe.

GM dashboard activity frequency and timeline charts for a pair of students

GM Dashboard · Mixed methods · From logs to decisions

The chart showed everything, and told the teacher nothing

What does a teacher need to act on 20 minutes of collaborative math, at a glance?

  • log analysis (448K events)
  • time-series clustering
  • classroom fieldwork
  • teacher co-design
  • LLM pipeline

Clustering found real collaboration patterns, but the teacher couldn’t act on them. An AI layer on top of the charts now flags breakthroughs and struggles, so she doesn’t have to decode them.

EDM 2024 (first author) · AIED 2025 (under 16% acceptance) · public dashboard with anonymized demo data

Case studyLive demo ↗Log study ↗Dashboard paper ↗

How I work

I build the instrument, then study the people using it

Building the product myself keeps the loop between a finding and a design change short. Every method below links to where I used it.

  1. FrameTurn a product decision into a research question someone can act on.
  2. BuildMake the instrument: the prototype, the consent flow, the telemetry.
  3. StudyWatch real use: think-alouds, field deployments, logs, audits.
  4. ChangeShip the decision, then measure whether it held.

AI in my research

I seed hypotheses with an LLM and then test them against people (SmarterFlash), run LLM pipelines with human audits (VETTING) or validation layers (GM Dashboard) on top, and build study instruments and analysis code with Claude Code and Codex every day.

Tools

Python (pandas, scikit-learn, PyTorch, Hugging Face), SQL (PostgreSQL, SQLite/D1), statistics from regression to bootstrapped confidence intervals, and Kaggle Competitions Master for the machine-learning side.

Instrumentation

I design telemetry schemas, consent flows, and logging standards into the products I study, so the data a study needs exists before the study starts.

Understand people

Measure behavior

Evaluate AI

Research infrastructure

Every study runs through one gateway I built

A shared logging standard means any study can be replayed: the exact prompt, model, response, cost, and latency of every turn, tagged by product, feature, and session.

Products

  • CHECKPOINT
  • SmarterFlash
  • VETTING studies
  • GM Dashboard
  • Class-Ready AI
  • ChronoForge
  • Rubber Ducking
  • Provoking Minds
  • LOGEN AI

VIABLE Lab Developer Platform

One OpenAI-compatible API, 46 models

Commercial and lab-hosted open-weight models behind one key. Requests are tagged with X-Platform-ID and an operation, and VETTING verification is one endpoint away.

developers.viablelab.org ↗

What researchers get

  • Full-text request logs for replay, or a metadata-only mode
  • Uploaded images and audio archived with hashes; every read audit-logged
  • Step-by-step traces of agent runs
  • Scoped CSV and JSONL exports that expire in 24 hours

It has two audiences, developers and their coding agents, so it ships a written interaction contract for the dashboard, plain-language error messages with request IDs, and an agent guide that tools like Claude Code can follow end to end.

Design notes

One decision each

Smaller projects where a single design choice carried the work.

Class-Ready AI

The AI coaches. It never rewrites.

Pre-service teachers write their own learning objectives; the AI responds with questions and guidance, never a finished rewrite. A quick 1–5 helpfulness rating appears at a natural pause, after feedback and before the next edit, so it samples experience in context.

Prompt changed from “offer a concrete suggestion” to question-style coachingClass-Ready AI ↗

ChronoForge

You choose, the LLM narrates.

Students live through historical turning points. An LLM narrates every scene from their choices, but on key rounds a deterministic engine swaps in authored options, so history stays anchored. The ending compares their path with the real record, with no shaming language.

6 scenarios · presented at AECT 2026ChronoForge ↗

Rubber Ducking

Three roles for an AI that listens to you code, and a team that took it further.

A silent duck that never calls a model, a “beginner friend” that plays a confused novice and asks one question at a time, and an expert that debugs. An undergraduate CS group joined to keep building it, found students wanted help inside Google Colab, and with our mentoring shipped it as a Chrome extension.

Web app plus Rubber Ducking for Colab on the Chrome Web StoreRubber Ducking ↗

takeback

Found by using it, not by guessing.

Before launch I ran my undo tool with real coding agents. That surfaced what no spec would: hooks the agent silently skipped, checkpoints saved twice, a stale lock that blocked later saves. The README now says plainly what it cannot undo.

Open source · 24 tests on Linux, macOS, Windowstakeback ↗

Everything I've shipped

Each product started as a question

Research systems I built or helped build at VIABLE Lab, with the question each one was made to answer.

  • CHECKPOINTAI-Powered Assessment Platform

    Can every wrong answer tell a teacher which misconception a student holds?

    Case study →
  • LOGEN AIAI-Powered Instructional Design Assistant

    How do instructional designers stay the decision-maker when AI drafts learning objectives?

    Case study →
  • VIABLE Lab Developer PlatformAgent-Native AI Platform for Learning

    Can one gateway make every study’s AI traffic replayable, for developers and coding agents alike?

    Lead System Architect, Full-Stack & DevOps Engineer
  • VIABLE Lab WebsiteResearch Laboratory Platform

    How does a research lab publish, review, and measure its own work without third-party tracking?

    Full-Stack Developer & UX Designer & DevOps Engineer
  • GraspableMath DashboardCollaborative Learning Analytics

    What should a teacher see from a 20-minute collaborative math session, at a glance?

    Case study →
  • Class-Ready AIInstructional Design for Pre-Service Teachers

    Can AI feedback coach new teachers to write better objectives, without writing them for them?

    Lead Developer
  • Provoking MindsAI Agent Debate Platform

    Can critiquing two AI agents that disagree build critical thinking?

    Product Manager & Lead Developer
  • Pro-CaREProfessional Career Recommendation Engine

    Which internships close the gap between an engineering student’s skills and the career they want?

    Case study →
  • VETTING ChatAI Assistant Validation & Safety Tool

    Can a second model enforce rules that a chatbot alone can be talked out of?

    Case study →
  • SmarterFlashAdaptive Self-Regulated Learning System

    Where should AI stop automating, so learners still own their studying?

    Case study →
  • PRODUCTIVEAI Math Problem Generator

    Can AI help teachers write productive-failure math problems at the right difficulty?

    Research Assistant & Full-Stack Developer
  • Creativity AssessmentAI-Powered Evaluation Tool

    Can an LLM apply a human creativity rubric to student-built game levels?

    Case study →
  • Rubber DuckingAI Problem-Solving Tool

    What role should an AI play while you explain your code out loud?

    Developer
  • ChronoForgeAI Narrative Simulations for Learning

    Can an LLM narrate history while students make the decisions and the facts stay put?

    Lead Developer & Researcher
  • NSF EAGEROrchestrating Productive Collaboration

    What does productive collaboration look like in students’ activity logs?

    Research Assistant

Let’s talk.

Open to UX research and HCI research roles on AI products. I can walk through any of these studies in depth.