# AI Evals: 7 Proven Fixes for Apps That Keep Breaking

URL: https://whattheweb.co.in/ai-evals-vibe-coding/
Published: 2026-11-08
Author: Vinita Mathreja
Topics: Vibe Coding
Updated: 2026-11-12

> AI evals stop your vibe-coded app from breaking after every prompt. Here are 7 proven fixes you can run inside Claude, Cursor and Gemini.

AI evals are the fastest way to stop a vibe-coded app from breaking every time you prompt. They are a plain-English checklist your AI tool must pass before it tells you the work is done. Without them, every new feature puts the last one at risk.

You know the pattern. You ask Claude, Cursor or Gemini for one small change. The change lands, and it looks great. But three prompts later your login form breaks, your navbar slides off-screen, or your API stops returning data.

So you paste error logs back and forth for an hour. That is The Loop. It eats your time, your tokens and your confidence in the tool.

This guide breaks The Loop. Below are seven proven fixes, written for builders who do not read code line by line.

On this page

1. [What are AI evals?](#what-are-ai-evals)
2. [Why AI evals matter for vibe coders](#why-ai-evals-matter-for-vibe-coders)
3. [Which projects actually need AI evals?](#which-projects-actually-need-ai-evals)
4. [How an AI eval runs, start to finish](#how-an-ai-eval-runs-start-to-finish)
5. [7 proven fixes to set up AI evals](#7-proven-fixes-to-set-up-ai-evals)
6. [5 mistakes that make AI evals useless](#5-mistakes-that-make-ai-evals-useless)
7. [AI evals FAQ](#ai-evals-faq)
8. [The bottom line](#the-bottom-line)

## What are AI evals?

> **Quick definition:**  An AI eval is a written test that checks whether an AI-generated app still meets your visual, functional and logical requirements. Traditional unit tests need code. Evals need only plain-language criteria, a short script, or a second AI model acting as the judge.

Think of an eval as a quality checklist your AI tool has to clear before it says “finished”.

When you prompt a change, the model does not only read your code. It also reads your eval file. Then it answers three questions:

- Did it ship the feature you asked for?
- Did it break anything that already worked?
- Does the app still follow your design and logic rules?

In short, AI evals turn “I think it works” into “I can prove it works.”

## Why AI evals matter for vibe coders

AI coding tools are non-deterministic. Prompt the same request twice, and you can get two different solutions. Neither one knows what the other did.

So building without evals feels like Jenga. Every new block risks the whole tower.

Traditional developers solve this with automated tests. But you did not write the code, so reading a test file helps you very little. Evals give you the same safety net in language you actually use.

| Without AI evals | With AI evals |
| --- | --- |
| You find bugs after you publish | The agent finds bugs before you publish |
| One fix quietly breaks another page | Working features stay locked in place |
| Tokens go on repeat debugging | Tokens go on new features |
| Adding auth or payments feels risky | Adding auth or payments feels routine |
| “It looks fine to me” | “Eleven of eleven checks passed” |

### Three benefits you feel in week one

- **No silent regressions.**  The model cannot fix one page while quietly breaking another.
- **Lower token costs.**  You stop paying twice for the same debugging loop.
- **Confident scaling.**  You can add auth, databases and payments without wrecking your MVP.

## Which projects actually need AI evals?

Not every page needs a test suite. A one-screen landing page certainly does not. But the moment your app stores data, takes user input or calls an API, evals start paying for themselves.

Use this ladder to decide how much rigour your project deserves:

1. **Simple landing page.**  A manual look on desktop and mobile is enough. Skip the evals.
2. **Login or multi-step form.**  Add structural and functional evals.
3. **E-commerce, SaaS or AI app.**  Run a full suite, including visual checks and edge cases.

![An AI-built WebGL site that needs AI evals to catch layout and data bugs](https://whattheweb.co.in/wp-content/uploads/2026/10/airasia-seat-22a-webgl-website-built-in-cursor.jpg)

Prompt-built sites look finished long before they are stable. That gap is exactly what evals close.

### 1. Login and multi-step form apps

**Why it matters:**  if sign-up breaks, your whole product stops. Nobody reaches the part you spent a month on.

**What to check:**  invalid emails get rejected, valid logins land on the dashboard, and the session survives a page refresh.

### 2. Database-driven apps

**Why it matters:**  apps that pull from Supabase, Airtable or an external API go blank when a single response key changes name.

**What to check:**  an empty table shows “No data found” instead of a white screen. Also confirm that a slow response shows a loading state. For a worked example of wiring data into an AI app, see our [beginner’s guide to building a RAG pipeline](https://whattheweb.co.in/how-to-build-a-rag-pipeline-a-beginners-guide/).

### 3. AI wrapper apps

**Why it matters:**  model output drifts. The same prompt returns three tidy paragraphs today and three words tomorrow.

**What to check:**  output length, output format such as JSON or Markdown, and your safety guardrails.

## How an AI eval runs, start to finish

Every eval moves through four stages, however simple it is:

1. **Input.**  A user action, a prompt or a test dataset starts the run.
2. **Execution.**  The agent builds the code or loads the live page.
3. **Evaluation.**  Each criterion gets a pass or a fail. No maybes.
4. **Report.**  The agent reads the result, then fixes whatever failed.

That fourth stage matters most. A test nobody acts on is just a log file.

### The 3 types of AI evals, compared

Evals come in three flavours. Most projects need all three, but you can start with the first and add the others later.

| Type | What it checks | Example | Best for |
| --- | --- | --- | --- |
| Assertion | Hard true or false facts | Does the page return a 200 status code? | Routes, APIs, build health |
| Visual | Layout, spacing, responsiveness | Does the drawer open without pushing the header away? | Mobile views, animations |
| LLM judge | Taste, tone, brand fit | Does this copy match our brand voice? | Generated text, design systems |

Assertion checks are cheap and instant, so run them constantly. LLM judges cost tokens, so save them for the work that actually ships.

### What a good eval report looks like

Ask your agent for a report card, not a paragraph of reassurance. A useful report names every check, its result, and the fix.

```
EVAL REPORT — 9/11 passed

PASS  auth.blank-email        inline error shown
PASS  auth.valid-login        redirected to /dashboard
PASS  auth.session-persist    session survived reload
FAIL  data.empty-state        rendered blank div, expected "No data found"
fix: add fallback in DataTable.tsx line 42
PASS  data.api-failure        retry button rendered
FAIL  layout.mobile-overflow  body scrolls 40px at 375px
fix: remove fixed width on .hero-card
```

Two failures, two named files, two suggested patches. That is a report you can act on in minutes, even if you never open the code yourself.

## 7 proven fixes to set up AI evals

### Fix 1: Write a gold-standard spec file

Create a file called `APP_EVALS.md` in your project root. Then write down what “working” means, in plain English.

Use the GIVEN / WHEN / THEN pattern. For example: GIVEN a logged-out visitor, WHEN they submit a blank form, THEN show an inline error and keep them on the page.

Keep every rule to one sentence. One rule, one check, one clear outcome. Ten sharp rules beat fifty fuzzy ones.

Here is a starter file you can copy today:

```
# APP_EVALS.md

## Auth
- GIVEN a blank email field, WHEN the user submits, THEN show an inline error.
- GIVEN valid credentials, WHEN the user submits, THEN redirect to /dashboard.
- GIVEN a logged-in user, WHEN the page reloads, THEN the session persists.

## Data
- GIVEN an empty results set, THEN render "No data found", never a blank screen.
- GIVEN a failed API call, THEN render a retry button and log the status code.

## Layout
- Below 768px, the nav collapses into a button with the ID #mobile-menu.
- No element overflows the viewport horizontally at 375px.
```

Notice the pattern. Every line names a trigger and an expected result. Nothing relies on judgement.

### Fix 2: Turn vague goals into binary thresholds

“Make the design look nice” is not a test. Nothing can pass it, and nothing can fail it.

- **Vague:**  make the mobile layout responsive.
- **Eval-ready:**  below 768px, the nav links collapse into a button with the ID `#mobile-menu`.

So write numbers, element IDs and hex codes. Then the result is never a matter of taste.

### Fix 3: Add assertion checks for hard yes or no

Assertion checks are deterministic. They return true or false, with nothing in between.

Two good starters: does the page return a 200 status code, and does `.submit-btn` exist in the DOM? Both take seconds to run. Both catch the ugliest failures.

### Fix 4: Add visual and UI checks

Layout breaks are the bugs you notice last, usually on a phone, usually after you share the link. So check them on purpose.

Ask your agent to confirm spacing, mobile breakpoints and overlap. For instance: does the checkout drawer open without pushing the header off-screen? We learned this the slow way while shipping scroll-heavy pages, which we wrote up in [8 mistakes to avoid when building a WebGL website in Cursor](https://whattheweb.co.in/how-to-build-a-webgl-website-in-cursor/).

![Visual AI evals catch layout breaks in scroll-driven sites built with AI tools](https://whattheweb.co.in/wp-content/uploads/2026/10/ferrari-world-webgl-website-built-in-cursor.jpg)

Visual evals catch the breaks that a passing build never reports.

### Fix 5: Add an LLM judge for taste

Some things resist true or false. Brand fit, tone and code cleanliness, for instance.

So hand those to a second model. Ask it to grade the output against your written guidelines, then list every miss.

Try this prompt: “Review this component. Does it use our brand palette, #3B82F6 and #0F172A? Score it out of 10 and name every element that strays.”

### Fix 6: Wire AI evals into Claude, Cursor or Gemini

Each platform reads project rules from a different place. Here is where to put yours.

#### Claude Code

Claude Code reads persistent project rules from a `CLAUDE.md` file, as described in the [official Claude Code documentation](https://docs.claude.com/en/docs/claude-code/overview). Drop this block into it:

```
## Eval protocol
Before you call a feature complete:
1. Read APP_EVALS.md and list the affected cases.
2. Run npm run test and read the terminal output.
3. If a case fails, fix the code and run it again.
4. Do not ask me to confirm until every case passes.
```

Then prompt: “Add a dark mode toggle, then run our eval protocol.” For more on working this way without a dev background, read [how I use Claude Code as a non-coder across 30+ projects](https://whattheweb.co.in/how-i-use-claude-code-as-a-non-coder-30-projects-across-content-design-and-code/).

#### Cursor

Cursor reads scoped rules from a `.cursor/rules` directory, documented in the [Cursor rules reference](https://docs.cursor.com/context/rules). Create `eval-suite.mdc` and add your checkpoints:

```
---
description: Core app evaluation rules
globs: src/**/*.tsx, src/**/*.ts
---
# Checkpoints
- Every interactive element needs a data-testid.
- Run npm run build after each change.
- If the build fails, revert and try another approach.
```

Then tag `@eval-suite.mdc` whenever you request a change. Cursor will hold the rules in context while it edits.

#### Gemini

Gemini handles machine-readable criteria well, especially through the [Gemini CLI](https://github.com/google-gemini/gemini-cli). Save your rules as `EVAL_CRITERIA.json`:

```
{
"visual": [
{ "target": "navbar", "rule": "stays sticky at the top" },
{ "target": "cta_button", "rule": "hex equals #FF5733" }
]
}
```

Then ask: “Read EVAL_CRITERIA.json, inspect src/App.jsx, and give me a report card with the exact patches I need.”

### Fix 7: Run light evals on every single prompt

Run a build check and a route check after each change. These take seconds, and they catch most of what breaks.

Then run the full suite before you deploy. That is the moment auth flows, payment triggers and edge cases deserve a proper look.

Finally, add a new rule every time a bug reaches production. Your eval file should grow with your scars.

## 5 mistakes that make AI evals useless

Most eval suites fail for the same handful of reasons. Avoid these, and yours will earn its keep.

### 1. Writing rules only you understand

“The hero should feel premium” means nothing to a model. Name the font size, the spacing and the colour instead.

### 2. Testing the happy path only

Real users paste emojis into phone fields and refresh mid-payment. So write at least one rule per failure mode: empty input, bad input, slow network and expired session.

### 3. Letting the agent mark its own homework silently

Models will claim success without running anything. Therefore demand evidence. Ask for the terminal output, the status codes and the count of checks that passed.

### 4. Never updating the file

A stale eval file protects last month’s app. Add a rule the same day a bug escapes you.

### 5. Making the suite so slow you skip it

If a full run takes ten minutes, you will stop running it by Thursday. So split the suite in two. Keep a fast set for every prompt, and a deep set for deploy day.

## AI evals FAQ

### What is the difference between AI evals and traditional software testing?

Traditional software testing checks fixed outputs using code a developer writes, such as unit tests or integration tests. AI evals check unpredictable output from a language model. So they grade work against plain-language criteria, pass or fail thresholds, and sometimes a second AI judge, rather than hard-coded assertions alone.

### How do I test my web app if I cannot write code?

Write your rules in plain English inside a markdown file such as `APP_EVALS.md`. Use GIVEN / WHEN / THEN sentences. Then ask your AI agent to run the checks, read the errors and fix them. You review the report card, not the code.

### Can AI agents run evals without me watching?

Yes. Claude Code, Cursor agent mode and Gemini CLI can all run terminal commands, read error output and loop until the checks pass. They will not do it by default, though. You have to write the instruction into your rules file.

### What is LLM-as-a-judge in app evaluation?

LLM-as-a-judge means a second model grades the first model’s output. Use it for the soft criteria that rules cannot capture: tone, brand alignment, formatting and code readability. Give the judge your written guidelines, and ask for a score plus a list of misses.

### What if my AI tool keeps failing the same eval?

Your criteria are probably too vague, or two rules contradict each other. So pause the agent and do three things:

- Read the raw failure output yourself.
- Split the test into smaller, atomic rules.
- Hand the model the exact docs or stylesheet it needs.

### How often should I run AI evals on a vibe-coded project?

Run lightweight checks, such as build verification and route checks, on every prompt iteration. Run the full suite before any production deploy. That rhythm costs almost nothing and catches almost everything.

## The bottom line

Vibe coding makes building fast. AI evals make it reliable. You need both.

So write the checklist once, then let Claude, Cursor or Gemini prove the work before you ship it. The tool stops being an unpredictable assistant and starts behaving like a disciplined builder.

Start small. One spec file and five rules will catch most of what breaks this week.

---

![Vinita Mathreja, founder and editor of WhatTheWeb](https://whattheweb.co.in/wp-content/uploads/2026/10/vinita-mathreja-founder-whattheweb.jpg)

**About the author.**  Vinita Mathreja is the founder and editor of WhatTheWeb. She builds web and mobile apps with AI tools, ships them, and writes about what actually survives contact with real users. Read more of her work on [AEO content writing](https://whattheweb.co.in/aeo-content-writing/).

**I build what I write about.**  Try the live experiments on the [Apps page](/apps/), or look up any term in the [AI Glossary](/ai-glossary/).