Reproducible evaluation method

How to evaluate an AI image-editing API

Evaluate an AI image-editing system on a fixed, versioned set of source images and instructions. Score instruction following, visual fidelity, and preservation separately; blind and position-swap pairwise reviews; and report failures, latency distributions, and cost per successful output. Pre-register the decision rule and show uncertainty. A small starter manifest can validate the pipeline, but it cannot establish a winning system.

Published by Peelaway Engineering. Sources checked August 11, 2026.

Download starter manifestRead the method

Method and smoke manifest, not benchmark results

This page publishes no model ranking, vendor comparison, output, score, customer result, or Peelaway quality claim. Twenty cases can test an evaluation pipeline; they cannot support a universal winner.

Evaluation sequence

Six steps from task coverage to a defensible decision

Freeze the experiment before generation, separate quality from operations, and publish enough detail for another team to repeat the comparison.

  1. 01

    Freeze task coverage

    Version the source, instruction, intended change, must-preserve content, use case, edit type, and difficulty before producing outputs.

  2. 02

    Record operational outcomes

    Count submitted, completed, failed, timed-out, and retried cases. Report end-to-end p50 and p95 latency plus cost per successful output, not only provider compute time.

  3. 03

    Score separate quality axes

    Judge instruction following, visual fidelity and artifacts, and preservation independently. A plausible image can still fail the requested edit or alter protected content.

  4. 04

    Blind and position-swap

    Hide system names, reverse A/B order, and count inconsistent preferences as ties. Keep refusals or unjudgeable cases separate from wins.

  5. 05

    Pre-register the decision

    Choose the primary comparison, minimum sample, uncertainty method, and critical no-regression slices before looking at aggregate results.

  6. 06

    Publish limits with results

    Report the manifest version, missing cases, judge protocol, uncertainty, task slices, and failure definitions so another team can reproduce the claim.

Quality axes

Do not collapse three different questions into one score

AxisQuestionTypical failure
Instruction followingDid the result make the requested change to the intended subject?Plausible output, wrong object, color, material, or extent.
Visual fidelityIs the result coherent and free from visible artifacts or implausible integration?Warped geometry, broken text, seams, halos, or inconsistent light.
PreservationDid identity, layout, text, and every declared untouched region remain faithful?The edit succeeds but changes protected content or recomposes the scene.

Blind, position-swapped pairwise review

Label systems A and B without revealing identity. Judge the same case again with positions reversed. Keep a preference only when both orders agree; otherwise record a tie. Record refusals and unjudgeable cases separately instead of turning them into wins for the other system.

Operational evidence

Measure the system around the image

A visually strong output does not erase failures, retries, long tails, or unusable results. Define one successful output first, then use the same boundary for every operational metric.

Reliability

Completion, failure, timeout, retry, and refusal rates with every submitted case in the denominator.

Latency

End-to-end p50 and p95 from submission to usable result, including polling, queueing, downloads, and retries.

Cost

Total billable and retry cost divided by successful usable outputs—not the advertised unit price alone.

Versioned starter data

A 20-case manifest for testing the pipeline

Version 1.0 spans ordinary production use cases and several edit types. It contains source links, license labels, instructions, must-preserve clauses, and difficulty labels. It intentionally contains no generated output, system identity, score, or winner.

Cases
20
Use cases
8
Edit types
7
Difficulty
3 / 12 / 5

easy / medium / hard

Per-case fields

id · use_case · edit_type · source_url · source_license · instruction · must_preserve · difficulty

The manifest links to third-party source images but does not redistribute them. Resolve the source record and comply with its attribution and license requirements before running a case.

Download the JSON manifest

Interpretation boundaries

What this framework cannot establish

  • Twenty cases are a smoke set, not a powered model or vendor comparison.
  • A model judge is not ground truth; human review and disagreement reporting remain necessary.
  • Results from one task distribution, geography, date, API version, or quality setting do not automatically generalize.
  • Pixel similarity alone cannot prove instruction following, semantic preservation, or visual quality.
  • Missing, refused, timed-out, and failed cases must remain visible rather than disappearing from the denominator.
  • This page does not measure or rank Peelaway or any other image-editing product.

Source ledger

Research behind the framework

Sources checked August 11, 2026. Peelaway adapted the practical workflow and starter manifest for production-system evaluation; the research below supports multidimensional scoring, human alignment, and the limitations of automated judges.

  • NeurIPS 2024 — I2EBench

    Instruction-based image editing needs multidimensional evaluation and human-perception alignment; I2EBench reports more than 2,000 source images, 4,000 instructions, and 16 evaluation dimensions.

  • Google Research — EditInspector

    Edit verification spans accuracy, artifacts, visual quality, scene integration, common sense, and change description. Its authors also report that current model judges can miss defects and hallucinate changes.

Evaluate Peelaway in your own harness

Use your own source rights, task distribution, success boundary, and reviewers. Peelaway publishes no score for itself on this starter manifest.

Create an API account