Tuning an AI rewriting prompt with an LLM judge: 4 rounds, $3, and why we stopped at v02

Lirnyk rewrites the text you select in any Mac app with one shortcut. Each profile is just a prompt. We stopped guessing whether our Corporate prompt was good and measured it instead.

Why measure a prompt at all

A Lirnyk profile is a short instruction that goes to the AI along with your selected text. The Corporate profile, the one you press when a draft needs to sound like work, shipped with this:

Fix typos and grammar. Be polite and professional, follow corporate communication style. Keep the original language. Output only the rewritten text.

It was written by hand and checked only by using it every day. That tells you when something is badly wrong, but it can't tell you whether one version is better than another. A bad prompt gives bad results on every model, so before comparing models we wanted to get the prompt right.

How the measurement works

We built a small harness in Python. One round takes a few minutes and goes like this:

  1. A fixed test set. Claude wrote 12 short texts, 6 in English and 6 in Ukrainian: chat messages, emails, bug reports, technical docs and opinion pieces. They are angry, anxious, enthusiastic, passive-aggressive or neutral, and they have the typos real drafts have. Two of them are already well written, to catch a prompt that makes good text worse. We reviewed the set and froze it, so every round is measured against the same texts.
  2. Written criteria. Claude also wrote what a good Corporate rewrite means, as 7 criteria scored from 0 to 10. Four apply to every profile: keeps the language, keeps the meaning and facts, outputs only the rewritten text, and fixes errors without over-editing. Three are for Corporate only: a polite professional tone, a business register, and a length and structure close to the original.
  3. Run the prompt the way the app does. Each text goes to a cheap, popular model on OpenRouter, DeepSeek V4.1 Flash, with the prompt as the system message, exactly as Lirnyk sends it. We record the output, tokens, cost and time.
  4. Judge every output. Claude Opus 5.5 scores each rewrite on each criterion and writes a one-line reason. Every output is judged in its own call, so one text can't bias another's score.
  5. Propose a better prompt. Claude reads the scores and the judge's comments and writes the next version. It sees criterion names, not their full text, and is told to keep the prompt short and general, so it can't simply copy the rubric or write rules for specific test texts.

A person reviews every round before the next one starts. The goal we set was 10 out of 10 on every criterion for every text, twice in a row.

Round 1: the shipped prompt scores 9.13, but its worst score is 3

An average of 9.13 out of 10 sounds fine. The lowest scores show what it hides:

  • Sarcasm passed through untouched (3/10 for tone). A Ukrainian email chasing a late report said, roughly, “I understand you have far more important things than our joint project.” The rewrite fixed the commas and kept the jab word for word.
  • Invented content (5/10 for meaning). An angry Ukrainian post about potholes got a greeting to the city council and a closing request, neither of which the author wrote.
  • Over-editing good text (6/10 for fixing errors). An English email with no mistakes came back with “I'm sorry” changed to “I apologize” and “The updated version is attached” changed to “Please find the updated version attached”.
InputСкільки ще мешканці Вербівки будуть терпіти цей жах на дорогах?! Вулицю Садову обіцяли відремонтувати ще два роки тому а там і досі ями по коліно. […] Ми платимо податки і маємо право на нормальні дороги!
v01 outputШановні представники міської ради! Скільки ще мешканці Вербівки будуть змушені терпіти незадовільний стан доріг? […] Ми сплачуємо податки й маємо право на належний стан доріг. Просимо вжити конкретних заходів для вирішення цього питання.
Round 1, prompt v01. The highlighted greeting and request were not in the original. The judge gave it 5/10 for meaning.

After the round, the improver kept the original prompt and added instructions to: neutralise sarcasm and hostility; leave wording that is already correct alone; keep every fact; and not add greetings, sign-offs, addressees, requests or claims.

Round 2: v02 lifts the worst score from 3 to 7

Fix typos and grammar. Be polite and professional, follow corporate communication style: rephrase rude, emotional, sarcastic or passive-aggressive wording as neutral, direct business language, and leave wording that is already correct and professional unchanged. Preserve all facts and specific details. Do not add greetings, sign-offs, addressees, requests or claims that are not in the original, and keep its structure and approximate length. Keep the original language. Output only the rewritten text.

The average rose to 9.31, and no criterion scored below 7 on any text. The pothole post kept its anger as facts, with no invented greeting or request. The sarcastic line in the email became “I understand you have a lot of work”. The judge still heard some edge in “if you can find a minute”, which earned the round's lowest score, a 7 for tone.

InputВулицю Садову обіцяли відремонтувати ще два роки тому а там і досі ями по коліно.
v02 outputВулицю Садову обіцяли відремонтувати ще два роки тому, але там досі ями завглибшки по коліно.
Round 2, prompt v02. The comma is fixed and the knee-deep potholes stay; v01 had turned them into “potholes that pose a danger”.

Rounds 3 and 4: more rules, no better results

The improver kept going. Each round it added rules for whatever scored lowest: keep the function of a phrase, moderate excessive courtesy instead of deleting it, don't swap correct words for synonyms, keep names and sign-offs. The prompt grew from 504 characters to 742 and then 1,079.

The scores didn't follow. v03 averaged 9.21, lower than v02. v04 reached 9.37, only 0.06 above v02, which is within what we'd expect from the judge's own variation, since each round was judged once. Meanwhile the worst score got worse: 6 in round 3 and 5 in round 4, when emoji slipped back into a cheerful Ukrainian team message.

The cost showed up somewhere we didn't expect. The rewritten text stayed the same size, about 4,000 characters per round, but the model's output tokens nearly doubled, from 5,964 to 10,757. DeepSeek reasons before it answers, and a longer list of rules gives it more to reason about. The median rewrite went from 2.2 seconds with v01 to 3.4 with v02 and 4.5 with v04, and the slowest v04 rewrite took 16 seconds.

Lowest score (out of 10)The worst single mark any text got on any criterion
v01: 3/103v01v02: 7/107v02v03: 6/106v03v04: 5/105v04
Average scoreMean of all 84 marks: 12 texts × 7 criteria. The scale starts at 9
9.009.259.50v01: 9.139.13v01v02: 9.319.31v02v03: 9.219.21v03v04: 9.379.37v04
Seconds per rewriteMedian time for one rewrite with DeepSeek V4.1 Flash
v01: 2.2 s2.2 sv01v02: 3.4 s3.4 sv02v03: 3.4 s3.4 sv03v04: 4.5 s4.5 sv04
Prompt lengthCharacters in the prompt
v01: 148 characters148v01v02: 504 characters504v02v03: 742 characters742v03v04: 1079 characters1079v04
Results by prompt version
Prompt Length Average Lowest Perfect 10s Cost per rewrite Median time
v01 (shipped)1489.13355%$0.000272.2 s
v02 (chosen)5049.31754%$0.000353.4 s
v037429.21656%$0.000433.4 s
v041,0799.37560%$0.000584.5 s
v051,476not tested

Why we didn't run v05

After round 4 the improver proposed v05, at 1,476 characters, ten times the original. Several new rules were aimed at particular texts in the test set: remove emoji, keep “a stated right” as a right, turn falsely polite phrases into sincere requests. That is the prompt learning the test instead of the task. Before we started, we wrote this down as a failure even if the scores went up, so we stopped there.

The decision: ship v02

We picked v02, not the version with the highest average:

  • It was the one real improvement. The worst score went from 3 to 7, and the two most embarrassing failures of v01 were fixed.
  • Nothing after it was clearly better. The average moved by less than the judge's likely noise, and the worst scores got worse.
  • It is as fast as any of the improved prompts. In an app where you wait for the rewrite after pressing a shortcut, a second matters more than a fraction of a cent.

v02 becomes the Corporate default in Lirnyk 0.4.0. If you never edited the Corporate prompt, it updates by itself. If you did, your prompt is left alone, and the new Reset to Default button under it in Settings → Profiles switches it to v02 in one click.

What we learned

  • Watch the worst score, not the average. The average barely moved, from 9.13 to 9.37 over four rounds. The lowest score is where the useful signal was: it marks the rewrite a user would actually be embarrassed to send.
  • Automatic prompt improvement levels off fast. After one useful step, every new version added rules that pulled against each other: neutralise the sarcasm, but don't change correct wording. The scores just moved around.
  • A longer prompt costs time, and with a reasoning model it costs more than you'd think. The reply stayed the same size while the hidden reasoning doubled.
  • 10 out of 10 everywhere was the wrong target for a cheap, fast model. A floor of 7 with no invented content is a target you can ship.
  • Judging is the expensive part. The whole experiment cost $3.26, including $0.22 to write the test set and the criteria. Almost all the rest was Claude reading and scoring 48 rewrites.

What's next

The same harness will tune the Friendly and Tech profiles. After that, with the prompts fixed, we can use it for the question it was really built for: which local Ollama model rewrites well enough, especially in Ukrainian, to run Lirnyk fully offline.

If you write work email, Slack messages or bug reports in more than one language, get Lirnyk and tell us on GitHub where the Corporate profile still gets it wrong.