Translation Quality: How to Measure and Improve It (With a Simple Scorecard)
By PageTranslate Team · · 8 min read
To measure translation quality, have a bilingual reviewer log every error in a sample by category and severity, add up weighted penalty points, scale the total to 1,000 words and compare it with a pass line you set before reading. That is the MQM method boiled down, and the scorecard below does the arithmetic for you.
Scores alone mislead. A translation can come out at 94.2 out of 100 and still fail, because one changed number (30 days becoming 3) counts as a critical error. The worked example shows how that happens, and how the same slips pass for a marketing team and fail for a legal one.
Exact price before you pay. Your own reviewer does the scoring.
Why "good enough" stops being good enough
A translation can read smoothly and still change a number. Readers rarely notice. The auditor does, and so does customer who followed the wrong instruction.
The damage tends to show up as support tickets and as a slogan that means something else in the target language. Or as a clause that says 3 days where the source said 30.
"Readable" and "high quality" are different bars. Readable means you can follow it. High quality means a bilingual expert would sign it, and that becomes measurable once you decide how.
What translation quality covers
Quality has a linguistic side and a functional side. Text can be flawless and still fail its job (wrong audience, wrong register, a layout nobody can use). A rough draft can do its job perfectly well if all you need is the gist.
What matters most also changes with the content.
| Content | Protect first | Usual reviewer |
|---|---|---|
| Contracts, compliance, medical | Accuracy, terminology, every number | Subject-matter linguist, sometimes a lawyer |
| Marketing and web pages | Voice, audience fit, locale | In-country marketer |
| Software and help text | Consistent terms, text that fits the screen | Bilingual product person |
| Fiction | Voice and rhythm | An editor who reads the target language well |
| Internal gist | Key facts | Whoever needs to act on it |
MQM in plain words
MQM stands for Multidimensional Quality Metrics. It grew out of EU research projects (QTLaunchPad and QT21, led by DFKI in Germany) and is now looked after by the MQM Council at themqm.org. The Council says MQM Core is used in ISO 5060 (2024), the standard for evaluating translation output, and Google's researchers used MQM to rate machine translation in the WMT benchmarks. Think of it as a shared vocabulary for errors.
Core has seven top-level dimensions.
| Dimension | The question it asks | Example (English to Spanish) |
|---|---|---|
| Accuracy | Does the target say what the source says? | "30 days" became "3 días" |
| Terminology | Is the approved term used, every time? | "Salvar cambios" where the glossary says "Guardar cambios" |
| Linguistic conventions | Are grammar, spelling and punctuation right? | "las problemas" |
| Style | Does it follow the tone and voice guidance? | Formal "usted" where the brand speaks to "tú" |
| Locale conventions | Do dates, numbers, units and currency fit the locale? | 10/07/2026 left in US order for a Spanish reader |
| Audience appropriateness | Does it suit these readers? | Slang in a letter about a pension |
| Design and markup | Do layout and tags survive? | Text spilling out of a button |
Older guides call Linguistic conventions "Fluency" and Audience appropriateness "Verity". Those are MQM 1.0 names.
Severity comes in four levels. Neutral means flag it but don't penalize it. Minor, major and critical carry penalties. MQM's sample multipliers for them are 1, 5 and 25. MQM labels those numbers non-normative, which makes them a reasonable default to start from. Critical means the content is unfit for its purpose or risks serious harm, and in many systems a single critical error fails the whole sample.
Build a one-page scorecard
Each error you log gets a category and a severity, and identical errors share a line with a count. The penalty for a line is points times count times a category weight (the weights are yours to set). Add the lines up and scale to 1,000 words, so samples of different sizes can be compared.
- Pick the sample. MQM says 500 to 20,000 words is typical and the full text is safest. My own habit is at least 1,000 words from three places, including the hardest section.
- Set the pass line first.
- Log errors as you read. One row each, with a category and a severity.
- Add up the penalty, then multiply by 1,000 and divide by the words you reviewed.
- Fail on any critical error. Otherwise compare the result to the line.
A raw score is 100 minus a tenth of the penalty per 1,000 words. So 5 per 1,000 words is 99.5 and 28 per 1,000 is 97.2, which are the figures MQM's scoring page uses as examples for legal content and technical help. I added 15 for marketing as a middle ground, and that one is purely my pick.
Translation quality scorecard
| Category | Weight | Minor (1) | Major (5) | Critical (25) |
|---|---|---|---|---|
| Accuracy | 1 | |||
| Terminology | 1 | |||
| Linguistic conventions | 1 | |||
| Style | 2 | |||
| Locale conventions | 1 | |||
| Audience appropriateness | 1 | |||
| Design and markup | 1 |
Pass. 0 penalty points per 1,000 words (raw score 100).
0 weighted points over 500 words, against a pass line of 15.
Points per error are MQM's sample multipliers (1, 5 and 25), which MQM calls non-normative. Weights and pass lines are example starting points. Agree your own before anyone reads the translation, and keep them constant if you want scores to be comparable over time.
You don't need all of MQM to start. Seven categories and three severities are enough for a useful review, and you can ignore any category your content doesn't have.
Score three samples
Here are three invented English to Spanish pairs, judged as if they came from a 500-word page that was otherwise clean.
| Source | Translation | What is wrong | Category | Severity |
|---|---|---|---|---|
| Your file is deleted after 30 days. | Su archivo se elimina después de 3 días. | A number in a retention promise changed | Accuracy | Critical |
| Save changes | Salvar cambios (twice on the page) | Glossary says "Guardar cambios" | Terminology | Minor, two of them |
| Don't let email run your day. | No deje que el correo corra su día. | "Corra" mistranslates "run" a day, and "deje" is formal where the brand says "tú" | Accuracy and Style | Minor each |
With every weight at 1, the penalty is 25 + 2 + 1 + 1 = 29 points. Scaled to 1,000 words that is 29 × 1,000 ÷ 500 = 58, a raw score of 94.2.
94.2 sounds respectable. It fails every pass line in this article, and the critical error would fail it alone. To reproduce the 58, set the scorecard above to Help articles and internal docs, where every weight is 1, and enter one critical Accuracy error, two minor Terminology errors, one minor Accuracy error and one minor Style error.
Now the other half of the story. Take two more invented logs, again on 500 words. Log X holds four tone slips, all minor Style errors. Log Y is one product term rendered inconsistently, logged once as a major Terminology error. A marketing team doubles the weight on Style. A legal team doubles Accuracy and Terminology and halves Style.
- Log X under marketing weights comes to 4 × 1 × 2 = 8 points, or 16 per 1,000. That is over a line of 15.
- Under legal weights the same log is 4 × 1 × 0.5 = 2 points, or 4 per 1,000, inside a line of 5.
- Log Y under marketing weights is 5 points, or 10 per 1,000. Inside the line.
- Under legal weights Log Y is 5 × 2 = 10 points, or 20 per 1,000, well over.
The same slips, scored by two teams. Both error logs and every weight are invented for this example.
Neither team is wrong. They protect different things, which is why the weights and the line belong in a document someone signed before the review. I suspect most quality disputes are really disagreements about the pass line, and nobody notices because it was never written down.
Use the scorecard in your workflow
Three moments justify the effort. Vendor onboarding, where every candidate translates the same 1,000-word test and you score them identically. A spot check on each release. And a periodic audit, say quarterly, plus one whenever something changes (new vendor, new machine translation engine, new glossary).
Reviewers matter as much as the form. An in-country reader who knows the audience catches what a dictionary can't. A bilingual project manager can triage. For anything high-risk, bring in a professional linguist, and keep the translator from marking their own work.
Then do something with the number.
| What the log shows | What to do |
|---|---|
| Any critical error | Retranslate that passage and find out how it got through. Check every number and unit nearby. |
| Terminology majors cluster | Fix the glossary, then rescore a fresh sample |
| Style minors cluster | Write a one-page style guide with three before-and-after examples |
| Accuracy majors from machine output | Post-edit the section (see the post-editing workflow) or change the engine |
| Scores swing between audits | Check that the sample, the weights and the reviewer stayed the same |
Track it in a spreadsheet with a row per review. Date, document, language, reviewer, words reviewed, penalty per 1,000 words, critical count and notes make a workable set of columns. MQM warns that scores built on different weights or calibration aren't directly comparable, so keep yours constant.
Improve quality before the review
Most fixes happen upstream. A style guide can be two pages long and still settle a dozen arguments. A glossary of approved terms settles the rest. Earlier approved translations and screenshots of the product help too.
Choose the process to match the stakes. Raw machine output is fine for gist. A machine draft plus human post-editing suits a lot of business content. Contracts and medical text deserve a human translator and a second linguist. The trade-offs are laid out in machine translation vs human translation and in the notes on Google Translate's limits.
Close the loop.. Share the error log with whoever produced the translation and update the glossary. Next time, score a new sample.
PageTranslate has no glossary upload, no translation memory and no reviewer workflow. It translates whole documents in one job and keeps names as written. The DOCX output stays editable in Word, and an EPUB can be edited in calibre (here is how to edit EPUB files). So a practical loop is to translate, hand the file to your reviewer, score a sample, fix recurring terms with find-and-replace in the returned file, and score a fresh sample. You can try one chapter first with a free account's starter words. For the product side, see the AI novel translator and how technical documents are handled.