Google Translate is the first place I turn for free machine translation (MT), mostly because it supports the greatest number of language pairs. I use Microsoft Translator as well, but usually only when I want to compare engines. I haven’t used Babel Fish in years.
But which engine offers the highest quality translations? I’m assuming Google, but this is only based on anecdotal feedback and personal experience.
Years ago, IBM developed an algorithmic method of measuring MT quality known as the BLEU score. Google scored well here, but the BLEU score is not without its critics.
Translation, like writing itself, is as much an art as it is a science.
Which is why translators are best positioned to judge the quality of machine translation engines. And although even translators are going to disagree as well, if you get enough of them together, perhaps you can begin to draw statistically significant conclusions.
Enter Ethan Shen and his start-up venture Gabble On.
Ethan has set out to recruit a few thousand volunteer translators to compare the three free translation engines. He asked me to help get out the word. He promises that he will publish the results for all to see. He’s also offering a free Apple iPad to one lucky volunteer. I have no financial interest in the project. I’m just curious to see what engine comes out on top.
Here are the details from Ethan:
We are seeking functional to fluent speakers of any two languages to take 5 minutes to judge and submit their opinion in our dynamic comparison engine (until March 29, 2010). At the end of the 6 week voting period, we will be publishing our results publicly in hopes that our research can to contribute meaningfully to the body of knowledge in this field.
In gratitude for your participation, we are awarding one new Apple iPad to a lucky participant. The survey can be found at: www.gabble-on.com/SurveySelector.aspx.
Which engine do you think is best?
Tweet this post on Twitter
Subscribe to Global by Design
2 responses so far ↓
1 Mike Unwalla @ TechScribe // Feb 22, 2010 at 10:04 am
@John: Which is why translators are best positioned to judge the quality of machine translation engines.
@John: But which engine offers the highest quality translations? I’m assuming Google, but this is only based on anecdotal feedback and personal experience.
For an evaluation of the quality of machine translation from Google, see http://www.international-english.co.uk/mt-evaluation.html. English to Spanish and English to Norwegian give satisfactory translations. English to Welsh is very low quality.
@John: And although even translators are going to disagree as well, if you get enough of them together, perhaps you can begin to draw statistically significant conclusions.
Yes, we had the problem that the translators disagreed with their evaluations of some sentences. For example, if an English term is in the translation, some translators gave a low rating to the sentence and other translators did not necessarily give a low rating. We did not want to give the translators too many guidelines for their evaluation, because we wanted the translators to use their judgement. However, we needed to give some guidelines.
2 Is Google the best machine translation engine? It depends… | Global by Design // Mar 12, 2010 at 3:12 pm
[...] weeks ago, I introduced Ethan Shen and his project to analyze the three major free machine translation (MT) engines — Google, Microsoft, and [...]
Leave a Comment