• Google Translate
• Bing (Microsoft) Translator
• Yahoo! Babelfish

Ethan Shen has wrapped up the project, soliciting input from more than 1,000 reviewers. He summed up his findings here.

Here are the findings that jumped out at me:

• Google wins, hands down, translating longer text passages. No big surprise here.
• Bing and Babelfish are competitive translating shorter texts (150 or fewer characters). Bing did quite well with Italian and German, while Babelfish did well with Chinese.
• Google’s brand trumps all. About halfway through his test, Ethan removed the brand names from the search engines, so the reviewers did not know which engine was doing which translation. The change in results was significant. Reviewers were 21% more likely to say Google was better than Microsoft when they knew the brand names. And reviewers were 136% more likely to say Google was better than Babelfish.

This last finding is what poses the greatest hurdle for Microsoft and Yahoo!

When it comes to machine translation — perception is (almost) everything. If people think you’re the best translation engine, then you are the best.

Integration is the other key element of success, and Google Translate is doing well here also — I absolutely love the Chrome browser integration.

Ethan is not done with his research. This is only stage one. To help him with stage two, click here.

Ethan has provided me with a mid-point summary of results, which I’ve included below. I was surprised to find that Microsoft and Babelfish are beating Google on some languages pairs, as well as on shorter text strings. Although Google is emerging the overall winner — and receiving some much-deserved attention from the media — it’s nice to see some healthy competition.

That said, quality is only one piece of the puzzle. The other piece — perhaps much more important — is usability. Now that Google has embedded its MT engine into Gmail and Reader — and now its Chrome client –I find I’m using Google exclusively as my MT engine.

Here are Ethan’s findings so far (emphasis mine):

At the highest level, it appears that survey participants prefer Google Translate’s results across the board.

In a few languages (Arabic, Polish, Dutch) the preference is overwhelming with votes for Google doubling its nearest competitor

However, once you remove voters that have self defined their fluency in the source or target language as “limited,” the contest becomes closer along some of the heavily trafficked languages. For example:

• Microsoft Bing Translator leads in German
• Yahoo! Babelfish leads in Chinese
• Google maintains its lead in Spanish, Japanese, and French

Observing only the self-defined “limited fluency” voter reveals a strong brand bias. If your fluency in the target translation language is limited, it would stand to reason your ability to assess the quality of the translation is very limited. And yet…

• Limited-fluency voters chose Google over Bing by 2 to 1
• They also chose Google over Yahoo! Babelfish by 5 to 1

As I had guessed, Yahoo! and Microsoft’s hybrid rules-based MT model performed better on shorter text passages

For phrases below 50 characters, Google’s lead in Spanish, Japanese, and French disappear. And Microsoft’s lead in German widens.

Beyond 50 characters, Google’s relative performance seems to improve across the board.

For passages that are only one sentence, the same effect is seen, though to a lesser extent than under 50 characters.

On March 4th, we made a few changes to our survey – hiding the brands and randomizing the positions of the text results before voting.  Since then, we have not yet collected enough data to draw conclusions, but Babelfish seems to be receiving the biggest boost, perhaps showing the effects of the recent neglect of that tool.

Clearly, Ethan needs more data to arrive at more concrete conclusions. If you’re a translator and you want to lend a hand, here is the voting site.

PS: Here’s an interview with Google’s MT guru Franz Josef Och.

But which engine offers the highest quality translations? I’m assuming Google, but this is only based on anecdotal feedback and personal experience.

Years ago, IBM developed an algorithmic method of measuring MT quality known as the BLEU score. Google scored well here, but the BLEU score is not without its critics.

Translation, like writing itself, is as much an art as it is a science.

Which is why translators are best positioned to judge the quality of machine translation engines. And although even translators are going to disagree as well, if you get enough of them together, perhaps you can begin to draw statistically significant conclusions.

Enter Ethan Shen and his start-up venture Gabble On.

Ethan has set out to recruit a few thousand volunteer translators to compare the three free translation engines. He asked me to help get out the word. He promises that he will publish the results for all to see. He’s also offering a free Apple iPad to one lucky volunteer. I have no financial interest in the project. I’m just curious to see what engine comes out on top.

Here are the details from Ethan:

We are seeking functional to fluent speakers of any two languages to take 5 minutes to judge and submit their opinion in our dynamic comparison engine (until March 29, 2010). At the end of the 6 week voting period, we will be publishing our results publicly in hopes that our research can to contribute meaningfully to the body of knowledge in this field.

In gratitude for your participation, we are awarding one new Apple iPad to a lucky participant. The survey can be found at: www.gabble-on.com/SurveySelector.aspx.

Which engine do you think is best?

It was a tough decision to make. I enjoyed working with the many people who are passionate about globalization. And I enjoyed being a part of one of the most global companies on the planet.

But the time was right for me to move on.

I approached Midge Raymond regarding the possibility of returning to Byte Level and, after much negotiation, she extended an offer.

I am now working with Midge on the 2010 Web Globalization Report Card. As Midge noted a few weeks ago, a lot has happened over the past 18 months. You’d think a global recession would have kept web globalization to a minimum, but the data is showing continued growth.

Shortly after the report publishes, I will be making time for consulting projects. If you need someone to evaluate your company’s web site, train your web and marketing teams, or conduct a conference call on web globalization best practices, please contact me at Byte Level.

Firefox 3.5 is now out and available in more than 70 languages.

Yes, 70 languages.

Naturally, I was curious to learn how many languages the other browsers currently support. Here is a rough list:

• Internet Explorer 8: 63 languages
• Chrome: 63 languages
• Opera 9.6: 36 languages
• Safari 4: 16 languages

I say “rough” because I didn’t verify every language and I might be off by one or two. I was surprised at how few languages Safari supports; 16 languages used to be a lot not that long ago. But not anymore.

And I think it’s safe to say that Firefox is the most global browser on the market today.

If you want to see how popular Firefox is around the world, check out the real-time download map here. The last I checked there were 23 million downloads with 5 million coming from the US, followed by Germany, Japan, and France. I particularly like how you can look up countries by ccTLD.

And on a separate note: If you enjoy watching real-time downloads, here’s another map that I found oddly hypnotizing — Zappos purchases as they happen: www.zappos.com/map/

Since Bing launched I’ve been thinking a lot about search engines and how I use them.

I’ve got two recent examples that illustrate why I think Bing might be onto something. Bing, for certain scenarios, cuts the clicks you must make to get the information you need — or at least the information I need.

Checking a sports score

I’m a St. Louis Cardinals fan. When I want to get a Cardinals score, I often go to ESPN, but that site loads so slowly and is so busy that I have been going to Bing lately and just entering “Cardinals” in the search window. Here’s what I get:

Bing gives me a nice summary of the Cardinals schedule. If there’s a game going on at the moment, I get a real-time sports score, which is nifty. Google, as shown here, only gives me a link to the Cardinals’ site. Another click, instead of a score.

Checking on a flight

My wife flew to Oakland recently and I wanted to check on her flight. So I entered the flight number into Google and Bing. Here’s what I found:

I got the arrival time so I knew when I could call her.

Google gives me a link to another site that will give me the details that Bing already gave me.

Google got to where it is today by prioritizing speed. Austere web design and massive data centers gave its search engine a massive advantage over everything else out there.

But speed isn’t just about how quickly a search page loads, it’s about how quickly you find what you’re looking for. If a search engine knows you’re looking for a sports score and not a sports team web site, it can save you a click and, as a result, save you time.

Time is clicks. You save people time by saving them clicks.

Granted, I’m probably not the most objective observer of the Bing vs. Google debate. So what do you think?

Are these two Bing innovations going to stick?

For now, I’d like you to try entering the following text strings into both Bing and Google (to save you time I created pre-loaded hyperlinks):

• Translate I love you: Bing vs. Google
• Translate I love you into Chinese: Bing vs. Google
• How do you say I love you in Italian: Bing vs. Google

Below are screen shots of the first text string in both Bing and Google. I’ll let the pictures speak for themselves:

Google, despite its massively powerful translation engine, doesn’t simply answer your translation question. Instead, it provides links.

I realize that this is a relatively minor feature and that it currently only supports a small number of very common text strings, but it’s still a very handy feature for a translation geek such as myself.

Now, I’m not saying Bing is perfect. When it comes to technical searches — or when I just need to look up a Wikipedia article quickly — Google still does better, sometimes far better.

But I’m glad to see Bing integrating translation in an intuitive way. It’s a feature that I’ll be using again.

PS: Here is the blog announcement of this feature from Microsoft Translate team.

Turns out, most companies haven’t even registered Twitter accounts for their primary brands.

Like who?

Apple, for one.

Here we have someone who apparently likes apples but isn’t Apple:

It appear that Microsoft reserved its account early on, though nothing is there. Microsoft does have about a dozen Twitter accounts that do include content.

Coke — someone who drinks Coke, but not the company.

While Pepsi does have a Twitter account.

The Wall Street Journal has an article out about this domain name rush.

So many questions come to mind:

• Will Twitter enforce trademarks for valid holders? Usually, the WIPO does this with domain names, but this isn’t actually a domain name in the traditional sense.
• What percentage of the millions of new Twitter accounts being registered every day simply squatters hoping to make a quick buck? That is, how much of Twitter’s growth actual growth?
• And what about third-party domain marketplaces — will we see them emerge? Or will Twitter start its own marketplace?

In the meantime, if you’re thinking about reserving a Twitter domain, do it now before getting Twit-jacked…

Does this icon say “language” to you?

It doesn’t to me.

But the OMC design studio feels so strongly about it that it has launched a web site to promote this icon as a global standard.

I applaud the effort and I fully agree that there is a need for such an icon, but I don’t believe that this one should be it. I find that this looks like a floppy disk (and, yes, I’m aware that there is an entire generation of computer users out there who don’t even know what a floppy disk looks like).

If I were to vote for an international icon, I would vote for a generic globe icon. Companies such as Panasonic, Dow Corning, and Microsoft have used a globe icon to denote either language or country/region (or both).

Other companies use tiny maps, such as John Deere and Caterpillar.

I prefer the globe, but either will do the trick.

What do you think?

PS: I just discovered that I wrote about the need for a standard icon way back in 2004. I preferred the globe icon even back then.

I’m happy to see Microsoft Research launch this new Web site widget.

I’ll have to test it out on my site when I get the time — and would love to hear from others who have tried it.

A demo site is here.

I really like how the widget creates a more seamless translation experience, which is a big step toward taking machine translation to the masses. I’m looking forward to the day when I no longer have to manually copy and paste URLs into Google Translate and the like.

However, I did get confused initially on figuring out how to return the page back to English. That bar at the top of the page blended in a bit too much at first. I’d like to see a language reset button on the widget itself.