Comments on: Ruby 1.9 Encodings: A Primer and the Solution for Rails

By: Arne Brasseur

Arne Brasseur — Mon, 15 Apr 2013 17:03:24 +0000

The reason the Japanese don’t like simply converting everything to Unicode is because of a process adopted by the Unicode consortium called Han Unification. Unicode encodes abstract “glyphs” rather than concrete “graphemes”. For example, the letter ‘a’ can be written as a circle with a line to the right, or it can have an extra curl at the top. These are two graphemes of the same glyph, so there is only one Unicode code point for both, the difference is in the font that renders it.

Chinese, Japanese, traditional Korean and to some extent Vietnamese all use Han characters (known as Chinese characters, Hanzi or Kanji). Often the same glyph is used in all four languages, but with small regional variants in the graphemes. So to correctly display a Japanese text encoded in Unicode, you need to use a font that’s specific for Japanese, or you might end up seeing Chinese variants for instance. These variations might seem very small to a western eye, but they are considered an important part of these nations’ culture and history.

The Japanese especially have always strongly opposed Han unification. It is however possible to do a lossless round trip convertion from JIS to UTF-8 and back, in fact Unicode has several code points that were added specifically to make these kind of round trip conversion lossless.

By: Serge Bedzhik

Serge Bedzhik — Tue, 13 Mar 2012 07:20:21 +0000

Hi there! Thanks for the great article :)

I have a little question. I’m trying to deal with ASCII-8BIT (as it detected), coming from TCP socket with Russian symbols like \xD1, \xD0 etc., but get scrambled output when force_encoding(‘UTF-8′) on the BINARY data, exactly like you say. What I can to do, to get normal UTF-8 output when I send this symbols over telnet? I’m talking about TCPServer from stdlib.

By: Jonas Elfström

Jonas Elfström — Sun, 05 Feb 2012 22:55:51 +0000

Please consider clarifying, or giving a source that explains, the complicated reasons of UTF-8 not mapping perfectly to SHIFT-JIS that you mentioned.

By: Mike Rose

Mike Rose — Wed, 24 Aug 2011 00:42:03 +0000

Just for the sanity of those visiting this blog article in hopes of dealing with MySQL Ascii8 & UTF-8, I found that the Mysql2 Gem actually solves all of the issues I had when using the default Mysql gem for ruby-mysql interactions. Give that a try if you get odd UnknownConversion errors :)

By: Maher Sllam

Maher Sllam — Wed, 10 Aug 2011 05:02:14 +0000

Thank you for this great article.

I think that ruby should add an encoding detection method. Most Arabic websites uses windows-1256. Some websites read about the benefits of UTF-8 and tried to switch to it… Can you guess what they did? They just changed the encoding in their html tag. It worked for websites with static content, but failed for the other websites which used database content.

Some were luck to revert back before new data got it’s way into the database (that’s because they saw weird text all over the place when opening their websites), others were not. Consequently, now support forums are filled with help requests asking about how to correct this kind of errors.

In these situations, an encoding detection method will be AWESOME! Writing such functionality might be impossible, but I think there should be something which can help in these situations.

By: Tim

Tim — Wed, 04 May 2011 08:43:54 +0000

I’d also love to see a reference for the part about SHIFT-JIS to UTF-8 being lossy, as I searched and couldn’t find anything (but perhaps that’s because I can’t speak japanese ;)).

By: Mark Towfiq

Mark Towfiq — Mon, 25 Apr 2011 18:08:05 +0000

“For a variety of complicated reasons, Japanese encoding, such as SHIFT-JIS, are not considered to losslessly encode into UTF-8.”

What are the complicated reasons? I can’t find references to them. They appear to be the root cause of a major “configuration over convention” choice which runs counter to core Ruby philosophy.

By: Pankaj

Pankaj — Sat, 16 Apr 2011 10:58:44 +0000

Hi Yehuda,
I have been searching for this from last so many days, Thanks for such a great article.
I have come across some other issue, my application has admin side and client side both using the same database, Database contains the name like “Crème Fraîche”.
When I do force encoding like r.name.force_encoding(“ISO-8859-1″).encode(“UTF-8″)
admin side it displays the data properly but on client side it gives error “incompatible character encodings: ISO-8859-1 and UTF-8″

Before doing force encoding when I see the name in logs they are proper both admin as well as client side,

Am I missing something on client side, some other change might be required, please help I have been seeing for this problem from last 5 days.

By: shevy

shevy — Sat, 05 Feb 2011 01:42:29 +0000

The encoding stuff bores me to no ends and is total crap anyway.

I want the default behaviour of Ruby 1.8.x

I don’t care about the change in 1.9

I am not japanese, I don’t even use UTF-8

Why am I forced to deal with encoding in Ruby 1.9 now???

By: shir

shir — Sun, 23 Jan 2011 12:04:54 +0000

hi yehuda
can you explain me how to use hebrew characters with ruby 1.9.2,
thanks