(4) Characters and Computers

• American Standard Code for Information Interchange (ASCII)
  • for the first time a basic set of characters had a universally accepted encoding
  • many Internet protocols (such as HTTP [Web Foundations (URI & HTTP); Hypertext Transfer Protocol (HTTP) (1)]) encode their information in ASCII commands
• ASCII is a very limited repertoire of characters
  • basic ASCII contains 128 characters (7 bit) with a number of control chars
  • no variants of characters (german umlauts, french accents) are supported
  • various code pages extending ASCII to 8 bit exist and are hard to distinguish
• Character is not a trivial concept when regarded globally
  • european languages have writing systems based on a small number of atoms
  • other languages and writing systems have different ideas of language atoms

(5) Characters

Character. (1) The smallest component of written language that has semantic value; refers to the abstract meaning and/or shape […]

The Unicode Standard, Version 4.0, Addison-Wesley, 2003 [http://dret.net/biblio/reference/unicode4]

• Different languages or writing systems use different characters
  • english: A B C D E F G H I J K L …
  • german: A Ä B C D E F G H I J K L …
  • russian: А Б В Г Д Е Ё Ж З И Й К …
  • greek: Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ …
  • hiragana: い ろ は に お え ど ち り ぬ る を …
  • kanji: 阪 熊 奈 岡 鹿 梨 阜 埼 茨 栃 媛 …

(6) Glyphs

[A Glyph is] a recognizable abstract graphic symbol which is independent of a specific design.

ISO/IEC 9541:1991, Information Technology – Font Information Interchange [http://dret.net/biblio/reference/iso9541]

• Visual rendering introduces the notion of a glyph.
• There is not a one-to-one correspondence between characters and glyphs
  • A A : a single character can be represented by multiple glyphs
  • ffi : a single glyph may represent a sequence of characters
  • a character may be rendered with different glyphs (old german script [http://www.tr62.de/script/letter4.html])
  • A : a single glyph may represent different characters (e.g. latin, greek, and cyrillic)

(7) Character Set Identification

• HTTP, XML, and HTML support character set identification
  • HTTP [Web Foundations (URI & HTTP); Hypertext Transfer Protocol (HTTP) (1)] supports the Content-Type header field [../services-fall06/web1#(17)]

    Content-Type: text/html; charset=utf-8

  • XML encodes the character set in the XML declaration [../xml-fall08/basics#(8)]

    <?xml version="1.0" encoding="utf-8"?>

  • HTML [Hypertext Markup Language (HTML)] supports the meta element in the document's head

    <meta http-equiv="Content-Type" content="text/html;charset=utf-8">

(8) History of Character Sets

• Text documents need ways to represent characters
  • computers handle bits, not characters
  • to handle characters, computers need a mapping from characters to bits
• For a long time, computers were doing their work in a very isolated way
  • I think there is a world market for maybe five computers. (¬ T. J. Watson [http://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_misquote])
• With more computers being used, more data is exchanged between computers
• Standardization of character sets started in the 1960's
  • ASCII was the first generally accepted character set
  • EBCDIC was invented and marketed by IBM (more than 128 characters)
  • ISO 8859 was the first attempt to better support character sets beyond ASCII
  • asian scripts were always a problem because of the number of characters they need

(9) ASCII 1967

(10) Beyond ASCII

• ASCII is called ASCII for a reason
  • it works well for english-speaking countries
  • the majority of other languages cannot be represented
• Character sets and the 8 bit computer start to collide
  • ASCII is very convenient because characters and bytes correspond 1:1
  • every character set expanding ASCII will make this more complicated
  • complications can occur within and/or outside of the character set
• Introducing a character set beyond 8 bit is a fundamental change
  • dealing with and counting bytes is a seductively simple idea
• Introducing several 8 bit character sets saves the 8 bit world
  • by introducing several character sets, each of them can remain 8 bit
  • the complexity has now been shifted to the handling of various character sets

(11) ISO 8859

• A family of character sets rather that a single character set
  • each ISO 8859 family member is an 8 bit character set (256 characters)
  • the lower half (128 characters) are always the same (ASCII)
  • the upper half is supporting different user groups and changes between versions
• ISO 8859 files cannot be identified by inspection
  • ASCII characters can always be safely interpreted (identical on all ISO 8859 code pages)
  • the upper half can only be interpreted if the code page is well-known
• ISO 8859 environments must carefully track the code pages being used
  • failure to do so results in misinterpretation of characters

  It also shows the Euro sign € which is part of ISO 8859-15 (Latin-9), but not included in ISO 8859-1 (Latin-1).

  • iso8859-15.txt (line 3-3)

  It also shows the Euro sign ¤ which is part of ISO 8859-15 (Latin-9), but not included in ISO 8859-1 (Latin-1).

  • iso8859-15.txt (line 3-3)

(12) ISO 8859-1 (Latin-1) & ISO 8859-15 (Latin-9)

Latin-1 (Western European)

Latin-9

(14) ISO 8859 Problems

• One document can only contain characters from one character set
  • mixing characters from different sets is impossible

  In Latin-9, Latin-1's currency symbol ¤ has been replaced with the Euro sign €.

  • currency-euro.txt
• An increasing number of character sets does not make life easier
  • in particular, if they sometime differ only slightly (e.g., Latin-1 vs. Latin-9)
• For bigger character sets, the 8 bit approach is not working at all
  • the ISO 8859 approach allows only 128 special characters (the lower half is ASCII)
• ISO 8859 is as good as it gets with 8 bit
  • to improve this approach, the 8 bit philosophy must be abandoned

(15) Unicode

• Generalization takes characters beyond one and even two bytes
  • Unicode has been designed to cover all characters of the world
  • Unicode recently added its 100'000^th character
  • for handling this character set, a more structured approach is required
• Unicode cleanly separates various conceptual steps
  • characters are collected and are then part of the character repertoire
  • characters are then identified by a unique code point (written as U+0041)
  • a Character Encoding Scheme (CES) then maps the Coded Character Set (CCS) based on a Character Encoding Form (CEF)
• XML is ASCII for the 21^st century [../xml-fall08/basics#(6)]
  • purists sometimes consider Unicode too big or dangerous
  • Unicode is well-established and is necessary in a globalized economy

(16) Unicode Character Count

• Unicode has the ability to encode 17 × 2¹⁶ = 1'114'112 characters
  • this means that currently ~10% of the available space is used
• Characters are organized into 17 planes of 2¹⁶ = 65'536 characters
• Planes are numbered from 0 to 16
• Plane 0 is the Basic Multilingual Plane (BMP)
  • all characters which in practical use today are part of the BMP (well, almost … [http://www.tlg.uci.edu/~opoudjis/unicode/unicode_astral.html])
• Planes beyond the BMP contain rare and historic characters
  • Old Italic [http://unicode.org/charts/PDF/U10300.pdf], Deseret [http://unicode.org/charts/PDF/U10400.pdf], Byzantine Musical Symbols [http://unicode.org/charts/PDF/U1D000.pdf]
  • most space within these astral planes is empty

(18) What is Language?

• Language is more than the encoding of individual words
  • languages and culture are deeply intertwined and inseparable
  • ideally, systems/solutions should adapt to culture, not only to language
  • on a superficial level, adapting to language is useful a first step
• Languages have properties which are beyond character sequences
  • for right-to-left languages, screen layout should be done right-to-left
  • if languages are mixed, one language has to be the preferred language
• Language identification and selection are basic I18N tasks
  • the one language fits all assumption is becoming increasingly inappropriate
  • the just switch the labels strategy also may be too little for true L10N [Localization (L10N) (1)]

(19) Beyond Language

• Directionality is the concept how written language is organized
  • early languages had no inherent directionality (even zigzag writing was possible)
  • the majority of today's languages are left-to-right languages
  • Arabic and Hebrew are two popular right-to-left languages
  • Chinese can be written top-to-bottom and right-to-left
  • Mongolian (using the Uighur alphabet [http://www.linguamongolia.co.uk/]) is written top-to-bottom and left-to-right
• Icons can be very culture-specific and often need to be localized as well
  • icons are pictorial metaphors (rooted in language and/or culture)
  • language-specific metaphors may not work across languages (e.g., OK gesture [http://customsholidays.suite101.com/article.cfm/international_ok_gestures])
  • culture-specific metaphors may not work across cultures (e.g., owls)

(20) Directionality and Screen Layout

(21) Definition

Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language.

• I18N starts at the design phase and influences the complete process
• The upfront costs of I18N are considerable (increased complexity)
• The costs of retroactive I18N are much higher than that

(22) I18N Tasks

1. UI elements (windows, menus) must be modified to accept translated text
2. Static text must be made configurable
3. Icons and graphics must be changed to be more culturally appropriate
4. Sound files that contain spoken language must be re-recorded
5. Online help must be translated
6. Dynamic text (dates, times) must be formatted using the locale
7. Text handling code must calculate word breaks using the locale
8. Tabular data must be sortable using the locale

(24) Definition

Localization refers to the adaptation of a product, application or document content to meet the language, cultural and other requirements of a specific target market (a locale).

• Without proper I18N, L10N is a very expensive and risky process
• L10N based on internationalized products should be straight-forward
• during L10N for specific locales, new areas for I18N might be identified
  • some non-internationalized part of the product is identified as inappropriate
• L10N should be regarded as an input for improved I18N

(25) L10N Tasks

1. Create translations for all interface elements
2. Translate all static texts
3. If necessary, create localized icons and graphics
4. Any spoken text must be recorded in the target language
5. Make sure that the localized product uses the localized online help
6. Formatting of data types must be treated locale-specific
7. If necessary, dictionaries and other language tools must be integrated
8. Sorting functions in the code must respect the locale