(4) Characters and Computers

• American Standard Code for Information Interchange (ASCII)
  • for the first time a basic set of characters had a universally accepted encoding
  • many Internet protocols (such as HTTP [../services-fall06/web1#(12)]) encode their information in ASCII commands
• ASCII is a very limited repertoire of characters
  • basic ASCII contains 128 characters (7 bit) with a number of control chars
  • no variants of characters (german umlauts, french accents) are supported
  • various code pages extending ASCII to 8 bit exist and are hard to distinguish
• Character is not a trivial concept when regarded globally
  • european languages all have writing systems based on a small number of atoms
  • other languages and writing systems have vastly different ideas of language atoms

(5) Characters

Character. (1) The smallest component of written language that has semantic value; refers to the abstract meaning and/or shape […]

The Unicode Standard, Version 4.0, Addison-Wesley, 2003 [http://dret.net/biblio/reference/unicode4]

• The alphabetic approach is only one of several possibilities
  • A character in Japanese hiragana and katakana scripts corresponds to a syllable (usually a combination of consonant plus vowel)
  • Korean Hangul combines symbols for individual sounds of the language into square blocks, each of which represents a syllable; depending on the user and the application, either the individual symbols or the syllabic clusters can be considered to be characters
  • In Indic scripts each consonant letter carries an inherent vowel that is eliminated or replaced using semi-regular or irregular ways to combine consonants and vowels into clusters; depending on the user and the application, either individual consonants or vowels, or the consonant or consonant-vowel clusters can be perceived as characters
  • Arabic and Hebrew vowel sounds are typically not written at all; when they are written they are indicated by the use of combining marks placed above and below the consonantal letters

(6) Glyphs

[A Glyph is] a recognizable abstract graphic symbol which is independent of a specific design.

ISO/IEC 9541:1991, Information Technology – Font Information Interchange [http://dret.net/biblio/reference/iso9541]

• Visual rendering introduces the notion of a glyph.
• There is not a one-to-one correspondence between characters and glyphs
  • A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character); these glyphs may be physically separated from one another
  • A single glyph may represent a sequence of characters (this is the case with ligatures, among others)
  • A character may be rendered with very different glyphs depending on the context
  • A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A)

(8) History of Character Sets

• Text documents need ways to represent characters
  • computers handle bits, not characters
  • to handle characters, computers need a mapping from characters to bits
• For a long time, computers were doing their work in a very isolated way
  • I think there is a world market for maybe five computers. (¬ T. J. Watson [http://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_misquote])
• With more computers being used, more data is exchanged between computers
• Data rot happens on all levels (media, formats, applications)
• Standardization of character sets started in the 60's
  • ASCII was the first generally accepted character set
  • EBCDIC was invented and marketed by IBM (and a terribly designed character encoding)
  • ISO 8859 was the first attempt to better support character sets beyond ASCII
  • asian scripts were always a problem because of the number of characters they need

(9) ASCII 1963

(10) ASCII 1965

(11) ASCII 1967

(12) Beyond ASCII

• ASCII is called ASCII for a reason
  • it works well for english-speaking countries
  • the majority of other languages cannot be represented
• Character sets and the 8 bit computer start to collide
  • ASCII is very convenient because characters and bytes correspond 1:1
  • every character set expanding ASCII will make this more complicated
  • complications can occur within and/or outside of the character set
• Introducing a character set beyond 8 bit is a fundamental change
  • dealing with and counting bytes is a seductively simple idea
• Introducing several 8 bit character sets saves the 8 bit world
  • by introducing several character sets, each of them can remain 8 bit
  • the complexity has now been shifted to the handling of various character sets

(13) ISO 8859

• A family of character sets rather that a single character set
  • each ISO 8859 family member is an 8 bit character set (256 characters)
  • the lower half (128 characters) are always the same (ASCII)
  • the upper half is supporting different user groups and changes between versions
• ISO 8859 files cannot be identified by inspection
  • ASCII characters can always be safely interpreted (identical on all ISO 8859 code pages)
  • the upper half can only be interpreted if the code page is well-known
• ISO 8859 environments must carefully track the code pages being used
  • failure to do so results in misinterpretation of characters

  It also shows the Euro sign € which is part of ISO 8859-15 (Latin-9), but not included in ISO 8859-1 (Latin-1).

  • iso8859-15.txt (line 3-3)

  It also shows the Euro sign ¤ which is part of ISO 8859-15 (Latin-9), but not included in ISO 8859-1 (Latin-1).

  • iso8859-15.txt (line 3-3)

(14) ISO 8859-1 (Latin-1) & ISO 8859-2 (Latin-2)

Latin-1 (Western European)

Latin-2 (Central European)

(15) ISO 8859-7 (Greek) & ISO 8859-15 (Latin-9)

Greek

Latin-9

(17) ISO 8859 Problems

• One document can only contain characters from one character set
  • mixing characters from different sets is impossible

  In Latin-9, Latin-1's currency symbol ¤ has been replaced with the Euro sign €.

  • currency-euro.txt
• An increasing number of character sets does not make life easier
  • in particular, if they sometime differ only slightly (e.g., Latin-1 vs. Latin-9)
• For bigger character sets, the 8 bit approach is not working at all
  • the ISO 8859 approach allows only 128 special characters (the lower half is ASCII)
• ISO 8859 is as good as it gets with 8 bit
  • to improve this approach, the 8 bit philosophy must be abandoned

(18) Unicode

• Generalization takes characters beyond one and even two bytes
  • Unicode has been designed to cover all characters of the world
  • Unicode recently added its 100,000^th character
  • for handling this character set, a more structured approach is required
• Unicode cleanly separates various conceptual steps
  • characters are collected and are then part of the character repertoire
  • characters are then identified by a unique code point (written as U+0041)
  • a Character Encoding Scheme (CES) then maps the Coded Character Set (CCS) based on a Character Encoding Form (CEF)
• XML is ASCII for the 21^st century [../xml-fall09/basics#(6)]
  • purists sometimes consider Unicode too big or dangerous
  • Unicode is well-established and is necessary in a globalized economy

(19) Unicode Character Count

• Unicode has the ability to encode 17 × 2¹⁶ = 1'114'112 characters
  • this means that currently ~10% of the available space is used
• Characters are organized into 17 planes of 2¹⁶ = 65'536 characters
• Planes are numbered from 0 to 16
• Plane 0 is the Basic Multilingual Plane (BMP)
  • all characters which in practical use today are part of the BMP (well, almost … [http://www.tlg.uci.edu/~opoudjis/unicode/unicode_astral.html])
• Planes beyond the BMP contain rare and historic characters
  • Old Italic [http://unicode.org/charts/PDF/U10300.pdf], Deseret [http://unicode.org/charts/PDF/U10400.pdf], Byzantine Musical Symbols [http://unicode.org/charts/PDF/U1D000.pdf]
  • most space within these astral planes is empty

(20) Unicode Encodings

	A	א	好
Code point	U+0041	U+05D0	U+597D	U+233B4
UTF-8	41	D7 90	E5 A5 BD	F0 A3 8E B4
UTF-16	00 41	05 D0	59 7D	D8 4C DF B4
UTF-32	00 00 00 41	00 00 05 D0	00 00 59 7D	00 02 33 B4

(21) UTF-8

• UTF-8 is one of the two standardized encodings for XML
  • every ASCII document by definition is a UTF-8 document
  • UTF-8 must be supported by every XML implementation
• UTF-8 is not trivial, but it is widely supported and easy to implement
  • there is no 1:1 correspondence between bytes and characters
  • each Unicode character is encoded by 1-6 bytes
  • UTF-8 is good for europeans (1 byte per ASCII character)
• Non-Unicode documents must be transcoded to UTF-8
  • keeping track of resource character encodings is a good idea

(22) Other UTFs

• UTF-16 stores every character as 2 or more bytes
  • BMP characters are stored as 2 bytes
  • astral plane characters are stored as 4 bytes
  • UTF-16 is the other encoding (in addition to UTF-8) required by XML
• UTF-32 stores every character as 4 bytes
  • very simple and very inefficient (requires four times more space than ASCII)
• Multi-byte formats introduce the problem of byte order
  • UTF-16/32BE and UTF-16/32LE are stored with guaranteed endian
  • UTF-16/32 may use a Byte Order Mark (BOM) (U+FEFF) to detect the endian

(23) Character Set Identification

• HTTP, XML, and HTML support character set identification
  • HTTP [../services-fall06/web1#(12)] supports the Content-Type header field [../services-fall06/web1#(17)]

    Content-Type: text/html; charset=utf-8

  • XML encodes the character set in the XML declaration [../xml-fall09/basics#(8)]

    <?xml version="1.0" encoding="utf-8"?>

  • HTML supports the meta element in the document's head

    <meta http-equiv="Content-Type" content="text/html;charset=utf-8">

• Different identifications serve different purposes
• Conflicting identifications are a sign of management problems

(24) Unicode is Complex

• Many languages have characters which are composed
  • german umlauts are vowels with a double dot
  • french accents and the cedilla are also added to regular characters
• Unicode contains composed as well as composing characters
  • for most languages, composed characters are considered to be regular characters
  • in some circumstances, it might be required to compose a character out of a base and a diacritical mark
  • as a result, the question arises how to define the equality of these variants
• Unicode defines normal forms which prescribe one variant
  • a complex field with different concepts of equivalence (canonical and compatibility)
  • based on the equivalence forms, there are four normalization forms [http://www.unicode.org/reports/tr15/]

<français>français ≠ français</français>

<français>français ≠ français</français>

(25) Transcoding

• Handling Unicode in an unconstrained environment is not easy
  • Unicode's size and variability make character processing harder then it used to be
  • for some applications with limited character need, this might be too much
• In all text-based environments, well-defined rules must be defined for:
  1. the encoding of documents (maybe if variants, such as Unicode normalization forms)
  2. accepted incoming encodings and how they are mapped to the internal encoding
  3. available outgoing encodings and how they can be requested and will be generated
• Transcoding is the activity of changing the encoding of data
  • ideally, transcoding should be lossless and round-trip proof
  • in any scenario with non-trivial encodings, this is not an easy goal
  • even staying with one encoding can be a problem (if there are variations allowed)
• Transcoding is essential for maintaining data quality

(27) What is Language?

• Language is more than the encoding of individual words
  • languages and culture are deeply intertwined and inseparable
  • ideally, systems/solutions should adapt to culture, not only to language
  • on a superficial level, adapting to language is useful a first step
• Languages have properties which are beyond character sequences
  • for right-to-left languages, screen layout should be done right-to-left
  • if languages are mixed, one language has to be the preferred language
• Language identification and selection are basic I18N tasks
  • the one language fits all assumption is becoming increasingly inappropriate
  • the just switch the labels strategy also may be too little for true L10N

(28) Beyond Language

• Directionality is the concept how written language is organized
  • early languages had no inherent directionality (even zigzag writing was possible)
  • the majority of today's languages are left-to-right languages
  • Arabic and Hebrew are two popular right-to-left languages
  • Chinese can be written top-to-bottom and right-to-left
  • Mongolian (using the Uighur alphabet [http://www.linguamongolia.co.uk/]) is written top-to-bottom and left-to-right
• Icons can be very culture-specific and often need to be localized as well
  • icons are pictorial metaphors (rooted in language and/or culture)
  • language-specific metaphors may not work across languages (e.g., OK gesture)
  • culture-specific metaphors may not work across cultures (e.g., owls)

(29) Definition

Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language.

• I18N starts at the design phase and influences the complete process
• The upfront costs of I18N are considerable (increased complexity)
• The costs of retroactive I18N are much higher than that

(30) I18N Tasks

1. UI elements (windows, menus) must be modified to accept translated text
2. Static text must be made configurable
3. Icons and graphics must be changed to be more culturally appropriate
4. Sound files that contain spoken language must be re-recorded
5. Online help must be translated
6. Dynamic text (dates, times) must be formatted using the locale
7. Text handling code must calculate word breaks using the locale
8. Tabular data must be sortable using the locale

(32) Definition

Localization refers to the adaptation of a product, application or document content to meet the language, cultural and other requirements of a specific target market (a locale).

• Without proper I18N, L10N is a very expensive and risky process
• L10N based on internationalized products should be straight-forward
• during L10N for specific locales, new areas for I18N might be identified
  • some non-internationalized part of the product is identified as inappropriate
• L10N should be regarded as an input for improved I18N

(33) L10N Tasks

1. Create translations for all interface elements
2. Translate all static texts
3. If necessary, create localized icons and graphics
4. Any spoken text must be recorded in the target language
5. Make sure that the localized product uses the localized online help
6. Formatting of data types must be treated locale-specific
7. If necessary, dictionaries and other language tools must be integrated
8. Sorting functions in the code must respect the locale

(35) Language Codes

• Language identification is required in many contexts
  • successful identification needs a standardized set of language tags
• RFC 4646 [http://dret.net/rfc-index/reference/RFC4646] defines Tags for Identifying Languages
  • two letter codes such as en are interpreted according to ISO 639-1 [http://dret.net/biblio/reference/iso639-1]
  • three letter codes such as eng are interpreted according to ISO 639-2 [http://dret.net/biblio/reference/iso639-2]
  • x- indicates a value which is not standardized (requires mutual agreement)
  • subtags such as en-US specify additional properties (regions, dialects, scripts, …)
• RFC 4647 [http://dret.net/rfc-index/reference/RFC4647] defines the Matching of Language Tags
  • matchmaking between requested and available languages
  • how to proceed if en-US or de are requested but available variants are en and de

(36) ISO 639-2 Code List

dum|||Dutch, Middle (ca.1050-1350)|néerlandais moyen (ca. 1050-1350)
dut|nld|nl|Dutch; Flemish|néerlandais; flamand
dyu|||Dyula|dioula
dzo||dz|Dzongkha|dzongkha
efi|||Efik|efik
egy|||Egyptian (Ancient)|égyptien
eka|||Ekajuk|ekajuk
elx|||Elamite|élamite
eng||en|English|anglais
enm|||English, Middle (1100-1500)|anglais moyen (1100-1500)
epo||eo|Esperanto|espéranto
est||et|Estonian|estonien
ewe||ee|Ewe|éwé
ewo|||Ewondo|éwondo
fan|||Fang|fang
fao||fo|Faroese|féroïen
fat|||Fanti|fanti
fij||fj|Fijian|fidjien
fil|||Filipino; Pilipino|filipino; pilipino
fin||fi|Finnish|finnois
fiu|||Finno-Ugrian (Other)|finno-ougriennes, autres langues
fon|||Fon|fon

(37) IANA Language Subtag Registry

%%
Type: region
Subtag: UA
Description: Ukraine
Added: 2005-10-16
%%
Type: region
Subtag: UG
Description: Uganda
Added: 2005-10-16
%%
Type: region
Subtag: UM
Description: United States Minor Outlying Islands
Added: 2005-10-16
%%
Type: region
Subtag: US
Description: United States
Added: 2005-10-16
%%

(39) Naming Language Variants

• URIs are intended to be names for resources (not for representations)
  • when does a representation become an individual resource?
  • the distinction is a gradual one (and it also applies to versioning over time)
  • if I send you a bookmark of a german page, should you get an english variant?
• Advantages when language variants are different resources
  • language variants can be identified (e.g., bookmarked) reliably
• Advantages when language variants are the same resource
  • when accessing the resource, a more appropriate variant can be negotiated dynamically
• Ideally, language variants should use a mix of both approaches
  • have their own URI so that they can be bookmarked and handled individually
  • remain identifiable as a variant of the more generic language-independent resource

(40) Variant Naming Variations

• URIs can be used in a number of different ways
  • any Web-based publishing should have a strategy for using URIs
  • being consistent is almost as important as doing it right
• For variations of URIs, various parts of the Web architecture can be used
  • DNS names for naming servers
  • various parts of the URI name
  • protocol mechanisms of HTTP or Cookies [State Management; Cookies (1)]
  • Designing URLs for Multilingual Web Sites [http://h3h.net/2007/01/designing-urls-for-multilingual-web-sites/] lists a number of possible variations

(41) DNS Domains

http://en.example.com/some/page

• Defines DNS subdomains for all supported languages
• Advantages
  • offers easy load-balancing to different servers (but code must kept in sync)
  • bookmarks identify the language variant
• Disadvantages
  • no easy way to get from one variant to another in terms of URI navigation
  • language management requires DNS updates
  • DNS names have not been designed for this kind of usage

(42) Constructed Paths

http://example.com/en/some/page

• Encodes the language as the first path segment of the URI path
• Advantages
  • bookmarks identify the language variant
• Disadvantages
  • logically, the URI path does not represent the hierarchy of resources
  • no easy way to get from one variant to another in terms of URI navigation
  • hard to maintain if this is used as the actual layout of documents in directories

(43) Query Strings

http://example.com/some/page?lang=en

• Use a query string to get the language in the required language variant
• Advantages
  • bookmarks identify the language variant
  • http://example.com/some/page is usable for the abstract resource
• Disadvantages
  • does not allow proper caching of pages
  • search engines ignore query strings and thus cannot link to the variants
  • against the idea of query strings which are meant for dynamic content
  • hard to combine with other query string information (if required)

(44) DNS TLDs

http://example.us/some/page

• Use DNS TLDs to identify the supported languages
• Advantages
  • bookmarks identify the language variant
• Disadvantages
  • requires registration of many TLDs (effort, costs, domain squatting)
  • countries do not identify languages (requires additional mechanism)
  • because of the country/language complexity, cross-language links are hard to maintain
  • DNS names have not been designed for this kind of usage

(45) Cookies

http://example.com/some/page

• Store the cookie with the language preference and use the cookie setting
• Advantages
  • language is not part of the URI and thus the URI identifies the resource
  • bookmarks (and any intra-page links) will automatically yield the preferred language variant
• Disadvantages
  • does not work if Cookies [State Management; Cookies (1)] are disabled or intercepted
  • it is not possible to create a bookmark for the language variant

(46) Content Negotiation

http://example.com/some/page

• Use browser settings and HTTP Content Negotiation [Web Foundations (URI & HTTP); HTTP Content Negotiation (1)] to get the variant
• Advantages
  • language is not part of the URI and thus the URI identifies the resource
  • bookmarks (and any intra-page links) will automatically yield the preferred language variant
• Disadvantages
  • does not work if the user agent does not support HTTP Content Negotiation [Web Foundations (URI & HTTP); HTTP Content Negotiation (1)]
  • does not work if the user has not configured the browser correctly
  • switching between languages requires configuration of the browser
  • it is not possible to create a bookmark for the language variant

(47) Path Segment Name

http://example.com/some/page.en

• Use . which uses the resource's extension
• Advantages
  • bookmarks identify the language variant
  • maps easily to extensions in file systems
• Disadvantages
  • does not mix well if additional properties (such as the format) are required
  • no easy way to get from one variant to another in terms of URI navigation
  • not officially recognized as URI structure (just simplifies parsing and recognition)

(48) URI Sub-Delimiter Comma

http://example.com/some/page,en

• Use , for specifying a parameter to a URI path segment
• Advantages
  • bookmarks identify the language variant
  • makes the language identifiable as a special part of the URI
  • http://example.com/some/page is usable for the abstract resource
• Disadvantages
  • does not mix well with parameters for other resource dimensions
  • not officially recognized as URI structure (just simplifies parsing and recognition)

(49) URI Sub-Delimiter Semicolon

http://example.com/some/page;lang=en

• Use ; for specifying a parameter to a URI path segment
• Advantages
  • bookmarks identify the language variant
  • makes the language identifiable as a special part of the URI
  • can be combined with parameters for other resource dimensions
  • http://example.com/some/page is usable for the abstract resource
• Disadvantages
  • not officially recognized as URI structure (just simplifies parsing and recognition)

(50) Now What?

• There is no perfect solution for naming language variants
  • the Web architecture does not provide support for variant preference in a URI
  • there is a strong separation between static URIs and dynamic content negotiation
• The name-based solutions (using ., ;, and ,) are very similar
  • any difference between them is how they are used in practice
  • from the URI point of view, path segments containing ., ;, and , are not treated in any special way
  • the only officially recognized special path segments are . and ..
• Theoretically, HTTP could define scheme-specific semantics
  • for http URIs, , and/or ; could interact with content negotiation
  • while this is an interesting idea with many useful applications, it is very unlikely to happen
• Use , [URI Sub-Delimiter Comma (1)] or ; [URI Sub-Delimiter Semicolon (1)] for language variants, but do not expect magic to happen

(52) Babelification

• I18N is a non-trivial task and requires some planning
• L10N involves much more than just translating text
• Producing fully localized products is a complex issue
• The locale is based on language and culture and customs
• Successful L10N requires the input of local people