Abstract

Many publishing environments need to support multiple languages. In many cases, the requirement to support multiple languages surfaces in later stages of a product development or publishing solution, which can cause major design changes, driving up costs. Internationalization (I18N) is the approach to design systems which can adapt to different locales. Localization (L10N) is the activity to identify, define, and encode locales, based on internationalized software.

I18N & L10N

  <arc xlink:type="arc" xlink:from="permalink" xlink:to="blog" xlink:title="Blog Home"/>
  <arc xlink:type="arc" xlink:from="blog" xlink:to="author" xlink:title="Blog Author"/>
  <arc xlink:type="arc" xlink:from="author" xlink:to="blog" xlink:title="Authored Blogs"/>

  <arc xlink:type="arc" xlink:from="permalink" xlink:to="blog">
   <title xlink:type="title" xml:lang="en">Blog Home</title>
   <title xlink:type="title" xml:lang="de">Blog Webseite</title>
  </arc>
  <arc xlink:type="arc" xlink:from="blog" xlink:to="author">
   <title xlink:type="title" xml:lang="en">Blog Author</title>
   <title xlink:type="title" xml:lang="de">Autor des Blogs</title>
  </arc>
  <arc xlink:type="arc" xlink:from="author" xlink:to="blog">
   <title xlink:type="title" xml:lang="en">Authored Blogs</title>
   <title xlink:type="title" xml:lang="de">Geschriebene Blogs</title>
  </arc>

What is Language?

• Language is more than the encoding of individual words
  • languages and culture are deeply intertwined and inseparable
  • ideally, systems/solutions should adapt to culture, not only to language
  • on a superficial level, adapting to language is useful a first step
• Languages have properties which are beyond character sequences
  • for right-to-left languages, screen layout should be done right-to-left
  • if languages are mixed, one language has to be the preferred language
• Language identification and selection are basic I18N tasks
  • the one language fits all assumption is becoming increasingly inappropriate
  • the just switch the labels strategy also may be too little for true L10N

Beyond Language

• Directionality is the concept how written language is organized
  • early languages had no inherent directionality (even zigzag writing was possible)
  • the majority of today's languages are left-to-right languages
  • Arabic and Hebrew are two popular right-to-left languages
  • Chinese can be written top-to-bottom and right-to-left
  • Mongolian (using the Uighur alphabet) is written top-to-bottom and left-to-right
• Icons can be very culture-specific and often need to be localized as well
  • icons are pictorial metaphors (rooted in language and/or culture)
  • language-specific metaphors may not work across languages (e.g., OK gesture)
  • culture-specific metaphors may not work across cultures (e.g., owls)

Directionality and Screen Layout

Definition

Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language.

• I18N starts at the design phase and influences the complete process
• The upfront costs of I18N are considerable (increased complexity)
• The costs of retroactive I18N are much higher than that

I18N Tasks

1. UI elements (windows, menus) must be modified to accept translated text
2. Static text must be made configurable
3. Icons and graphics must be changed to be more culturally appropriate
4. Sound files that contain spoken language must be re-recorded
5. Online help must be translated
6. Dynamic text (dates, times) must be formatted using the locale
7. Text handling code must calculate word breaks using the locale
8. Tabular data must be sortable using the locale

Definition

Localization refers to the adaptation of a product, application or document content to meet the language, cultural and other requirements of a specific target market (a locale).

• Without proper I18N, L10N is a very expensive and risky process
• L10N based on internationalized products should be straight-forward
• during L10N for specific locales, new areas for I18N might be identified
  • some non-internationalized part of the product is identified as inappropriate
• L10N should be regarded as an input for improved I18N

L10N Tasks

1. Create translations for all interface elements
2. Translate all static texts
3. If necessary, create localized icons and graphics
4. Any spoken text must be recorded in the target language
5. Make sure that the localized product uses the localized online help
6. Formatting of data types must be treated locale-specific
7. If necessary, dictionaries and other language tools must be integrated
8. Sorting functions in the code must respect the locale

Language Codes

• Language identification is required in many contexts
  • successful identification needs a standardized set of language tags
• RFC 4646 defines Tags for Identifying Languages
  • two letter codes such as en are interpreted according to ISO 639-1
  • three letter codes such as eng are interpreted according to ISO 639-2
  • x- indicates a value which is not standardized (requires mutual agreement)
  • subtags such as en-US specify additional properties (regions, dialects, scripts, …)
• RFC 4647 defines the Matching of Language Tags
  • matchmaking between requested and available languages
  • how to proceed if en-US or de are requested but available variants are en and de

ISO 639-2 Code List

dum|||Dutch, Middle (ca.1050-1350)|néerlandais moyen (ca. 1050-1350)
dut|nld|nl|Dutch; Flemish|néerlandais; flamand
dyu|||Dyula|dioula
dzo||dz|Dzongkha|dzongkha
efi|||Efik|efik
egy|||Egyptian (Ancient)|égyptien
eka|||Ekajuk|ekajuk
elx|||Elamite|élamite
eng||en|English|anglais
enm|||English, Middle (1100-1500)|anglais moyen (1100-1500)
epo||eo|Esperanto|espéranto
est||et|Estonian|estonien
ewe||ee|Ewe|éwé
ewo|||Ewondo|éwondo
fan|||Fang|fang
fao||fo|Faroese|féroïen
fat|||Fanti|fanti
fij||fj|Fijian|fidjien
fil|||Filipino; Pilipino|filipino; pilipino
fin||fi|Finnish|finnois
fiu|||Finno-Ugrian (Other)|finno-ougriennes, autres langues
fon|||Fon|fon

IANA Language Subtag Registry

%%
Type: region
Subtag: UA
Description: Ukraine
Added: 2005-10-16
%%
Type: region
Subtag: UG
Description: Uganda
Added: 2005-10-16
%%
Type: region
Subtag: UM
Description: United States Minor Outlying Islands
Added: 2005-10-16
%%
Type: region
Subtag: US
Description: United States
Added: 2005-10-16
%%

Language Identification in XML

• XML itself has (almost) no semantics
  • applications create or use schemas and the semantics are associated with the schema
  • XML is the serialization of the semantics which are embodied in the schema
• XML has a very small set of built-in semantics
  • xml:lang is a reserved attribute for encoding the language of content
  • xml:id is a reserved attribute for identifying elements (without requiring a schema)

  <arc xlink:type="arc" xlink:from="permalink" xlink:to="blog">
   <title xlink:type="title" xml:lang="en">Blog Home</title>
   <title xlink:type="title" xml:lang="de">Blog Webseite</title>
  </arc>
  <arc xlink:type="arc" xlink:from="blog" xlink:to="author">
   <title xlink:type="title" xml:lang="en">Blog Author</title>
   <title xlink:type="title" xml:lang="de">Autor des Blogs</title>
  </arc>

Naming Language Variants

• URIs are intended to be names for resources (not for representations)
  • when does a representation become an individual resource?
  • the distinction is a gradual one (and it also applies to versioning over time)
  • if I send you a bookmark of a german page, should you get an english variant?
• Advantages when language variants are different resources
  • language variants can be identified (e.g., bookmarked) reliably
• Advantages when language variants are the same resource
  • when accessing the resource, a more appropriate variant can be negotiated dynamically
• Ideally, language variants should use a mix of both approaches
  • have their own URI so that they can be bookmarked and handled individually
  • remain identifiable as a variant of the more generic language-independent resource

Variant Naming Variations

• URIs can be used in a number of different ways
  • any Web-based publishing should have a strategy for using URIs
  • being consistent is almost as important as doing it right
• For variations of URIs, various parts of the Web architecture can be used
  • DNS names for naming servers
  • various parts of the URI name
  • protocol mechanisms of HTTP or Cookies
  • Designing URLs for Multilingual Web Sites lists a number of possible variations

DNS Domains

http://en.example.com/foo/bar

• Defines DNS subdomains for all supported languages
• Advantages
  • offers easy load-balancing to different servers (but code must kept in sync)
  • bookmarks identify the language variant
• Disadvantages
  • no easy way to get from one variant to another in terms of URI navigation
  • language management requires DNS updates
  • DNS names have not been designed for this kind of usage

Constructed Paths

http://example.com/en/foo/bar

• Encodes the language as the first path segment of the URI path
• Advantages
  • bookmarks identify the language variant
• Disadvantages
  • logically, the URI path does not represent the hierarchy of resources
  • no easy way to get from one variant to another in terms of URI navigation
  • hard to maintain if this is used as the actual layout of documents in directories

Query Strings

http://example.com/foo/bar?lang=en

• Use a query string to get the language in the required language variant
• Advantages
  • bookmarks identify the language variant
  • http://example.com/foo/bar is usable for the abstract resource
• Disadvantages
  • does not allow proper caching of pages
  • search engines ignore query strings and thus cannot link to the variants
  • against the idea of query strings which are meant for dynamic content
  • hard to combine with other query string information (if required)

DNS TLDs

http://example.us/foo/bar

• Use DNS TLDs to identify the supported languages
• Advantages
  • bookmarks identify the language variant
• Disadvantages
  • requires registration of many TLDs (effort, costs, domain squatting)
  • countries do not identify languages (requires additional mechanism)
  • because of the country/language complexity, cross-language links are hard to maintain
  • DNS names have not been designed for this kind of usage

Cookies

http://example.com/foo/bar

• Store the cookie with the language preference and use the cookie setting
• Advantages
  • language is not part of the URI and thus the URI identifies the resource
  • bookmarks (and any intra-page links) will automatically yield the preferred language variant
• Disadvantages
  • does not work if Cookies are disabled or intercepted
  • it is not possible to create a bookmark for the language variant

Content Negotiation

http://example.com/foo/bar

• Use browser settings and Content Negotiation to get the variant
• Advantages
  • language is not part of the URI and thus the URI identifies the resource
  • bookmarks (and any intra-page links) will automatically yield the preferred language variant
• Disadvantages
  • does not work if the user agent does not support Content Negotiation
  • does not work if the user has not configured the browser correctly
  • switching between languages requires configuration of the browser
  • it is not possible to create a bookmark for the language variant

Path Segment Name

http://example.com/foo/bar.en

• Use . which uses the resource's extension
• Advantages
  • bookmarks identify the language variant
  • maps easily to extensions in file systems
• Disadvantages
  • does not mix well if additional properties (such as the format) are required
  • no easy way to get from one variant to another in terms of URI navigation
  • not officially recognized as URI structure (just simplifies parsing and recognition)

URI Sub-Delimiter Comma

http://example.com/foo/bar,en

• Use , for specifying a parameter to a URI path segment
• Advantages
  • bookmarks identify the language variant
  • makes the language identifiable as a special part of the URI
  • http://example.com/foo/bar is usable for the abstract resource
• Disadvantages
  • does not mix well with parameters for other resource dimensions
  • not officially recognized as URI structure (just simplifies parsing and recognition)

URI Sub-Delimiter Semicolon

http://example.com/foo/bar;lang=en

• Use ; for specifying a parameter to a URI path segment
• Advantages
  • bookmarks identify the language variant
  • makes the language identifiable as a special part of the URI
  • can be combined with parameters for other resource dimensions
  • http://example.com/foo/bar is usable for the abstract resource
• Disadvantages
  • not officially recognized as URI structure (just simplifies parsing and recognition)

Now What?

• There is no perfect solution for naming language variants
  • the Web architecture does not provide support for variant preference in a URI
  • there is a strong separation between static URIs and dynamic content negotiation
• The name-based solutions (using ., ;, and ,) are very similar
  • any difference between them is how they are used in practice
  • from the URI point of view, path segments containing ., ;, and , are not treated in any special way
  • the only officially recognized special path segments are . and ..
• Theoretically, HTTP could define scheme-specific semantics
  • for http URIs, , and/or ; could interact with content negotiation
  • while this is an interesting idea with many useful applications, it is very unlikely to happen
• Use , or ; for language variants, but do not expect magic to happen

Babelification

• I18N is a non-trivial task and requires some planning
• L10N involves much more than just translating text
• Producing fully localized products is a complex issue
• The locale is based on language and culture and customs
• Successful L10N requires the input of local people