/[ddp]/manuals/trunk/intro-i18n/intro-i18n.sgml

Diff of /manuals/trunk/intro-i18n/intro-i18n.sgml

Parent Directory | Revision Log | View Patch Patch

-revision 1037 by kubota,
Mon Nov 13 07:27:21 2000 UTC
+revision 1038 by kubota,
Wed Nov 15 10:00:55 2000 UTC
 Line 60 
 This document describes the basic ideas
  programmers and package maintainers of Debian GNU/Linux and
  other UNIX-like platforms.
  The aim of this document is to offer an introduction to
  basic concepts, character codes, and points of which care
  should be taken when one writes an I18N-ed software or
- an I18N patch for an existing software.  This document
+ an I18N patch for an existing software.  There are many
- also tries to introduce the real state and existing
+ know-hows and case-studies on internationalization of
- problems for each language and country.
+ softwares.  This document also tries to introduce the
+ real state and existing problems for each language and country.
  </P>
  <P>
-Line 425 
 and so on.
+Line 426 
 and so on.
  <sect id="coding-general"><heading>General Discussions</heading>
- <sect1 id="codeset"><heading>Basic Terminology</heading>
+ <sect1 id="coding-general-term"><heading>Basic Terminology</heading>
  <P>
  At first I begin this chapter by defining a few very important word.
-Line 1750 
 these functions have became newly availa
+Line 1751 
 these functions have became newly availa
  here.)
  </p>
+ <p>
+ Note that very simple softwares such as <tt>echo</tt> doesn't
+ have to care about multibyte character. and wide characters.
+ Such software can input and output multibyte character as is.
+ </p>
  <sect id="locale_unicode">Unicode and LOCALE technology</heading>
-Line 1934 
 in the distribution of GNU libc for usag
+Line 1941 
 in the distribution of GNU libc for usag
  </p>
- <p>
- <strong>****** I HAVE REWRITTEN THIS DOCUMENT UNTIL HERE ******</strong>
- </p>
  <chapt id="output"><heading>Output to Display</heading>
  <P>
- Here 'Output to Display' does not mean I18N of messages using
+ Here 'Output to Display' does not mean translation of messages using
  <prgn>gettext</prgn>.
  I will concern on whether characters are correctly outputed so that
  we can read it.  For example, install <package>libcanna1g</package>
-Line 1955 
 people can not read such a row of strang
+Line 1958 
 people can not read such a row of strang
  prefer if you were a Japanese speaker, an English message which can be read
  with a dictionary or such a row of strange characters which is
  a result of <prgn>gettext</prgn>ization?
- (Yes, there <em>is</em> a way to display
+ <footnote>
- Japanese characters correctly -- <prgn>kon</prgn> (in <package>kon2</package>
+ (Yes, there <em>are</em> ways to display Japanese characters
- package)  for console and <prgn>kterm</prgn> for X, and
+ correctly -- <prgn>kon</prgn> (in <package>kon2</package> package)
- Japanese people are happy with <prgn>gettext</prgn>ized Japanese messages.)
+ for console and <prgn>kterm</prgn> for X, and Japanese people are
+ happy with <prgn>gettext</prgn>ized Japanese messages.)
+ </footnote>
  </P>
  <P>
- Problems on displaying non-English characters are discussed below.
+ Problems on displaying non-English (non-ASCII) characters
- Since the mother tongue of the author is Japanese, the content may
+ are discussed below.
- be biased to Japanese.
  </P>
-Line 1972 
 be biased to Japanese.
+Line 1976 
 be biased to Japanese.
  <sect id="output-console"><heading>Console Softwares</heading>
  <P>
- Softwares running on the console are not responsible for displaying.
+ In this section, problems on displaying characters on
- The console itself is responsible.  There are terminal emulators
+ <strong>console</strong> are discussed.
- which can display non-English languages such as <prgn>kterm</prgn>
+ <footnote>
- (EUC-JP, Shift JIS Japanese, ISO 2022 international),
+ This section does not include problems on developing console;
- <prgn>krxvt</prgn>, <prgn>grxvt</prgn>, and <prgn>crxvt</prgn>
+ This section includes problems on developing softwares which run
- (Japanese, Greek, and Chinese, included
+ on console.
- in <package>rxvt-ml</package> package), <prgn>cxterm</prgn>
+ </footnote>
- (Chinese, Korean, and Japanese, non-free), <prgn>hanterm</prgn>
+ Here, console includes a bare <strong>Linux console</strong> including
- (Korean)
+ framebuffer and conventional version, special consoles such as
- and so on and softwares with which non-English characters can be
+ <strong>kon2</strong>, <strong>jfbterm</strong>, <strong>chdrv</strong>,
- displayed on console such as <package>kon2</package> (Japanese)
+ and so on constructed by special softwares, and X terminal emulators
- and <package>jfbterm</package> (EUC-based character codes and
+ such as <strong>xterm</strong>, <strong>kterm</strong>,
- ISO2022-based character codes).
+ <strong>hanterm</strong>, <strong>xiterm</strong>, <strong>rxvt</strong>,
+ <strong>xvt</strong>, <strong>gnome-terminal</strong>,
+ <strong>wterm</strong>, <strong>aterm</strong>, <strong>eterm</strong>,
+ and so on.  Remote environments via telnet and secure shell such as
+ <strong>NCSA telnet</strong> for Macintosh and <strong>Tera Term</strong>
+ for Windows are also regarded as consoles.
  </P>
  <P>
- All what a software on console/terminal-emulators
+ The feature of console is that:
- has to do is that output a correct code to the console.
+ <list>
+   <item>All what a software has to do is to send a correct character
+         code to standard output.  Softwares on console don't need to
+         care about fonts and so on.
+   <item>Fonts with fixed sizes are used.  The unit of the width
+         of the font is called 'column'.  'Doublewidth' fonts, i.e.,
+         fonts whose width is 2 columns, are used for CJK ideograms,
+         Japanese Hiragana and Katakana, Korean Hangul, and related
+         symbols.  Combined characters used for Thai and so on can be
+         regarded as 'zero'-column characters.
+ </list>
  </P>
- <P>
+ <sect1 id="output-console-code"><heading>Character Code</heading>
- At first, it is important not to destroy string data.
- Sometimes it can be done only by 8bit-clean-ize.
- '8bit-clean' means that the software does not destroy the
- most significant bit (MSB) of data the software treats.
- </P>
  <P>
- Next, be careful for a software which sends control codes such
+ Softwares running on the console are not responsible for displaying.
- as location every time it output 1 byte.  Such codes destroy
+ The console itself is responsible.  There are consoles
- the continuity of multibyte character.
+ which can display character codes other than ASCII such as
+ <taglist>
+  <tag>kon2
+       <item>EUC-JP, Shift-JIS, and ISO-2022-JP
+  <tag>jfbterm
+       <item>EUC-JP, ISO-2022-jp, and ISO-2022 (including any 7bit and
+bit character sets whose fonts are available)
+  <tag>kterm
+       <item>EUC-JP, Shift-JIS, ISO-2022-JP, and ISO-2022 (including
+             ISO8859-{1,2,3,4,5,6,7,8,9}, JISX0201, JISX0208, JISX0212,
+             GB2312, and KSC5601)
+  <tag>krxvt
+       <item>EUC-JP
+  <tag>crxvt-gb
+       <item>CN-GB
+  <tag>crxvt-big5
+       <item>Big5
+  <tag>hanterm
+       <item>EUC-KR, Johab, and ISO-2022-KR
+  <tag>xiterm+thai
+       <item>TIS620
+  <tag>xterm
+       <item>UTF-8
+ </taglist>
+ However, there are no way for a software on console to know which
+ character code is available.  I think it is a responsibility for
+ a user to properly set LC_CTYPE locale (i.e. LC_ALL, LC_CTYPE, or LANG
+ environmental variable).  Provided LC_CTYPE locale is set properly,
+ a software can use it to know which character code to be supported
+ by the console.
  </P>
  <P>
- Be also careful for destruction of multicolumn characters.
+ Concerning the translated messages by <prgn>gettext</prgn>,
- For example, when a string exceeds the width of the console,
+ the software does not need anything.  It works well if the
- the string is divided at the end of the line.  Terminal emulators
+ user properly set LC_CTYPE and LC_MESSAGES locale.
- should have a faculty to avoid such a 'excess of line width' type
- destruction of character but so far no terminal emulators
- have such a faculty.  (Only one exception --- shell mode of Emacs.
- However, unfortunately shell mode of Emacs is a dumb terminal and
- many softwares cannot be run on it.)  Thus each software on
- console should be careful.
  </P>
  <P>
- There is another reason to destroy multicolumn characters.
+ If you are handling a string in non-ASCII encoding (using
- When a message is overwritten on another string, a part
+ multibyte character, UTF-8 directly, and so on), you will have
- of a character which is a part of a previous string can be
+ to care about points which you don't have to care about if you are
- left not overwritten.  This may be more troublesome than many
+ using ASCII.
- people would think because multicolumn character can be
+ <list>
- written at every columns, not only at the multiple of the
+   <item>8-bit cleanness.  I think everyone understand this.
- width of the character.
+   <item>Continuity of multibyte characters.  In multibyte character
+         codes such as EUC-JP and UTF-8, one character may consist
+         from more than two bytes.  These bytes should be outputed
+         continued.  Insertion of additional codes between the
+         continuing bytes can break the character.  I have seen a
+         software which outputs location control code everytime
+         it outputs one byte.  It breaks multibyte character.
+ </list>
  </P>
+ <sect1 id="output-console-column"><heading>Number of Column</heading>
  <P>
- These destruction of continuity of multibyte characters may
+ Internationalized console software cannot assume that a character
- be a cause of the destruction of the whole line following
+ always occupy one column.  You can get the number of column of a
- the character.  Whether this can occur depends on the internal
+ character of a string using <tt>wcwidth()</tt> and
- implementation of console program.  This can occur if the
+ <tt>wcswidth()</tt>.  Note that you have to use
- terminal emulator does not treat columns, bytes and characters
+ <tt>wchar_t</tt>-style programming since these functions have
- properly separately.  The shell mode of Emacs is the only example
+ a <tt>wchar_t</tt> parameter.
- doing that but there are no chance to overwrite character on
- the shell mode of Emacs, because it is a dumb terminal.
  </P>
  <P>
- There are no standards for number of columns a character occupies.
+ Additional cares have to be taken not to destroy multicolumn
- This can be a large problem for softwares with <tt>ncurses</tt>.
+ characters.  For example, imagine your software displayed a
- There is no 'right' way to solve this.  Each software has to
+ double-column character at (row, column) = (1, 1).  What will occur
- have an information for each character set.  Consult section
+ when your software then display a single-column character at (row, column)
- <ref id="languages">
+ = (1, 2) or at (1, 1) ?  The single-column character erases
- for each language.  Take care of the distinction between number
+ the half of the double-column character?  Nobody knows the answer.
- of columns, bytes, and characters.  For subset of EUC-JP
+ It depends on the implementation of the console.  All what I can
- (ASCII alphabets and JIS X 0208 kanji), number of bytes and columns
+ tell is that your software should avoid such cases.
- are equal (1-byte character occupy 1 column and 2-byte character
- occupy 2 columns).  Note that cursor-moving control characters
- such as 'BS' (0x08) moves cursor one COLUMN, not one CHARACTER.
  </P>
  <P>
- Another important point is that the string has to be converted
+ If your software inputs a string from keyboard,  you will have to
- into a character code which the console can understand.  So far there
+ take more cares.  All of numbers of characters, bytes, and columns
- are no consoles which understand Unicode.
+ differ.  For example, in UTF-8 character code, one character of
+ 'a' with acute accent occupies two bytes and one column.  One
+ character of CJK-ideograph occupies three bytes and two columns.
+ For example, if the user types 'Backspace', how many backspace
+ code (0x08) should the software outputs?  How many bytes should
+ the software erase from the internal buffer?
+ Don't be nervous; you can use <tt>wchar_t</tt> which assures
+ one character occupy one <tt>wchar_t</tt> everytime and you can
+ use <tt>wcwidth()</tt> to know the number of columns.
+ Note that control codes such as 'backspace' (0x08) and so on are
+ column-oriented everytime.  It backs 'one' column even if the
+ character at the position is a doublewidth character.
  </P>
  <sect id="output-x"><heading>X Clients</heading>
  <P>
-Line 2070 
 functions.
+Line 2121 
 functions.
  </P>
  <P>
- An X font is related to a specific <em>character set</em>.  The
+ The most important part for internationalization of displaying
- conventional font-related functions can use one font at the same time.
+ for X clients is the usage of internationalized
- However, text is expressed in a specific <em>character code</em> and
+ <strong>XFontSet</strong>-related functions introduced since
- some character codes need multiple character sets.  Chinese, Japanese,
+ X11R5 instead of conventional <strong>XFontStruct</strong>-related
- and Korean are languages which need multiple character sets.
+ functions.
- These languages cannot be displayed using the conventional
- font-related functions.
  </P>
  <P>
- 'fontset' is an idea that multiple fonts are selected and construct
+ The main feature of XFontSet is that it can handle multiple fonts
- a set of fonts.  Using fontset enables to display international texts.
+ at the same time.  This is related to the distinction between
+ coded character set (CCS) and character encoding scheme (CES)
+ which I wrote at the section of <ref id="coding-general-term">.
+ Some character codes in the world use multiple coded character
+ sets at the same time.  This is the reason we have to handle
+ multiple X fonts at the same time.
+ <footnote>
+ Though UTF-8 is a character code with single CCS, the current
+ version of XFree86 (4.0.1) needs multiple fonts to handle UTF-8.
+ </footnote>
  </P>
  <P>
- Here is a list of structure and functions of conventional 'Font'-related
+ Another significant feature of XFontSet is that it is
- and internationalized 'FontSet'-related.  Consult manpages for detail.
+ locale (LC_CTYPE)-sensible.  This means that you have to
- <example>
+ call <tt>setlocale()</tt> before you use XFontSet-related
- Font              | FontSet
+ functions.  And more, you have to specify the string you want
- ==================+====================
+ to draw as a mulbibyte character or a wide character.
- XFontStruct       | XFontSet
- ------------------+--------------------
- XLoadFont()       | XCreateFontSet()
- ------------------+--------------------
- XUnloadFont()     | XFreeFontSet()
- ------------------+--------------------
- XQueryFont()      | XFontsOfFontSet()
- ------------------+--------------------
- XDrawString() and | XmbDrawString() or
- XDrawString16()   | XwcDrawString()
- ------------------+--------------------
- XDrawText() and   | XmbDrawText() or
- XDrawText16()     | XwcDrawText()
- ------------------+--------------------
- </example>
  </P>
  <P>
- If a software uses the left-hand functions it have to be rewritten
+ In the conventional <tt>XFontStruct</tt> model, an X client
- using the corresponding right-hand functions in the table.  Note that
+ opens a font using <tt>XLoadQueryFont()</tt>, draw a string
- this table is not perfect but only for an example.  Since these
+ using <tt>XDrawString()</tt>, and close the font using
- right-hand functions use wide characters and multibyte characters
+ <tt>XFreeFont()</tt>.  On the other hand, in the internationalized
- in C, <tt>setlocale()</tt> has to be called in advance.
+ <tt>XFontSet</tt> model, an X client opens a font using
+ <tt>XCreateFontSet()</tt>, draw a string using <tt>XmbDrawString()</tt>,
+ and close the font using <tt>XFreeFontSet()</tt>.
+ The following are a concise list of substitution.
+ <list>
+   <item><tt>XFontStruct</tt> -&gt; <tt>XFontSet</tt>
+   <item><tt>XLoadQueryFont()</tt> -&gt; <tt>XCreateFontSet()</tt>
+   <item>both of <tt>XDrawString()</tt> and <tt>XDrawString16</tt>
+         -&gt; either of <tt>XmbDrawString()</tt> or <tt>XwcDrawString()</tt>
+   <item>both of <tt>XDrawImageString()</tt> and <tt>XDrawImageString16</tt>
+         -&gt; either of <tt>XmbDrawImageString()</tt> or
+         <tt>XwcDrawImageString()</tt>
+ </list>
+ Note that <tt>XFontStruct</tt> is usually used as a pointer, while
+ <tt>XFontSet</tt> itself is a pointer.
  </P>
  <P>
  Some people (ISO-8859-1-language speakers) may think that
- XFontSet is not 8-bit clean.  This is wrong.  XFontSet-related
+ <tt>XFontSet</tt>-related functions are not 8-bit clean.
- functions work according to LC_CTYPE locale.  The default LC_CTYPE
+ This is wrong.  <tt>XFontSet</tt>-related
- locale uses ASCII.  Thus, if a user doesn't set <tt>LANG</tt>,
+ functions work according to <tt>LC_CTYPE</tt> locale.  The default
+ LC_CTYPE locale uses ASCII.  Thus, if a user doesn't set <tt>LANG</tt>,
  <tt>LC_CTYPE</tt>, nor <tt>LC_ALL</tt> environmental variable,
- XFontSet will use ASCII, i.e., not 8-bit clean.  The user
+ <tt>XFontSet</tt>-related functions will use ASCII, i.e., not 8-bit
- has to set <tt>LANG</tt>, <tt>LC_CTYPE</tt>, or <tt>LC_ALL</tt>
+ clean.  The user has to set <tt>LANG</tt>, <tt>LC_CTYPE</tt>, or
- environmental variable properly (for example, <tt>LANG=en_US</tt>).
+ <tt>LC_ALL</tt> environmental variable properly (for example,
+ <tt>LANG=en_US</tt>).
  </P>
  <P>
  The upstream developers of X clients sometimes hate to enforce
- users to set such environmental variables.  In such a case,
+ users to set such environmental variables.
+ <footnote>
+  IMO, all users will have to set LANG properly when UTF-8 will
+  become popular.
+ </footnote>
+ In such a case,
  The X clients should have two ways to output text, i.e.,
- XFontStruct-related conventional way and XFontSet-related
+ <tt>XFontStruct</tt>-related conventional way and
- internationalized way.  If <tt>setlocale()</tt> returns
+ <tt>XFontSet</tt>-related internationalized way.
- <tt>NULL</tt>, <tt>"C"</tt>, or <tt>"POSIX"</tt>, use
+ If <tt>setlocale()</tt> returns <tt>NULL</tt>, <tt>"C"</tt>,
- XFontStruct way.  Otherwise use XFontSet way.  This algorithm
+ or <tt>"POSIX"</tt>, use
- is adopted by <package>Blackbox</package> (0.60.1 or later).
+ <tt>XFontStruct</tt> way.  Otherwise use <tt>XFontSet</tt> way.
+ The author implemented this algoritym to a few window managers
+ such as TWM (version 4.0.1d), Blackbox (0.60.1), IceWM (1.0.0),
+ sawmill (0.28), and so on.
  </P>
  <P>
- The same problem exists for softwares using toolkits such as
+ Window managers need more modifications related to inter-clients
- athena, GTK+, Qt, and so on.
+ communication.  This topic will be described later.
  </P>
+ <p>
+ <strong>****** I HAVE REWRITTEN THIS DOCUMENT UNTIL HERE ******</strong>
+ </p>

 Legend:



Removed from v.1037
 


changed lines


 
Added in v.1038
 Legend:



Removed from v.1037
 


changed lines


 
Added in v.1038
-Removed from v.1037
+Added in v.1038

	ViewVC Help
Powered by ViewVC 1.1.5