trunk/intro-i18n/japanese-japan.sgml


<sect id="japanese"><heading>Japanese language / used in Japan</heading>


<P>
This section is the text written by
Tomohiro KUBOTA <email>kubota@debian.org</email> (no more reachable).
</P>

<P>
Japanese is the only official language used in Japan.
People in Okinawa islands and Ainu ethnic group in Hokkaido region
have each language, though they are used among few number
of people and they don't have own letters.
</P>

<P>
Japan is the only region where Japanese language is widely used.
</P>


<sect1 id="japanese-character"><heading>Characters used in Japanese</heading>

<P>
There are three kinds of characters used in Japan,
Hiragana, Katakana, and Kanji.
Arabic numerical characters (same as European languages) are 
widely used in Japanese, though we have Kanji numerical characters.
Though Latin alphabets are not a part of Japanese characters,
they are widely used for proper nouns for companies and so on.
</P>

<P>
Hiragana and Katakana are phonogram derived from Kanji.
Hiragana and Katakana characters have one-to-one correspondence
each other like upper and lower case of Latin alphabets.
However, <tt>toupper()</tt> and <tt>tolower()</tt> should not 
convert Hiragana and Katakana each other.
Hiragana contains about 100 characters and of course Katakana does.
(FYI: about 50 regular characters, 20 characters with voiced 
consonant symbol, 5 characters with semi-voiced consonant symbol,
and 9 small characters.)
</P>

<P>
Kanji is ideogram imported from China roughly about 1 - 2 thousands
years ago.
Nobody knows the whole number of Kanji and almost all of adult Japanese
people know several thousands of Kanji characters.
Though the origin of Kanji is Chinese character, shapes are
changed from original ancient Chinese Kanji.
Almost all Kanji have several ways to read, according to the
word the Kanji is contained.
</P>


<sect1 id="japanese-sets"><heading>Character Sets</heading>

<P>
JIS (Japan Industrial Standards) is an organization responsible
for coded character sets (CCS) and encodings used in Japan.
The major coded character sets in Japan are:
<list>
 <item>JIS X 0201-1976 Roman characters (Almost same to ASCII but 0x5c 
       is Yen mark instead of backslash and 0x7e is upper bar instead of tilde)
 <item>JIS X 0201-1976 Kana (about 60 KATAKANA characters),
 <item>JIS X 0208-1997 1st and 2nd levels (about 7000 characters 
       including symbols, numeric characters, Latin, Cyrillic and 
       Greek alphabets, Japanese HIRAGANA, KATAKANA, and KANJI),
 <item>JIS X 0212 (about 6000 characters including KANJI, which are not 
       included in JIS X 0208), and
 <item>JIS X 0213:2000 (aka JIS 3rd and 4th levels).
</list>
</P>

<P>
<strong>JIS X 0201 Roman</strong> is the Japanese version of ISO 646.
Though JIS X 0201 is included in SHIFT-JIS encoding (explained later) and
widely used for Windows/Macintosh, usage of this is not encouraged in UNIX.
</P>

<P>
<strong>JIS X 0201 Kana</strong> defines about 60 KATAKANA characters.
This is widely used by old 8bit computers.
In deed, SHIFT-JIS encoding was designed to be upward-compatible
with 8-bit encoding of JISX 0201 Roman and JISX 0201 Kana.
Note this CCS is not included in ISO 2022-JP encoding which is
used for e-mail and so on.
</P>

<P>
<strong>JIS X 0212</strong> is not widely used, probably because it cannot be 
included in SHIFT-JIS, the standard encoding for Japanese version
of Windows and Macintosh.  And more, this CCS may be obsolete 
when JIS X 0213 will be popular, since JIS X 0213 has many
characters which are included in JIS X 0212.
However, the advantage of JIS X 0212 over JIS X 0213 is that
all characters in JIS X 0212 are included in the current
Unicode (version 3.0.1) while not all characters in JIS X 0213
are.
</P>

<P>
<strong>JIS X 0208</strong> (aka JIS C 6226) is the main standard 
for Japanese characters. 
Strictly speaking, it was originally defined in 1978 and
revised on 1983, 1990, and 1997.
Though 1997 version has 77 more characters than original 1976 version
and shape of more than 200 characters are changed,
almost softwares don't have to care about the difference between them.
However, be careful of that ISO-2022-JP encoding (explained below)
contains both JIS X 0208-1978 and JIS X 0208-1983.
1978 version is called 'old JIS' and later is called 'new JIS'.
Characters in JIS X 0208 are divided into two levels, 1st and 2nd.
Old 8bit computers rarely implemented the 2nd level.
</P>

<P>
Usage of numeric characters and Latin alphabets in JIS X 0208 is 
not encouraged because these characters are also included in ASCII
and JIS X 0201 Roman, either of which is included in all encodings.
When converting into Unicode, these characters are mapped into
'fullwidth forms'.
</P>

<P>
All of these coded character sets (except for JIS X 0213) are 
included in Unicode 3.0.1.  A part of JIS X 0213 characters are not
included in Unicode 3.0.1.
</P>

<P>
There are a few different tables for conversion between non-letter 
characters in JIS X 0208 and Unicode.  This is a problem because
this may deny 'round-trip compatiblilty'.
<url id="http://www.opengroup.or.jp/jvc/cde/ucs-conv-e.html"
name="Problems and Solutions for Unicode and User/Vendor Defined Characters">
discusses this problem in detail.
</P>


<sect1 id="japanese-encodings"><heading>Encodings</heading>

<P>
There are three popular encodings widely used in Japan.
<list>
 <item>ISO-2022-JP (aka JIS code or JUNET code)
    <list>
     <item>stateful
     <item>subset of 7bit version of ISO-2022, where ASCII, 
           JIS X 0201-1976 Roman, JIS X 0208-1978,
           and JIS X 0208-1983 are supported.
     <item>7bit, which means the most significant bit (MSB) of each
           byte is always zero.
     <item>used for e-mail and net-news and preferred for HTML.
     <item>Determined in RFC 1468.
    </list>
 <item>EUC-JP (Japanese version of Extended UNIX Code)
    <list>
     <item>stateless
     <item>an implementation of EUC where G0, G1, G2, and G3 are
           ASCII, JIS X 0208, JIS X 0201 Kana, and JIS X 0212
           respectively.  There are many implementation which cannot
           use JIS X 0201 Kana and JIS X 0212.
     <item>8bit
     <item>preferred encoding for UNIX. For example, almost all Japanese 
           message catalogs for gettext is written in EUC-JP.
     <item>Japanese code is mapped in <tt>0xa0</tt> - <tt>0xff</tt>.  
           This is important
           for programmer because one doesn't need to care there are
           fake '\' or '/' (which can be treated in a special way in
           various context) in the Japanese code.
    </list>
 <item>SHIFT-JIS (aka Microsoft Kanji Code)
    <list>
     <item>stateless
     <item>NOT subset of ISO-2022
     <item>8bit
     <item>JIS X 0201 Roman, JIS X 0201 Kana, and JIS X 0208
           can be expressed, but JIS X 0212 cannot.
     <item>The standard encoding for Windows/Macintosh.  This makes
           SHIFT-JIS the most popular encoding in Japan.  Though MS 
           is thinking about transition to UNICODE, it is suspicious
           that it can be done successfully.
    </list>
</list>
</P>

<P>
<strong>ISO-2022-JP</strong> is a subset of 7bit version of ISO 2022,
where only G0 is used and G0 is assumed to be invoked into GL.
Character sets included in ISO-2022-JP are:
<list>
 <item>ASCII (ESC 0x28 0x42),
 <item>JIS X 0201-1976 Roman (ESC 0x28 0x4a),
 <item>JIS X 0208-1978 (old JIS) (ESC 0x24 0x40), and
 <item>JIS X 0208-1983 (new JIS) (ESC 0x24 0x42).
</list>
Note that JIS X 0208-1978 and JIS X 0208-1983 are almost identical
and ASCII and JIS X 0201-1976 Roman are also almost identical.
A line (stream of bytes between 'newline' control code) must
start by ASCII status and to end by ASCII status.
See <ref id="iso-2022"> for detail.
</P>

<P>
<strong>ISO-2022-JP-2</strong> (RFC 1554) is a subset of 7bit version
of ISO 2022 and superset of ISO-2022-JP.  Difference between ISO-2022-JP
and ISO-2022-JP-2 is that ISO-2022-JP-2 has more coded character sets
than ISO-2022-JP. Character sets included in ISO-2022-JP-2 are:
<list>
 <item>ASCII (ESC 0x28 0x42)
 <item>JIS X 0201-1976 Roman (ESC 0x28 0x4a),
 <item>JIS X 0208-1978 (old JIS) (ESC 0x24 0x40),
 <item>JIS X 0208-1983 (new JIS) (ESC 0x24 0x42),
 <item>GB2312-80 (simplified Chinese) (ESC 0x24 0x41),
 <item>KS C 5601 (Korean) (ESC 0x24 0x28 0x43),
 <item>JIS X 0212-1990 (ESC 0x24 0x28 0x44),
 <item>ISO 8859-1 (Latin-1) (ESC 0x2e 0x41), and
 <item>ISO 8859-7 (Greek) (ESC 0x2e 0x46).
</list>
Though JIS X 0212-1990 may sometimes be used, ISO-2022-JP-2
is rarely used.
</P>

<P>
<strong>ISO-2022-INT-1</strong> is a superset of ISO-2022-JP-2 which has
CNS 11643-1986-1 and CNS 11643-1986-2 (traditional Chinese).
</P>

<P>
<strong>EUC-JP</strong> is a version of EUC, where 
G0 is ASCII, G1 is JIS X 0208, G2 is JIS X 0201 Kana, and
G3 is JIS X 0212.  G2 and G3 are sometimes not implemented.
This is the most popular encoding for Linux/Unix.
See <ref id="euc"> for detail.
</P>

<P>
<strong>SHIFT-JIS</strong> is designed to be a superset of
encodings for old 8bit computers which includes JIS X 0201
Roman and JIS X 0201 Kana.  <tt>0x20</tt> - <tt>0x7f</tt>
is JIS X 0201 Roman and <tt>0xa0</tt> - <tt>0xdf</tt> is
JIS X 0201 Kana.  <tt>0x80</tt> - <tt>0x9f</tt> and <tt>0xe0</tt>
- <tt>0xff</tt> is the first byte of doublebyte characters.
The second byte is <tt>0x40</tt> - <tt>0x7e</tt> and 
<tt>0x80</tt> - <tt>0xfc</tt>.  This code space is used for JIS X 0208.
</P>

<P>
UNICODE is not popular in Japan at all, probably because
conversion from these codes into Unicode is a bit difficult.
However MS Windows uses Unicode in a limited field, for example,
internal code for file names.
I guess more and more softwares will come to support
Unicode in the future.
</P>

<P>
You can convert files written in these encodings one another using
<package>nkf</package> or <package>kcc</package> package.  
Using options <tt>-j</tt>, <tt>-s</tt>, and <tt>-e</tt>, 
<prgn>nkf</prgn> convert a file into ISO-2022-JP (aka JIS), 
SHIFT-JIS (aka MS-KANJI), and EUC-JP, respectively.  Note that 
difference between JIS X 0201 Roman and ASCII is ignored.
Though <prgn>nkf</prgn> can guess the encoding of
the input file, you can specify the encoding by command option.
This is because there are no algorithm to completely distinguish
EUC-JP and SHIFT-JIS, though <prgn>nkf</prgn> usually guesses
correctly.  <prgn>tcs</prgn> can also convert these encodings,
though without guessing input encoding.
Conversion between these encodings can be done with a simple
algorithm since all of them are based on the same character sets.
You need a table for code conversion between these encodings and Unicode.
</P>


<sect1 id="japanese-how"><heading>How These Encodings Are Used --- Information for Programmers</heading>

<P>
Since EUC-JP is widely used for UNIX,
EUC-JP should be supported. Exceptions are shown below.
Of course direct implementation of knowledge on EUC-JP is not
encouraged.  If you can implement without the knowledge by use
of <tt>wchar_t</tt> and so on, you should do so.
<list>
 <item>the body of mail and news messages must be written in ISO-2022-JP.
 <item>De-facto standard of ICQ is SHIFT-JIS.
 <item>WWW browser must recognize all encodings.
 <item>Softwares which communicate with Windows/Macintosh should use
       SHIFT-JIS.
 <item>SHIFT-JIS is widely used for BBS.  (BBS is a service like Compuserve).
 <item>File names for Joliet-format CD-ROM used for Windows is written 
       in Unicode.
</list>
</P>


<sect1 id="japanese-columns"><heading>Columns</heading>

<P>
In consoles which are able to display Japanese characters
(kon, jfbterm, kterm, krxvt, and so on), characters in JIS X 0201 
(Roman and Kana) occupy 1 column 
and characters in JIS X 0208, JIS X 0212, and JIS X 0213 occupy 2 columns.
</P>


<sect1 id="japanese-direction"><heading>Writing Direction and Combined Characters</heading>

<P>
Japanese language can be written in vertical direction.  A line goes
downward and the row of lies goes from right to left.  This direction
is the traditional style.  For example, most Japanese books, magazines
and newspapers except for in the field of natural science 
(or ones containing many Latin words or equations) are written
in vertical direction.  Thus a word processor is strongly recommended
to support this direction.  DTP systems which don't support this
direction are almost useless.
</P>

<P>
Japanese language can also written in the same direction to Latin
languages.  Japanese books and magazines on science and technology
are written in this direction.  It is enough for almost usual softwares
to support this direction only.
</P>

<P>
A few Japanese characters have to have different fonts for vertical
direction.  They are reasonable characters --- parentheses and 
'long syllable' symbol whose shape is like dash in English or
mathematical 'minus' sign.  Symbols equivalent to
period and comma also have different style for horizontal and vertical
direction.
</P>

<P>
In Japan, Arabic numerical characters are widely used, like European 
languages, though we have Kanji (ideogram) numerical characters.  
Latin characters
can also appear in Japanese texts.  If a row of 1 - 3 (or 4) characters of 
Arabic and Latin appear in Japanese vertical text, these characters
can be crowded into one column.  If more characters appear (large numbers
or long words), the paper is rotated 90 degree in anticlockwise and
the characters are written in European way.  Sometimes Latin characters
which appears in vertical text are written in the same way as Japanese
character, i.e., vertical direction.  This is not so strong
custom.  Arabic and Latin characters can always be written in both
normal and rotated way in vertical text.
<footnote>
I HAVE TO SHOW EXAMPLE USING GRAPHICS.
</footnote>
DTP system should support all of them.
</P>

<P>
A version of Japanized TeX (developed by ASCII, a publishing company
in Japan) can use vertical direction.  This can also
treat a page containing both vertical and horizontal texts.
</P>


<sect1 id="japanese-layout"><heading>Layout of Characters</heading>

<P>
In Japanese language, words are not separated by space and
a line can be broken anywhere, with a few exceptions, unlike
European languages.  Thus hyphenation is not needed for Japanese.
</P>

<P>
Characters like open parentheses cannot come to the end
of a line. Characters like close parentheses and sorts of
sentence-separating marks such as period and comma cannot
come to the top of a line.  This rule and processing is
called 'kinsoku' in Japanese.
</P>

<P>
In European languages, a break of line is equivalent to a space.
In Japanese language, a break of line should be neglected.
For example, when rendering an HTML file, line-breaking character
in the HTML source should not be converted into whitespace.
</P>


<sect1 id="japanese-lang"><heading>LANG variable</heading>

<P>
Different value of <tt>LANG</tt> used for different encodings.
</P>

<P>
Following values are used for EUC-JP.
<list>
  <item><tt>LANG=ja_JP.eucJP</tt> (major for Linux and *BSD)
  <item><tt>LANG=ja_JP.ujis</tt> (used to be major for Linux)
  <item><tt>LANG=ja_JP</tt> (because EUC-JP is the de-facto standard for UNIX;
        not recommended)
  <item><tt>LANG=ja</tt> (because EUC-JP is the de-facto standard for UNIX;
        not recommended)
</list>
</P>

<P>
<tt>LANG=ja_JP.jis</tt> is used for ISO-2022-JP (aka JIS code or JUNET code).
</P>

<P>
<tt>LANG=ja_JP.sjis</tt> is used for SHIFT-JIS (aka Microsoft Kanji Code).
</P>

<P>
Setting LANG is not sufficient for a Japanese user who has just installed
Linux to get a minimal Japanese environment.  There are several
books on establishing Japanese environment on Linux/BSD and
magazines on Linux often have feature articles on how to establish
Japanese environment.  Nowadays many Japanized Linux distributions
which are optimized so that many basic software can display and
input Japanese are popular.
Debian GNU/Linux has <package>user-ja</package> (for potato) and 
<package>language-env</package> (for woody and following versions)
packages to establish basic Japanese environment.
</P>


<sect1 id="japanese-input"><heading>Input from Keyboard</heading>

<P>
Since Japanese characters cannot be inputed directly from a keyboard,
a software is needed to convert ASCII characters into Japanese.
<prgn>WNN</prgn>, <prgn>Canna</prgn>, and <prgn>SKK</prgn> are popular 
free softwares to input Japanese language.  Though 
<prgn>T-Code</prgn> is also available, it is difficult to use.
Since these adopt server/client model and implement their own protocols,
we cannot input Japanese only with <package>wnn</package>,
<package>canna</package>, or <package>skk</package>
(and their depending packages).
</P>

<P>
In X Window System environment, <package>kinput2-*</package> and
<package>skkinput</package> packages 
connects these protocols and XIM, which is the standard input
protocol for X.  Kinput2 also has an original protocol and
<package>kterm</package> and so on can be a client of kinput2 protocol.
Kinput2 protocol was developed before international standards such
as XIM (or Ximp or Xsi) became available.
</P>

<P>
On console, there are no standard and each software has to support
wnn and/or canna protocol.  For example, <package>jvim-canna</package>,
<package>xemacs21-mule-canna</package>, and emacs20 with
<package>emacs-dl-canna</package> or <package>emacs-dl-wnn</package>.
Thus the ways to operate are different between softwares.
<package>skkfep</package> provides a general way to input Japanese
on console.
</P>

<P>
Then the way to input Japanese is explained.
</P>

<P>
Since almost Hiraganas and Katakanas represents a pair of a vowel 
and a consonant with one character, we can input one Hiragana or
one Katakana with two Latin alphabets.  A few Hiraganas and Katakanas
need one or three alphabets.
</P>

<P>
Kanji is obtained by converting from Hiragana. 
There are many Japanese words which are expressed by two or more Kanjis
and almost recent converting softwares can convert such words at a time.
(Old softwares can convert one Kanji at a time.  You must be patient
to use this way.)
Softwares with good grammar/context analyzer and large dictionary 
can convert longer phrases or even a whole sentence at a time.
However, we usually have to select one Kanji or word from
candidates the software shows, because Japanese language has
many homophones.  For example, 61 Kanjis whose readings are 'KAN'
and 6 words whose readings are 'KOUKOU' are registered in 
dictionary of <tt>canna</tt>.
(Today, 2 Oct 1999, I saw a TV advertisement film of Japanese word processor
which insists the software can correctly convert an input into
'a cafe which opened today', not 'a cafe which rotated today'.
Though Japanese word 'KAITEN' means both 'open (a shop)' and 'rotate',
the software knows it is more usual for a cafe to open than to rotate.)
</P>

<P>
The conversion from Hiragana to Kanji needs a large dictionary which
contains the Kanji spelling and readings of Japanese major words and
conjugation or inflection.  Thus proprietary softwares tend to
efficiently convert.  They usually have dictionaries larger than 
few megabytes. Some of these recent proprietary softwares
even analyze the topic or meaning of the inputed Hiragana sentence 
and choose the most appropriate homophone, though they often choose 
wrong ones.
</P>

<P>
Nowadays several proprietary conversion softwares such as ATOK, WNN6,
and VJE for Linux are sold in Japan.
</P>

<P>
Since it is complex and hard work for users to input Japanese characters,
we don't want to input Y (for YES) or N (for NO) in Japanese.
We prefer learning such basic English words to inputing Japanese
words by invoking conversion software, inputing Latin alphabetic
expression of Japanese, converting it into Hiragana, converting
it into Kanji, choosing the correct Kanji, determining the correct
Kanji, and ending the conversion software each time we need to
input yes or no or similar words.
</P>


<sect1 id="japanese-more"><heading>More Detailed Discussions</heading>

<sect2 id="japanese-width"><heading>Width of Characters</heading>

<P>
Different from European languages, Japanese characters should 
written in a fixed width.  Exceptions arises when two symbols 
such as parentheses, periods and commas continue.  Kerning
should be done for such cases if the software is a word processor.
A text editor need not.
</P>

<sect2 id="japanese-ruby"><heading>Ruby</heading>

<P>
<strong>Ruby</strong> is a small (usually 1/2 in length and 1/4 in 
area or a bit smaller) 
characters written above (in horizontal direction) or at right side
(in vertical direction) of the main text.  This is usually used to show 
a reading of difficult Kanji.  
</P>

<P>
Japanized TeX can use ruby by using an extra macro.  Word processors should
have Ruby faculty.
</P>


<sect2 id="japanese-case"><heading>Upper And Lower Cases</heading>

<P>
Japanese character does not have upper and lower case although
there two sets of phonograms, Hiragana and Katakana.
</P>

<P>
Thus <tt>tolower()</tt> and <tt>toupper()</tt> should not convert
between Hiragana and Katakana.
</P>

<P>
Hiragana is used for usual text.  Katakana is used mainly for
express foreign or imported words, for example, KONPYU-TA
for computer, MAIKUROSOFUTO for Microsoft, and AINSYUTAIN for Einstein.
</P>


<sect2 id="japanese-sort"><heading>Sorting</heading>

<P>
Phonograms (Hiragana and Katakana) have sorting order.
The order is same to defined in JIS X 0208, with a few exceptions.
</P>

<P>
Ideograms (Kanji) sorting is difficult.  They should be sorted
by their reading but almost all kanji have a few readings according
to the context.  So if you want to sort Japanese text, you will need
a dictionary of whole Japanese Kanji words.  And more, a few
Japanese words written in Kanji have different readings with
exactly same series of Kanjis, this can occur especially for names of 
person.
So it is usual that addressbook databases have two 'name' columns,
one for Kanji expression and the other for Hiragana.
</P>

<P>
I know no softwares which can sort Japanese words in perfect way,
including free and proprietary softwares.
</P>


<sect2 id="japanese-romaji"><heading> Ro-ma ji (Alphabetic expression of Japanese)</heading>

<P>
We have a phonetic alphabetic expression of Japanese, Ro-ma ji.
It has almost one-to-one correspondence to Japanese phonogram.
It can be used to display Japanese text on Linux console and
so on.  Since Japanese have many homophones this expression
can be crabbed.
</P>

<P>
There are several variants of Ro-ma ji.  
</P>

<P>
The first distinguishing point is on handling of long syllable.
For example, long syllable of 'E' is expressed in:
<list>
 <item>'E' with caret,
 <item>'E' with upper bar,
 <item>only 'E' in which long syllable mark is ignored,
 <item>'EE',
 <item>and so on.
</list>
</P>

<P>
The second distinguishing point is some special pairs 
of vowel and consonant.
For example, Hiragana character for combination of 'T' and 'I' is 
pronounced like 'CHI'.
<list>
 <item>TI or CHI, as described above,
 <item>TU or TSU,
 <item>SI or SHI,
 <item>HU or FU,
 <item>WO or O,
 <item>TYA or CHA, and
 <item>N or M.
</list>
</P>
