Now we will concentrate on LC_CTYPE, which is the most important category in six locale categories.

Many encodings such as ASCII, ISO 8859-*, KOI8-R, EUC-*, ISO 2022-*, TIS 620, UTF-8, and so on are used widely in the world. It is inefficient and a cause of bugs, even not impossible, for every softwares to implement all these encodings. Fortunately, we can use LOCALE technology to solve this problem. Usage of UCS-4 is the second best solution for this problem. Sometimes LOCALE technology cannot be used and UCS-4 is the best. I will discuss this solution later.

Multibyte characters is a term to call characters encoded in locale-specific encoding. It is nothing special. It is mere a word to call our daily encodings. In ISO 8859-1 locale, ISO 8859-1 is multibyte character. In EUC-JP locale, EUC-JP is multibyte character. In UTF-8 locale, UTF-8 is multibyte character. In short, multibyte character is defined by LC_CTYPE locale category. Multibyte characters is used when your software inputs or outputs text data from/to everywhere out of your software, for example, standard input/output, display, keyboard, file, and so on, as you are doing everyday. There are a few exceptions. Compound text should be used for communication between X clients. UTF-8 would be the standard for file names in Linux.

You can handle multibyte characters using ordinal char or unsigned char types and ordinal character- and string-oriented functions. It is just like you used to do for ASCII and 8bit encodings.

Then why we call it with a special term of multibyte character? The answer is, ISO C specifies a set of functions which can handle multibyte characters properly. On the other hand, it is obvious that usual C functions such as strlen() cannot handle multibyte characters properly.

Then what is these functions which can handle multibyte characters properly? Please wait a minute. Multibyte character may be stateful or stateless and multibyte or non-multibyte, since it includes all encodings ever used and will be used on the earth. Thus it is not convenient for internal processing. It needs complex algorithm even for, for example, character extraction from a string, addition and division of a string, or counting of number of character in a string. Thus, wide characters should be used for internal processing. And, the main part of these C functions which can handle multibyte characters are functions for interconversion between multibyte characters and wide characters. These functions are introduced later. Note that you may be able to do without these functions, since ISO C supplies I/O functions with conversion.

Wide character is defined in ISO C that all characters are expressed in fixed width of bits. that it is stateless, i.e., it doesn't have shift states.

There are two types for wide characters: wchar_t and wint_t. wchar_t is a type which can contain one wide character. It is just like 'char' type can be used for contain one character. wint_t can contain one wide character or WEOF, an substitution of EOF.

A string of wide characters is achieved by an array of wchar_t, just like a string of characters is achieved by an array of char.

There are functions for wchar_t, substitute for functions for char. strcat(), strncat() -> wcscat(), wcsncat() strcpy(), strncpy() -> wcscpy(), wcsncpy() strcmp(), strncmp() -> wcscmp(), wcsncmp() strcasecmp(), strncasecmp() -> wcscasecmp(), wcsncasecmp() strcoll(), strxfrm() -> wcscoll(), wcsxfrm() strchr(), strrchr() -> wcschr(), wcsrchr() strstr(), strpbrk() -> wcsstr(), wcspbrk() strtok(), strspn(), strcspn() -> wcstok(), wcsspn(), wcscspn() strtol(), strtoul(), strtod() -> wcstol(), wcstoul(), wcstod() strftime() -> wcsftime() strlen() -> wcslen() toupper(), tolower() -> towupper(), towlower() isalnum(), isalpha(), isblank(), iscntrl(), isdigit(), isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(), isxdigit() -> iswalnum(), iswalpha(), iswblank(), iswcntrl(), iswdigit(), iswgraph(), iswlower(), iswprint(), iswpunct(), iswspace(), iswupper(), iswxdigit() (isascii() doesn't have its wide character version). memset(), memcpy(), memmove, memmove(), memchr() -> wmemset(), wmemcpy(), wmemmove, wmemmove(), wmemchr() There are additional functions for wchar_t. wcwidth(), wcswidth() wctrans(), towctrans()

You cannot assume anything on the concrete value of wchar_t, besides 0x21 - 0x7e are identical to ASCII. Some of you may know GNU libc uses UCS-4 for the internal expression of wchar_t. However, you should not use the knowledge. It may differ in other systems. You may feel this limitation is too strong. If you cannot do under this limitation, you can use UCS-4 as the internal encoding. In such a case, you can write your software emulating the locale-sensible behavior using setlocale(), nl_langinfo(CODESET), and iconv(). Consult the section of . Note that it is generally easier to use wide character than implement UCS-4 or UTF-8.

You can write wide character in the source code as L'a' and wide string as L"string". Since the encoding for the source code is ASCII, you can only write ASCII characters. If you'd like to use other characters, you should use gettext.

There are two ways to use wide characters: I/O is described using multibyte characters. Inputed data are converted into wide character immediately after reading and data for output are converted from wide character to multibyte character immediately before writing. Conversion can be achieved using functions of mbstowcs(), mbsrtowcs(), wcstombs(), wcsrtombs(), mblen(), mbrlen(), mbsinit(), and so on. Please consult the manual pages for these functions. Wide characters are directly used for I/O, using wide character functions such as getwchar(), fgetwc(), getwc(), ungetwc(), fgetws, putwchar(), fputwc(), putwc(), and fputws(), formatted I/O functions for wide characters such as fwscanf(), wscanf(), swscanf(), fwprintf(), wprintf(), swprintf(), vfwprintf(), vwprintf(), and vswprintf(), and wide character identifier of %lc, %C, %ls, %S for conventional formatted I/O functions. By using this approach, you don't need to handle multibyte characters at all. Please consult the manual pages for these functions. Though latter functions are also determined in ISO C, these functions have became newly available since GNU libc 2.2. (Of course all UNIX operating systems have all functions described here.)

Note that very simple softwares such as echo doesn't have to care about multibyte character. and wide characters. Such software can input and output multibyte character as is. Of course you may modify these softwares using wide characters. It may be a good practice of wide character programming. Examples of a fragment of source codes will be discussed in .

There is an explanation of multibyte and wide characters also in Ken Lunde's "CJKV Information Processing" (p25). However, the explanation is entirely wrong.