string - Handling multibyte (non-ASCII) characters in C

Question

I am trying to do my own version of wc (unix filter), but I have a problem with non-ASCII characters. I did a HEX dump of a text file and found out that these characters occupy more than one byte. So they won't fit to char. Is there any way I can read these characters from file and handle them like single characters (in order to count characters in a file) in C? I've been googling a little bit and found some wchar_t type, but there were not any simple examples how to use it with files.

Answer 1

I've been googling a little bit and found some wchar_t type, but there was not any simple example how to use it with files.

Well met. There weren't any simple examples because, unfortunately, proper character set support isn't simple.

Aside: In an ideal world, everybody would use UTF-8 (a Unicode encoding that is memory-efficient, robust, and backward-compatible with ASCII), the standard C library would include UTF-8 encoding-decoding support, and the answer to this question (and dealing with text in general) would be simple and straightforward.

The answer to the question "What is the best unicode library for C?" is to use the ICU library. You may want to look at ustdio.h, as it has a u_fgetc function, and adding Unicode support to your program will probably take little more than typing u_ a few times.

Also, if you can spare a few minutes for some light reading, you may want to read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know about Unicode and Character Sets (No Excuses!) from Joel On Software.

I, personally, have never used ICU, but I probably will from now on :-)