The pitfalls of UTF-8 encoding and Unicode

Since my beginnings in the early 2000s, I have hit several encoding problems. In fact, these are the kinds of bugs that can drive a developer crazy! An application works perfectly for months, then a simple accented character, an emoji, or some text is enough to break everything, causing a major production incident...

Yet these problems often come from confusion between Unicode, character sets, and encodings. Understanding these concepts helps avoid many bugs that can sometimes be difficult to diagnose.

Unicode is not an encoding

A common mistake is to think that Unicode is an encoding. I also made this mistake when I started.

In reality, Unicode is a standard that assigns a unique identifier (called a code point) to each character.

For example:

A        U+0041

é        U+00E9

€        U+20AC

😀       U+1F600

Unicode defines characters, but not how they are stored in memory. For that, we need to use an encoding such as UTF-8 or UTF-16.

UTF-8 is an encoding

UTF-8 is now the most widely used encoding on the Web. It encodes each character using one to four bytes.

Some examples:

A      → 1 byte

é      → 2 bytes

€      → 3 bytes

😀     → 4 bytes

One of its main advantages is its compatibility with ASCII. All ASCII characters occupy exactly one byte in UTF-8. This certainly helped with its adoption around the world.

Accented characters that become unreadable

Who has never encountered this?

François

becoming:

François

or:

é

I have encountered this problem many times over the past twenty years. It happens when text is saved with one encoding but read with another.

For example:

• file saved in UTF-8;
• read as ISO-8859-1.

The bytes are identical, but their interpretation is different.

Be careful with databases

Encoding problems very often appear when using a database. It is not enough for the table to use UTF-8.

You also need:

• the database to use UTF-8;
• tables to use UTF-8;
• columns to use UTF-8;
• the connection between the application and the database to use UTF-8.

A single forgotten step can produce corrupted characters. Depending on the database system, it is more or less easy to make mistakes.

Emojis do not always work

Many developers discover this problem when a user adds an emoji:

😀

Some old MySQL configurations use: utf8

Unfortunately, this name is misleading. MySQL's `utf8` character set only supports characters encoded using up to three bytes. Emojis often require four bytes.

You therefore need to use: utf8mb4

which correctly supports the full range of Unicode characters.

Bytes are not characters

Another mistake is thinking that a character always corresponds to one byte. With UTF-8, this is not true.

Let's take:

Hello

5 characters.

But:

Bonjour 😀

contains a character that uses four bytes.

Counting bytes and counting characters are therefore two different operations. This is a common mistake when limiting text length.

Be careful with string manipulation functions

Some functions manipulate bytes rather than characters.

Let's take this PHP example, a mistake I made in the past:

strlen("é")

The result is: 2

because `é` uses two bytes in UTF-8.

To count characters, you need to use multibyte functions:

mb_strlen("é")

which returns: 1

Most programming languages now provide functions adapted to Unicode.

The BOM

The *Byte Order Mark* (BOM) is a sequence of bytes sometimes added at the beginning of a file. In some cases, it helps indicate the encoding being used. However, it can also cause problems.

In PHP, a BOM placed before:

<?php

can produce the error:

Headers already sent

Many developers therefore choose to save their UTF-8 files without a BOM.

Unicode normalization

The same character can sometimes be represented in several different ways.

For example:

é

can be stored:

• as a single character;
• or as the letter `e` followed by an accent.

Visually, the result is identical.

However, string comparisons can fail.

Some libraries allow strings to be normalized before comparing them.

Conclusion

Encoding problems are rarely bugs in the programming language or the database. They almost always come from mixing encodings or misunderstanding Unicode.

By using UTF-8 from end to end, choosing the right string manipulation functions, and paying attention to exchanges between applications, it is possible to avoid most of these problems.

Even after several years of development, I still encounter encoding-related bugs. Fortunately, once you understand the difference between Unicode, UTF-8, and character sets, they become much easier to solve.