Lua UTF-8 Guide — Unicode Support and String Encoding

In this tutorial, you will learn about Lua UTF. We cover key concepts, practical examples, and best practices to help you master this topic.

Lua UTF-8 library (utf8) in Lua 5.3+ handles Unicode strings with functions for character iteration, codepoint conversion, and offset calculation -- working directly on UTF-8 encoded byte sequences without transcoding.

Basic UTF-8 Functions

local s = "Hello, 世界!"

-- String length in bytes
print(#s)  --> 15

-- UTF-8 character length
print(utf8.len(s))  --> 10

-- Codepoint at position
local cp = utf8.codepoint(s, 8)
print(string.format("U+%04X", cp))  --> U+4E16

Iterating Over Characters

local s = "Hello 世界"

-- Iterate over all characters
for pos, cp in utf8.codes(s) do
    local char = utf8.char(cp)
    print(pos, string.format("U+%04X", cp), char)
end

-- Output:
-- 1      U+0048    H
-- 2      U+0065    e
-- ...
-- 7      U+4E16   世
-- 10     U+754C   界

Offset Calculations

local s = "Hello 世界"

-- Convert byte offset to character offset
print(utf8.offset(s, 7))  --> 7 (byte position of 7th char)
print(utf8.offset(s, 8))  --> 10 (byte position of 8th char)

-- Character offset from byte position
print(utf8.offset(s, 1, 7))  --> shows char offset from byte 7

Validating UTF-8

local valid = "Hello 世界"
local invalid = "Hello\xFE\xFF"

print(utf8.len(valid))     --> 8
print(utf8.len(invalid))   --> nil + error (invalid continuation byte)

-- Check validity
local ok = pcall(utf8.len, invalid)
print(ok)  --> false

Common Mistakes

1. Using # for UTF-8 strings

The length operator # counts bytes, not characters. Use utf8.len for character count.

2. Assuming one byte per character

ASCII characters are 1 byte, but 世界 takes 3 bytes each. Never assume fixed byte widths.

3. Mixing byte and character indices

string.sub uses byte positions. Use utf8.offset to convert character positions to byte positions.

Practice Questions

1. How do you get the number of UTF-8 characters in a string? utf8.len(s) returns the number of UTF-8 characters, or nil if the string is invalid.

2. How do you iterate over Unicode codepoints? Use utf8.codes(s) in a for loop: for pos, cp in utf8.codes(s) do ... end.

3. What is the difference between #s and utf8.len(s)? #s counts bytes. utf8.len counts UTF-8 characters.

FAQ

{{< faq question="Does Lua support all Unicode normalization forms?" >}} No. The utf8 library provides basic iteration and codepoint access. For normalization (NFC, NFD), use a library like slnunicode. {{< /faq >}}

{{< faq question="Can I use string.match with UTF-8 strings?" >}} Yes, but patterns work on bytes. A pattern like %a matches ASCII letters only. Use utf8.codes for Unicode-aware matching. {{< /faq >}}

{{< faq question="What happens if I pass invalid UTF-8?" >}} utf8.len returns nil and an error message. utf8.codes raises an error on invalid sequences. {{< /faq >}}

What's Next

Now learn about goto and labels in Lua.