Perl Unicode Guide — Encoding, Decoding, and Character Handling

In this tutorial, you will learn about Perl Unicode Guide. We cover key concepts, practical examples, and best practices to help you master this topic.

Perl Unicode handling requires explicit encoding/decoding between external byte strings and internal character strings -- using the Encode module, utf8 pragma for source code, and three-argument open with encoding layers for file I/O.

Unicode in Source Code

# Source code is UTF-8
use utf8;
my $greeting = "Hello, 世界!";  # valid with use utf8
print length($greeting);        # character count: 10

Encoding Layers

# Open file with UTF-8 encoding layer
open(my $fh, "<:encoding(UTF-8)", "input.txt")
    or die "Cannot open: $!";
    
# Write with UTF-8
open(my $out, ">:encoding(UTF-8)", "output.txt")
    or die "Cannot open: $!";

# Default encoding for all I/O
use open ':encoding(UTF-8)';
use open ':std';

Encode Module

use Encode;

my $string = "Hello 世界";

# Encode to bytes
my $utf8_bytes = encode("UTF-8", $string);
print length($utf8_bytes);  # byte length

# Decode from bytes
my $decoded = decode("UTF-8", $utf8_bytes);
print length($decoded);      # character length

Unicode in Regex

use feature 'unicode_strings';

# \w matches Unicode word characters
my $text = "cafe\u0301";  # é as combining character
if ($text =~ /\w+/) {
    print "Word characters found\n";
}

# \X matches Unicode grapheme clusters
while ($text =~ /(\X)/g) {
    print "Grapheme: $1\n";
}

Common Mistakes

1. Mixing byte and character lengths

length() returns characters. Use length() on decoded strings, bytes::length() on encoded bytes.

2. Wide character warnings

print $unicode_string without encoding causes "Wide character" warning. Set binmode or encoding layer.

3. Assuming one byte = one character

Only ASCII fits one byte. Multibyte characters need UTF-8 encoding. Always decode input, encode output.

Practice Questions

1. What does use utf8 do? Allows UTF-8 literals in source code. Without it, non-ASCII characters cause a parse error.

2. How do you read a UTF-8 file? open(my $fh, "<:encoding(UTF-8)", "file.txt") reads with automatic decoding.

3. What causes "Wide character in print" warnings? Printing a decoded character string without setting the output encoding layer.

FAQ

{{< faq question="What is the difference between bytes and characters?" >}} Bytes are raw 8-bit values. Characters are Unicode codepoints. UTF-8 encodes one character as 1-4 bytes. {{< /faq >}}

{{< faq question="How do I check if a string is valid UTF-8?" >}} eval { decode("UTF-8", $bytes, Encode::FB_CROAK) } returns true if valid, dies on invalid. {{< /faq >}}

{{< faq question="What does the -C flag do?" >}} The -C command-line flag enables Unicode features: perl -CSDA enables UTF-8 for STDIN/STDOUT/STDERR and @ARGV. {{< /faq >}}

What's Next

Now learn about signal handling in Perl.