What is String Encoding

Computer hardware and operating system memory operate exclusively on bits and bytes (values from 0 to 255). A character encoding serves as a two-way translation dictionary:

🟢 Junior Level

Character Encoding is a standardized system of rules and lookup tables that defines how abstract textual characters (letters, digits, punctuation marks, emojis) are converted into sequences of physical binary bytes for storage on disk or transmission across a network, and vice versa.

Computer hardware and operating system memory operate exclusively on bits and bytes (values from 0 to 255). A character encoding serves as a two-way translation dictionary:

Encoding: Textual representation (String) $\to$ physical byte array (byte[]).
Decoding: Physical byte array (byte[]) $\to$ textual representation (String).

In a Nutshell (30 Seconds)

The golden rule of enterprise Java development: never rely on platform-implicit default encodings. Whenever performing I/O, serialization, socket communication, or file operations, always explicitly pass StandardCharsets.UTF_8.

String text = "Hello, Java 21! 🔥";

// 1. Encoding: Explicitly specify UTF-8
byte[] bytes = text.getBytes(StandardCharsets.UTF_8);

// 2. Decoding: Reconstruct the string using the identical encoding
String restored = new String(bytes, StandardCharsets.UTF_8);

System.out.println(restored.equals(text)); // true

If an application encodes a string with one character set (e.g., Windows-1251) and decodes it with another (e.g., UTF-8 or ISO-8859-1), the result is data corruption known as Mojibake (garbled characters, question marks ???, or Unicode replacement symbols \uFFFD).

🟡 Middle Level

Unicode: Code Points vs. Code Units

To understand character encoding in Java, distinguish between two foundational concepts:

Unicode Code Point: An abstract, unique numerical identifier assigned to a character by the Unicode standard (denoted as U+XXXX, e.g., 'A' is U+0041, while '🔥' is U+1F525). The Unicode standard defines over 1.1 million possible code points.
Code Unit: The minimum bit/byte storage unit used by a specific encoding format to represent a code point:
- In UTF-8, a code unit is 1 byte (a single Unicode character spans 1 to 4 bytes).
- In UTF-16, a code unit is 2 bytes (corresponding to Java’s primitive char type).

The Surrogate Pairs Trap in Java

Java’s primitive char type was designed in 1995 as a 16-bit unsigned integer, under the assumption that 65,536 code points (the Basic Multilingual Plane, BMP: U+0000 through U+FFFF) would suffice for all world languages.

When Unicode expanded up to U+10FFFF (encompassing emojis, mathematical symbols, and historic scripts), UTF-16 represented characters outside the BMP using Surrogate Pairs: two consecutive 16-bit char code units (High Surrogate \uD800..\uDBFF + Low Surrogate \uDC00..\uDFFF):

String emoji = "🔥";

// String.length() returns the count of 16-bit UTF-16 code units, NOT Unicode characters!
System.out.println(emoji.length()); // 2! (one surrogate pair)

// Correct count of actual Unicode characters (Code Points):
System.out.println(emoji.codePointCount(0, emoji.length())); // 1

Popular Character Encodings Comparison

Encoding	Code Unit Size	Bytes per Character	ASCII Compatibility	Key Characteristics & Use Cases
UTF-8	8 bits (1 byte)	1 to 4 bytes (variable)	✅ 100% compatible (1 byte)	Universal standard for Internet, JSON, REST APIs, databases
UTF-16	16 bits (2 bytes)	2 or 4 bytes	❌ Incompatible (min 2 bytes)	JVM internal string representation, Windows APIs (BE and LE variants)
ISO-8859-1 (Latin-1)	8 bits (1 byte)	Exactly 1 byte	✅ Compatible in low 128	Western European scripts. Foundation for Compact Strings
Windows-1251	8 bits (1 byte)	Exactly 1 byte	✅ Compatible in low 128	Legacy Windows Cyrillic codepage. Incompatible with UTF-8
US-ASCII	7 bits (stored in 1 byte)	Exactly 1 byte	—	Base 128 English characters and control codes

JEP 400 (Java 18+): Standardized UTF-8 Default

Prior to Java 18, parameterless methods such as str.getBytes() or new FileReader("file.txt") defaulted to the host operating system’s locale encoding (file.encoding):

On Linux, this was virtually always UTF-8.
On Windows, it defaulted to Windows-1252 (Western) or Windows-1251 (Eastern Europe). Consequently, code tested on a Windows developer machine frequently corrupted characters when deployed into a Linux container.

Beginning in Java 18 (JEP 400), Charset.defaultCharset() is standardized to UTF-8 across all platforms. Nevertheless, explicitly supplying StandardCharsets.UTF_8 remains mandatory enterprise best practice, as administrators can still override -Dfile.encoding=COMPAT on legacy infrastructure.

🔴 Senior Level

Internal JVM Architecture: `StringCoding` and Compact Strings (Java 9+)

Starting with Java 9 (JEP 254), java.lang.String abandoned char[] in favor of an internal byte[] value paired with a single-byte coder flag:

public final class String {
    @Stable
    private final byte[] value;
    private final byte coder; // 0 = LATIN1, 1 = UTF16
}

When text.getBytes(StandardCharsets.UTF_8) is invoked, HotSpot delegates to java.lang.StringCoding:

When coder == LATIN1:
- The HotSpot C2 compiler leverages SIMD hardware intrinsics (AVX-512 / AVX2 / ARM NEON) to check the high bit of all bytes in the array in parallel.
- If all bytes are $<128$ (pure ASCII), UTF-8 encoding requires zero transformation: raw bytes are copied directly in 64-byte chunks per CPU cycle.
When coder == UTF16:
- HotSpot dispatches to a vectorized transcoder StringUTF16.toUtf8, converting 2-byte UTF-16 units into 1, 2, or 3 UTF-8 bytes.

The Byte Order Mark (BOM) Hazard

When reading UTF-8 or UTF-16 data from external sources (e.g., Windows Notepad exports or CSV files), the payload often starts with a Byte Order Mark:

In UTF-8, the BOM signature consists of three bytes: 0xEF, 0xBB, 0xBF.
Standard Java decoding via new String(bytes, StandardCharsets.UTF_8) decodes these bytes into an invisible Unicode zero-width character: \uFEFF (Zero Width No-Break Space).

byte[] bomBytes = {(byte) 0xEF, (byte) 0xBB, (byte) 0xBF, 'O', 'K'};
String str = new String(bomBytes, StandardCharsets.UTF_8);

System.out.println(str.equals("OK"));      // false! Hidden \uFEFF prefix!
System.out.println(str.startsWith("OK"));  // false!
System.out.println(str.length());          // 3 (instead of 2!)

Production Fix: Strip the leading \uFEFF manually or wrap streams in Apache Commons IO BOMInputStream.

Corrupted Bytes Handling: `CodingErrorAction`

The standard convenience constructor new String(byte[] bytes, Charset charset) silently suppresses corruption:

Malformed or incomplete byte sequences are automatically replaced by the Unicode replacement character '\uFFFD' (REPLACE policy).
In mission-critical financial, cryptographic, or network protocol parsing, silent replacement leads to undetected data loss.
Strict Decoding Pattern: Enforce immediate failure on corrupted input using an explicit CharsetDecoder:

CharsetDecoder decoder = StandardCharsets.UTF_8.newDecoder()
    .onMalformedInput(CodingErrorAction.REPORT)     // Throws MalformedInputException
    .onUnmappableCharacter(CodingErrorAction.REPORT);

ByteBuffer buffer = ByteBuffer.wrap(corruptedBytes);
CharBuffer result = decoder.decode(buffer); // Fails fast on invalid bytes

4 Tricky Questions

1. Why does `"👋".length()` return 2 instead of 1 in Java, and how do you properly iterate over Unicode characters?

Answer: Java’s String.length() historically returns the count of 16-bit UTF-16 code units (char), not the count of perceived Unicode characters (grapheme clusters / code points). The wave emoji "👋" (U+1F44B) belongs to the Supplementary Multilingual Plane (above U+FFFF) and cannot fit into a single 16-bit char. In UTF-16, it is encoded as a surrogate pair: High Surrogate (\uD83D) and Low Surrogate (\uDC4B). To process actual Unicode characters correctly, use the Code Point streaming API:

String str = "Hi 👋!";
str.codePoints().forEach(cp -> {
    System.out.println(Character.toString(cp));
});

2. What happens if you slice a string directly in the middle of a surrogate pair using `substring()`?

Answer: It creates an isolated, invalid surrogate (Unpaired Surrogate):

The first slice retains only the High Surrogate (0xD800..0xDBFF), while the second slice retains only the Low Surrogate (0xDC00..0xDFFF).
No StringIndexOutOfBoundsException is thrown, because substring() operates strictly on char offsets.
However, when attempting to transmit, persist, or encode that string via str.getBytes(StandardCharsets.UTF_8), the UTF-8 encoder detects the unpaired surrogate and replaces it with \uFFFD (Replacement Character, ). The original emoji data is permanently lost.

3. Why is it strictly prohibited to convert arbitrary binary payloads (AES keys, image data, SHA hashes) to `String` via `new String(bytes)` and back?

Answer: Because character encodings are not a bijective (one-to-one and onto) mapping for all 256 possible byte values:

In UTF-8, certain byte sequences are invalid (e.g., standalone bytes 0xFF, 0xFE, or invalid multi-byte continuation headers). When encountering arbitrary binary values, the UTF-8 decoder replaces invalid sequences with \uFFFD (3 bytes 0xEF, 0xBB, 0xBF).
Calling getBytes(StandardCharsets.UTF_8) subsequently will produce those replacement bytes instead of the original raw data, corrupting cryptographic keys or binary headers.
Binary data must always be stored in byte[] / ByteBuffer, or converted to a safe text representation using Base64 (Base64.getEncoder().encodeToString(bytes)) or Hex.

4. Are `CharsetEncoder` and `CharsetDecoder` instances thread-safe in Java?

Answer: No, they are strictly NOT thread-safe. While Charset instances and StandardCharsets.UTF_8 singletons are immutable and thread-safe, CharsetEncoder and CharsetDecoder maintain mutable internal state, including incomplete multi-byte buffers, partial decoding states, and reset flags. Invoking .encode() or .decode() concurrently on the same encoder/decoder instance across multiple threads causes race conditions, IllegalStateException crashes, or silent data corruption. Encoders and decoders must be created per invocation, pooled, or stored in a ThreadLocal.

🎯 Interview Cheat Sheet

30-Second Summary

“Character encoding defines how Unicode code points map to binary bytes. Java internally stores strings using Compact Strings (Latin-1 at 1 byte/char or UTF-16 at 2 bytes/char), while UTF-8 is the universal standard for all external I/O. Key engineering rules: always explicitly specify StandardCharsets.UTF_8, never store raw binary data in String (use Base64), remember that length() counts 16-bit char units so emojis have length 2 (surrogate pairs), and note that CharsetDecoder is mutable and not thread-safe.”

Key Takeaways for Senior Interviews

JEP 400 (Java 18+): UTF-8 is the default platform charset across all operating systems, but explicit parameterization remains mandatory.
Surrogate Pairs: Supplementary code points ($> \text{U+FFFF}$) require two char units. Use codePoints() and codePointCount() for accurate Unicode manipulation.
BOM Hazards: UTF-8 BOM (0xEF, 0xBB, 0xBF) decodes into \uFEFF, breaking prefix and equality checks.
Silent Corruption: new String(bytes, UTF_8) replaces invalid bytes with \uFFFD. Use CharsetDecoder with CodingErrorAction.REPORT for strict validation.
Thread Safety: StandardCharsets.* is thread-safe; CharsetEncoder / CharsetDecoder are stateful and not thread-safe.

Red Flags to Avoid

❌ Calling str.getBytes() or new String(bytes) without an explicit Charset argument.
❌ Assuming that “1 character always equals 1 char or 1 byte” (emojis = 2 char = 4 bytes in UTF-8).
❌ Storing encrypted byte arrays or image binaries in a String without Base64 encoding.
❌ Sharing a single CharsetDecoder instance across concurrent threads.

How to Properly Convert String to byte[] and Back — Production-grade conversion techniques
What are Compact Strings in Java 9+ — Latin-1 vs UTF-16 heap internals
How to Find Out How Much Memory a String Occupies — Exact memory calculation
Why String is Immutable — Security and design implications