0xhexforgeCTF utility layer
hexforge / guides / binary-text

GUIDE 12 / ENCODING

How to Identify Binary Text

A sequence of zeroes and ones becomes meaningful only after its boundaries and interpretation are justified. Start with byte alignment, then test the output.

Measure before decoding

Remove only presentation whitespace and count the remaining bits. A length divisible by eight supports a byte-oriented interpretation, but it does not prove one. Groups of seven may indicate ASCII without a leading zero; groups of four may represent binary nibbles; a fixed width such as 16 or 32 may represent integers rather than text. Keep separators because their original spacing can reveal the intended framing.

A byte-aligned example

01100110 01101100 01100001 01100111 01111011 01100010 01101001 01110100 01110011 01111101

Each group contains eight bits. Converting each group to a byte and decoding the bytes as UTF-8 yields flag{bits}. The output has readable syntax and the transformation can be reversed exactly, so the interpretation is supported by multiple signals.

Do not jump directly from binary to characters when the values include bytes that are invalid in UTF-8. They may describe a compressed stream, an image header, encrypted data, or another encoding layer. Display the bytes as hex and compare signatures before declaring failure.

Investigate missing boundaries

A continuous bitstream may begin at an unknown offset. Instead of padding it silently, test offsets from zero through seven and record which alignments produce valid bytes. Readability is useful evidence, but a single short word can occur by chance. Longer structure, valid JSON, a known file header, or a flag boundary provides stronger confirmation.

Bit order can also vary. Most text examples write the most significant bit first within each byte. Embedded protocols and puzzle constructions may reverse bits or bytes. Only reverse them when another clue supports that operation, and retain the unreversed value for comparison.

Account for UTF-8

ASCII characters occupy one byte and have a leading zero bit. Many Unicode characters require multiple UTF-8 bytes. The character é is encoded as bytes c3 a9, or 11000011 10101001. Interpreting each byte as an independent character produces misleading output. A decoder should reconstruct the byte array first and pass the complete sequence to a UTF-8 decoder.

Do not add zeroes just to make the text decode

Padding changes the evidence. If the length is unexpected, investigate framing, truncation, or a non-byte representation before modifying the stream.

Common mistakes

Discarding original separators

Spaces, commas, and line breaks may indicate fields or code words. Save both the original and compact forms.

Assuming every eight-bit group is text

A byte sequence can encode an integer, machine instruction, pixel, or file. Inspect both text and hexadecimal views.

Accepting replacement characters

A decoder may replace invalid UTF-8 with �. Use strict decoding when validation matters so invalid bytes produce an explicit error.

How this guide was verified

Fixtures included empty input, ASCII A, the published flag example, a two-byte UTF-8 character, an invalid UTF-8 sequence, a seven-bit stream, and streams shifted by one through seven positions. Valid text fixtures were round-tripped to their original bit sequences.

Decision workflow

  1. Preserve separators and count data bits.
  2. Test eight-bit grouping without adding padding.
  3. Convert bytes to hex and strict UTF-8.
  4. Inspect structure rather than readability alone.
  5. Test offsets or bit order only when evidence supports them.
  6. Round-trip the final interpretation.