Protobuf & Avro Decoder

About Protobuf & Avro Decoder

Two binary formats that turn up as an unexplained blob in a log, a queue or a captured request: protocol buffer wire bytes, and Avro container files. Paste them as base64 or hex and see what is inside.

The two are in completely different situations, and the results differ accordingly. An Avro container carries its own schema in its header — field names, types and order — so it decodes into exactly the records it was written from, and nothing outside the file is needed. Protobuf carries no names at all. The wire format has only field *numbers* and one of six wire types; the names live in the .proto file, which is not part of the message. So a protobuf result is numbered rather than named, and that is not a shortcoming of this tool but a fact about the format.

What a schema-less protobuf read does give you is the shape: which fields are present, how they nest, how long each one is, and what type each was written as. That is usually enough to match a payload against the .proto you already have, or to tell whether a field you expected was sent at all.

One genuine ambiguity is worth understanding, because it is the only place this guesses. Wire type 2 means "this many bytes follow" and is used for strings, byte arrays, nested messages and packed number lists — all four identical on the wire. Each one is tried as a nested message first and accepted only if it parses completely, then as text, then reported as hex. The guess is shown in its own column rather than hidden, so a nested message that is really a string can be spotted and overruled.

Neither format can be written back. Protobuf cannot be built without the .proto that names the fields, and Avro cannot be written without deciding a schema, so this tool reads and does not convert. Take the JSON it produces into the Structure Converter if you want it somewhere else.

Where it helps

• Reads protobuf with no .proto file at all, which is the situation somebody staring at an unexplained payload is actually in.
• Reads Avro containers using the schema inside them, so the records come back with their real field names and types.
• Shows the wire type and the interpretation of every field separately, so a guess can be seen and disagreed with.
• Handles deflate-compressed Avro blocks as well as uncompressed ones.
• Takes base64 or hex, and works out which one was pasted rather than making you say.
• Keeps a 64-bit integer too large for a JavaScript number as a string instead of rounding it, so the low bits survive.
• Checks the sync marker after every Avro block, so a damaged file is reported rather than decoded into plausible-looking nonsense.
• Runs entirely in the browser: a captured payload is often the most sensitive thing you have, and this one never leaves the machine.

Limits

• Protobuf field names cannot be recovered. They are in the .proto file, not in the message, and no decoder can produce them from the bytes alone.
• A protobuf field of wire type 2 is guessed at, and the guess can be wrong — short byte arrays sometimes parse as a message by coincidence.
• Signed protobuf integers written as sint32 or sint64 are zig-zag encoded, and without the schema there is no way to know to undo it, so they appear as their raw unsigned value.
• A protobuf double or float is reported as the 64-bit or 32-bit pattern it is, because nothing on the wire says it was meant as a number rather than as a fixed-width integer.
• Avro files compressed with snappy are not read: it needs a decompressor and a checksum this does not carry.
• Neither format can be written back out, so this reads and never converts.
• Deprecated protobuf groups — wire types 3 and 4 — are refused rather than decoded.

Frequently asked questions

Why are my protobuf fields numbered instead of named?

Because the names were never sent. A protobuf message on the wire is a series of field numbers, wire types and values — the names exist only in the .proto file used to build and read it. That is the compression: not repeating "customer_id": on every message is most of why the format is smaller than JSON. Match the numbers against your .proto and the names come back.

How do I know whether a field is a string or a nested message?

Often you cannot, and the result says which way it guessed in the "read as" column. Both are wire type 2 — a length followed by that many bytes — and the only difference is what the bytes mean. The bytes are tried as a message first and accepted only if they parse completely, which is right nearly always and occasionally wrong on short byte arrays that happen to look like valid tags.

Why did my number come out far larger than it should be?

Two likely causes. If it was written as sint32 or sint64, protobuf zig-zag encodes it so small negative numbers stay small — -1 is sent as 1, -2 as 3 — and without the schema there is no way to know to undo that. If it was a double or float, what you are seeing is the bit pattern read as an integer, because nothing on the wire distinguishes the two.

Why does Avro give me field names when protobuf does not?

Because an Avro object container file begins with its own schema, written out as JSON. That is the whole argument for the container format: a file is self-describing, and a reader needs nothing from the system that produced it. Protobuf made the opposite trade, keeping the bytes as small as possible and requiring the schema to be shared out of band.

Can I turn this back into protobuf or Avro?

Not here. Writing either one needs a schema — the .proto for protobuf, a declared record type for Avro — and inventing one from the decoded values would produce a file that happens to parse rather than a file the other side would accept. Take the JSON into the Structure Converter if you want it in another text format.

Does the payload get sent anywhere?

No. All of it runs in the browser, which matters more for this tool than for most: a captured request body or a queue message is frequently the most sensitive thing on your screen.

Related tools

• Structure Converter — Convert between JSON, XML, YAML, TOML, TOON, HCL and CSV
• Tree Viewer — Explore JSON, XML, YAML, TOML, TOON or HCL as a collapsible tree
• Table Viewer — Show the records in a JSON, XML, YAML, TOML or TOON document as a table
• Base64 — Encode or decode UTF-8 text and URL-safe Base64
• String Escaper — Escape a JSON, XML, YAML, TOML or TOON document for embedding, or read one back
• Structure Formatter — Pretty-print JSON, XML, YAML, TOML, TOON or HCL with a chosen indent

More Structured Formats tools

• Structure Converter — Convert between JSON, XML, YAML, TOML, TOON, HCL and CSV
• String Escaper — Escape a JSON, XML, YAML, TOML or TOON document for embedding, or read one back
• Structure Formatter — Pretty-print JSON, XML, YAML, TOML, TOON or HCL with a chosen indent
• Structure Validator — Check JSON, XML, YAML, TOML, TOON or HCL and report structural problems
• Tree Viewer — Explore JSON, XML, YAML, TOML, TOON or HCL as a collapsible tree
• Code Generator — Turn a JSON, YAML, XML or other document into TypeScript, Go, C#, Python, Rust and more
• Schema Generator — Write a JSON Schema, XSD, Avro schema or .proto from a document you already have
• Schema Validator — Check a document against a JSON Schema, .proto, Avro schema, XSD or RELAX NG
• Table Viewer — Show the records in a JSON, XML, YAML, TOML or TOON document as a table
• Structure Compare — Diff two JSON, XML, YAML, TOML, TOON or HCL documents by path
• Structure Repair — Fix broken JSON, YAML, XML, TOML, TOON, INI, .env, .properties and HCL, listing every repair
• Structure Minifier — Strip formatting from JSON, XML, YAML, TOML or TOON
• Path Finder — Select nodes with JSONPath or XPath
• Expression Evaluator — Reduce a JSONPath or XPath match to a single answer