Quickstart

nim-protobuf-serialization is a Nim library that helps you serialize your Nim objects to Protobuf 2 or 3.

• Repository →
• API docs:
  • API Index →
  • protobuf_serialization
  • proto_parser
• Contributor's Guide →
• Issues →

Installation

Add the dependency to your .nimble file:

requires "protobuf_serialization"

Or install directly:

nimble install protobuf_serialization

Basic Usage

1. Define Your Types

Annotate your Nim types with protobuf pragmas:

import protobuf_serialization
type
  Person {.proto3.} = object
    name {.fieldNumber: 1.}: string
    age {.fieldNumber: 2, pint.}: int32
    email {.fieldNumber: 3.}: string

2. Encode and Decode

Use Protobuf.encode and Protobuf.decode to serialize your objects:

let person = Person(
  name: "Alice",
  age: 30,
  email: "alice@example.com"
)

# Encode to bytes
let encoded = Protobuf.encode(person)

# Decode back to object
let decoded = Protobuf.decode(encoded, Person)

assert decoded == person

Proto2 vs Proto3

Both Protobuf 2 and 3 semantics are supported. Add the proto2 or proto3 pragma to your object type:

type
  MessageV2 {.proto2.} = object
    id {.fieldNumber: 1, required, pint.}: int32
    text {.fieldNumber: 2.}: seq[string]

type
  MessageV3 {.proto3.} = object
    id {.fieldNumber: 1, pint.}: int32
    text {.fieldNumber: 2.}: string

Importing .proto Files

.proto files are Protocol Buffers schema definition files. They define the structure of your messages using a language-neutral, platform-neutral syntax. These files are commonly used to share message definitions across different programming languages and systems.

You might want to import .proto files when:

• Working with existing protobuf schemas from other projects or teams
• Ensuring compatibility with services that use standard .proto definitions
• Avoiding manual annotation of Nim types when a schema already exists

This library can generate Nim types directly from .proto files at compile-time:

import protobuf_serialization/proto_parser

# This generates Nim types from your .proto file at compile-time
import_proto3 "my_protocol.proto3"

Building Documentation

Build the documentation site:

nimble docs

The docs will be generated in docs/ directory. The API docs are in docs/apidocs. Serve with any static file server, e.g.:

python3 -m http.server -d docs

Next Steps

• Learn how to annotate your objects with protobuf pragmas
• Explore repeated and packed fields
• Understand oneof fields for union types
• Handle optional fields with PBOption
• Import existing .proto files
• Use type extensions for custom serialization

Annotating Objects

To make your Nim types protobuf-serializable, you need to annotate them with pragmas that define the protobuf schema.

Message Types

Every protobuf message must be annotated with either {.proto2.} or {.proto3.}:

type
  MyMessage {.proto3.} = object
    # fields here

This corresponds to the syntax declaration in .proto files:

syntax = "proto3";

message MyMessage {
  // fields here
}

Field Numbers

Every field must have a unique field number using the {.fieldNumber: N.} pragma:

type
  Person {.proto3.} = object
    name {.fieldNumber: 1.}: string
    age {.fieldNumber: 2, pint.}: int32
    email {.fieldNumber: 3.}: string

Field numbers are permanent—once assigned, they should never change for a given field.

Scalar Types

Protobuf supports several scalar types. Here's how they map to Nim types:

Integer Encoding Strategies

Choose the right encoding for your integers:

pint (p stands for "plain")—uses standard varint encoding. Best for positive values or small negative values

type
  Counter {.proto3.} = object
    count {.fieldNumber: 1, pint.}: int32

sint (s stands for "signed")—uses zig-zag encoding. Best for negative values

type
  Temperature {.proto3.} = object
    celsius {.fieldNumber: 1, sint.}: int32

fixed: Best for large values or when size predictability matters

type
  LargeNumber {.proto3.} = object
    value {.fieldNumber: 1, fixed.}: uint64

Nested Messages

You can use other protobuf messages as field types:

type
  Address {.proto3.} = object
    street {.fieldNumber: 1.}: string
    city {.fieldNumber: 2.}: string
    zipCode {.fieldNumber: 3.}: string

  PersonWithAddress {.proto3.} = object
    name {.fieldNumber: 1.}: string
    age {.fieldNumber: 2, pint.}: int32
    address {.fieldNumber: 3.}: Address

Proto2 vs Proto3 Differences

Required Fields (Proto2 only)

In proto2, every field must be explicitly marked as required, be a repeated field (seq), be a PBOption, or be an extension type:

1. required pragma — the field must be present in the encoded message:

# 1. required pragma
type
  Message1 {.proto2.} = object
    id {.fieldNumber: 1, required, pint.}: int32

2. Repeated field (seq) — repeated fields are always allowed without required:

# 2. Repeated field (seq)
type
  Message2 {.proto2.} = object
    tags {.fieldNumber: 1.}: seq[string]

3. PBOption — explicitly optional field that can be absent:

# 3. PBOption
type
  Message3 {.proto2.} = object
    name {.fieldNumber: 1.}: PBOption[default(string)]

4. Extension type (ext) — a custom type with user-defined serialization logic:

# 4. Extension type
type
  MyCustomType = object
    value: int32

Protobuf.extensionDefaults(MyCustomType, pint32)

func computeFieldSize(
    field: int,
    value: MyCustomType,
    ProtoType: type ProtobufExt,
    skipDefault: static bool
): int =
  computeFieldSize(field, value.value, pint32, skipDefault)

proc writeField(
    stream: OutputStream,
    field: int,
    value: MyCustomType,
    ProtoType: type ProtobufExt,
    skipDefault: static bool = false
) {.raises: [IOError].} =
  writeField(stream, field, value.value, pint32, skipDefault)

proc readFieldInto(
    stream: InputStream,
    value: var MyCustomType,
    header: FieldHeader,
    ProtoType: type ProtobufExt
): bool {.raises: [SerializationError, IOError].} =
  readFieldInto(stream, value.value, header, pint32)

type
  Message4 {.proto2.} = object
    data {.fieldNumber: 1, required, ext.}: MyCustomType

Complete Example

Here's a complete example with various field types:

import protobuf_serialization

type
  Person {.proto3.} = object
    name {.fieldNumber: 1.}: string
    age {.fieldNumber: 2, pint.}: int32
    email {.fieldNumber: 3.}: string

type
  Counter {.proto3.} = object
    count {.fieldNumber: 1, pint.}: int32

type
  Temperature {.proto3.} = object
    celsius {.fieldNumber: 1, sint.}: int32

type
  LargeNumber {.proto3.} = object
    value {.fieldNumber: 1, fixed.}: uint64

type
  Address {.proto3.} = object
    street {.fieldNumber: 1.}: string
    city {.fieldNumber: 2.}: string
    zipCode {.fieldNumber: 3.}: string

  PersonWithAddress {.proto3.} = object
    name {.fieldNumber: 1.}: string
    age {.fieldNumber: 2, pint.}: int32
    address {.fieldNumber: 3.}: Address

let person = Person(name: "Bob", age: 25, email: "bob@example.com")
let encoded = Protobuf.encode(person)
let decoded = Protobuf.decode(encoded, Person)
assert decoded == person

echo "Annotating objects example passed!"

Next Steps

• Learn about repeated and packed fields for arrays
• Explore oneof fields for union types

Repeated & Packed Fields

Protobuf supports repeated fields (arrays/lists) with two encoding modes: unpacked and packed.

Repeated Fields (Unpacked)

In proto2, repeated fields are encoded as separate entries for each element.

In proto3, packed pragma with false value is used to mark fields as unpacked:

type
  Numbers {.proto3.} = object
    values {.fieldNumber: 1, sint, packed: false.}: seq[int32]
    names {.fieldNumber: 2, packed: false.}: seq[string]
    flags {.fieldNumber: 3, packed: false.}: seq[bool]

Example

let nums = Numbers(
  values: @[5'i32, -3, 300, -612],
  names: @["zero", "one", "two"],
  flags: @[true, false, true]
)

let encoded = Protobuf.encode(nums)
let decoded = Protobuf.decode(encoded, Numbers)
assert decoded == nums

The encoded bytes for the values field look like this:

08 0a 08 05 08 d8 04 08 c7 09

Each value has its own field tag (08 for field 1, wire type 0 = varint).

Packed Fields

Packed encoding is more efficient for scalar numeric types. All elements are encoded as a single length-delimited field. Use the packed pragma with true value to enable it in proto2:

type
  PackedNumbers {.proto2.} = object
    values {.fieldNumber: 1, sint, packed: true.}: seq[int32]
    flags {.fieldNumber: 2, packed: true.}: seq[bool]
    scores {.fieldNumber: 3, fixed, packed: true.}: seq[int32]
    weights {.fieldNumber: 4, packed: true.}: seq[float32]

Benefits of Packed Encoding

• Smaller size: Single length prefix instead of one per element
• Faster parsing: Elements are contiguous in memory
• Better for large arrays: Especially beneficial for numeric data

When to Use Packed

✅ Use packed for:

• Numeric arrays (int, uint, float, bool)
• Large sequences
• Performance-critical code

❌ Don't use packed for:

• String arrays (not supported)
• Message arrays (not supported)
• Small arrays (overhead may not be worth it)

Example

let packed = PackedNumbers(
  values: @[5'i32, -3, 300],
  flags: @[true, false, true],
  scores: @[100'i32, 200, 300],
  weights: @[1.5'f32, 2.5, 3.5]
)

let encodedPacked = Protobuf.encode(packed)
let decodedPacked = Protobuf.decode(encodedPacked, PackedNumbers)
assert decodedPacked == packed

The encoded bytes for the values field look like this:

0a 04 0a 05 d8 04

All values are grouped together after a single field tag (0a for field 1, wire type 2 = length-delimited) and a length prefix (04 = 4 bytes).

Proto2 vs Proto3

Proto2: Repeated fields are unpacked by default

type
  Message {.proto2.} = object
    values {.fieldNumber: 1.}: seq[int32]  # Unpacked by default

Proto3: Scalar numeric types are packed by default

type
  Message {.proto3.} = object
    values {.fieldNumber: 1.}: seq[int32]  # Packed by default

You can explicitly control this with the packed pragma:

type
  Message {.proto3.} = object
    unpacked {.fieldNumber: 1, packed: false.}: seq[int32]
    packed {.fieldNumber: 2, packed: true.}: seq[int32]

Empty Sequences

Empty sequences are omitted from the encoded output entirely. When decoded, they become empty sequences:

let empty = DataSet()
let encoded = Protobuf.encode(empty)
# encoded is empty - no bytes at all

let decoded = Protobuf.decode(encoded, DataSet)
assert decoded.temperatures.len == 0
assert decoded.labels.len == 0

Next Steps

• Learn about oneof fields for union types
• Explore optional fields with PBOption

Oneof Fields

The oneof field allows you to define a union type where only one field can be set at a time. This is useful for representing variant data.

Defining Oneof Types

A oneof is defined as a separate object type with the {.oneof.} pragma, using a Nim case object:

type
  ContactKind {.pure.} = enum
    notSet
    email
    phone
    address

  ContactInfo {.proto3, oneof.} = object
    case kind: ContactKind
    of ContactKind.notSet:
      discard
    of ContactKind.email:
      email {.fieldNumber: 1.}: string
    of ContactKind.phone:
      phone {.fieldNumber: 2.}: string
    of ContactKind.address:
      address {.fieldNumber: 3.}: string

  Person {.proto3.} = object
    name {.fieldNumber: 10.}: string
    contact {.oneof.}: ContactInfo

How Oneof Works

Only one field in a oneof can be set at a time. When you set a field, all other fields are cleared:

# Create a person with email contact
let person1 = Person(
  name: "Alice",
  contact: ContactInfo(kind: ContactKind.email, email: "alice@example.com")
)

assert person1.contact.kind == ContactKind.email
assert person1.contact.email == "alice@example.com"

# Encode and decode
let encoded = Protobuf.encode(person1)
let decoded = Protobuf.decode(encoded, Person)

assert decoded.name == "Alice"
assert decoded.contact.kind == ContactKind.email
assert decoded.contact.email == "alice@example.com"

# Create a person with phone contact
let person2 = Person(
  name: "Bob",
  contact: ContactInfo(kind: ContactKind.phone, phone: "+1234567890")
)

assert person2.contact.kind == ContactKind.phone
assert person2.contact.phone == "+1234567890"

Default State

When a oneof is not set, it defaults to the first value of the discriminator enum. In the example above, that's ContactKind.notSet, but you can name it whatever makes sense for your use case:

let person = Person(name: "Bob")
assert person.contact.kind == ContactKind.notSet

let encoded = Protobuf.encode(person)
let decoded = Protobuf.decode(encoded, Person)
assert decoded.contact.kind == ContactKind.notSet

Oneof with Different Types

Oneof fields can have different types:

import protobuf_serialization

type
  ValueKind {.pure.} = enum
    notSet
    intValue
    stringValue
    boolValue

  Value {.proto3, oneof.} = object
    case kind: ValueKind
    of ValueKind.notSet:
      discard
    of ValueKind.intValue:
      intValue {.fieldNumber: 10, sint.}: int32
    of ValueKind.stringValue:
      stringValue {.fieldNumber: 11.}: string
    of ValueKind.boolValue:
      boolValue {.fieldNumber: 12.}: bool

  Config {.proto3.} = object
    key {.fieldNumber: 1.}: string
    value {.oneof.}: Value

# Usage
let config1 = Config(
  key: "timeout",
  value: Value(kind: ValueKind.intValue, intValue: 30)
)

let config2 = Config(
  key: "name",
  value: Value(kind: ValueKind.stringValue, stringValue: "Alice")
)

let config3 = Config(
  key: "enabled",
  value: Value(kind: ValueKind.boolValue, boolValue: true)
)

# Encode and decode
let encoded1 = Protobuf.encode(config1)
let decoded1 = Protobuf.decode(encoded1, Config)
assert decoded1.key == "timeout"
assert decoded1.value.kind == ValueKind.intValue
assert decoded1.value.intValue == 30

echo "Oneof with different types example passed!"

Oneof with Nested Messages

Oneof fields can contain nested message types:

import protobuf_serialization

type
  Error {.proto3.} = object
    code {.fieldNumber: 1, pint.}: int32
    message {.fieldNumber: 2.}: string

  Success {.proto3.} = object
    data {.fieldNumber: 1.}: string

  ResultKind {.pure.} = enum
    notSet
    success
    error

  Result {.proto3, oneof.} = object
    case kind: ResultKind
    of ResultKind.notSet:
      discard
    of ResultKind.success:
      success {.fieldNumber: 10.}: Success
    of ResultKind.error:
      error {.fieldNumber: 11.}: Error

  Response {.proto3.} = object
    id {.fieldNumber: 1.}: string
    result {.oneof.}: Result

# Usage
let response = Response(
  id: "req-123",
  result: Result(
    kind: ResultKind.success,
    success: Success(data: "Operation completed")
  )
)

# Encode and decode
let encoded = Protobuf.encode(response)
let decoded = Protobuf.decode(encoded, Response)
assert decoded.id == "req-123"
assert decoded.result.kind == ResultKind.success
assert decoded.result.success.data == "Operation completed"

echo "Oneof with nested messages example passed!"

Next Steps

• Learn about optional fields with PBOption
• Explore type extensions for custom serialization

Optional Fields

In protobuf, distinguishing between "field not set" and "field set to default value" can be important. This library provides two ways to handle optional fields: Opt[T] for proto3/proto2 and PBOption for proto2.

The Problem with Default Values

In proto3, fields have implicit default values:

# Problem: In proto3, you can't distinguish between "not set" and "set to default"
type
  MessageDefault {.proto3.} = object
    count {.fieldNumber: 1.}: int32  # Defaults to 0
    text {.fieldNumber: 2.}: string  # Defaults to ""

When you decode a message, you can't tell if count was explicitly set to 0 or just not set at all.

Using Opt[T] (Proto3 or Proto2)

For proto3, the recommended approach is to use Opt[T] from the results library (re-exported via protobuf_serialization/pkg/results). This wraps a value to make it explicitly optional:

# Solution: Use Opt[T] to make fields explicitly optional
type
  MessageOpt {.proto3.} = object
    count {.fieldNumber: 1, ext.}: Opt[int32]
    text {.fieldNumber: 2, ext.}: Opt[string]

Creating Optional Values

Use Opt.some() to create a value that is present, and Opt.none() to create a value that is absent:

# Creating optional values with Opt.some() and Opt.none()
let msg1 = MessageOpt(
  count: Opt.some(42'i32),
  text: Opt.some("hello")
)

let msg2 = MessageOpt(
  count: Opt.none(int32),
  text: Opt.none(string)
)

Checking if a Value is Present

Use isSome() and isNone() to check presence:

# Checking if a value is present
if msg1.count.isSome():
  echo "Count is set to: ", msg1.count.get()
else:
  echo "Count is not set"

Getting the Value

Use get() to retrieve the value:

# Getting the value
let count = msg1.count.get()  # Returns the int32 value
echo "Count: ", count

Use valueOr() to provide a default:

# Providing a default value
let count2 = msg2.count.valueOr(0'i32)  # Returns 0 if not set
echo "Count2: ", count2

Using PBOption (Proto2)

For proto2, use PBOption to make fields explicitly optional:

# For proto2, use PBOption to make fields explicitly optional
type
  Settings {.proto2.} = object
    username {.fieldNumber: 1, required.}: string
    theme {.fieldNumber: 2.}: PBOption[default(string)]
    fontSize {.fieldNumber: 3, pint.}: PBOption[0'i32]
    notifications {.fieldNumber: 4.}: PBOption[false]

Creating Optional Values

Use pbSome() to create a value that is present, and pbNone() to create a value that is absent:

# Creating optional values with pbSome() and pbNone()
let settings1 = Settings(
  username: "alice",
  theme: pbSome("dark"),
  fontSize: pbSome(14'i32),
  notifications: pbSome(true)
)

let settings2 = Settings(
  username: "bob",
  theme: pbSome("light"),
  fontSize: pbNone(0'i32),
  notifications: pbNone(false)
)

Encoding and Decoding

# Encoding and decoding
let encoded = Protobuf.encode(settings2)
let decoded = Protobuf.decode(encoded, Settings)

assert decoded.username == "bob"
assert decoded.theme.isSome
assert decoded.theme.get == "light"
assert decoded.fontSize.isNone
assert decoded.notifications.isNone

Getting the Value

Use get to retrieve the value, and valueOr to provide a default:

# Providing a default value
let fontSize = decoded.fontSize.valueOr(12'i32)
assert fontSize == 12
echo "Font size: ", fontSize

When to Use Each Approach

Use Opt[T] when:

• Working with proto3
• You need to distinguish "not set" from "set to default"
• Building APIs where presence matters

Use PBOption when:

• Working with proto2
• You need to distinguish "not set" from "set to default"
• You need to set a default value other than default(T)

Don't use either when:

• Default values are acceptable
• You don't need presence tracking
• Performance is critical (adds overhead)

Next Steps

• Learn about importing .proto files
• Explore type extensions for custom serialization

Importing .proto Files

Instead of manually annotating Nim types, you can generate them directly from .proto files using the import_proto3 template.

Basic Usage

import protobuf_serialization/proto_parser

# Generate Nim types from a .proto file
import_proto3 "my_protocol.proto3"

This template reads the .proto file at compile time and generates equivalent Nim types.

Example .proto File

person.proto3:

syntax = "proto3";

message Person {
  string name = 1;
  int32 age = 2;
  string email = 3;
}

message AddressBook {
  repeated Person people = 1;
}

Nim code:

import protobuf_serialization
import protobuf_serialization/proto_parser

# This macro generates Person and AddressBook types at compile time
import_proto3 "person.proto3"

# Now you can use the generated types as if they were manually defined
let person = Person(
  name: "Alice",
  age: 30,
  email: "alice@example.com"
)

let encoded = Protobuf.encode(person)
let decoded = Protobuf.decode(encoded, Person)

assert decoded.name == "Alice"
assert decoded.age == 30
assert decoded.email == "alice@example.com"

echo "Import proto basic example passed!"

Supported Features

The proto parser supports most proto3 features:

Messages

message User {
  string username = 1;
  int32 score = 2;
}

Enums

enum Status {
  UNKNOWN = 0;
  ACTIVE = 1;
  INACTIVE = 2;
}

message User {
  string name = 1;
  Status status = 2;
}

Nested Messages

message Outer {
  message Inner {
    string value = 1;
  }
  Inner data = 1;
}

Repeated Fields

message Numbers {
  repeated int32 values = 1;
  repeated string names = 2;
}

Oneof

message Contact {
  oneof contact_info {
    string email = 1;
    string phone = 2;
    string address = 3;
  }
}

Packages

package mypackage;

message Request {
  string query = 1;
}

Field Type Mapping

The parser automatically maps proto types to appropriate Nim types with correct pragmas:

Complete Example

protocol.proto3:

syntax = "proto3";

package example;

enum Priority {
  LOW = 0;
  MEDIUM = 1;
  HIGH = 2;
}

message Task {
  string id = 1;
  string title = 2;
  string description = 3;
  Priority priority = 4;
  repeated string tags = 5;
  bool completed = 6;
}

message TaskList {
  repeated Task tasks = 1;
  string owner = 2;
}

Nim code:

import protobuf_serialization
import protobuf_serialization/proto_parser

# This macro generates Task, TaskList, and Priority types at compile time
import_proto3 "protocol.proto3"

# Use the generated types as if they were manually defined
let task = Task(
  id: "task-001",
  title: "Implement feature",
  description: "Add new functionality",
  priority: Priority.HIGH,
  tags: @["urgent", "backend"],
  completed: false
)

let taskList = TaskList(
  tasks: @[task],
  owner: "alice"
)

# Encode and decode
let encoded = Protobuf.encode(taskList)
let decoded = Protobuf.decode(encoded, TaskList)

assert decoded.tasks[0].title == "Implement feature"
assert decoded.tasks[0].priority == Priority.HIGH
assert decoded.tasks[0].tags == @["urgent", "backend"]
assert decoded.owner == "alice"

echo "Import proto complete example passed!"

File Paths

The .proto file path is relative to the Nim source file:

# If your Nim file is in src/main.nim
# and proto file is in src/protocol.proto3
import_proto3 "protocol.proto3"

# Or use absolute/relative paths
import_proto3 "../protos/protocol.proto3"

Services and RPCs

Services and RPCs are fully parsed from .proto files. The library provides a hook mechanism that allows you to generate custom code for services. This is useful for generating RPC client/server stubs.

The import_proto3 template accepts an optional protoHook parameter that receives the parsed proto definitions and can generate additional Nim code. See the test suite for an example of generating service proc definitions and path constants: tests/test_proto_file.nim.

For a real-world example of using this hook to generate gRPC client/server code, see nim-grpc.

Maps

Maps are supported and are mapped to repeated fields rather than Nim Table types. This is because protobuf spec allows repeated keys, which Table doesn't support. In the future, there may be a way to provide custom type mappings (proto-type → nim-type), but this requires a more general solution for any type.

Other Limitations

• Custom options
• Extensions (proto2 feature)

Next Steps

• Learn about type extensions for custom serialization
• Read the contributor's guide

Type Extensions

Type extensions allow you to define custom serialization logic for types that aren't directly annotated with protobuf pragmas. This is useful when you want to serialize external types or types with custom encoding requirements.

When to Use Type Extensions

Use type extensions when:

• You need to serialize types from external libraries
• You want custom encoding logic (e.g., compression, encryption)
• You're working with types that can't be annotated directly
• You need to serialize wrapper types

Basic Structure

A type extension consists of three procedures:

1. computeFieldSize—calculates the encoded size
2. writeField—encodes the value
3. readFieldInto—decodes the value

Simple Example

Let's create a custom type and make it serializable:

# Custom wrapper type (not annotated with proto2/proto3)
type
  IntWrapper = object
    value: int32

Now we define a message that uses this custom type:

# Message that uses the custom type
type
  Container {.proto3.} = object
    name {.fieldNumber: 1.}: string
    data {.fieldNumber: 2, ext.}: IntWrapper

To make IntWrapper serializable, we need to implement the three required procedures:

# Define type extension for IntWrapper
Protobuf.extensionDefaults(IntWrapper, pint32, defaultSeq = true)

func computeFieldSize(
    field: int,
    value: IntWrapper,
    ProtoType: type ProtobufExt,
    skipDefault: static bool
): int =
  computeFieldSize(field, value.value, pint32, skipDefault)

proc writeField(
    stream: OutputStream,
    field: int,
    value: IntWrapper,
    ProtoType: type ProtobufExt,
    skipDefault: static bool = false
) {.raises: [IOError].} =
  writeField(stream, field, value.value, pint32, skipDefault)

proc readFieldInto(
    stream: InputStream,
    value: var IntWrapper,
    header: FieldHeader,
    ProtoType: type ProtobufExt
): bool {.raises: [SerializationError, IOError].} =
  readFieldInto(stream, value.value, header, pint32)

Using the Extension

Now you can use the custom type in your protobuf messages:

let container = Container(
  name: "Test",
  data: IntWrapper(value: 42)
)

let encoded = Protobuf.encode(container)
let decoded = Protobuf.decode(encoded, Container)

assert decoded.data.value == 42

Extension with Sequences

The extensionDefaults macro can generate default handlers for sequences:

import protobuf_serialization

type
  CustomType = object
    value: string

# Generate defaults for seq[CustomType]
Protobuf.extensionDefaults(CustomType, pstring, defaultSeq = true)

func computeFieldSize(
    field: int,
    value: CustomType,
    ProtoType: type ProtobufExt,
    skipDefault: static bool
): int =
  computeFieldSize(field, value.value, pstring, skipDefault)

proc writeField(
    stream: OutputStream,
    field: int,
    value: CustomType,
    ProtoType: type ProtobufExt,
    skipDefault: static bool = false
) {.raises: [IOError].} =
  writeField(stream, field, value.value, pstring, skipDefault)

proc readFieldInto(
    stream: InputStream,
    value: var CustomType,
    header: FieldHeader,
    ProtoType: type ProtobufExt
): bool {.raises: [SerializationError, IOError].} =
  readFieldInto(stream, value.value, header, pstring)

# Now you can use seq[CustomType]
type
  Container {.proto3.} = object
    items {.fieldNumber: 1, ext.}: seq[CustomType]

let container = Container(
  items: @[
    CustomType(value: "first"),
    CustomType(value: "second"),
    CustomType(value: "third")
  ]
)

let encoded = Protobuf.encode(container)
let decoded = Protobuf.decode(encoded, Container)

assert decoded.items.len == 3
assert decoded.items[0].value == "first"
assert decoded.items[1].value == "second"
assert decoded.items[2].value == "third"

echo "Extension with sequences example passed!"

Complete Example

Here's a complete example with a custom timestamp type:

import protobuf_serialization

# Custom timestamp type
type
  Timestamp = object
    seconds: int64

# Message using the custom type
type
  Event {.proto3.} = object
    name {.fieldNumber: 1.}: string
    timestamp {.fieldNumber: 2, ext.}: Timestamp

# Type extension for Timestamp
Protobuf.extensionDefaults(Timestamp, pint64, defaultSeq = false)

func computeFieldSize(
    field: int,
    value: Timestamp,
    ProtoType: type ProtobufExt,
    skipDefault: static bool
): int =
  computeFieldSize(field, value.seconds, pint64, skipDefault)

proc writeField(
    stream: OutputStream,
    field: int,
    value: Timestamp,
    ProtoType: type ProtobufExt,
    skipDefault: static bool = false
) {.raises: [IOError].} =
  writeField(stream, field, value.seconds, pint64, skipDefault)

proc readFieldInto(
    stream: InputStream,
    value: var Timestamp,
    header: FieldHeader,
    ProtoType: type ProtobufExt
): bool {.raises: [SerializationError, IOError].} =
  readFieldInto(stream, value.seconds, header, pint64)

# Usage
let event = Event(
  name: "UserLogin",
  timestamp: Timestamp(seconds: 1234567890'i64)
)

let encoded = Protobuf.encode(event)
let decoded = Protobuf.decode(encoded, Event)

assert decoded.timestamp.seconds == 1234567890

echo "Complete type extension example passed!"

Key Points

• Mark fields using extensions with {.ext.} pragma
• The extended type itself should NOT be annotated with {.proto2.} or {.proto3.}
• Use extensionDefaults to generate default sequence handlers
• Implement all three procedures: computeFieldSize, writeField, readFieldInto
• Extensions work with both single values and sequences

Next Steps

• Read the contributor's guide to learn how to contribute to the library

Enums

Protobuf enums map naturally to Nim enums. Support for enum fields is not part of the default import—it lives in a separate module that you have to import explicitly.

Importing std/enums

The core protobuf_serialization module does not know how to serialize enum fields. To use them, import protobuf_serialization/std/enums:

import protobuf_serialization
import protobuf_serialization/std/enums

Without this import, the compiler has no serialization handlers for your enum type and the code will not compile.

Enum fields are annotated with the {.ext.} pragma, because enum support is implemented as a type extension:

type
  Status = enum
    Unknown  # ord 0
    Active   # ord 1
    Banned   # ord 2

  Account {.proto3.} = object
    status {.fieldNumber: 1, ext.}: Status

Encoding and decoding then works like any other field:

let account = Account(status: Active)
let encoded = Protobuf.encode(account)
let decoded = Protobuf.decode(encoded, Account)
assert decoded == account

The Zero-Value Requirement

In proto3, an enum must contain a constant that maps to 0. If it does not, compilation fails with a {.fatal.} error. In the example above, Unknown satisfies this requirement because it is the first value and therefore has ordinal 0.

Closed Enum Semantics

Nim enums are closed: a value of an enum type can only be one of the constants defined for that type. There is no way to hold a Status whose value is 3 when only 0, 1, and 2 are defined.

Because protobuf enums are mapped onto Nim enums, they inherit this closed behavior. This is not conformant with proto3's open-enum semantics, where an unknown value is preserved so it can be re-serialized unchanged. In nim-protobuf-protobuf_serialization, unknown values are not stored at all and therefore cannot be serialized back.

If you need proto3-conformant behavior, use an int32 field instead—see Preserving proto3-Compatible Semantics below.

Unknown Values

How an unknown value is handled on decode depends on the syntax:

proto3—the unknown value is dropped and the field keeps its zero value:

# Nim enums are closed: a decoded value that is not part of the enum cannot
# be represented, so it is dropped and the field keeps its zero value.
# The bytes "0803" encode field 1 (tag 08) with the value 3, which is not a
# valid Status. Decoding maps it to the zero value (Unknown) and drops the 3.
block:
  let encoded = "0803".hexToSeqByte
  let decoded = Protobuf.decode(encoded, Account)
  assert decoded.status == Unknown

proto2 with required—the unknown value is a decode error:

type
  RequiredAccount {.proto2.} = object
    status {.fieldNumber: 1, required, ext.}: Status

# In proto2 with a required enum, an unknown value is a decode error.
block:
  let encoded = "0803".hexToSeqByte
  var raised = false
  try:
    discard Protobuf.decode(encoded, RequiredAccount)
  except ProtobufReadError:
    raised = true
  assert raised

Enums With Holes

Enum ordinals do not have to be contiguous, and they may be negative:

# Enum ordinals do not have to be contiguous.
type
  Priority = enum
    Low = -10
    Normal = 0
    High = 10
    Critical  # ord 11

  Task {.proto3.} = object
    priority {.fieldNumber: 1, ext.}: Priority

Optional Enums

In proto2, wrap an enum field in a PBOption to distinguish "not set" from "set to the default value". Use pbSome and pbNone to construct the values:

type
  OptionalAccount {.proto2.} = object
    status {.fieldNumber: 1, ext.}: PBOption[default(Status)]

let present = OptionalAccount(status: pbSome(Active))
assert Protobuf.decode(Protobuf.encode(present), OptionalAccount) == present

# The PBOption parameter is the field's default value, not its type, so
# pbNone takes the default value too: default(Status), i.e. Unknown.
let absent = OptionalAccount(status: pbNone(default(Status)))
assert Protobuf.decode(Protobuf.encode(absent), OptionalAccount) == absent

Repeated & Packed Enums

Repeated enum fields follow the same packing rules as other repeated numeric fields (see Repeated & Packed Fields): proto2 defaults to unpacked, proto3 defaults to packed, and the {.packed.} pragma overrides the default:

type
  # proto2 repeated fields are unpacked by default.
  StatusLogP2 {.proto2.} = object
    statuses {.fieldNumber: 1, ext.}: seq[Status]

  # proto3 repeated numeric/enum fields are packed by default.
  StatusLogP3 {.proto3.} = object
    statuses {.fieldNumber: 1, ext.}: seq[Status]

  # The packed pragma overrides the default in either syntax.
  StatusLogPacked {.proto2.} = object
    statuses {.fieldNumber: 1, ext, packed: true.}: seq[Status]

Unknown entries in a repeated enum field are dropped; the valid entries are kept:

# Unknown entries in a repeated enum field are dropped; valid ones are kept.
# "080008030802" is three unpacked entries: 0, 3 (invalid), 2.
let encoded = "080008030802".hexToSeqByte
let decoded = Protobuf.decode(encoded, StatusLogP2)
assert decoded.statuses == @[Unknown, Banned]

Preserving proto3-Compatible Semantics

Because Nim enums are closed, they cannot preserve unknown values. If you need the proto3-conformant behavior—where unknown values survive a decode/encode round-trip—store the value as an int32 field with the {.pint.} pragma instead of a Nim enum. Named values become plain constants:

# To get proto3-conformant, open-enum behavior, store the value as an int32
# instead of a Nim enum. Named values become constants, and any unknown value
# is preserved on decode instead of being dropped.
const
  StatusUnknown = 0'i32
  StatusActive = 1'i32
  StatusBanned = 2'i32

type
  OpenAccount {.proto3.} = object
    status {.fieldNumber: 1, pint.}: int32

Now an unknown value is preserved rather than dropped:

# The same "0803" bytes now round-trip: the unknown value 3 is preserved.
let encoded = "0803".hexToSeqByte
let decoded = Protobuf.decode(encoded, OpenAccount)
assert decoded.status == 3'i32
assert Protobuf.encode(decoded) == encoded

Compare this with the proto3 unknown-value behavior of a Nim enum, where the value 3 was dropped and replaced with the zero value.

Next Steps

• Learn about repeated and packed fields for arrays
• Explore optional fields with PBOption

Proto Editions

Protobuf editions are the successor to the proto2/proto3 split. Instead of picking a syntax for the whole file, an edition (identified by a year, such as 2023) defines a set of default behaviors, and individual features—like field presence—are configured per field. The goal is smoother, incremental evolution of the format while preserving backward compatibility.

nim-protobuf-serialization exposes editions through the {.proto.} pragma.

The {.proto.} Pragma

Annotate a message with {.proto.} instead of {.proto2.} or {.proto3.}:

# With {.proto.}, presence is chosen per field:
#   - PBExplicit: explicit presence (like proto2 optional)
#   - required:   must be present in the encoded message
#   - implicit:   no presence tracking (like proto3 scalars)
type
  Mixed {.proto.} = object
    a {.fieldNumber: 1, pint.}: PBExplicit[0'i32]
    b {.fieldNumber: 2, pint, required.}: int32
    c {.fieldNumber: 3, pint, implicit.}: int32

Unlike proto2 and proto3, {.proto.} lets you choose presence for each field:

• PBExplicit—explicit presence: the field can distinguish "not set" from "set to the default value" (this is an alias of PBOption).
• required—the field must be present in the encoded message.
• implicit—no presence tracking: the field behaves like a proto3 scalar, where the zero value is indistinguishable from "not set".

Every field in a {.proto.} message must be exactly one of implicit, required, PBExplicit, or a repeated field (seq[T]).

Encoding and decoding work as usual:

let msg = Mixed(a: pbSome(1'i32), b: 7'i32, c: 3'i32)
let encoded = Protobuf.encode(msg)
let decoded = Protobuf.decode(encoded, Mixed)
assert decoded == msg

A required field that is missing from the encoded message is a decode error:

# A required field that is absent from the encoded message is a decode error.
block:
  var raised = false
  try:
    discard Protobuf.decode(default(seq[byte]), Mixed)
  except ProtobufReadError:
    raised = true
  assert raised

Implicit by Default

Adding implicit at the type level makes fields implicit by default, so they no longer need a per-field presence pragma (except required):

# {.proto, implicit.} makes fields implicit by default, so they no longer
# need a per-field presence pragma (except required).
type
  Settings {.proto, implicit.} = object
    host {.fieldNumber: 1.}: string
    port {.fieldNumber: 2, pint.}: int32
    token {.fieldNumber: 3, pint.}: PBExplicit[0'i32]

The Edition Year

The edition parameter selects the protobuf edition the type conforms to. It must be one of the supported editions—2023, 2024, or 2026—and bare {.proto.} defaults to 2023:

# The edition year selects the protobuf edition the type conforms to. It must
# be one of the supported editions (2023, 2024, 2026); bare {.proto.} defaults
# to 2023. The year is validated at compile time but does not currently change
# the wire format, so these three types serialize identically.
type
  Config2023 {.proto: 2023, implicit.} = object
    value {.fieldNumber: 1, pint.}: int32

  Config2024 {.proto: 2024, implicit.} = object
    value {.fieldNumber: 1, pint.}: int32

  Config2026 {.proto: 2026, implicit.} = object
    value {.fieldNumber: 1, pint.}: int32

# All three editions produce the same bytes.
let a = Protobuf.encode(Config2023(value: 5'i32))
let b = Protobuf.encode(Config2024(value: 5'i32))
let c = Protobuf.encode(Config2026(value: 5'i32))
assert a == b
assert b == c

Next Steps

• Learn about optional fields with PBOption
• Read the annotating objects tutorial for proto2 and proto3 basics

Contributing

Thank you for your interest in contributing to nim-protobuf-serialization! This guide will help you get started.

Getting the Source

git clone https://github.com/status-im/nim-protobuf-serialization.git
cd nim-protobuf-serialization

Prerequisites

• Nim >= 1.6.20
• mdBook (for building documentation)

Project Structure

├── protobuf_serialization/       # Library source code
│   ├── codec.nim                 # Core codec implementation
│   ├── extension.nim             # Type extension support
│   ├── format.nim                # Format definition
│   ├── internal.nim              # Internal utilities
│   ├── reader.nim                # Decoding logic
│   ├── writer.nim                # Encoding logic
│   ├── sizer.nim                 # Size computation
│   ├── types.nim                 # Type definitions and pragmas
│   ├── proto_parser.nim          # Proto file parser entry
│   └── files/
│       ├── type_generator.nim    # Code generation from .proto files
│       └── proto_parser.nim      # PEG-based proto parser
├── tests/                        # Test suite
├── book/                         # mdBook documentation source
│   ├── book.toml                 # mdBook configuration
│   └── src/                      # Markdown source files
└── docs/                         # Generated documentation (output)

Running Tests

Run the full test suite:

nimble test

This runs tests with multiple configurations:

• --threads:off and --threads:on
• -d:release and -d:danger
• Address sanitizer (on Linux/amd64 with Nim >= 2.2)

The test suite also includes compile-fail tests in tests/fail/. These are negative tests that verify the library correctly rejects invalid code—for example, using proto2-only features in proto3, or missing required pragmas. The test suite automatically attempts to compile each file in this directory and expects compilation to fail.

Running Individual Tests

nim c -r tests/test_objects.nim
nim c -r tests/test_oneof.nim
nim c -r tests/test_extension.nim

Conformance Tests

Run the official protobuf conformance test suite:

nimble conformance_test

Building Documentation

Build the book documentation:

nimble book

Generate API documentation:

nimble apidocs

To preview the book locally with live reload:

mdbook serve book

Code Style

• Follow the Status Nim Style Guide
• Use {.push raises: [], gcsafe.} at the top of modules

Adding Tests

Tests live in the tests/ directory. Each test file should:

1. Import unittest2 and the library
2. Define test types with protobuf annotations
3. Use the roundtrip helper from tests/utils.nim for encode/decode verification

import unittest2
import ../protobuf_serialization
import ./utils

type
  MyType {.proto3.} = object
    field {.fieldNumber: 1.}: string

suite "My feature":
  test "roundtrip":
    roundtrip(MyType(field: "hello"), "0a0568656c6c6f")

Submitting Changes

1. Create a feature branch from master
2. Make your changes
3. Run the test suite: nimble test and build docs: nimble docs
4. Commit with a clear message
5. Open a pull request

Reporting Issues

Open an issue on GitHub with:

• A clear description of the problem
• Steps to reproduce (if applicable)
• Nim version and platform information
• Minimal code example (if applicable)

License

By contributing, you agree that your contributions will be licensed under the same terms as the project (MIT / Apache 2.0, at your option).