Introduction

nim-json-serialization is a library in the nim-serialization family for turning Nim objects into JSON documents and back. Features include:

Efficient coding of JSON documents directly to and from Nim data types
- Full type-based customization of both parsing and formatting
- Flavors for defining multiple JSON serialization styles per Nim type
- Efficient skipping of tags and values for partial JSON parsing
Flexibility in mixing type-based and dynamic JSON access
- Structured JsonValueRef node type for DOM-style access to parsed document
- Flat JsonString type for passing nested JSON documents between abstraction layers
- Seamless interoperability with std/json and JsonNode
Full RFC8259 spec compliance including the notorious JSON number
- Passes JSONTestSuite
- Customizable parser strictness including support for non-standard extensions
Well-defined handling of malformed / malicious inputs with configurable parsing limits
- Fuzzing and comprehensive manual test coverage

Installation

As a nimble dependency:

requires "json_serialization"

Via nimble install:

nimble install json_serialization

Getting started

json_serialization is used to parse JSON documents directly into Nim types and to encode them back as JSON efficiently.

Let's start with a simple JSON-RPC example:

{"jsonrpc": "2.0", "method": "subtract", "params": [42, 3], "id": 1}

Imports and exports

Before we can use json_serialization, we have to import the library.

If you put your custom serialization code in a separate module, make sure to re-export json_serialization:

{.push gcsafe, raises: [].} # Encourage exception handling hygiene in procedures!

import json_serialization
export json_serialization

A common way to organize serialization code is to use a separate module named either after the library (mylibrary_json_serialization) or the flavor (myflavor_json_serialization).

For types that mainly exist to interface with JSON, custom serializers can also be placed together with the type definitions.

Re-exports

When importing a module that contains custom serializers, make sure to re-export it or you might end up with cryptic compiler errors or worse, the default serializers being used!

Simple reader

Looking at the example, we'll define a Nim object to hold the request data, with matching field names and types:

type Request = object
  jsonrpc: string
  `method`: string # Quote Nim keywords
  params: seq[int] # Map JSON array to `seq`
  id: int

Json.decode can now turn our JSON input into a Request:

# Decode the string into our Request type
let decoded = Json.decode(
  """{"jsonrpc": "2.0", "method": "subtract", "params": [42, 3], "id": 1}""", Request
)

echo decoded.id

Replace decode/encode with loadFile/saveFile to read and write a file instead!

Encoding and pretty printing

Having parsed the example with Json.decode, we can pretty-print it back to the console using Json.encode that returns a string:

# Now that we have a `Request` instance, we can pretty-print it:
echo Json.encode(decoded, pretty = true)

Handling errors

Of course, someone might give us some invalid data - json_serialization will raise an exception when that happens:

try:
  # Oops, a string was used for the `id` field!
  discard Json.decode("""{"id": "test"}""", Request)
except JsonError as exc:
  # "<string>" helps identify the source of the document - this can be a
  # filename, URL or something else that helps the user find the error
  echo "Failed to parse document: ", exc.formatMsg("<string>")

The error message points out where things went wrong:

Failed to parse document: <string>(1, 8) number expected

Happy we averted a crisis by adding the forgotten exception handler, we go back to the JSON-RPC specification and notice that strings are actually allowed in the id field - further, the only thing we have to do with id is to pass it back in the response - we don't really care about its contents.

We'll define a helper type to deal with this situation and attach some custom parsing code to it that checks the type. Using JsonString as underlying storage is an easy way to pass around snippets of JSON whose contents we don't need.

The custom code is added to readValue/writeValue procedures that take the stream and our custom type as arguments:

type JsonRpcId = distinct JsonString

proc readValue*(
    r: var JsonReader, val: var JsonRpcId
) {.raises: [IOError, JsonReaderError].} =
  let tok = r.tokKind
  case tok
  of JsonValueKind.Number, JsonValueKind.String, JsonValueKind.Null:
    # Keep the original value without further processing
    val = JsonRpcId(r.parseAsString())
  else:
    r.raiseUnexpectedValue("Invalid RequestId, got " & $tok)

proc writeValue*(w: var JsonWriter, val: JsonRpcId) {.raises: [IOError].} =
  w.writeValue(JsonString(val)) # Preserve the original content

Flavors and strictness

While the defaults that json_serialization offers are sufficient to get started, implementing JSON-based standards often requires more fine-grained control, such as what to do when a field is missing, unknown or has high-level requirements for parsing and formatting.

We use createJsonFlavor to declare the new flavor passing to it the customization options that we're interested in:

createJsonFlavor JrpcSys,
  automaticObjectSerialization = false,
  requireAllFields = true,
  omitOptionalFields = true, # Don't output `none` values when writing
  allowUnknownFields = false

Required and optional fields

In the JSON-RPC example, both the jsonrpc version tag and method are required while parameters and id can be omitted. Our flavor required all fields to be present except those explicitly optional - we use Opt from results to select the optional ones:

type Request = object
  jsonrpc: string
  `method`: string
  params: Opt[seq[int]]
  id: Opt[JsonRpcId]

Automatic object conversion

The default Json flavor allows any object to be converted to JSON. If you define a custom serializer and someone forgets to import it, the compiler might end up using the default instead resulting in a nasty runtime surprise.

automaticObjectSerialization = false forces a compiler error for any type that has not opted in to be serialized:

# Allow serializing the `Request` type - serializing other types will result in
# a compile-time error because `automaticObjectSerialization` is false!
JrpcSys.useDefaultSerializationFor Request

With all that work done, we can finally use our custom flavor to encode and decode the Request:

const json = """{"jsonrpc": "2.0", "method": "subtract", "params": [42, 3], "id": 1}"""

echo JrpcSys.encode(JrpcSys.decode(json, Request))

...almost there!

While we've covered a fair bit of ground already, our Request parser is still not fully standards-compliant - in particular, the list of parameters must be able to handle both positional and named arguments and the values can themselves be full JSON documents that need custom parsing based on the method value.

A more mature JSON-RPC parser can be found in nim-json-rpc which connects the json_serialization library to a DSL that conveniently allows mapping Nim procedures to JSON-RPC methods, featuring automatic parameter conversion and other nice conveniences..

Furtyher examples of how to use json_serialization can be found in the tests folder.

Read that spec!

Not only did we learn to about json_serialization, but also that examples are no substitute for reading the spec!

Reference

Parsing

Common API

JSON parsing uses the common serialization API, supporting both object-based and dynamic JSON documents:

const rawJson = """{"name": "localhost", "port": 42}"""
type
  NimServer = object
    name: string
    port: int

  MixedServer = object
    name: JsonValueRef[uint64]
    port: int

  StringServer = object
    name: JsonString
    port: JsonString

var conf = defaultJsonReaderConf
conf.nestedDepthLimit = 0

let native =
  Json.decode(rawJson, NimServer, flags = defaultJsonReaderFlags, conf = conf)

# decode into native Nim
#let native = Json.decode(rawJson, NimServer)

# decode into mixed Nim + JsonValueRef
let mixed = Json.decode(rawJson, MixedServer)

# decode any value into nested json string
let str = Json.decode(rawJson, StringServer)

# decode any valid JSON, using the `json_serialization` node type
let value = Json.decode(rawJson, JsonValueRef[uint64])

# decode any valid JSON, using the `std/json` node type
let stdjson = Json.decode(rawJson, JsonNode)

# read JSON document from file instead
let file = Json.loadFile("filename.json", NimServer)

Standalone Reader

A reader can be created from any faststreams-compatible stream:

var reader = JsonReader[DefaultFlavor].init(memoryInput(rawJson))
let native2 = reader.readValue(NimServer)

# Overwrite an existing instance
var reader2 = JsonReader[DefaultFlavor].init(memoryInput(rawJson))
var native3: NimServer
reader2.readValue(native3)

Parser options

Parser options allow you to control the strictness and limits of the parser. Set them by passing to Json.decode or when initializing the reader:

let flags = defaultJsonReaderFlags + {allowUnknownFields}

var conf = defaultJsonReaderConf
conf.nestedDepthLimit = 0

let native = Json.decode(
  rawJson, NimServer, flags = flags, conf = conf)

Flavors can be used to override the defaults for some these options.

Flags

Flags control aspects of the parser that are not all part of the JSON standard, but commonly found in the wild:

allowUnknownFields [=off]: Skip unknown fields instead of raising an error.
requireAllFields [=off]: Raise an error if any required field is missing.
escapeHex [=off]: Allow \xHH escape sequences, which are not standard but common in some languages.
relaxedEscape [=off]: Allow escaping any character, not just control characters.
portableInt [=off]: Restrict integers to the safe JavaScript range (-2^53 + 1 to 2^53 - 1).
trailingComma [=on]: Allow trailing commas after the last object member or array element.
allowComments [=on]: Allow C-style comments (//... and /* ... */).
leadingFraction [=on]: Accept numbers like .123, which are not valid JSON but often used.
integerPositiveSign [=on]: Accept numbers like +123, for symmetry with negative numbers.

Limits

Parser limits are passed to decode, similar to flags:

You can adjust these defaults to suit your needs:

nestedDepthLimit [=512]: Maximum nesting depth for objects and arrays (0 = unlimited).
arrayElementsLimit [=0]: Maximum number of array elements (0 = unlimited).
objectMembersLimit [=0]: Maximum number of key-value pairs in an object (0 = unlimited).
integerDigitsLimit [=128]: Maximum digits in the integer part of a number.
fractionDigitsLimit [=128]: Maximum digits in the fractional part of a number.
exponentDigitsLimit [=32]: Maximum digits in the exponent part of a number.
stringLengthLimit [=0]: Maximum string length in bytes (0 = unlimited).

Special types

JsonString: Holds a JSON fragment as a distinct string.
JsonVoid: Skips a valid JSON value.
JsonNumber: Holds a JSON number, including fraction and exponent.
- This is a generic type supporting uint64 and string as parameters.
- The parameter determines the type for the integer and exponent parts.
- If uint64 is used, overflow or digit limits may apply.
- If string is used, only digit limits apply.
- The fraction part is always a string to preserve leading zeros.
JsonValueRef: Holds any valid JSON value, similar to std/json.JsonNode, but uses JsonNumber instead of int or float.

Writing

Common API

Similar to parsing, the common serialization API is used to produce JSON documents.

# Convert object to string
echo Json.encode(native)

# Write JSON to file
Json.saveFile("filename.json", native)

# Pretty-print a tuple
echo Json.encode((x: 4, y: 5), pretty = true)

Standalone Writer

var output = memoryOutput()
var writer = JsonWriter[DefaultFlavor].init(output)
writer.writeValue(native)
echo output.getOutput(string)

Flavors

Flags and limits are runtime configurations, while a flavor is a compile-time mechanism to prevent conflicts between custom serializers for the same type. For example, a JSON-RPC-based API might require that numbers are formatted as hex strings while the same type exposed through REST should use a number.

Flavors ensure the compiler selects the correct serializer for each subsystem. Use useDefaultSerializationIn to assign serializers of a flavor to a specific type.

# Parameters for `createJsonFlavor`:

  FlavorName: untyped
  mimeTypeValue = "application/json"
  automaticObjectSerialization = false
  requireAllFields = true
  omitOptionalFields = true
  allowUnknownFields = true
  skipNullFields = false

type
  OptionalFields = object
    one: Opt[string]
    two: Option[int]

createJsonFlavor OptJson
OptionalFields.useDefaultSerializationIn OptJson

automaticObjectSerialization: By default, all object types are accepted by json_serialization - disable automatic object serialization to only serialize explicitly allowed types
omitOptionalFields: Writer ignores fields with null values.
skipNullFields: Reader ignores fields with null values.

Custom parsers and writers

Parsing and writing can be customized by providing overloads for the readValue and writeValue functions. Overrides are commonly used with a flavor that prevents automatic object serialization, to avoid that some objects use the default serialization, should an import be forgotten.

# Custom serializers for MyType should match the following signatures
proc readValue*(r: var JsonReader, v: var MyType) {.raises: [IOError, SerializationError].}
proc writeValue*(w: var JsonWriter, v: MyType) {.raises: [IOError].}

# When flavors are used, add the flavor as well
proc readValue*(r: var JsonReader[MyFlavor], v: var MyType) {.raises: [IOError, SerializationError].}
proc writeValue*(w: var JsonWriter[MyFlavor], v: MyType) {.raises: [IOError].}

The JsonReader provides access to the JSON token stream coming out of the lexer. While the token stream can be accessed directly, there are several helpers that make it easier to correctly parse common JSON shapes.

Objects

Decode objects using the parseObject template. To parse values, use helper functions or readValue. The readObject and readObjectFields iterators are also useful for custom object parsers.

proc readValue*(r: var JsonReader, table: var Table[string, int]) =
  parseObject(r, key):
    table[key] = r.parseInt(int)

Sets and List-like Types

Sets and list/array-like structures can be parsed using the parseArray template, which supports both indexed and non-indexed forms.

Built-in readValue implementations exist for regular seq and array. For set or set-like types, you must provide your own implementation.

type
  HoldArray = object
    data: array[3, int]

  HoldSeq = object
    data: seq[int]

  WelderFlag = enum
    TIG
    MIG
    MMA

  Welder = object
    flags: set[WelderFlag]

proc readValue*(r: var JsonReader, value: var HoldArray) =
  # parseArray with index, `i` can be any valid identifier
  r.parseArray(i):
    value.data[i] = r.parseInt(int)

proc readValue*(r: var JsonReader, value: var HoldSeq) =
  # parseArray without index
  r.parseArray:
    let lastPos = value.data.len
    value.data.setLen(lastPos + 1)
    readValue(r, value.data[lastPos])

proc readValue*(r: var JsonReader, value: var Welder) =
  # populating set also okay
  r.parseArray:
    value.flags.incl r.parseInt(int).WelderFlag

Custom Iterators

Custom iterators provide access to sub-token elements:

customIntValueIt(r: var JsonReader; body: untyped)
customNumberValueIt(r: var JsonReader; body: untyped)
customStringValueIt(r: var JsonReader; limit: untyped; body: untyped)
customStringValueIt(r: var JsonReader; body: untyped)

Convenience Iterators

readArray(r: var JsonReader, ElemType: typedesc): ElemType
readObjectFields(r: var JsonReader, KeyType: type): KeyType
readObjectFields(r: var JsonReader): string
readObject(r: var JsonReader, KeyType: type, ValueType: type): (KeyType, ValueType)

Helper Procedures

When writing a custom serializer, use these safe and intuitive parsers. Avoid using the lexer directly.

tokKind(r: var JsonReader): JsonValueKind
parseString(r: var JsonReader, limit: int): string
parseString(r: var JsonReader): string
parseBool(r: var JsonReader): bool
parseNull(r: var JsonReader)
parseNumber(r: var JsonReader, T: type): JsonNumber[T: string or uint64]
parseNumber(r: var JsonReader, val: var JsonNumber)
toInt(r: var JsonReader, val: JsonNumber, T: type SomeInteger, portable: bool): T
parseInt(r: var JsonReader, T: type SomeInteger, portable: bool = false): T
toFloat(r: var JsonReader, val: JsonNumber, T: type SomeFloat): T
parseFloat(r: var JsonReader, T: type SomeFloat): T
parseAsString(r: var JsonReader, val: var string)
parseAsString(r: var JsonReader): JsonString
parseValue(r: var JsonReader, T: type): JsonValueRef[T: string or uint64]
parseValue(r: var JsonReader, val: var JsonValueRef)
parseArray(r: var JsonReader; body: untyped)
parseArray(r: var JsonReader; idx: untyped; body: untyped)
parseObject(r: var JsonReader, key: untyped, body: untyped)
parseObjectWithoutSkip(r: var JsonReader, key: untyped, body: untyped)
parseObjectSkipNullFields(r: var JsonReader, key: untyped, body: untyped)
parseObjectCustomKey(r: var JsonReader, keyAction: untyped, body: untyped)
parseJsonNode(r: var JsonReader): JsonNode
skipSingleJsValue(r: var JsonReader)
readRecordValue[T](r: var JsonReader, value: var T)

JsonWriter Helper Procedures

beginRecord(w: var JsonWriter, T: type)
beginRecord(w: var JsonWriter)
endRecord(w: var JsonWriter)

writeObject(w: var JsonWriter, T: type)
writeObject(w: var JsonWriter)

writeFieldName(w: var JsonWriter, name: string)
writeField(w: var JsonWriter, name: string, value: auto)

iterator stepwiseArrayCreation[C](w: var JsonWriter, collection: C): auto
writeIterable(w: var JsonWriter, collection: auto)
writeArray[T](w: var JsonWriter, elements: openArray[T])

writeNumber[F,T](w: var JsonWriter[F], value: JsonNumber[T])
writeJsonValueRef[F,T](w: var JsonWriter[F], value: JsonValueRef[T])

Enums

type
  Fruit = enum
    Apple = "Apple"
    Banana = "Banana"

  Drawer = enum
    One
    Two

  Number = enum
    Three = 3
    Four = 4

  Mixed = enum
    Six = 6
    Seven = "Seven"

nim-json-serialization automatically detects the expected representation for each enum based on its declaration.

Fruit expects string literals.
Drawer and Number expect numeric literals.
Mixed (with both string and numeric values) is disallowed by default. If the JSON literal does not match the expected style, an exception is raised. You can configure individual enum types:

configureJsonDeserialization(
    T: type[enum], allowNumericRepr: static[bool] = false,
    stringNormalizer: static[proc(s: string): string] = strictNormalize)

# Example:
Mixed.configureJsonDeserialization(allowNumericRepr = true) # Only at top level

You can also configure enum encoding at the flavor or type level:

type
  EnumRepresentation* = enum
    EnumAsString
    EnumAsNumber
    EnumAsStringifiedNumber

# Examples:

# Flavor level
Json.flavorEnumRep(EnumAsString)   # Default flavor, can be called from non-top level
Flavor.flavorEnumRep(EnumAsNumber) # Custom flavor, can be called from non-top level

# Individual enum type, regardless of flavor
Fruit.configureJsonSerialization(EnumAsNumber) # Only at top level

# Individual enum type for a specific flavor
MyJson.flavorEnumRep(Drawer, EnumAsString) # Only at top level

Updating this book

This book is built using mdBook, which in turn requires a recent version of rust and cargo installed.

# Install correct versions of tooling
nimble mdbook

# Run a local mdbook server
mdbook serve docs

A CI job automatically published the book to GitHub Pages.

Keyboard shortcuts

json_serialization