unittest2

Search:
Group by:
Source   Edit  

Authors:Zahary Karadjov, Ștefan Talpalaru, Status Research and Development

This module makes unit testing easy.

nim c -r testfile.nim

exits with 0 or 1.

Running individual tests

Specify the test names as command line arguments.

nim c -r test "my test name" "another test"

Multiple arguments can be used.

Running a single test suite

Specify the suite name delimited by "::".

nim c -r test "my suite name::"

Selecting tests by pattern

A single "*" can be used for globbing.

Delimit the end of a suite name with "::".

Tests matching any of the arguments are executed.

nim c -r test fast_suite::mytest1 fast_suite::mytest2
nim c -r test "fast_suite::mytest*"
nim c -r test "auth*::" "crypto::hashing*"
# Run suites starting with 'bug #' and standalone tests starting with '#'
nim c -r test 'bug #*::' '::#*'

Command line arguments

The unit test runner recognises serveral parameters that can be specified either via environment or command line, the latter taking precedence.

Several options also have defaults that can be controlled at compile-time.

--help
Print short help and quit
--xml:file
Write JUnit-compatible XML report to file
--console
Write report to the console (default, when no other output is selected)
--output-lvl:level Verbosity of output [COMPACT, VERBOSE, FAILURES, NONE] (env: UNITTEST2_OUTPUT_LVL)
--verbose, -v
Shorthand for --output-lvl:VERBOSE

Command line parsing can be disabled with -d:unittest2DisableParamFiltering.

Running tests in parallel

Early versions of this library had rudimentary support for running tests in parallel - this has since been removed due to safety issues in the implementation and may be reintroduced at a future date.

Example

suite "description for this stuff":
  echo "suite setup: run once before the tests"
  
  setup:
    echo "run before each test"
  
  teardown:
    echo "run after each test"
  
  test "essential truths":
    # give up and stop if this fails
    require(true)
  
  test "slightly less obvious stuff":
    # print a nasty message and move on, skipping
    # the remainder of this block
    check(1 != 1)
    check("asd"[2] == 'd')
  
  test "out of bounds error is thrown on bad access":
    let v = @[1, 2, 3]  # you can do initialization here
    expect(IndexError):
      discard v[4]
  
  suiteTeardown:
    echo "suite teardown: run once after the tests"

Types

ConsoleOutputFormatter = ref object of OutputFormatter
  ## Have test results printed in color.
  ## Default is `auto` depending on `isatty(stdout)`, or override it with
  ## `-d:nimUnittestColor:auto|on|off`.
  ## 
  ## Deprecated: Setting the environment variable `NIMTEST_COLOR` to `always`
  ## or `never` changes the default for the non-js target to true or false respectively.
  ## Deprecated: the environment variable `NIMTEST_NO_COLOR`, when set, changes the
  ## default to true, if `NIMTEST_COLOR` is undefined.
  ## Set the verbosity of test results.
  ## Default is `VERBOSE`, or override with:
  ## `-d:nimUnittestOutputLevel:VERBOSE|FAILURES|NONE`.
  ## 
  ## Deprecated: the `NIMTEST_OUTPUT_LVL` environment variable is set for the non-js target.
  when collect:
Source   Edit   
OutputFormatter = ref object of RootObj
Source   Edit   
OutputLevel = enum
  VERBOSE,                  ## Print as much as possible.
  COMPACT,                  ## Print failures and compact success information
  FAILURES,                 ## Print only failures
  NONE                       ## Print nothing.
The output verbosity of the tests. Source   Edit   
TestResult = object
  suiteName*: string         ## Name of the test suite that contains this test case.
  testName*: string          ## Name of the test case
  status*: TestStatus
  duration*: Duration
  output*: string
  errors*: string
Source   Edit   
TestStatus = enum
  OK, FAILED, SKIPPED
The status of a test when it is done. Source   Edit  

Vars

abortOnError {.threadvar.}: bool
Set to true in order to quit immediately on fail. Default is false, or override with -d:nimUnittestAbortOnError:on|off. Source   Edit  

Consts

unittest2Static {.booldefine.} = false
Run tests at compile time as well - only a subset of functionality is enabled at compile-time meaning that tests must be written conservatively. suite features (setup etc) in particular are not supported. Source   Edit  

Procs

proc addOutputFormatter(formatter: OutputFormatter) {....raises: [], tags: [],
    forbids: [].}
Source   Edit   
proc defaultConsoleFormatter(): ConsoleOutputFormatter {....raises: [],
    tags: [ReadEnvEffect], forbids: [].}
Source   Edit   
proc disableParamFiltering() {....deprecated: "Compile with -d:unittest2DisableParamFiltering instead",
                               raises: [], tags: [], forbids: [].}
Deprecated: Compile with -d:unittest2DisableParamFiltering instead
Source   Edit   
proc newConsoleOutputFormatter(outputLevel: OutputLevel = outputLevelDefault;
                               colorOutput = true): ConsoleOutputFormatter {.
    ...raises: [], tags: [], forbids: [].}
Source   Edit   
proc newJUnitOutputFormatter(stream: Stream): JUnitOutputFormatter {....raises: [],
    tags: [WriteIOEffect], forbids: [].}
Creates a formatter that writes report to the specified stream in JUnit format. The stream is NOT closed automatically when the test are finished, because the formatter has no way to know when all tests are finished. You should invoke formatter.close() to finalize the report. Source   Edit   
proc parseParameters(args: openArray[string]) {....raises: [],
    tags: [ReadEnvEffect, WriteIOEffect], forbids: [].}
Source   Edit   
proc resetOutputFormatters() {....raises: [], tags: [], forbids: [].}
Source   Edit  

Methods

method failureOccurred(formatter: ConsoleOutputFormatter;
                       checkpoints: seq[string]; stackTrace: string) {.
    ...raises: [], tags: [], forbids: [].}
Source   Edit   
method failureOccurred(formatter: JUnitOutputFormatter;
                       checkpoints: seq[string]; stackTrace: string) {.
    ...raises: [], tags: [], forbids: [].}
stackTrace is provided only if the failure occurred due to an exception. checkpoints is never nil. Source   Edit   
method failureOccurred(formatter: OutputFormatter; checkpoints: seq[string];
                       stackTrace: string) {.base, ...gcsafe, raises: [], tags: [],
    forbids: [].}
stackTrace is provided only if the failure occurred due to an exception. checkpoints is never nil. Source   Edit   
method suiteEnded(formatter: ConsoleOutputFormatter) {....raises: [],
    tags: [WriteIOEffect, ReadIOEffect], forbids: [].}
Source   Edit   
method suiteEnded(formatter: JUnitOutputFormatter) {....raises: [], tags: [],
    forbids: [].}
Source   Edit   
method suiteEnded(formatter: OutputFormatter) {.base, ...gcsafe, raises: [],
    tags: [], forbids: [].}
Source   Edit   
method suiteStarted(formatter: ConsoleOutputFormatter; suiteName: string) {.
    ...raises: [], tags: [WriteIOEffect], forbids: [].}
Source   Edit   
method suiteStarted(formatter: JUnitOutputFormatter; suiteName: string) {.
    ...raises: [], tags: [], forbids: [].}
Source   Edit   
method suiteStarted(formatter: OutputFormatter; suiteName: string) {.base,
    ...gcsafe, raises: [], tags: [], forbids: [].}
Source   Edit   
method testEnded(formatter: ConsoleOutputFormatter; testResult: TestResult) {.
    ...raises: [], tags: [ReadIOEffect, WriteIOEffect], forbids: [].}
Source   Edit   
method testEnded(formatter: JUnitOutputFormatter; testResult: TestResult) {.
    ...raises: [], tags: [], forbids: [].}
Source   Edit   
method testEnded(formatter: OutputFormatter; testResult: TestResult) {.base,
    ...gcsafe, raises: [], tags: [], forbids: [].}
Source   Edit   
method testRunEnded(formatter: ConsoleOutputFormatter) {....raises: [],
    tags: [WriteIOEffect], forbids: [].}
Source   Edit   
method testRunEnded(formatter: JUnitOutputFormatter) {....raises: [],
    tags: [WriteIOEffect], forbids: [].}
Completes the report and closes the underlying stream. Source   Edit   
method testRunEnded(formatter: OutputFormatter) {.base, ...gcsafe, raises: [],
    tags: [], forbids: [].}
Source   Edit   
method testStarted(formatter: ConsoleOutputFormatter; testName: string) {.
    ...raises: [], tags: [WriteIOEffect], forbids: [].}
Source   Edit   
method testStarted(formatter: JUnitOutputFormatter; testName: string) {.
    ...raises: [], tags: [], forbids: [].}
Source   Edit   
method testStarted(formatter: OutputFormatter; testName: string) {.base, ...gcsafe,
    raises: [], tags: [], forbids: [].}
Source   Edit  

Macros

macro check(conditions: untyped): untyped
Verify if a statement or a list of statements is true. A helpful error message and set checkpoints are printed out on failure (if outputLevel is not NONE).

Example:

import std/strutils

check("AKB48".toLowerAscii() == "akb48")

let teams = {'A', 'K', 'B', '4', '8'}

check:
  "AKB48".toLowerAscii() == "akb48"
  'C' notin teams
Source   Edit   
macro expect(exceptions: varargs[typed]; body: untyped): untyped
Test if body raises an exception found in the passed exceptions. The test passes if the raised exception is part of the acceptable exceptions. Otherwise, it fails.

Example:

import std/[math, random, strutils]
proc defectiveRobot() =
  randomize()
  case rand(1..4)
  of 1: raise newException(OSError, "CANNOT COMPUTE!")
  of 2: discard parseInt("Hello World!")
  of 3: raise newException(IOError, "I can't do that Dave.")
  else: assert 2 + 2 == 5

expect IOError, OSError, ValueError, AssertionDefect:
  defectiveRobot()
Source   Edit  

Templates

template checkpoint(msg: string) {..}
Set a checkpoint identified by msg. Upon test failure all checkpoints encountered so far are printed out. Example:
checkpoint("Checkpoint A")
check((42, "the Answer to life and everything") == (1, "a"))
checkpoint("Checkpoint B")

outputs "Checkpoint A" once it fails.

Source   Edit   
template dualTest(nameParam: string; body: untyped) {..}
Similar to test but run the test both compuletime and run time, no matter the unittest2Static flag Source   Edit   
template fail() {..}
Print out the checkpoints encountered so far and quit if abortOnError is true. Otherwise, erase the checkpoints and indicate the test has failed (change exit code and test status). This template is useful for debugging, but is otherwise mostly used internally. Example:
checkpoint("Checkpoint A")
complicatedProcInThread()
fail()

outputs "Checkpoint A" before quitting.

Source   Edit   
template require(conditions: untyped)
Same as check except any failed test causes the program to quit immediately. Any teardown statements are not executed and the failed test output is not generated. Source   Edit   
template runtimeTest(nameParam: string; body: untyped) {..}
Similar to test but runs only at run time, no matter the unittest2Static setting Source   Edit   
template skip() {..}
Mark the test as skipped. Should be used directly in case when it is not possible to perform test for reasons depending on outer environment, or certain application logic conditions or configurations. The test code is still executed.
if not isGLContextCreated():
  skip()
Source   Edit   
template staticTest(nameParam: string; body: untyped) {..}
Similar to test but runs only at compiletime, no matter the unittest2Static flag Source   Edit   
template suite(nameParam: string; body: untyped) {.dirty.}

Declare a test suite identified by name with optional setup and/or teardown section.

A test suite is a series of one or more related tests sharing a common fixture (setup, teardown). The fixture is executed for EACH test.

suite "test suite for addition":
  setup:
    let result = 4
  
  test "2 + 2 = 4":
    check(2+2 == result)
  
  test "(2 + -2) != 4":
    check(2 + -2 != result)
  
  # No teardown needed

The suite will run the individual test cases in the order in which they were listed. With default global settings the above code prints:

[Suite] test suite for addition
  [OK] 2 + 2 = 4
  [OK] (2 + -2) != 4
Source   Edit   
template test(nameParam: string; body: untyped) {..}
Define a single test case identified by name.
test "roses are red":
  let roses = "red"
  check(roses == "red")

The above code outputs:

[OK] roses are red
Source   Edit  

Made with Nim. Generated: 2024-10-07 19:38:01 UTC