Dynamic Test Cases

Dynamic tests generate fresh inputs on every run: you ship a reference solution and an input generator, and the runner computes the expected outputs itself. Students can't hardcode answers, and you never hand-maintain a JSON file.

Two files, both scaffold files on the problem:

File	Role
_solution.<ext>	The reference implementation — must define the problem's function name
_generate_inputs.<ext>	Exposes generate_inputs() taking no arguments, returning a list of argument tuples

When both are present they take priority over _expected_outputs.json. On each submission the runner:

1. Calls generate_inputs() → a list of argument lists
2. Runs the reference on each → the expected outputs
3. Runs the student's code on the same inputs and compares

Python example

Problem: add(a, b), function name add.

_solution.py:

def add(a, b):
    return a + b

_generate_inputs.py:

import random

def generate_inputs():
    return [(random.randint(-100, 100), random.randint(-100, 100)) for _ in range(8)]

Each entry from generate_inputs() is unpacked as the call's arguments: (3, -7) becomes add(3, -7).

C# example

All C# files compile into one assembly, so each role needs its own class name — this is the part people miss:

Role	Class	Method
Student code	Solution	the function name
Reference solution (_solution.cs)	Reference	the function name
Generator (_generate_inputs.cs)	Generator	public static IEnumerable<object[]> generate_inputs()

_solution.cs:

class Reference
{
    public int add(int a, int b) => a + b;
}

_generate_inputs.cs:

class Generator
{
    public static IEnumerable<object[]> generate_inputs()
    {
        var random = new Random();
        for (int i = 0; i < 8; i++)
            yield return new object[] { random.Next(-100, 100), random.Next(-100, 100) };
    }
}

Tips

• Keep the generator fast and bounded — it runs inside the same sandboxed container as the student's code, under the same resource limits.
• Generate a spread: small values, edge values (0, negatives, empty collections), and a couple of large ones.
• Make the reference boring. It defines correctness; cleverness belongs in the student's solution, not yours.
• A wrong reference grades every student wrong — submit the placeholder yourself once to sanity-check the problem end to end.