Custom Resolver

The resolver is the script that runs inside the sandbox container: it loads the student's submission, runs the tests, and writes the results. Each language ships a standard one — but a problem can override it by adding a _resolve.<ext> file (_resolve.py, _resolve.cs).

Use an override when the defaults don't fit: float tolerance, order-insensitive comparison, grading stdout, multi-step scenarios. For ordinary problems, static or dynamic tests are all you need.

You take over completely

When _resolve.<ext> is present, the standard resolver is not included. Your script is the whole test run: it must load the student's code, run the cases, and write _output.json itself.

What your script gets

Inside the container, the working directory holds:

File	Contents
userCode.py / UserCode.cs	The student's submission. In C# the student's class is Solution
_solution.<ext>, _generate_inputs.<ext>, _expected_outputs.json	The problem's scaffold files, whichever exist
env var funcName	The problem's function name

The container has no network and runs under the platform's resource limits.

What your script must write — _output.json

The file is returned to the student exactly as you write it — there is no reshaping step. The schema:

{
  "results": [
    { "runTime": 0.0001, "correct": true, "input": [1, 2],
      "output": 3, "expected": 3, "stdout": "", "stderr": null }
  ],
  "correct": 1.0,
  "err": null
}

Field	Type	Meaning
results	array	One object per case, in input order
correct	float	Pass ratio across cases, 0.0–1.0 — you compute it
err	string | null	Top-level failure (e.g. the submission failed to load); else null

Each entry in results:

Field	Type	Meaning
runTime	float	Seconds spent in the call
correct	bool	Did this case pass — by your definition
input	array	The arguments passed
output	any	What the student's function returned
expected	any	What you expected
stdout	string	Anything the case printed
stderr	string | null	Exception message, else null

On any uncaught error, still write the file — {"results": [], "correct": 0.0, "err": "<message>"} — rather than crashing. If _output.json is missing entirely, the platform falls back to returning the container's logs as the error.

Worked example — float tolerance

_resolve.py for a problem whose answers are floats:

import json
from os import getenv
from time import time

import userCode

func = getattr(userCode, getenv("funcName", "main"))
cases = [([2.0], 1.4142135), ([9.0], 3.0)]   # or generate them here

results = []
for args, expected in cases:
    before = time()
    try:
        output, stderr = func(*args), None
    except Exception as error:
        output, stderr = None, str(error)
    results.append({
        "runTime": time() - before,
        "correct": stderr is None and abs(output - expected) < 1e-6,
        "input": list(args),
        "output": output,
        "expected": expected,
        "stdout": "",
        "stderr": stderr,
    })

passed = sum(1 for case in results if case["correct"])
with open("_output.json", "w") as file:
    json.dump({"results": results,
               "correct": passed / len(results) if results else 0.0,
               "err": None}, file)

The only line that differs from the standard resolver is the correct comparison — keep the rest of the shape intact and the student's results view works unchanged.

The standard resolvers are good starting points: resolvers/python/resolve.py and resolvers/csharp/resolve.cs.