Rewards for RL#

Training or testing an AI model on code needs a checker: something that runs every answer the model writes against its test cases and says how well it did, without the model's code reaching anything it should not. POST /api/judge is that checker. Each answer runs in a locked-down box with no internet, and the share of test cases it passes is its reward, from 0 to 1.

A reward function#

import requests

RUSTBOX = "https://api.rustbox.sh/api/judge"
API_KEY = "rb_live_your_key_here"


def reward(code: str, cases: list[dict], language: str = "python") -> float:
    response = requests.post(
        RUSTBOX,
        headers={"X-API-Key": API_KEY},
        json={"language": language, "code": code, "cases": cases},
        timeout=120,
    )
    response.raise_for_status()
    results = response.json()["results"]
    return sum(case["passed"] for case in results) / len(results)


cases = [
    {"stdin": "1 2", "expected": "3"},
    {"stdin": "2 2", "expected": "4"},
]
print(reward("a, b = map(int, input().split())\nprint(a + b)", cases))  # 1.0
print(reward("a, b = map(int, input().split())\nprint(a - b)", cases))  # 0.0

For a trainer that takes reward functions of the form reward(completions, **dataset_columns) -> list[float], map that function over the completions and read each sample's cases from a dataset column.

What a completion cannot do#

A completion that...	Result
Reads the expected output	Impossible: `expected` is compared outside the sandbox and never sent into it
Fetches answers from the internet	Impossible: no network, no loopback, no DNS
Exits early, crashes or hangs	The case does not pass (`WA`, `RE` or `TLE`)
Prints the right answer	Passes: that is the test

Limits#

Limit	Value
Cases per request	25; send larger test sets in several requests
Per case	The per-language limits
Billing	Each case is one run against your plan

Running evals or RL at volume? Talk to us.