Configuration file

Empirical uses a JSON configuration file, called empiricalrc.json, to describe the test to run. This configuration is declarative, in the sense that you define what you want to test, and Empirical will internally implement the expected behavior.

​

Configuration reference

The empiricalrc.json configuration file has two high-level properties:

• runs: Use this to define model providers and scoring functions
• dataset: Use this to define the scenarios to test

​

Code editor integration

Your code editor can give you auto-completions and detect linting errors for this configuration file. This uses a JSON Schema definition which is hosted by Empirical.

There are two ways to configure the schema definition.

​

$schema property

Use the $schema property in the configuration file to specify the JSON schema URL.

{
  "$schema": "https://assets.empirical.run/config/schema/latest.json",
  "runs": [
    // ...
  ],
  "dataset": {
    // ...
  }
}

The above URL will always point to the JSON schema for the latest CLI version. If you want to specify JSON schema for a particular CLI version, you can replace “latest” with the version number.

{
  // https://assets.empirical.run/config/schema/v{CLI_VERSION_NUMBER}.json
  "$schema": "https://assets.empirical.run/config/schema/v0.4.0.json", 
  "runs": [
    // ...
  ],
  "dataset": {
    // ...
  }
}

The easiest way to detect the CLI version is by running npx empiricalrun -V which will print the currently installed version number. You can then use that version number in the schema URL.

​

Visual Studio Code

Add the json.schemas property to your VS Code configuration (user or workspace). This maps the empiricalrc.json file to use the JSON schema.

{
  "json.schemas": [
    {
      "fileMatch": [
        "empiricalrc.json"
      ],
      "url": "https://assets.empirical.run/config/schema/latest.json"
    }
  ]
}