JSON

For simpler configurations, you can use a JSON file to configure your tests, instead of JavaScript or TypeScript.

To set this up, create the config file at empiricalrc.json which describes the test to run.

​

Configuration reference

{
  "runs": [
    // Define model providers to run against
  ],
  "dataset": {
    // Define dataset with scenarios to test
  },
  "scorers": [
    // Define scoring functions that evaluate model outputs
  ]
}

The empiricalrc.json configuration file has three high-level properties:

• runs: Use this to define model providers
• dataset: Use this to define the scenarios to test
• scorers: Use this to define scoring functions

​

Editor helpers

Your code editor can give you auto-completions and detect linting errors for this configuration file. This uses a JSON Schema definition which is hosted by Empirical.

There are two ways to configure the schema definition.

​

$schema property

Use the $schema property in the configuration file to specify the JSON schema URL.

{
  "$schema": "https://assets.empirical.run/config/schema/latest.json",
  "runs": [
    // ...
  ],
  "dataset": {
    // ...
  }
}

The above URL will always point to the JSON schema for the latest CLI version. If you want to specify JSON schema for a particular CLI version, you can replace “latest” with the version number.

{
  // https://assets.empirical.run/config/schema/v{CLI_VERSION_NUMBER}.json
  "$schema": "https://assets.empirical.run/config/schema/v0.4.0.json", 
  "runs": [
    // ...
  ],
  "dataset": {
    // ...
  }
}

The easiest way to detect the CLI version is by running npx empiricalrun -V which will print the currently installed version number. You can then use that version number in the schema URL.

​

Visual Studio Code

Add the json.schemas property to your VS Code configuration (user or workspace). This maps the empiricalrc.json file to use the JSON schema.

{
  "json.schemas": [
    {
      "fileMatch": [
        "empiricalrc.json"
      ],
      "url": "https://assets.empirical.run/config/schema/latest.json"
    }
  ]
}