Empirical bundles together a test runner and a web app, as the test reporter. These can be used through the CLI in your terminal window.

Empirical relies on a configuration file, typically located at empiricalrc.js which describes the test to run. This configuration is declarative, which means that you define what you want to test, and Empirical will internally implement the expected behavior.

Required: Node.js 20+ needs to be installed on your system.

​
Start with a basic example

In this example, we will ask an LLM to extract entities from user messages and give us a structured JSON output. For example, “I’m Alice from Maryland” will become {name: 'Alice', location: 'Maryland'}.

Our test will succeed if the model outputs valid JSON.

1

Set up Empirical

Use the CLI to create a sample configuration file in empiricalrc.js.

npm init empiricalrun

# For TypeScript
npm init empiricalrun -- --using-ts
2

Run the test

Run the example dataset against the selected models.

npx empiricalrun

This step requires the OPENAI_API_KEY environment variable to authenticate with OpenAI. This run will cost $0.0026, based on the selected models.

3

See results

Use the ui command to open the reporter web app and see side-by-side results.

npx empiricalrun ui

​
Make it yours

Edit the empiricalrc.js file to make Empirical work for your use-case.

IntroductionExamples