Run tests

Tests can be run via the command line manually or in CI automatically.

Test components


Run your first test to establish baselines

Before we begin, make sure you have an <app-code> by logging to Chromatic and creating a project.

Your first test will setup continuous UI testing for your Storybook. Run the following command in your project directory.

./node_modules/.bin/chromatic test --app-code=<your-app-code>
If you are currently running your Storybook in a process, pass --do-not-start. If you customized the Storybook port or start script you may need to add options to our test command.

After the first test completes, you’ll have new test baselines for each story on this branch. In other words, screenshots of the last know good state. Any future changes to your stories will be compared to the baselines.

Setup Success

Run tests to check for UI changes

Continuous UI testing relies on comparing images of the new rendered UI code to the baseline images. If a UI change is caught you get notified.

Let’s demo how that works. In your project, make a change to the UI and save it (you can always undo later).

Then use the same test command you ran earlier. Add options as needed.

./node_modules/.bin/chromatic test --app-code=<your-app-code>

Follow the link to the Chromatic app where you’ll see the changes.

Build changes

There are changes – perhaps even some you didn’t expect! The component hierarchy enables small changes to snowball into major regressions. Continuous UI testing with Chromatic helps you catch these changes in development –as they happen– instead of during QA or production (5-10x more costly).


Ensure test consistency to prevent false positives

It’s essential that your components and stories render in a consistent fashion between tests to prevent false positives. Some reasons your stories might not render consistently and ways you can avoid this include:

  • Randomness in stories: It’s not uncommon to use random number generators to generate data for complex component inputs. To avoid this, you can hard-code the input data, but often a more convenient solution is to use a tool like seedrandom which you can use to make your “random” number generator consistent.

  • Use of the current date/time: Dates and times are a testers bane! To get consistency in components or tests that use the current time, you can use a tool to also “seed” the time, like timemachine for the Date object.

  • Animations and video: As it is difficult to precisely set the time that a snapshot is taken, Chromatic makes an effort to pause all video and disable animation. However, some times this is not possible (for instance JavaScript driven animations). In such cases, you can use a delay to set a minimum time before screenshotting the component, or if necessary, use an ignore region to stop Chromatic from considering such parts of your component.

  • Unpredictable resource hosts: Resources that load from unpredictable or flaky sources may not load in time (15s) to capture. Workaround this by serving resources as static files in Storybook or using a placeholder service.

  • Skip stories: Some stories may render unpredictably intentionally. If this is the case you may want to ignore the story from testing and move on.


Create an npm script for future test builds

The chromatic test command will also give you the option of adding an npm script to your package.json so you can run future builds with npm run chromatic. If you want to add it manually, it should look something like:

{
  "scripts": {
    "chromatic": "chromatic test"
  }
}

The above script command will pick up your app code by reading the CHROMATIC_APP_CODE environment variable. After adding the above, ensure you set CHROMATIC_APP_CODE when you run builds—such as in your CI config.

If you allowed chromatic to add the above line, it will also have written the environment variable to your package.json. Depending on your level of trust of your code hosting provider, you should consider removing the environment variable and setting via your CI config.


Next: Review workflow

📸You know how to trigger tests! Now catch bugs and update baselines with the review workflow.

Read next chapter


Available options

If you have customized the way your Storybook runs, you may need to pass additional options to the test command.

Option Use case
--app-code The unique code for your app – note you can just pass this via the CHROMATIC_APP_CODE environment variable.
--script-name The npm script that starts your Storybook we should take snapshots against (defaults to storybook). Use this if your Storybook startup script is named differently.
--storybook-port The port that Storybook serves on (autodetected by --script-name, override with this if that fails)
--do-not-start Use this if your Storybook is already running (for instance your Storybook is running in a different terminal already).
--exec Alternatively, start your storybook with an arbitrary command. If you use this option, pass --storybook-port to configure the port Storybook runs on.
--auto-accept-changes If there are any changes to the build, automatically accept them. This is useful in some branching situations. See more in the branching docs.
--exit-zero-on-changes If all snapshots render but there are visual changes, exit with a 0 exit code, rather than the usual 1.
--no-interactive Don’t ask interactive questions about your setup.
--debug Output extra debugging information.
CI=true Tell Chromatic that you’re running in CI. This will hide the “Setup CI / Automation” messages in the UI. Add before the test command like so: CI=true yarn chromatic...

Troubleshooting

Test build failures

A build will fail if any of the snapshots fail to render (i.e. in rendering the latest version of the component, the snapshot throws a JavaScript exception). You’ll need to fix the code for errored components before we can pass the build.

Errored builds

Chromatic builds and runs Storybook flawlessly most of the time, but we’re not perfect (we wish). Sometimes builds don’t run due to the rare infrastructure hiccup. If this happens, try to re-run the build. Rest assured, we keep track of errors and continue to work to improve the service every day.

Timed out

Chromatic takes snapshots very quickly. However, if we lose the connection to your server (for instance if you stop your server mid-build, or your internet connection goes down), builds can time out. Simply restart the build—perhaps with a more stable connection.

Failed to evaluate your stories when running tests

To make a list of Chromatic specs from your Storybook stories, we evaluate your story code from a node script, using JSDOM to simulate a browser environment. We don’t render your stories but just gather a list of them by including your story files. You may need to avoid calling various browser-only constructs at the top-level or mock them out. Pass --debug to the script command to get extra info if it fails.

No Storybook specs found

To get a list of specs from your stories, we evaluate your Storybook with JSDOM. This is a slightly different environment to a normal browser and can sometimes have problems. We will try to output errors if we see them; using the --debug flag to chromatic test may help if we didn’t catch any errors.