Mattermost Logo
We're hiring!
Edit on GitHub

How to Run E2E Tests

Environment Variables 

We use several environment variables for Cypress testing in order to:

  • Easily change when running in CI.
  • Cater to different values across developer machines.

Environment variables are defined in cypress.json under the env key. In most cases you don’t need to change the values, because it makes use of the default local developer setup. If you do need to make changes, the easiest method is to override by exporting CYPRESS_*, where * is the key of the variable, for example: CYPRESS_adminUsername. See Cypress documentation for details.

Variable Description
CYPRESS_adminUsername Admin’s username for the test server.

Default: sysadmin when server is seeded by make test-data.
CYPRESS_adminPassword Admin’s password for the test server.

Default: Sys@dmin-sample1 when server is seeded by make test-data.
CYPRESS_dbClient The database of the test server. It should match the server config SqlSettings.DriverName.

Default: postgres
Valid values: postgres or mysql
CYPRESS_dbConnection The database connection string of the test server. It should match the server config SqlSettings.DataSource.

Default: "postgres://mmuser:mostest@localhost/mattermost_test?sslmode=disable\u0026connect_timeout=10"
CYPRESS_enableVisualTest Use for visual regression testing.

Default: false
Valid values: true or false
CYPRESS_ldapServer Host of LDAP server.

Default: localhost
CYPRESS_ldapPort Port of LDAP server.

Default: 389
CYPRESS_runLDAPSync Option to run LDAP sync.

Default: true
Valid values: true or false
CYPRESS_resetBeforeTest When set to true, it deletes all teams and their channels where sysadmin is a member except eligendi team and its channels.

Default: false
Valid values: true or false
CYPRESS_storybookUrl Host for common components or widget testing.

Default: http://localhost:6006/ when initiated npm run storybook from the root folder.
CYPRESS_webhookBaseUrl A server used for testing webhook integration.

Default: http://localhost:3000 when initiated npm run start:webhook.

On Your Local Development Machine 

  1. Launch a local Mattermost instance by running make run in the mattermost-server directory. Confirm that the Mattermost instance has started successfully.
    • Run make test-data to preload your server instance with initial seed data.
    • Each test case should handle the required system or user settings, but in case you encounter an unexpected error while testing, you may want to run the server with default config based on mattermost-server/config/default.json.
    • In another terminal, in the mattermost-webapp directory, run npm run storybook. This is required for the Cypress tests which run against the Storybook widgets to pass.
  2. Change directory to mattermost-webapp/e2e, and install npm dependencies by running npm i.
    • Initiate npm run cypress:run in the command line to start full E2E testing. This excludes the specs in the /cypress/integration/enterprise folder as they require an Enterprise license to run successfully.
    • Initiate npm run cypress:open in the command line to start partial testing depending on which spec is selected via Cypress’s desktop app.
    • Initiate via node CLI to selectively run specs based on test metadata, e.g. node run_tests.js --group='@accessibility' which will run all specs with @accessibility metadata.
  3. Tests are executed according to your selection and will display whether the tests passed or failed.

In Continuous Integration Pipeline 

We run all tests in our Continuous Integration (CI) pipeline. However, they are grouped according to test stability.

  1. Daily production tests against development branch (master) - Initiated on master branch by node run_tests.js --stage='@prod' in the command line. These are tests, known as production tests, which were selected and labeled with @prod test metadata. See link for an example test run posted in our community channel.
  2. Daily production tests against release branch - Same as above except the test is initiated against the release branch. See link for an example test run.
  3. Daily unstable tests against development branch (master) - Initiated on the master branch by node run_tests.js --stage='@prod' --invert in the command line to run all except production tests. We call these unstable tests as they either consistently or intermittently fail due to automation bugs and not product bugs.