A flaky test is one that exhibits both passing and failing results when run multiple times without any code changes. When our automation detects a flaky test on your PR:
Check if the Test is Newly Introduced
For Existing Flaky Tests
Create a JIRA ticket titled “Flaky Test: {TestName}”, e.g. “Flaky Test: TestGetMattermostLog”
Copy the test failure message into the JIRA ticket description
Add the flaky-test
and triage-global
labels
Create a PR to skip the test by adding:
t.Skip("https://mattermost.atlassian.net/browse/MM-XXXXX")
where MM-XXXXX is your JIRA ticket number
Link the JIRA ticket in the skip message for tracking
This process helps us track and systematically address flaky tests while preventing them from blocking development work.
Leveraging parallel tests can drastically reduce execution time for entire test packages, such as api4
and app
, which are notably heavy with hundreds of tests. However, careful implementation is essential to ensure reliability and prevent flakiness. Follow these guidelines when writing parallel tests:
In api4
, app
, platform
, email
, jobs
packages:
func TestExample(t *testing.T) {
mainHelper.Parallel(t)
...
}
// OR
func TestExample(t *testing.T) {
th := Setup(t)
th.Parallel(t)
...
}
// OR
func TestExample(t *testing.T) {
if mainHelper.Options.RunParallel {
t.Parallel()
}
...
}
If sqlstore
package:
func TestExample(t *testing.T) {
if enableFullyParallelTests {
t.Parallel()
}
...
}
To enable parallel execution, you should set the ENABLE_FULLY_PARALLEL_TESTS
environment variable. Example:
ENABLE_FULLY_PARALLEL_TESTS=true go test -v ./api4/...
Avoid reliance on global variables and registrations such as:
LicenseValidator
platform.RegisterMetricsInterface
platform.PurgeLinkCache
model.BuildEnterpriseReady
jobs.DefaultWatcherPollingInterval
Avoid using os.Chdir
(or t.Chdir
) and relative paths tied to the test executable, as they may introduce inconsistencies when tests run in parallel. When possible, rely on temporary directories such as th.tempWorkspace
which are dedicated to the test.
Using os.Setenv
for feature flags and other settings can cause interference between parallel tests. Instead, use the configuration API:
// UNSAFE for parallel tests:
os.Setenv("MM_FEATUREFLAGS_CUSTOMFEATURE", "true")
defer os.Unsetenv("MM_FEATUREFLAGS_CUSTOMFEATURE")
// SAFE for parallel tests:
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.FeatureFlags.CustomFeature = true
})
Be cautious with methods affecting the entire process, such as pprof.StartCPUProfile
, which can introduce contention between tests.
Did you find what you were looking for?