Mattermost Logo
Edit on GitHub

Sustained Engineering Team

The Sustained Engineering Team (SET) is responsible for improving and maintaining quality.

Team Members 

SET is a rotating team that is comprised of engineers from the different feature teams. The rotation is from the 16th of one month to the 16th of the next month, lining up with Mattermost releases.

Each feature team will commit a single engineer to SET for each rotation. While on SET, that engineer should attend their feature team’s sprint planning but should not be assigned any work from their feature team. For the month that engineer should primarily be working on issues for SET. If SET has no tickets to work on, then the engineer may pick up some work for their feature team.

At any one time SET will have a lead engineer from one of the feature teams who will lead the team. This person should be identified as the SET Lead for each rotation in the channel header. The support team can contact this person when they are unsure who else to escalate to.

Workflow Process 

SET works in a continuous style using this Kanban JIRA board. Tickets in the TO DO column should be organized from highest priority to lowest priority, based on the Priority of Work section. The SET lead should make sure that the tickets are priority ordered but team members can move tickets around to meet the correct priorities as necessary.

Team members on SET should pull tickets off the top of TO DO queue, work on them to completion and then pull another one off the TO DO queue.

Discussion related to SET should occur in the ~Sustained Engineering channel on

SET meets twice per week, after the Tuesday and Thursday triage meetings. During these meetings SET members should provide status updates and talk about any blockers that have been encountered.

After the Tuesday meeting, the SET lead should post a team status update in the ~Sustained Engineering channel using the following template:

#set-update #set-update-YYYY-MM-DD

Bugs resolved in the last 7 days: <X> (<X> reported by customers)
Bug fix PRs submitted in the last 7 days: <X> (<X> reported by customer)

In progress fixes:
* <list of tickets that are currently being worked on>

Next in the queue:
* <list of 3-5 tickets next in the queue>

Total bugs in backlog: <X>

* <any notes or qualifiers for the week>

See for an example.

You can get the bugs resolved in the last 7 days from here:

And the customer bugs resolved in the last 7 days:

To see bugs submitted, look at the Submitted column of the Kanban board.

Priority of Work 

Below is how SET prioritizes what is worked on.

  1. Quality issues affecting
  • When a high impact issue affects the community server, SET responds and is responsible to coordinate fixing
  • Primary goal is to restore stability, whether this is fixing or reverting code
  1. Handles customer support escalation
  • Primary goal is to help with issues that require code change
  • Be helpful with training/knowledge sharing for the support team
  • Identify customer support requests that are features and route to PM as necessary
  1. General bug fixes
  • Pick up any bugs in triage without an obvious owning feature team
  • Work on bugs in this order: hotfix, release, customer reported, other
  • Close any low priority bugs that are not worth the mana
  1. Identify technical debt issues
  • Help identify areas of the codebase that cause quality problems


SET attends all triage meetings. During triage, SET should:

  • Help route tickets to the appropriate feature team
  • Assign bugs to SET when there is no clear feature team owner
  • Make sure reported bugs are in fact bugs and recommend turning non-bugs into stories or feature requests

During non-release weeks, triage is held twice a week. Near the end of the release cycle, triage is held daily.

Customer and Pre-Sales Support Escalation Process 

Part of SET’s responsibility is to interface with the customer support team. SET’s primary goal in respect to support is to triage escalated support issues, work on any high priority customer bugs requiring code change, as well as provide training and knowledge share with the customer support team. When the pre-sales and customer support teams require engineering support, they should:

  1. Post the issue in Sustained Engineering channel. If the message contains confidential information about a customer, then post in Customer Support channel in private Staff team.
  2. At-mention SET team lead, identified as the first person in the Sustained Engineering channel header, and provide the relevant information listed in the support handbook.
  3. If no solution or next steps (such as a Jira ticket) are presented within 24 hours, create a ticket in Mattermost Jira project and assign “Sustained Engineering” as the Mattermost Team. The ticket will be reviewed by a SET team during Tuesday or Thursday triage. If you need a resolution sooner than triage time, escalate to @chris.overton on

This process helps increase accountability and traceability across teams.

Support Response Templates 

SET members should use the following responses when receiving issue escalations from the customer or pre-sales support teams:

Requests that are new features and not bugs 
Hi [insert reporter name], thanks for the report. After looking into it a bit this is working as designed and any changes in behavior would be a new feature. Is this something we get a lot of requests from different customers for? If so, I’d suggest talking to [insert PM name] to see if it makes sense to put on the product roadmap. If you think this observation is incorrect, could you give us the following so we can look into it further:

1. Information about the customer’s environment (MM version, DB type/version, cloud provider, etc.)
2. Logs around the time of the issue
3. Reproduction steps
4. What you have tried to resolve the issue
5. [insert other needs as necessary]
For configuration issues reported as bugs 
Hi [insert reporter name], thanks for the report. This looks like a configuration issue that shouldn’t require any code change to resolve. Have you seen the docs for this [insert link]? If so, can you point to where the docs are deficient and we can work to get those updated.

In the meantime, here’s some things I’d suggest trying: [insert list of troubleshooting steps]
For configurations we don’t support 
Hi [insert reporter name], thanks for the report. This seems to be a configuration that we don’t support. I’d recommend steering the customer to use the supported configuration as documented here [insert link]. If you’re seeing lots of requests for this configuration, I’d suggest talking to [insert PM name] to see if we should add support for it.
We need more information 
Hi [insert reporter name], to help best could we get some more information? Particularly looking for the following:

1. Information about the customer’s environment (MM version, DB type/version, cloud provider, etc.)
2. Logs around the time of the issue
3. Reproduction steps
4. What you have tried to resolve the issue
5. [insert other needs as necessary]
Issues we are looking into 
Hi [insert reporter name], thanks for the report. This looks like a bug, I’ve created [insert ticket link] to track and have put it into the queue to be worked on. Just a reminder that unless this is a hotfix worthy issue the fix won’t make it into the product until the next release. We’ll try to find a workaround that can be applied without code change but that isn’t always possible.