In larger engineering organizations, there is often a team dedicated to ensuring a flow from an engineer’s commit to an artifact repository or a running service. Today it’s often called a platform engineering team, though it goes by many names including Developer Productivity/Experience, DevOps, Release/Build Engineering.  In smaller organizations this is often handled by the SRE, Cloud, or Infrastructure teams. In the old days it might have been part of the QA team. I’ll call it the platform team for the rest of this post for simplicity. They typically own the CI/CD infrastructure, and sometimes the production environment too.

When an engineer has a problem, especially one related to their development environment, builds or deployments, they need a place to turn. The company’s IT helpdesk is rarely (never?) able to help. These days the first stop is likely an LMM and maybe a team mate. If there aren’t quick answers there, the engineer typically tracks down someone they know on the Platform Engineering team and asks for help. If they’re in a bad mood, they start by blaming the team’s tools and never consider that the problem might be on their end.

Protect the Team from Interrupts

Interrupts are expensive. People have to stop what they’re doing, provide assistance, and then swap back in the context of what they were working on. Some people are able to context-switch quickly, but for many it’s hard and slow.

To avoid randomizing the team (pulling people off their work at unpredictable times), we found it best to have one person at a time play a “frontline” role. The role does two things:

1. Triages requests. The person on frontline duty would identify the urgency and size of the work being requested. Many helpdesk requests could be quickly handled by the frontline person. If a request was a small amount of work that required one of their teammates, they would notify that teammate and ask them to pick it up at a natural break point in their own work. Our expectation was that this would happen within 2 hours. Anything bigger went into the team’s backlog and was prioritized like any other work. Requests related to a customer-visible incident jumped the queue.
2. Solves the requestor’s problem. Often this is a collaborative debugging session. Our goal was to have people wait less than 5 minutes from request to engagement, and have most issues resolved within 15 minutes. When there were multiple requests, they were handled in first-in, first-out (FIFO) order.
3. Clear low hanging fruit. When the person on frontline duty wasn’t helping people or triaging new requests, they pulled small tasks from the team’s work queue. We generally encourage them not to pick items in the current sprint (scrum) on deck (kanban) but rather something in the backlog they would like to do. We wanted them to feel productive and to work on something that interested them during quiet periods. The work had to be small enough to drop the moment a new request came in and not on the critical path of an immediate team deliverable.

Virtual Frontline

These days, frontline duty is often virtual. It could be a Slack channel: the person on frontline duty turns on notifications, and the rest of the team mutes the channel unless someone @-mentions them. Or it could be a paging or on-call tool that routes requests based on a schedule and/or the nature of the request. Often the system supports escalation, triggered automatically when a response-time SLO is missed or manually when someone asks for it.

Increasingly, the first responder in the channel is an AI assistant. It answers common questions from the team’s docs and runbooks, and hands off to the person on frontline duty when it can’t help.

Physical Frontline

Teams with more resources typically have a frontline desk at the edge of their space. Whoever has frontline duty temporarily moves to the frontline desk. Besides handling requests, their job is to act as a gatekeeper. They keep random people from wandering into the team’s space and interrupting their teammates.

Doing this has two costs. The first is physical space, which is often at a premium in a growing company. The second is that when they aren’t handling requests, the person at the frontline desk is typically less productive, because they’re away from the workspace they’ve set up for themselves.

When there’s no dedicated desk and frontline duty moves between people’s own desks, it’s good to have some way to indicate who is on duty. You can use a sign, or a behavioral cue like looking for whoever isn’t wearing headphones. Many teams adopt an object that sits on or next to the desk of the person in the frontline role. Often the object’s name becomes the name of the role. At 23andMe we called it Bacon. We had a large plush strip of bacon. The joke was that whoever had frontline duty was in the skillet, just like a piece of bacon.

Empower Requestors

Often people actually knew the answers to their own questions. They were preoccupied at best and lazy at worst; either way, they wanted someone else to fix their problem.

The first step in helping people is discovering what the problem is. We have found that people often find their own answers when they take the time to accurately describe the problem.

Prepare to be Helped

My teams typically had a way to record “walk up” requests which captures what they were trying to do, what happened instead, and links to the failing build or logs, etc. Sometimes it was a separate Jira project, a field, or a workflow. We created these tickets for two reasons. First, it is an attempt to collect the relevant information to deliver assistance. Second, it lets us see what issues are trending and helps the team to identify changes that might reduce the number of requests.

If the frontline person is free, they will collaboratively record the information with the requestor.

If the frontline person is in the midst of helping someone else, we encourage the person asking for help to take a minute or two to record relevant information. Often this process enables the requestor to solve their own problem. See the next section “Talk to the Duck”. Even if they haven’t solved their own problem, they have recorded information that will help the frontline person work on the problem.

Talk to the Duck

Everyone has had the experience of realizing the solution while describing a problem to a colleague. This has been called many things, such as “cardboard debugging” and “teddy bear debugging,” but the phrase that has stuck is “rubber duck debugging,” or in the imperative, “Talk to the duck.” The name comes from The Pragmatic Programmer, in which a programmer debugs by explaining their code, line by line, to a rubber duck.

I have worked at several places where there is a rubber duck sitting on the frontline desk. It is a visual reminder to the requestor that they might be able to solve their own problem without bothering anyone else or being blocked if no one is available. If the frontline person is already helping someone, they’ll often tell the next requestor when they expect to be free, then say, “While you’re waiting, talk to the duck.”

These days, many engineers’ first duck is an AI assistant. Unlike the rubber one, it talks back. That helps, as long as the requestor still does the work of explaining the problem clearly.

Self Service

The best frontline service is one that isn’t required. In the old days we would build documentation, runbacks, FAQs, etc to cover the most common issues. This would often evolve into purpose built tools that allow people to correct the most common problems in a secure manner. These days, AI chatbots are becoming useful.

Bring Other Engineers In

Shared Responsibility

It can be quite helpful to have senior engineers who use the development and CI/CD systems, but don’t build them, take frontline shifts. This does three things:

1. Builds empathy in the senior engineers toward the people who own the developer experience.
2. Gives the senior engineers greater insight into what the platform team is working on. This can lead to breakthroughs: they may notice that some planned work isn’t really needed, or spot missing pieces that the people building the system can’t see.
3. Keeps the occasional lazy engineer honest. When they come for help with an “it’s your job to help me” attitude, they get pushback from their own team’s leaders: “You should know this. Do your job; don’t fob your problems off on someone else.”

Reverse Embedding

Each quarter we had an exchange: one product engineer would embed with the platform team. This is the reverse of the more common arrangement, where SREs embed in product teams. In return, one of the SREs was often dedicated to the embedded engineer’s home team, to make up for some of the lost capacity.

Selecting the embedded engineer was a two-phase process. First, engineers were encouraged to propose projects that would take 1–2 FTEs a quarter. Second, all the engineers voted on the proposals. This happened outside the normal planning process. The author of the winning proposal had the right of first refusal for the reverse embedding. If they didn’t take the embedding, another engineer would be selected. People could volunteer, but most of the time people were recommended by their manager as a wall to round out an engineers experience.

Further Reading

• DevEx: What Actually Drives Productivity – ACM Queue Article
• Developer Productivity tagged articles on Pragmatic Programmer
• What I Talk About When I Talk About Platforms – Idea piece from Thoughtworks (2018)
• Dealing with Interrupts – Classic chapter from Google’s SRE Book
• Hero Rotation – Adding a frontline to any development organizations
• Designing and Managing Support Processes for Internal Platform Users – Guidewires pivot to the cloud. Some interesting ideas about capturing what is interrupt work.
• Impact of Programmer Interrupts – Christopher Parnin thoughtful reflections of the impact of interrupts
• The Cost of Interrupted Work: More Speed and Stress – Classic 2008 paper