Dimitris Platis

Mob programming for autonomous cars

18 April 2022

For the past year and a half, my team and I have been practicing mob programming every day, the entire day. It is our default way of working which means that we take “individual” tasks rarely and by exception. Mob programming has been working well for us, decreased the pressure on individual members, increased the team’s effectiveness, and the frequency we deliver our sprint’s commitment to the fullest. In this article, I will dive into the practicalities of how we practice mob programming, the benefits as well as the drawbacks we have observed.

Background

Before delving deeper into the subject, let me share with you a bit of the background.
I work at Zenseact which is a Volvo Cars spinoff focusing on autonomous driving. Our goal is to provide the software stack for autonomous driving vehicles under a Platform-as-a-Service scheme. The company develops everything down from fetching sensor data, to how the car should move and transmitting information to the cloud. The vertical is pretty extensive and there are strict safety requirements for the software development.

In this large stack, my team and I work all the way down at the “bottom”, very close to the cameras. We integrate them into the rest of our stack and ensure that images, as well as metadata, reach other components in a reliable and highly performant manner. The team is not involved in any features related to the “driving” of the actual vehicle in other words. Nonetheless, our code does end up on the actual vehicles and we are glad for that. We are currently 8 developers, up from 3 that we used to be approximately 6 months ago.

Have you heard of mob programming? If not, here’s what Wikipedia says on it:

Mob programming (sometimes informally called mobbing or ensemble programming) is a software development approach where the whole team works on the same thing, at the same time, in the same space, and at the same computer. This is similar to pair programming where two people sit at the same computer and collaborate on the same code at the same time. With mob programming, the collaboration is extended to everyone on the team, while still using a single computer for writing the code and inputting it into the codebase.

We have taken the core concept, namely multiple developers working on the same task at the same time, and tweaked it to the team’s needs and preferences. One could say that we supercharged it, doing mob programming on steroids since we work like this every day, all day. Keep reading to learn more.

Why mob programming in the first place?

About a year and a half ago, the team was challenged to deliver a particularly intricate feature within a relatively short period. The problem? Only one out of the five team members had prior experience with this. Be reminded that up to that point, the team was working in a “classic” setup, where there’s a backlog and each member picks a task/story and starts working on it.
The team started planning how to move forward but it was not obvious. We had to figure out how to decrease the pressure on the single member with the past involvement and at the same time disseminate the knowledge throughout the rest. This was when a great idea came up: Mob programming.
Instead of attempting to sweeten the pill, we would tackle the root cause of the problem. Everyone works on the problem together, taking turns. The more experienced member guides the rest but tries to mostly be in the backseat. Following this principle, 1.5 years later, we believe we have benefited from this approach.

How we mob

We mob every day all day. However, we do not do it “by the book” since:

1. The entire team does not typically work on the same thing (this was the case only when we were just 3)
2. The team is not in the same space, we work 99% remotely

So, how do we mob?
Unlike other cases I have heard of where teams do mob programming for part of their day or when there’s some kind of emergency at hand, this is our normal way of working for 8 hours straight. OK this is untrue, we do take breaks.

We start our day with a typical standup, where we go through the sprint backlog items, what we worked on during the previous day and what will we work on until the next. Remember, we work remotely, so this is essentially a call. The team typically works on 2, at most 3 items at the same time. Each task gets a separate call created in our team channel. We have named these calls constellations or tracks. Each constellation operates in parallel with others and is usually comprised of 3 to 4 members. A diverse backlog with stories independent from each other is essential for this to function.
Each track has a keyboard warrior who is the person with the keyboard and the shared screen. Unlike the traditional mob programming setting, the keyboard warrior is not necessarily passive but has the freedom to drive the development. The rest of the constellation members are commenting on the activities taking place on the shared screen. Their involvement varies from telling the keyboard warrior exactly what to type or try out to intermittently expressing opinions on what is currently being written by the person with the keyboard.
We strive to change the keyboard warrior often, but not as frequently as often found in mob programming “instruction manuals”. For us, this ideally occurs every hour or approximately when a work package has been completed. We are not so “religious” about switching drivers on specific intervals, since we believe that establishing a flow is important.

Constellations are fluid, meaning that occasionally members jump between them. This can be due to:

• The expertise or passion of a particular member is required in the other call
• Someone feels underutilized or bored in the call they are in and fancies a switch
• Consensus of the entire team is required on a decision, e.g. the design or strategy to follow
• Code-review walkthroughs, especially more complex patches

While not so common, a constellation may temporarily get divided into smaller parallel tracks. Individual members may undertake relatively trivial subtasks if that is deemed faster than working sequentially on them as a mob.
Furthermore, the tracks are often joined by our Product Owner in case we need input or he has some questions for us. Despite working remotely, we are very easy to get hold of. At least if you are a member of our team. 😉

Working in a mob, one should abolish any notion of personal code ownership. Multiple individuals will work on the same feature until it gets merged and since the composition of the constellations changes, everyone implicitly ends up working with everyone and everything. No cherry-picking tasks either. The most prioritized item is the one to always pick up next from the backlog, no exceptions.
It may not be obvious at a first glance, but changing the mentality of individuals who are accustomed to establishing their dominion over parts of the code base is crucial for mob programming to work. Seniority does not matter. It is all about the mindset.

Despite the conscious effort to avoid distractions and occupy individual members with “personal” tasks, they are sometimes inevitable. For example, as a SCRUM master, I have to attend some extra meetings or be involved in additional communication channels. Moreover, there may be relatively minor issues that do not justify an entire mob being devoted to them, so we may have an individual member tending to them as a background task. While these “side activities” always exist, when working as a mob, their frequency and distraction potential have observably decreased. That being said, nowadays, when individuals need to engage themselves with tasks outside the mobs, it has little effect on the team. The mob can typically afford to lose a member temporarily, especially once the development has gotten into a flow.
Compare this to the classic setup where parts of the sprint may be completely blocked due to individuals not being able to attain focus.

Benefits

The positive effects of mob programming were imminent and the response from the team was particularly enthusiastic.
When writing this article, I did not remember when, exactly, we started with mob programming. However, I was confident I would be able to discern this based on the log we keep of our past retrospective meetings. This was indeed the case.

It was heartwarming rediscovering this. I will elaborate further on the specifics but on the 6th of November, 2020, the the team was extraordinarily excited about the new way of working and the value it brought to the team. Keep in mind this took place amid the second coronavirus wave! 😷

The positive impact of mob programming was not a firework. 🎆
It continued with the next sprint’s “What we did well” sticky notes followed the same pattern. Nowadays, a year and a half later we are less enthusiastic about the benefits of mob programming as we have grown accustomed to them. Nonetheless, they are still apparent and we still pat ourselves on the back when we do exceptionally well when mobbing. These days we focus on optimizing our way of working even further, as the team has gotten bigger and merely 3 of the original team members that started mobbing remained.

Burndown

One of the first effects of mob programming was reflected on the burndown chart; it zeroed out by the end of the sprint. Since then, we have delivered or even over-delivered our sprint forecasts on numerous occasions, something unprecedented. I can see several reasons behind this:

1. The team is more focused and experiences fewer distractions. I will elaborate more on this soon.
2. The estimates used to plan our work may have become more accurate as we are more aligned on what the current status is and what needs to be done. That being said, they are still not as accurate as we would like them to be.
3. Getting “all hands on deck” is more efficient than it used to be. Quickly assembling the entire team to work on the remaining tasks is simpler and easier than before since the members are accustomed to working in this manner and are already aligned to a great extent. When a constellation is done with their task, their first concern is whether the rest need help, not to move on to the next task.

I can not claim mob programming helped us always deliver on time, however, it did increase the frequency we manage to do so.

Focus

Mob programming has increased our ability to focus. The benefits were two-fold. On one hand, we decrease the number of tasks taking place at the same time. Instead of having approximately one potential source of distraction for each team member, we now have one per constellation. I refer to different tasks as distractions since they inherently require the team to synchronize on design decisions, code reviews and to resolve cross-dependencies.

On the other hand, it feels like we have fewer meetings whether they are internal or external. Internal meetings have almost vanished since the need to coordinate the team on various subjects has decreased. Alignment inherently takes during the mob sessions, whether that is done consciously, e.g. we initiate a discussion on a particular topic as soon as it arises, or without actively pursuing it, during the shared screen sessions.
Furthermore, we are exhibiting increased resistance to requests for external meetings, especially recurring ones without agendas that concretely concern us. I can not say if this was because of mob programming, but it seems the team’s mentality has shifted and we are more “possessive” of our precious time. If a meeting is requested and its value is not clear to us, we either try to simply say no or send a representative. Of course, we could have done this before mob programming, but the representative would then have to brief the team in a new separate meeting. Now, if the rest of the team needs to catch up, this happens during the ongoing mob call.
The overhead of setting up internal meetings has been eliminated.

Knowledge sharing & Eliminating silos

A natural outcome of mob programming is immense and continuous knowledge sharing. A welcome side-effect if you wish. The team members are communicating, debating, and reaching conclusions or compromises constantly. Interestingly, the knowledge flow is not always in a single direction. The more “experienced” members on a particular subject, through their continuous engagement with their colleagues, are gaining a deeper knowledge on topics they already knew to an extent but also get introduced to new perspectives, as different sets of eyes and brains revisit the same issues.

Additionally, when we realize a team member has become disproportionately more experienced on a topic compared to the rest, they are consciously staying in the background and avoid driving the work. Their role is limited to helping the mob get unstuck. This results in knowledge sharing on steroids 💪 and eliminating any notion of silos. In our team, no one is “responsible” for a particular part of the codebase. The team has an area of responsibility and that’s all. We actively try to steer clear of situations where we would have a task that requires the expertise or involvement of a specific individual. Everyone is not only encouraged but also expected to work on everything. We are not all as good at everything, however, we collectively work to minimize any “gaps” to sustainable levels.

Working together, everlasting knowledge sharing, and a lack of silos allow us to conduct blazing fast code reviews. Don’t get me wrong, we do code-review a lot and we are often very picky. However, the time typically spent during a thorough and proper code review, is now distributed throughout the implementation phase. In other words, we do not wait for the formal code review to settle small or especially bigger concerns that exist over the code. Moreover, the time it would take for someone to get up to speed with the code and the context is minimized since they are already familiar to a good extent with what needs to be done.
Some might say that since everyone has been working on the code, we must be having different developers review our code. I recognize this opinion does have some merit. Being involved in a solution, one may be biased and be unable to see how it may be problematic. From our perspective, there is a trade-off. The efficiency of a “new” reviewer who needs their time to get acclimated, compared to the mob members who are already familiar with what is going on. So far, we have not experienced any decrease in the effectiveness of the code reviews when conducted by mob members. To be completely frank, we suspect code reviews by “outsiders” were less likely to detect serious issues, as they would not necessarily fully comprehend the code review.
We still involve members of different mobs in reviews, especially in larger pull requests. In that case, the mobs temporarily converge to conduct a code walkthrough together.

Low pressure on individuals

No silos mean no individual ownership. No individual ownership means there are a few dependencies or bottlenecks on specific individuals to take out the fires. The pressure is dissipated on all team members, who may plan and decide on how to tackle the emergency.
We no longer have individuals firefighting, but an entire team. This results in lower stress and fewer distractions.

Socializing

Despite working remotely, we are extremely social. I believe we are probably talking more to each other than if we were working in a traditional setting seated near each other. In fact, I have noticed that when we are in the office, it feels like we talk less! 😅
When being in a mob, it is not always about code, or even if it is, it is not necessarily work-related. During the mob sessions, a variety of subjects are brought up, varying from memes, food, traveling, real estate, finance, and developments around the globe. On top of that, we organize “virtual fika” every other week where work-related talking subjects are severely frowned upon.

Utilizing remote work to the fullest

We started with mob programming during the pandemic and by doing so we removed a good portion of the typical disadvantages of remote work. Sweden never had any strict measures against CoViD, so we were always free to visit the office and work from there if we wanted. Nevertheless, we have realized that remote working is a prerequisite to working as a mob, 8 hours per day and 5 days per week.
From what I know, typical mob programming is emergent when a need arises or is practiced for part of the day. I can imagine that it is very intense and naturally teams would not cope with it. When mob programming remotely, team members are more relaxed, they can afford to take their attention away briefly and we have not observed fatigue.

Furthermore, we have three team members who are abroad. They do not even live in the same city. Mob programming has been instrumental in making everyone feel part of the same team; enhancing belongingness. Even if the majority of the members still live in Sweden, we work in the same way as everyone else. To be fair, from time to time we do enjoy lunch together. However, we are all “equals” when we work, as everyone is online.

Disadvantages and mitigation measures

We have come to realize that mob programming is not for everyone and I would hate to sugarcoat it. I imagine it will not work for most teams out of the box. It takes a special kind of an individual to be compatible with mob programming. Let’s take a look at some of the drawbacks of this technique.

Too high or slow pace

When mobbing the pace can be too high or too slow. It depends. Unless you are the one heavily driving it, chances are it will not be at the pace you would prefer. As you may imagine, being too slow can decrease engagement while the opposite can result in members falling behind. A decent compromise needs to be made to ensure that the pace is both sustainable and intrigues the members enough to not lose attention.
To keep members concentrated and engaged, one trick is for the keyboard warrior to describe what they are doing, especially if they have taken the initiative to drive the mob. Moreover, at least two mob members should try to actively ask questions, express remarks, or provide suggestions.
On the other hand, if someone is falling behind, they are always encouraged to halt the mob “train” and ask their questions. The “keyboard warrior” should regularly ensure everyone’s on board.

Aside from the need to tolerate a different pace, it takes a specific mindset to thrive in a mob. There are two schools of thought, (a) the “do first think later” who like to quickly dive into a problem and figure things out while working towards a solution and (b) those who prefer to first thoroughly understand the context and the problem before trying to dive in. I find the former mentality more compatible with mob programming. Of course, I could be wrong/biased, and perhaps as long as everyone’s mentality matches they can mob well together.

Inflexibility

Mob programming is very inflexible. One may not have imagined this, especially when working remotely, but there is not much room for members to have their own schedules or pace. The mob calls start after our morning standup meeting, we have our breaks together, we go to lunch at the same time, we continue after lunch and sign off around 17:00.
Additionally, if someone has a meeting and needs to temporarily depart from a mob, then they may need to put in some extra effort to catch up. Of course, we recognize this risk and actively try to accommodate returning members, by pausing the mob and doing a recap of the progress since they left. Moreover, we try to serve the need for syncing by coordinating our schedules, but ultimately this is an inherent issue of mob programming.
If you adopt a full-time mob programming setup like us, working remotely will be your primary choice. It is very tiring to physically seat together in a mob for long periods. For example, when some members of the team happen to be in the office, they still prefer not to sit together.

Little satisfaction by “personal achievements”

Tight collaboration and lack of individual ownership can result in a decrease in satisfaction due to personal achievements, simply because there are very few personal achievements. One may rarely seat down with a problem all by themselves, excessively contemplate over it, and have a eureka moment. There are multiple such occasions during the mob sessions, however, they are primarily a result of collective thinking. As a software engineer, I sometimes find myself being nostalgic over when I would be a “one-man team” and have the product being the outcome of “MY” hard effort. My “brain-child”.
To this avail, we try to split the mobs apart when subtasks can be parallelized. We also have a backlog for more trivial/background tasks that can be a good source of individual engagement if time permits. That being said, my opinion is the benefits of no silos and tight collaboration far outweigh the drawback.

Extroverts needed

It does take a special kind of person to thrive in a mob. Members need to be extrovert enough to effectively engage with the rest of the constellation. If you ask me, most people should naturally be extroverted enough to manage collaboration in a mob, however, not everyone is. Especially software engineers. 😂
Engagement is necessary to keep the attention high and ensure that everyone is aligned and knowledge is spread as intended. A mob being mostly mute with someone silently typing is super boring and should be avoided. When engagement levels fall, it is the responsibility of everyone, especially the “keyboard warrior”, to spark and rejuvenate the discussion. This can be awkward or uncomfortable but has to be done for the mob to thrive.
Mob programming is about collaboration, thus extroverts facilitate the process. Not much more one could do about it.

Will this work for you?

Considering the above, I hope by now you have realized whether this could work for you or not. I do not want to discourage you, but this practice is not for everyone. If you manage to make mob programming work, the way it has for us, then you will see an increased burndown rate, sharper focus, quicker code reviews, and stronger team spirit.
I advise you to try it out and see the extent to which your team is compatible with it. Just remember to formulate mobs with both experienced and inexperienced members. This will be the most effective.