In the first of a new series of guides, offering founders concrete advice on growing their B2B SaaS businesses from Series A and beyond, Constanza Diaz and Jonathan Durnford-Smith tackle the tricky topic of measuring a development team’s output. 

From the founder and CEO to the most junior hire, everyone in a B2B SaaS start-up has an instrumental role to play in turning the business from a great idea to a world-changing solution. But in these early days, the developers are indispensable. Put simply, they build a SaaS business’s core offering and turn a founder’s vision into reality. Recruiting and maintaining a world-class dev team is, unquestionably, a major competitive advantage. But there’s a problem. 

At every stage of a business, the leadership team needs a clear picture of how efficiently – and effectively – their different departments are working. It helps answer questions about when to scale and how and where to allocate resources. But when it comes to a business’s tech organisation – the dev team – getting that clear picture isn’t always easy.  

In fact, the question of how you measure your dev team’s output is one we hear asked over and over again across our networks. Why? There are a few key challenges that make assessing the output of this uniquely important department harder than assessing the output of other departments. Difficult – but not impossible.  

In this blog, we are going to explore some of the most common frameworks for measuring a dev team’s output and offer some tips on how to go about building one that works for your organisation. (Spoiler alert: as is so often the case, there is no single answer and iteration is key.)  

First, though, why is measuring a dev team’s productivity so hard?  

The challenges

It’s difficult to even define developer’s productivity in the first place. Software development isn’t straightforward. It’s not an old-fashioned, industrial-age production line. The cadence of work, with all its breakthroughs and setbacks, isn’t always clear, and software quality is hard to measure.

Developers’ work is a creative and subjective process. Yes, there are some repetitive tasks – but many more are intricate and complex, requiring collaboration, exploration and experimentation. Understanding this is important, because it explains why typical productivity metrics found in other disciplines aren’t much use when it comes to the dev team.

With some organisations misusing developers productivity metrics, setting up systems to track individual developer’s activities, suspicion towards measurement is widespread. While it’s entirely understandable, this doesn’t much help a founder or CEO seeking to understand how their dev team is performing – and if the time is right to grow it. So, what can be done?

Measuring output

A number of different frameworks for measuring a dev team’s output have been established.

1. DORA (DevOps Research and Assessment), from Google. DORA focusses on software development and operations (DevOps). The framework was established by the researchers behind the groundbreaking book, Accelerate, who drew on six years of research (and over 30,000 data points) to identify four core metrics that correlate to the performance of a dev team. These are:

• Deployment frequency: how often an organisation successfully deploys new features or software updates to production.

• Lead time for changes: the time it takes for a code to go from being committed to a version control system to being in a deployable state. Note DORA’s focus on DevOps rather than the full development cycle.

• Change failure rate: the percentage of changes or deployments that fail in production.

• Mean time to recovery: the average time it takes to restore the service to full functionality following an unexpected failure or incident.  

2. SPACE, developed by Nicole Forsgren (who was also part of the team that created DORA) and her colleagues at GitHub. SPACE offers a more comprehensive framework than DORA, built around:

• Satisfaction and wellbeing: using satisfaction surveys, it explores staffing-related metrics, including turnover and absenteeism.

• Performance: the outcome of the development process, including diverse metrics such as quality scores, code review velocity, features usage, NPS, etc.

• Activity:  a count of actions performed by the dev team. It can include metrics such as the number of pull requests, commits, number of tests run and the number of issues created and resolved.  

• Communication: a straightforward measurement of how teams communicate with each other and with stakeholders.

• Efficiency (or flow): this explores the team’s capacity to complete work seamlessly, with minimal disruptions.

3. DevEx, another stab at the topic by the creators of SPACE. This framework focuses on the experience of developers and the points of friction they encounter in their day-to-day work lives, with a view to maximising their effectiveness. It breaks developers’ experience out into three key dimensions:

• Cognitive load: the mental effort by developers to understand, process, and work with the codebase and tooling.

• Flow state:  the psychological state in which developers are fully immersed in their work.

• Feedback loops:  the effective and timely exchange of information in the development process.

• Given the more subjective nature of the DevEx framework, the measurement of these dimensions is typically done using surveys to gather direct input from developers.

Each of these frameworks is underpinned by in-depth research, and they can all be extremely useful. But they were conceived for larger, more mature organisations – not start-ups or young scale-ups.  With greater resource constraints, start-ups can’t just hire extra DevOps and quality assurance (QAs) if they need to support developers and accelerate product development. They have to do more, with less. Efficiency is key – an axiom that’s never been truer than it is today.

So what do we recommend?

As ever, there’s no one-size-fits-all playbook. You might find it useful to take pieces of each framework and combine them. You should also be prepared to adapt – and keep adapting – as your company scales.

Typically, we see two types of founder/CEO/CTO in start-ups. There’s the analyser type, who wants to pick the perfect framework (or build it), in order to measure each and every metric flawlessly. The other is the ‘doer,’ who just wants things to get done – and doesn’t want to spend time on deciding which framework to use, or what metric to measure and how.

Every founder, CEO or CTO will know where they fall on that spectrum and can adjust their decision-making process accordingly. Importantly, frameworks are supposed to make things easier (if they’re making them harder, or more confusing, they’re not working and need to be reviewed!). Stagnation, or analysis paralysis, are an undesirable consequence. Just as in product development, it’s essential to experiment, find out what works and keep iterating as your company grows.

It’s also worth noting that at the very early stages, with few employees (and only a handful in the dev team), you can get away with not measuring anything because it’s easy to communicate openly and directly with the developers. But as your business grows, you will need to start measuring your dev team’s output. Here’s what we recommend:

General thoughts

• Measure at the team level, not the level of the individual. Many of the complaints developers have about being measured relate to these metrics. Having someone looking over your shoulder the whole time is demotivating, which is why a broader view is preferable. It’s about striking the balance between understanding what your development team is capable of and showing good leadership by proving that you genuinely trust their capacity to deliver.

• Sometimes it will not be possible to compare the metrics of different teams if their work is too dissimilar. But we suggest you try as it can help you to identify bottlenecks, or areas of improvements

• Look at trends (not just one point in time). If a metric is getting worse over a series of sprints, that’s a sign that there’s a problem. When measuring development output, it’s the long view that’s important, not just one-offs.

• Be clear on what you’re measuring and why. You need to set clear goals and expectations, not expect your team to read your mind. This will also help to avoid individuals “gaming the system” and attempting to improve a metric rather than improving the process holistically.

What to measure

Timings

• You need a measure of lead or cycle times. These should consider the whole development cycle, from task creation to production. Make sure you measure the time of different stages in the development cycle too – so it is easier to identify where the delays are.

• If you can’t measure times accurately, maybe because task sizes are incomparable across sprints, try for ‘forecast accuracy’. This means the number of goals your dev team commits to each sprint, versus the number of goals they actually accomplish. Setting clear goals is key: your team need to understand why it’s important to accomplish a sprint’s goals within that sprint, and how they relate to the bigger picture – providing customers with a solution they need.

Quality

• This is highly subjective, and there’s no single KPI that can measure quality holistically, but it’s an important thing to keep in sight.

• To keep things simple, we think a KPI that measures the number of bugs found in production by a user in the first release is a good place to start.

• You could also examine mean time to recovery or down-time.

Team satisfaction

• Make sure you gather feedback frequently. For a small organisation, regular but informal direct chats will do – but anonymous feedback tends to be franker.

• To understand which tools and processes are working best, measure feedback across different stages of the development cycle.

• At the end of a sprint cycle, make sure to carry out retrospectives or feedback sessions. Both of us (Jonathan at Deep Mind) have used the simple yet powerful framework of stop-start-continue to guide feedback sessions.

Ratios and other points to consider

• A few rules of thumb ratios are useful to keep in mind, and may illuminate anything that’s off in your tech organisational structure.

• 3:1 – this is the number of developments per number of testers/QA full-time equivalents (FTEs) . At early stages, we strongly recommend that developers perform tests on each other’s work.

• 1:6 – product managers to developers. That team of six should always include a tech lead.

• In terms of direct management, we’d suggest an engineer manager should have no more than six direct reports. As a team grows, it’s natural to have transitional points where managers have changing numbers of direct reports. If you find your managers have more than six, it might be time to rethink the structure, or hire a new manager/lead, to reduce the other managers’ burden and lower the risk of burnout.

There are other important aspects of dev team productivity we haven’t covered: communication, architecture and tooling, and collaboration all matter. But with the suggestions above, you should be able to start building a framework that suits the demands of your start-up – and dev team.

Ultimately, it’s important to remember – and communicate – that measuring the team isn’t just about keeping tabs on them. It’s essential to mapping out your strengths and weaknesses, identifying gaps that might need to be hired for, and ensuring your dev team is in the best possible place, with the best possible support, to bring your revolutionary new solution to market.

If you’ve found or designed a framework that’s working wonders for your B2B SaaS business and you’d like to tell us about it, please get in touch. And if you’re working with CTOs on refining productivity and streamlining development in their organisation, let us know. You can reach Constanza at [email protected] and Jonathan at [email protected].