How to measure & improve developer productivity

Are your developers reaching their full productivity potential? Can we even determine this reliably? More importantly, who wants to know, and what are their motives?

Many argue that developer productivity is in fact measurable and should be quantified. There are numerous models for this, including DORA, SPACE, and various proprietary frameworks, such as the one introduced by McKinsey & Company in late August 2023. This particular framework sparked significant debate within the DevOps sphere.

However, several commentators contend that focusing on the incorrect aspects or misapplying these measurement frameworks doesn’t provide meaningful insights into your development teams. Instead, it may have the opposite effect and decrease productivity by encouraging manipulation of the system or lead to increased turnover.

The challenge is due to the inherent complexity of development tasks, which cannot be easily quantified by numbers, unlike sales or recruiting roles which can be evaluated based on revenue and recruitment success, respectively. Thus, both qualitative and quantitative evaluations are vital, supported by growing evidence that developer welfare significantly impacts their productivity.

This article explores the common measures of developer productivity used today, discusses the negative reactions to imposing a surveillance culture on developers, highlights the downsides of popular metrics, and explains how modern observability tools can eliminate major productivity barriers in cloud-native environments, thereby enhancing developer output.

Let’s start by defining key terms. What do we even mean by developer productivity? It generally signifies the capability of a developer team to consistently write and ship high-quality code that adds value to the organization.

Some questions that immediately start popping up include:

• What defines efficiency?
• What criteria determine high-quality code?
• Should the focus be on individual developers or the entire team?
• How is value to the organization measured?

Considering developers are human beings, not mere data points, it’s crucial that any productivity assessment includes both qualitative and quantitative elements.

Defining what can be quantified

To begin answering these and other questions, it’s important to understand that four different aspects of work – any type of work – can be quantified.

• Inputs: Sometimes referred to as effort in software development, this encompasses the time, energy, thought, and creativity invested in activities like design, coding, testing, and debugging.
• Outputs: These are the tangible deliverables from the inputs, which could be a specific software feature, the code itself, or accompanying documentation.
• Outcomes: These are the changes that occur as a result of the inputs and outputs. This could mean altered employee job functions due to reengineered business processes or shifts in customer behavior.

Impacts:This refers to the value generated for the business, such as increased employee efficiency or heightened customer purchases.

The DORA and SPACE frameworks are among the most widely utilized.

DORA

DORA (DevOps Research and Assessment), today stewarded by Google, has done extensive research and found that an organization’s level of software delivery performance does predict overall performance, team performance, and employee well-being. To gauge outcomes effectively, DORA outlines four key metrics you can use to understand your development teams’ ability to quickly deliver high-quality software.

The metrics are categorized into two groups—velocity and stability—to ensure that teams balance rapid delivery with high-quality outputs:

• Deployment frequency: How often a team deploys code changes to production
• Lead time for changes: The duration from committing code to running in production
• Change failure rate: The proportion of deployments that result in degraded service and require remediation work
• Time to restore service: The time it takes to restore service after a production change or release results in degraded service.

DORA metrics help categorize teams into elite, high, medium, or low performers, aiming to foster improvements. According to the DORA report, elite teams are significantly more likely to achieve or surpass their performance targets compared to other groups.

Here’s how Elite organizations fall for the key metrics from Google Cloud.

See how you stack up:

SPACE

The SPACE framework—co-created by GitHub and Microsoft—complements DORA by focusing on less quantifiable aspects of developer satisfaction and productivity. SPACE stands for Satisfaction and Well-being, Performance, Activity, Communication and Collaboration, and Efficiency and Flow, each addressing crucial, albeit hard-to-measure, facets of development work:

• Satisfaction and well-being: Satisfaction refers to how fulfilled a developer is with their work, team, tooling or culture. Well-being refers to how happy and healthy a developer is and how their work impacts it. The importance of developer satisfaction and well-being are key productivity drivers and best measured via surveys. These surveys often inquire if developers would recommend their teams, if they have the necessary tools, and if they face burnout risks. Such insights are pivotal in understanding the health and satisfaction levels within developer teams.
• Performance: Performance is typically assessed based on the outcomes—whether the software fulfills its intended function, rather than merely the outputs or even the broader impacts. Measuring performance in software development is complex — it’s not only about the volume of code produced but also its quality and effectiveness.
• Activity: Activity is the measure of actions or outputs involved in performing development work from planning and design to shipping code and level of operational issues. While these provide some indicators of productivity, their value is contentious, particularly when taken out of context. High activity levels, such as numerous pull requests, don’t necessarily equate to productivity, especially if many are aimed at correcting previous errors or reverting unwanted changes.
• Communication and collaboration: Effective communication and collaboration within and between software teams is vital for organizational success and are the most challenging to measure. Suggested measurements that serve as proxies for assessing how well teams work together and share information include: discoverability of documentation and expertise, quality of reviews of work contributed by team members, network metrics that show who is connected to whom, and the onboarding length and experience for new team members.
• Efficiency and flow: Flow, or the state of uninterrupted work, is a key aspect of individual developer efficiency while efficiency for the team and system level relates to value-stream mapping. Literature suggests various methods to help developers maintain this state, and metrics can include the number of interruptions (including quantity, timing, how they’re spaced, and the impact to work and flow), the frequency of handoffs between teams in a process, and a developers’ perception of their ability to stay in a flow state These measures help assess how well developers and teams can work together efficiently.

Additional developer productivity metrics

The following metrics, whether integrated within frameworks like DORA and SPACE or used independently, provide valuable insights into various aspects of developer productivity, allowing teams to optimize processes and align their efforts more closely with business outcomes:

• Cycle time: Cycle time tracks the amount of time code is “in flight” and commonly measured from the initial commit to when the code is released into production, or from the start to the completion of any given task. Typically, a shorter cycle time is preferred as it indicates quicker delivery of changes. However, it’s vital to balance this with maintaining code quality; rushing through this process can compromise the integrity and functionality of the software.
• PR size: Pull Request (PR) size measures the volume of changes introduced with each pull request. Smaller PRs are generally easier to manage, review, and integrate without disrupting the existing system. This metric expresses the granularity with which developers are submitting their work, which can influence both the stability of the application and the efficiency of the development process.
• Investment profile: This metric helps visualize how much time a development team is spending on different types of work such as contributing new functionality/features, fixing bugs, increasing reliability

The metrics introduced by McKinsey have sparked significant controversy within the developer community, reflecting ongoing tensions between evolving measurement frameworks and the practical realities of software development. McKinsey’s metrics, which include the “developer velocity index benchmark”, “contribution analysis”, and “talent capability scores” aim to enhance both team and individual productivity assessments. These metrics are posited as necessary adjustments to keep pace with the rapid evolution of software development, notably due to the influence of generative AI tools like ChatGPT, which McKinsey dubiously claims can double the speed of task completion.

This approach, however, has been met with skepticism and concern. Ken Beck, a prominent figure in software development known for pioneering extreme programming, has vocally criticized the McKinsey metrics on LinkedIn, saying: “The report is so absurd and naive that it makes no sense to critique it in detail.” In a later post, he added, “Why would I take the risk of calling out a big, influential organization. It’s because what they published damages people I care about. I’m here to help geeks feel safe in the world. This kind of surveillance makes geeks feel less safe.”

In collaboration with Gergely Orosz, who blogs at “The Pragmatic Engineer,” Beck articulated a broader critique in a two-part rebuttal, worth a read, emphasizing the need for metrics that enhance accountability without compromising the autonomy or morale of development teams. They argue for a focus on fostering high-performing teams defined not by rigid metrics but by their ability to satisfy customers, maintain a positive workplace atmosphere, and avoid counterproductive measures of productivity.

The debate highlights a fundamental issue with performance metrics in software development: the risk that quantifying performance too rigidly can lead developers to optimize for metrics rather than genuine quality or innovation. This is an instance of Goodhart’s Law, which tech journalist Bill Doerrfeld cites, suggesting that “when a measure becomes a target, it ceases to be a good measure.” Such a focus can degrade both the culture of development teams and the quality of the code they produce.

Leaders in the tech industry are thus challenged to discern what truly constitutes valuable measurements for developer productivity. Rather than narrowly focusing on output, such as lines of code, they should aim to measure and encourage practices and create a culture that leads to high-quality, impactful software development. Google’s approach serves as a benchmark, with the company finding a correlation between improved code quality and enhanced developer productivity through a comprehensive analysis of developer inputs and outputs.

Ultimately, the industry must navigate between the benefits of new measurement tools and the potential downsides of misapplied metrics, striving to maintain an environment that fosters both innovation and satisfaction among developers.

Most successful CTOs understand that the focus should be on measuring the productivity of team dynamics and outcomes. This approach recognizes the intricate nature of software development, where collaboration and interaction are key to delivering high-quality products and services. Industry leaders and developers often agree that the collective effort of a team is more indicative of success than isolated individual contributions.

Measuring individuals

Software development is inherently collaborative, involving constant interaction among team members who contribute in varied ways. For instance, while some developers may not produce large quantities of code, their contributions in terms of guidance, problem-solving, and technical expertise are crucial to the team’s overall success. Such contributions, rich in knowledge sharing and support, are difficult to quantify but have a significant impact on project outcomes.

Measuring teams

Focusing on team productivity rather than individual output allows organizations to capture the full spectrum of a team’s effectiveness. This method acknowledges that the sum of collaborative efforts often results in a greater output than the individual parts. Metrics for team productivity might include the completion of project milestones, the quality of the final product, or the ability to meet or exceed customer expectations.

Best practices for managing and measuring team performance:

Regular one-on-one meetings: Regular interactions between managers and team members help in understanding individual challenges and achievements, providing support where needed, and aligning individual goals with team objectives.

Anonymous feedback mechanisms: Implementing anonymous feedback systems can encourage honest communication and provide insights into team dynamics, individual contributions, and areas needing improvement without fear of retribution.

Cultural considerations: A DevOps culture plays a fundamental role in an organization’s productivity. A culture that values transparency, continuous learning, and mutual support tends to enhance team, system and overall organizational performance.

By prioritizing these management practices, you can effectively assess and enhance both individual and team productivity without relying solely on quantitative metrics that may not capture the full story. This holistic approach not only supports the professional growth of individual team members but also boosts the collective productivity and morale of the team, leading to more successful and sustainable outcomes.

Here’s a breakdown of the issues associated with various types of productivity measurements:

Issues with input measurements

Input-based metrics, such as hours logged in front of a computer, often incentivize the wrong behaviors. In environments where time spent is overly valued, developers might prioritize clocking hours over delivering quality work. This can lead to unproductive competition, where the goal becomes appearing busy rather than being effective, potentially resulting in subpar work outcomes.

Issues with output measurements

Metrics like lines of code or the number of commits can encourage quantity over quality. Developers may produce excessive or substandard code just to boost these numbers, which doesn’t necessarily contribute to the project’s success and can lead to technical debt.

Issues with outcome and impact measurements

Assessing a team’s performance based on broader business outcomes, such as profit increases or expanded market share, is challenging. These outcomes are influenced by myriad factors beyond any one developers’ control, making it difficult to attribute them directly to team performance.

Improving developer productivity involves a holistic approach that considers various factors, from workplace culture to the tools and processes used. Here’s a detailed look at the key levers that can significantly boost productivity:

Build a positive culture

Creating a conducive work environment is crucial. This involves fostering a culture that emphasizes collaboration over competition, encouraging the sharing of knowledge and expertise and learning from – not punishing mistakes. A transparent environment helps in minimizing misunderstandings and aligning goals. Prioritizing work-life balance, maintaining sufficiently staffed teams and on-call rotations, reducing stress, and focusing on physical and mental health are essential. Supportive management, realistic project timelines, appropriate task assignments, and constructive feedback all contribute to a productive work environment that minimizes distractions.

Equip your team with the right tools

Equipping developers with the right tools significantly affects their efficiency. Integrated development environments (IDEs), advanced debuggers, and the latest programming languages and frameworks are fundamental. Observability tools are also critical as they help in quickly identifying and resolving issues, thereby speeding up the troubleshooting process. Whenever possible, investing in comprehensive platforms that integrate multiple tools can reduce the learning curve associated with using disparate solutions.

Implement proven processes

Adopting effective development methodologies like Agile, Scrum, or Kanban can streamline the development process and enhance productivity. These methodologies, when properly implemented, provide a structured yet flexible framework that can adapt to project needs and changes, facilitating faster and more efficient project completion. What is most important is finding the methodology that works for your teams and your company and not being afraid to adapt them for your needs.

Utilize automation appropriately

Automation plays a large role in increasing productivity by reducing toil, or the time spent on repetitive tasks. Implementing pre-commit hooks, automated testing, continuous integration, and continuous deployment helps in speeding up the development cycle, allowing developers to focus more on fulfilling creative problem-solving and less on frustrating mundane tasks.

Provide training and learning opportunities

Continuous learning is key to keeping developers up to date with the latest technologies and methodologies, which in turn can boost productivity. Providing access to training sessions, workshops, and seminars allows developers to acquire new skills and refine existing ones, fostering a culture of continuous improvement. Offer a learning and development budget so developers can select resources tailored to their learning style. Consider starting Communities of Practice for developers to share learnings, announcements and conference reports within a given area like Front End, Reliability, Infrastructure, etc.

Prioritize code quality

Prioritizing code quality over speed or volume ensures that the codebase remains clean, tested, well-documented, and easier to maintain. This approach helps in reducing technical debt and simplifies future enhancements or modifications, leading to increased long-term productivity.

Chronosphere’s cloud native observability platform offers several strategic advantages that enhance developer productivity in cloud native environments. Here are four ways Chronosphere is driving improvements in DevOps efficiency according to insights from a Market Insight Report by 451 Research, part of S&P Global Market Intelligence:

1. Tailored observability tools: Chronosphere provides developers with observability tools that are specifically aligned with their operational workflows. By automatically filtering out irrelevant data, the platform ensures that each team receives only the data that is pertinent to them. This reduction in noise is crucial for teams in modern environments that operate in small, interdependent units, enhancing focus and efficiency.
2. Data analysis and refinement at scale: The platform enables SRE teams to analyze their observability data and usage to distinguish between useful data and waste. This capability allows teams to refine their data streams to enhance relevance and utility without the need to modify source code or perform redeployments, thereby avoiding disruptions in their workflow.
3. Optimization for speed and performance: Chronosphere emphasizes rapid loading of real-time dashboards and swift responses to queries, minimizing the time developers spend waiting for data. The platform is designed to facilitate quick remediation processes, constantly refining these operations to boost overall performance and productivity.

Approachable for all engineers: Unlike many observability tools that cater predominantly to power users, Chronosphere is built to be user-friendly for engineers at all levels. This empowers developers to lead their own investigations and reduces the need for escalating to more experienced engineers freeing up their time for more strategic work.

While measuring developer productivity is important, it is crucial to approach it thoughtfully to avoid counterproductive consequences. Metrics based on efforts or inputs, such as hours worked or lines of code, should never become the primary targets; rather, they should provide context to more meaningful outcomes and impacts.

For example, the relationship between the size and frequency of pull requests (PRs) and the incidence rate of coding issues can offer insights into potential areas for improvement without making these metrics targets in themselves. Don’t jump to quick conclusions about a developer or team and consider the broader context when interpreting these measurements – could a recent slate of reliability issues be tied to aggressive mandated deadlines or mounting technical debt?

Ultimately, the goal is to enhance productivity not just for its own sake but to foster the creation of valuable, high-quality software by happy and healthy developers and operators sustainably. Chronosphere’s observability solution supports this goal by equipping developers with the tools and insights needed to achieve efficient and effective outcomes.