The rapid adoption of AI-assisted tools in software development has rekindled interest in engineering productivity. However, a holistic understanding of these tools’ real-world impact on engineering productivity remains caught between marketing claims of substantial gains 1, 2, 22 and initial studies focused on task-level improvements. 5, 8, 10, 20, 23 While the marketing narrative is challenged by emerging real-world evidence, 11 the community still lacks deep, non-anecdotal studies that not only observe but also explain the disconnect. In this article, we aim to do just that. We introduce and investigate the efficiency-throughput gap, a phenomenon where individual developer efficiency gains fail to translate into measurable organizational throughput. Grounded in the Organizational Ohm’s Law (OOL) framework, our real-world field study provides a methodology to both empirically document this gap and offer a data-driven explanation.
Research motivation and principles. The motivation of this work is to understand the impact of GitHub Copilot on engineering productivity to inform the strategic integration of generative AI into existing product development. Specifically, we aim to understand how engineers use GitHub Copilot, and whether its adoption yields measurable engineering productivity gains. Furthermore, this research seeks to identify the requirements for effective measurement of engineering productivity in corporate settings.
To facilitate practical business decisions, our field study was designed to reduce disruption to established engineering practices and to reflect authentic real-world conditions rather than artificial laboratory settings. Our experimental design adheres to three guiding principles: first, to generate actionable business insights while satisfying academic curiosity; second, to minimize the burden on participating engineers and avoid introducing unnecessary research-only processes; and third, to measure relevant data and behaviors as accurately as possible to reflect real-life operational contexts.
The act of measurement can inherently alter the state of a system, a phenomenon observed across diverse domains, from electrical circuits to quantum mechanics. This effect is particularly pronounced in social and management contexts, where the establishment of specific metrics can influence behavior, potentially leading to tactical actions or even the gaming of systems to achieve favorable results. 6, 13 Recognizing the importance of minimizing such reactive effects and obtaining data reflective of genuine practices, throughout our study we prioritized transparency with participants regarding data-collection objectives, data linkage, and confidentiality. To further mitigate behavioral changes induced by the study, and unlike many observational studies that actively monitor participant activities prospectively, our primary analysis relied mainly on historical engineering metric data from recruited participants.
Organizational Ohm’s Law. The study’s experimental design and methodology are informed by the OOL framework, a concept we previously developed to analyze engineering productivity. 15, 16 The framework draws an analogy between electrical circuits and organizational systems, stating that outcome current, and hence organizational productivity, is proportional to outcome-output efficiency and organizational potential, and inversely proportional to organizational resistance. By approximating organizational resistance through quantifiable time allocations, the law shows that engineering productivity is positively correlated with the percentage time spent on core software engineering activities (i.e., coding and testing), the relative average engineer motivation, and the relative average engineer skill levels.
Leveraging this framework, we measured engineer time allocation before and after GitHub Copilot adoption across various areas, including coding, testing, communication, waiting for decisions and dependencies, documentation, engineering process, company process, and inefficiency of engineering infrastructure, 14, 21 alongside relative employee motivation 12, 17, 18 and skill ratings. The data allowed us to comprehensively assess the impact of GitHub Copilot adoption using a method grounded in a theoretically sound and intuitively sensible framework.
Our contribution. First, we present a holistic, real-world enterprise assessment of GitHub Copilot’s impact, identifying and providing empirical evidence for the efficiency-throughput gap. Our findings challenge the prevailing narrative of instant AI productivity gains: Despite observing improvements in engineer motivation and perceived skills, as well as reduced working hours, we found no immediate increase in key engineering metrics such as monthly pull requests and lines of code. This result underscores that effectively realizing the potential of AI-powered tools requires continued innovation and optimization of software engineering operations. Second, we introduce a multi-method framework for holistically assessing the impact of AI tools in an enterprise setting. By integrating the OOL framework, our human-centric survey, objective engineering metrics, and participant information, our study provides a methodology that organizations can adapt to continuously evaluate the impact of AI.
Our Approach
Experimental design. This field study was conducted within the engineering organization of Okta, Inc., a publicly traded U.S. enterprise software company providing cloud-native identity and access management for modern security. Aiming for research with direct practical impact, we integrated our study with the GitHub Copilot rollout within our company. This real-world context facilitated a comprehensive evaluation of Copilot’s effect on multiple engineering teams using a multi-method quantitative design. This approach involved triangulating data from subjective Copilot productivity surveys, objective GitHub Copilot and engineering metrics, and participant background information.
To control for confounding variables from other AI tools, we leveraged a preexisting policy that discouraged exposing company code to external, non-sanctioned LLMs. A preliminary survey before the rollout confirmed this policy’s effectiveness, reflecting minimal use of other AI tools and giving us high confidence in our baseline measurements (see Appendix A.1 for details). As GitHub Copilot was the only AI-powered developer tool officially deployed at scale during the study, we can attribute the observed changes primarily to its introduction.
Engineering productivity survey. To understand GitHub Copilot’s impact on engineering productivity following its initial organizational rollout, we conducted a survey several months after the GitHub Copilot rollout using Qualtrics, a commercial Web-based survey platform. The survey instrument gathered data on participants’ background, their average weekly time allocation across various work activities, Likert-scale ratings of their motivation and skills, Likert-scale assessments of code quality and test frequency both before and after Copilot usage, the frequency of Copilot utilization, areas of perceived Copilot usefulness, a Likert-scale rating of Copilot’s perceived productivity enhancement, concerns regarding its use, and open-ended comments on engineering productivity or GitHub Copilot experiences (see Figure A1 in the appendix for an example of the survey instrument). The survey was designed to be brief, requiring approximately five minutes to complete.
Participant recruitment was conducted in partnership with various engineering teams as part of their GitHub Copilot deployment. Between November 18 and December 20, 2024, we used Slack and email to invite 261 individuals who had been granted Copilot licenses between July and September 2024 (see Figure A2 in the appendix for the recruitment message). We received 97 responses, 95 of which included email addresses, enabling us to link their survey data with their GitHub Copilot and engineering metrics. Participation in the survey was voluntary and uncompensated. (The distribution of professional experience among survey respondents is shown in Figure A3 in the appendix.)
GitHub Copilot and engineering metrics. Transparency regarding data linkage was maintained throughout the study. Both the survey instrument and participant recruitment communications (via Slack and email) explicitly stated that participants’ email addresses would be used solely to correlate survey responses with GitHub Copilot and engineering metrics for this research on engineering productivity. Participants were also assured that all data would be analyzed anonymously and kept confidential. Consequently, we used participants’ email addresses as unique identifiers to link their subjective survey input with objective metrics.
We extracted relevant metrics using Faros.ai, a a commercial engineering data platform that integrates with our company’s suite of engineering software applications (such as GitHub and JIRA) via representational state transfer APIs (REST APIs). Faros.ai transforms the retrieved data into its proprietary schema, enabling us to query and extract key metrics through its dashboards. The metrics included Copilot activation date, Copilot usage statistics, pull request (PR) count, PR comment count, PR review count, and lines of code (LOC) contributed. To establish a before-Copilot baseline, we used data from April, May, and June 2024. After-Copilot data was represented by the period of September, October, and November 2024.
Participant information. The engineering productivity survey included questions about participants’ engineering experience, capturing variables such as years of experience in software development and their primary programming language. To provide further context, we enriched the survey data with information from our company’s Workday system, including variables such as hire date, job title, job level, job location, and engineering team.
Data preparation.
Data cleaning. The collected datasets underwent a data cleaning and preprocessing phase to address potential errors and inconsistencies across both the survey responses and metrics data. This process involved correcting participant identifiers, verifying and adjusting anomalies in self-reported data, and handling incomplete responses. We adopted a data strategy of retaining all responses and metric data for further analysis. While this necessitates more detailed processing and analytics, it allows us to maximize our dataset, which is crucial given our moderate sample sizes. More importantly, this approach more accurately reflects the complexities of real-world engineering productivity measurement, identifying edge cases and addressing them through subsequent analyses. This resulted in a final dataset of 95 valid responses, of which 74 were complete for the before-and-after analyses. (See Appendix A.3 for a detailed description of the data-cleaning procedures.)
The raw GitHub Copilot and engineering metrics, extracted monthly, exhibited inherent variability. To mitigate these fluctuations and account for onboarding and departures, we employed a three-month rolling average for both the before-Copilot (April to June 2024) and after-Copilot (September to November 2024) periods. We then calculated the number of valid monthly data points for each participant and computed the mean and standard deviation based on these valid months. The analysis comparing the three-month average after-Copilot metrics with metrics calculated only for the months after individual Copilot license assignment yielded similar conclusions. Consequently, the subsequent analysis used the three-month average engineering metric data from September to November 2024 to ensure consistency and reduce variability.
Data integration. To integrate data obtained from multiple sources and systems, we used participants’ email addresses as a unique identifier for record linkage. Data cleaning, joining procedures, and insight analyses were implemented using Python within Jupyter Notebook. Following data preparation, we generated a unified dataset in CSV format, encompassing the survey responses, GitHub Copilot and engineering metrics, and enriched participant information. This integrated dataset comprised up to 150 direct and derived attributes for each participant, enabling multifaceted analyses of engineering productivity for survey participants. For clarity, we focus on the analyses and results related to GitHub Copilot’s impact.
Assessing the Impact of GitHub Copilot
GitHub Copilot utilization. The survey included a question asking participants about their recent GitHub Copilot usage: “How often have you used GitHub Copilot recently?” We also collected objective GitHub Copilot usage metrics for the same period. Our analysis showed substantial variability in Copilot usage among participants, with 7% reporting no usage during the three months following Copilot deployment, illustrated in Figure 1. To assess the consistency between self-reported (orange bars) and measured (blue bars) usage, we compared these two datasets. Despite inherent differences in measurement methodology, data source, and data type, statistical tests indicated a statistically significant (p 0.05), we applied the paired t-test. The magnitude of the change was measured using Cohen’s d as the effect size.
-
If the differences were not normally distributed but were symmetric, we used the Wilcoxon signed-rank test. The magnitude of the change was measured using rank-biserial correlation (r) as the effect size.
-
If the differences were not symmetric, we applied a sign test, implemented as a two-sided binomial test on the count of positive vs. negative differences. The magnitude of the change was measured using Cohen’s g as the effect size.
The results of these statistical tests are visualized in Figures 2 and 3. For each response or metric, the corresponding subplot displays the before-and-after means, the type of statistical test performed, the resulting p-value, the effect size test, and the corresponding value, as well as the data pair count. If the selected statistical test yielded a statistically significant result (p ≤ 0.05), the before-and-after bars in the subplot are colored green; otherwise, they are colored pink.
Engineer time allocation. To understand engineer time allocation, we categorized engineering activities into several distinct categories and asked the survey participants to report their weekly time spent on each. Figure 2 (a–j) illustrates the mean reported time allocation and corresponding 95% confidence intervals before and after GitHub Copilot usage. Following Copilot adoption, the participants reported a reduction in the average time spent across all activity categories. The largest time savings were reported in coding (1.06 hours per week) and testing (1.56 hours per week). Overall, participants reported an improved work-life balance, with average weekly work hours decreasing from 44.77 to 40.83. Statistically significant reductions (p ≤ 0.05) were observed in time spent on context shifting, coding, documentation, testing, and total working hours. The effect sizes for these reductions were large, with rank-biserial r ranging from -0.51 to -0.90, 9 indicating a meaningful impact. In contrast, time allocated to more human-centric areas, such as communication, waiting for decisions and dependencies, company processes, engineering processes, and inefficiency of engineering infrastructure, did not show statistically significant changes.
Figure 2. Comparison of engineer time allocation, motivation, and skills before and after GitHub Copilot for Copilot users.
The data also illustrated that approximately 30% of participants’ work time was allocated to coding, 10–15% to testing, and slightly more than 50% to other activities. An analysis of the work time allocation by job level showed a statistically significant dependency (analysis of variance (ANOVA), p ≤ 0.05). Junior engineers reported spending 50–70% of their time on coding and testing, while senior engineers allocated less than 30% to these tasks (see Figure A4 in the appendix). This pattern aligns with typical expectations for engineering career progression, where more senior roles involve leading initiatives, contributing to technical strategy, cross-organizational collaboration, and mentorship, rather than direct code writing and testing.
Engineer motivation and skills. Figure 2 (k–l) presents the before and after ratings for engineer motivation and skills with 95% confidence intervals. Participants reported statistically significant improvements in both their motivation and perceived skill levels after adopting GitHub Copilot. These improvements were accompanied by very large effect sizes (rank-biserial r of 0.95 for motivation and 1.00 for skills). 9
Pull requests. Figure 3 (a–h) presents the before and after three-month average monthly PR count, PR cycle time, monthly PR comment count (comments received), monthly PR review count (reviews performed), and their corresponding 95% confidence intervals. Contrary to findings in existing research and marketing, and perhaps unexpectedly, only the monthly PR review count demonstrated a statistically significant increase; the other PR metrics showed no significant improvements. While we treated each PR equally in our analysis of the monthly PR counts, it is important to acknowledge that PRs can vary significantly in effort. Therefore, incorporating proper PR metadata, such as PR sizing, in future analyses could further improve the representation of engineering output.
Figure 3. Comparison of engineering metrics before and after GitHub Copilot for Copilot users.
Monthly lines of code. Monthly LOC metrics were calculated as the mean of the three-month average LOC for each participant. We extracted several LOC variants, including lines added, lines deleted, diff size, and net lines added. In practice, raw LOC proved an unreliable proxy for individual coding effort, with before Copilot averages exceeding an implausibly high 30,000 lines added per month. Analysis of individual contributions uncovered that the LOC consisted of diverse code development activities beyond feature coding. For example, feature development teams may primarily engage in writing application logic (the type of coding often studied in controlled environments); engineering platform teams may generate extensive configuration code with minimal manual input; and tech debt teams can produce large LOC changes during refactoring.
To isolate LOC indicative of direct engineering authoring, we considered focusing solely on feature development teams or repositories. However, this approach was insufficient due to the colocation of feature code with dependency, configuration, and metadata code within the same repositories, often without consistent directory or naming conventions that would allow for reliable automated separation. As an alternative, we explored filtering PRs based on the magnitude of LOC changes, hypothesizing that very large changes were less likely to represent direct, line-by-line coding effort. Drawing upon industry best practices for PR reviews and interviews with our engineers regarding common organizational practices, we developed a heuristic rule to identify PRs potentially containing engineering-authored code: (Lines added and deleted 5 comments) OR (Lines added and deleted 7 comments) OR (Lines added and deleted 10 comments). 3, 4, 7 LOC contributions from PRs not meeting this heuristic were excluded from participants’ monthly LOC totals. These recalculated LOC metrics values were not overly sensitive to variations in the heuristic rule. Moreover, recalculating the various LOC metrics after this filtering yielded values within a more reasonable range for direct coding effort, 16, 19 as shown in Figure 3 (e–h). Nevertheless, consistent with our findings for PR metrics, neither the raw LOC metric variants nor the recalculated ones showed statistically significant changes following Copilot adoption.
Additionally, for participants who were assigned Copilot licenses but did not use them, we observed no reported changes in survey responses and no statistically significant changes in any engineering metrics, as shown in Figures A5 and A6 in the appendix. However, the small sample size of this non-user group suggests that these null findings should be interpreted with caution.
Pareto insights. We also investigated the hypothesis, consistent with existing literature, 20 that AI tools benefit less experienced engineers more. The goal of this analysis was to see if Copilot led to a more even distribution of work output. To test this, we conducted a Pareto analysis of PR count before and after GitHub Copilot for participants who used it (Figure A7). Prior to Copilot, 46% (34 out of 74) of participants contributed to 80% of the total PR output. Following Copilot implementation, this proportion increased to 53% (39 out of 74) of participants, accounting for 80% of the slightly increased total PR output. The observed percentage shift implies that Copilot may have led to a more even distribution of PR contributions among engineers, potentially benefiting those with lower initial PR output. However, statistical analysis found that the difference between the before and after distributions was not statistically significant.
GitHub Copilot’s benefits and concerns. To assess GitHub Copilot’s benefits and concerns, we surveyed participants with multiple-choice questions. The first, “Which areas do you find GitHub Copilot useful?” offered categories aligned with engineer time allocation, motivation, and skills, including a detailed breakdown of coding tasks from simple to complex (Figure 4 a). Participants reported Copilot to be most useful for boilerplate code, function templates, queries, configuration files, and testing. They found it less capable, however, of developing front-end components and back-end services, writing complex logic or algorithms, and supporting human-centric enterprise development activities. The second question, “What are the reasons you don’t use GitHub Copilot after being assigned a seat?” highlighted that the primary reasons for non-use were concerns about Copilot’s accuracy and lack of time and knowledge to use Copilot (Figure 4 b). Despite the limitations, 94 participants rated GitHub Copilot’s impact on overall productivity at an average of 3.95 out of 5 on a Likert scale, indicating a reasonably positive perception.
Figure 4. Engineering perspectives on GitHub Copilot: (a) Benefits (n=92) and (b) Concerns (n=79).
We then sought to validate whether the self-reported time reductions observed in activities like coding and testing (Figure 2) were statistically linked to these perceived benefits. Our hypothesis was that engineers who perceived Copilot as useful for a specific activity (e.g., “testing”) would be the same engineers who reported a time reduction for that activity. To test the hypothesis, for each engineer time-allocation activity, we calculated both value and binary Pearson correlations, along with their corresponding p-values, between time-allocation changes and GitHub Copilot benefit mentions, to test the basic association. Additionally, we performed a Chi-square test (χ2 test) to assess the association between binary time-allocation changes and benefit mentions. Finally, we used a two-sample t-test to compare the mean time-allocation changes between engineers who mentioned Copilot benefits for a specific activity and those who did not.
Our analyses demonstrated relatively strong negative value correlations between time-allocation changes and benefit mentions for context shifting, engineering process, testing, and inefficiency of engineering infrastructure. Relatively strong binary correlations were found for context shifting, coding, documentation, engineering process, testing, and inefficiency of engineering infrastructure. The Chi-square test indicated a significant association between the binary changes in context shifting, coding, documentation, engineering process, testing, and inefficiency of engineering infrastructure, and their corresponding benefit mentions. Furthermore, the t-test showed significant differences in the mean metric change values of context shifting, coding, engineering process, and testing between the benefit mentioned and not-mentioned groups (see Appendix Table A1 for details).
Collectively, these analyses suggest a reasonable relationship between the four time-allocation activities that showed statistically significant reductions before and after GitHub Copilot adoption (context shifting, coding, documentation, and testing) and the perceived benefits of Copilot.
Discussion
This work provides a multifaceted examination of GitHub Copilot adoption, yielding insights into software productivity measurement and the tool’s impact on enterprise software development. On the one hand, we demonstrate the utility of OOL as a practical and holistic framework for understanding engineering productivity by integrating engineer activity (time allocation), mental state (motivation), and capability (skills) alongside engineering metrics. On the other hand, our real-world investigation highlights the inherent gaps in accurately measuring engineering productivity, suggesting the potential value of more rigorous practices, such as consistent repository directory labeling, pull-request sizing, and the identification of AI-generated code to improve future assessments.
Regarding GitHub Copilot’s impact, our findings reveal a critical distinction between individual efficiency and organizational throughput. While GitHub Copilot usage varied significantly among engineers, the tool demonstrably fostered positive changes in perceived engineer value, reduced time spent on various engineering activities, and boosted motivation and perceived skills. Yet we observed no immediate direct correlation with gains in key engineering metrics. This resulted in an individual efficiency gain (the same work in less time) but no statistically significant organizational throughput gain (no increase in monthly PR count or lines of code). According to the OOL framework, we hypothesize that productivity improvements from such tools occur in three stages:
-
Efficiency gains: The tool boosts average individual potential (motivation, skills) and reduces resistance on specific tasks such as coding and testing. However, the overall outcome current remains unchanged. The efficiency gains benefit individuals with a better work-life balance. Our data suggests early signs of this first stage, but not yet the second.
-
Productivity gains: To increase the outcome current, the organization must consciously reduce the total resistance and further increase individual potential. This requires upskilling developers to leverage the tool’s functionalities, adjusting productivity expectations, and optimizing development workflows to align with AI-augmented developer capabilities.
-
Organizational optimization: The organization can redesign its organizational circuit to lower human-centric resistance. This involves structural changes, such as creating more autonomous teams or increasing individual scope to reduce the overhead from communication and decision making, as well as exploring effective modes of human-AI teaming.
This work highlights that to truly achieve organizational productivity gains, deploying an AI-powered tool is not enough. It requires a deliberate effort from engineering and management to adapt, learn what the tool can and cannot do for their specific problems, as the tool’s functionalities evolve quickly, and reinvent teams, engineering processes, and infrastructure to capture the potential efficiencies. The findings informed our organization’s post-study strategy. Recognizing the strong positive impact on individuals, we accelerated Copilot adoption by removing management approval while simultaneously monitoring usage. We began sharing best practices to improve tool utilization, building a shared understanding of its current constraints, and identifying new challenges. Furthermore, we are embedding AI impact measurement into our regular engineering process and introducing a broader range of AI-powered tools, such as Copilot agent mode and various vibe-coding applications. The endeavor represents the beginning of a co-invention of the tool, the team, the process, and the infrastructure required to fully explore and realize the potential of AI-driven productivity gains.
While this study provides comprehensive analyses and insightful findings, its scope is subject to several limitations. First, the research was conducted within a single company and involved a moderate number of participants, potentially limiting the generalizability of the results. Furthermore, the sample size means some observations of statistical insignificance could be due to a lack of statistical power rather than an absence of effect. Second, as the study captured the initial rollout phase of GitHub Copilot, the observed impacts may not fully reflect its longer-term effects. However, our subsequent monitoring of PRs and LOC for participants from December 2024 to May 2025 showed no statistical improvements, supporting the point that time alone is insufficient to translate individual efficiency into organizational throughput. Third, the objective output metrics we used (e.g., monthly PR count and monthly LOC) are, at best, proxies for engineering productivity and do not capture the full picture of engineering output. Fourth, this work primarily focused on the impact on engineering output metrics, rather than broader business outcomes such as accelerated time to market or product quality improvements. Fifth, the analysis was based on the organization’s adoption of GitHub Copilot’s AI-assisted coding functionalities available at the time, including autocomplete and chat. Given the rapid evolution of AI-powered development tools, the implications of more advanced capabilities, such as retrieval-augmented generation (RAG) over enterprise codebases or deeper engineering workflow integrations, warrant separate investigation. Ongoing advances in these AI-powered tools could alter their impact. For future work, we recommend prioritizing longitudinal studies across diverse organizations to examine the long-term effects of AI-powered tools on engineering productivity, thereby providing valuable technical, business, and economic insights for researchers and practitioners.
Summary
This field study of GitHub Copilot productivity impact moves beyond the hype of instant gains by identifying an efficiency-throughput gap, revealing a more nuanced reality than that often presented in existing research and marketing. Our findings underscore the challenges that organizations face in translating AI-driven individual efficiency gains, which we observed as improved engineer sentiment and a reduction in working hours, into immediate and quantifiable organizational productivity throughput, where we found no statistically significant improvement in output metrics. By integrating the Organizational Ohm’s Law framework, our human-centric survey, and objective engineering metrics, this study also provides a robust and repeatable methodology. It offers a practical toolkit for organizations to move beyond one-time snapshots toward a continuous assessment process of AI’s evolving impact, paving the way for iterative co-invention and, ultimately, tangible engineering productivity gains.
Acknowledgments
We thank Bhawna Singh, JC Tierney, Robert Lucero, Demi Dina, Pablo Terradillos, Steve Workman, Vikas Gupta, and Chanaradee Leelamanthep for their valuable collaboration. We also extend our appreciation to Shubha Nabar and Bill Farrakh from Faros.ai for their support, and to the anonymous reviewers for their insightful feedback.
Submit an Article to CACM
CACM welcomes unsolicited submissions on topics of relevance and value to the computing community.
You Just Read
Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot
View in the ACM Digital Library
This work is licensed under a Creative Commons Attribution International 4.0 license.
© 2026 Copyright held by the owner/author(s).
DOI
10.1145/3797488
Join the Discussion (0)
Become a Member or Sign In to Post a Comment
BLOG@CACMSep 21 2026
The Promulgation of Values by Generative AI
BLOG@CACMSep 18 2026
Artificial Intelligence and Machine Learning
BLOG@CACMSep 18 2026
Can Synthetic Training Data Survive Its Own Regulation?
Dharmateja Priyadarshi Uddandarao
Artificial Intelligence and Machine Learning
Shape the Future of Computing
ACM encourages its members to take a direct hand in shaping the future of the association. There are more ways than ever to get involved.
Communications of the ACM (CACM) is now a fully Open Access publication.
By opening CACM to the world, we hope to increase engagement among the broader computer science community and encourage non-members to discover the rich resources ACM has to offer.
