Episode 35 · 26 minutes

Accelerate: The Science of Lean Software and DevOps

by Nicole Forsgren, Jez Humble and Gene Kim, 2018

The four measures, the two pairs, the twenty four capabilities and the three clusters, held to the standard the book's own title claims: the sampling, the cross sectional design, and the authors' own statement that they cannot show cause. For neurodivergent leaders: four measures of what shipped, and none of who was at a desk.

Transcript

The whole episode in text, 117 lines. Miles and Nora are synthetic voices reading a script approved by Logan Williamson, LPC.

Read the full transcript

Miles: Every year for four years, a link goes out. It lands in a mailing list, it gets posted on social media, and then it travels wherever the people who received it decide to send it.

Miles: Twenty three thousand people fill it in. Two thousand organizations, from firms with under five employees to ones past ten thousand.

Miles: And what comes back contradicts the rule the whole trade was built on. Go fast or be safe, pick one. In this data the teams shipping most often were the ones breaking things least.

Miles: Today: Accelerate, by Nicole Forsgren, Jez Humble and Gene Kim. In two sentences: four numbers describe how well a software team delivers, and not one of them is about a person. Five days on how work moves, and this is the only one that went out and gathered data on whether any of it was true.

Miles: Sixty Books. Presented by Stoar Consulting and Coaching.

Miles: Number thirty five, and it closes a run. Sixty Books, I'm Miles. The bargain doesn't shift: one management title an episode, sixty of the best of the last century, each handed the same two questions. What's in it for somebody leading, or answering to, a brain that came off a different drawing. And what stands up now AI works alongside you all day. Nora keeps the research, and she's been waiting for this one.

Miles: Nora, we ended yesterday on a surgeon who put a card into an operating theatre and then counted what happened to the patients. Today somebody does that to software.

Nora: With a questionnaire, which isn't the same instrument, and that difference runs through the whole episode. What they share is the intent. Both went looking for a number instead of a testimonial.

Miles: And you parked this three days ago.

Nora: I did. I named the programme, gave you one sample size and spent nothing else, because it belongs here. There's also a chapter in this book called Introduction to Psychometrics.

Miles: In a management book.

Nora: Yeah. In a management book. Which tells you what the authors think they're doing, and it's why this one gets a harder look from me than the four before it, not a gentler one.

Miles: Tell me about the programme, because that's what the book is standing on.

Nora: It begins as an industry survey run out of a company selling infrastructure automation, and the first large one goes out in December of twenty thirteen. Over nine thousand two hundred answers, from a hundred and ten countries, which for this field was unheard of.

Miles: And then a researcher turns up.

Nora: Forsgren joins, and it stops being a poll and starts being built like a study. Constructs defined before the data arrives. Questions written so each measures a single idea. Reliability and validity checked before anybody looks at a result. The group takes the name DORA.

Miles: And how much feeds the book.

Nora: Four annual rounds, twenty fourteen through twenty seventeen. Pooled, that's the twenty three thousand responses and the two thousand organizations. The last year alone was about three thousand two hundred people.

Miles: And how did they find them?

Nora: It's in the book, in the authors' own words, and I want it out here rather than saved for the evidence segment. Each year they emailed their own mailing lists, put the survey on social media, and asked people to pass it along. The book names the method. It's called snowball sampling.

Miles: So the sample is whoever they could reach.

Nora: Whoever was in reach and chose to answer. Hold that, because it isn't a scandal and it isn't nothing, and the book stating it on the page is what makes the rest of this conversation possible.

Miles: The authors.

Nora: Nicole Forsgren is the lead investigator. A doctorate in management information systems and a master's in accounting, both from the University of Arizona. Before the research she'd worked as a developer, a system administrator and a performance engineer, and taught as a professor at more than one university. So the person running the statistics had done the job the statistics are about.

Miles: Second.

Nora: Jez Humble. Physics and philosophy at Balliol College, Oxford, then a master's in ethnomusicology in London. Twenty years of code and infrastructure across three continents, a spell inside the United States federal government's own digital team, and lecturing at Berkeley. His first book, Continuous Delivery, came out in twenty ten and took a Jolt award, and it's the practice half of everything this one measures.

Miles: Which is the college yesterday's author read philosophy, politics and economics at.

Nora: It is. And the third you've met: Gene Kim, who sat in this chair three days ago as a novelist. Same man, and the two books make a pair. One dramatizes the practices. This one tries to test them.

Miles: And the book's own facts.

Nora: March of twenty eighteen, from the press Kim founded. Two hundred and eighty eight pages, roughly a third of it Part Two, which is nothing but the method. It won the Shingo publication award, and Martin Fowler wrote the foreword.

Miles: Walk me through it, in the book's order.

Nora: Four measures, arranged as two pairs. The first pair is tempo. Deployment frequency: how often does this team put code into production for the service they work on. Delivery lead time: from a change being committed to that change running in production for real users.

Miles: And the second pair.

Nora: Stability. Time to restore service: something breaks in production, how long until it's working again. Change failure rate: what share of changes to production either degrade the service or need remediating afterwards.

Miles: And what's deliberately absent.

Nora: Everything you'd expect to find. No hours. No lines of code. No story points, no tickets closed, no utilization, no headcount. All four belong to a stream of work rather than to any individual. Keep that, because our segment is built on it.

Miles: Right.

Miles: And the pairing is the argument.

Nora: The pairing is the entire argument. For decades the trade was taken as settled. You could ship quickly or you could ship safely, so every control anybody built was a brake.

Miles: So the trade-off everybody believed in.

Nora: Yeah. The trade-off everybody believed in turned out, in this data, to be mostly a change board. The two pairs move together. Teams deploying most often had the lowest failure rates and recovered quickest, and external change approval came out negatively associated with tempo and with stability both.

Miles: That's a lot of meetings.

Nora: It is. And it's the finding with the most riding on the method, so I'll hand that over properly in a minute.

Miles: Then the groups.

Nora: Cluster analysis, run afresh each year. You give the four measures to a procedure that sorts answers by similarity without being told what to look for, and every year it came back with three: high, medium, low. Those are the book's. The word everybody now uses, elite, isn't in it. It arrives the year after, in the next report, when the top group divided.

Miles: And the distance between top and bottom.

Nora: In the twenty seventeen data the book puts high performers at forty six times the deployment frequency, lead times four hundred and forty times shorter, recovery a hundred and seventy times faster, changes a fifth as likely to fail. Every one of those is a ratio between two self reported brackets. The year before, the same programme's figures were two hundred times, two and a half thousand times, twenty four times and three times.

Miles: So the ratios move.

Nora: Enormously, and the direction never does. Which is how I'd read all four. The ordering is the finding. The multiplier is a headline.

Miles: And the practices underneath.

Nora: Twenty four capabilities in five families, running from version control over everything including the infrastructure, through test automation and small batches, out to a culture measure taken from a patient safety journal.

Miles: And at the far end.

Nora: Organizational performance, and burnout. The first one is people rating their own organization against its competitors on profitability, market share and productivity, so that's the same questionnaire again. The second is the finding I'd keep if I could only keep one. Continuous delivery went with less burnout and less deployment pain, and nobody had wired the technical argument to the human one before, in a book this industry would read.

Miles: Now the part the book asked for by calling itself science.

Nora: And that's the correct frame. This one gets held to what it claims rather than to what the shelf usually gets. Four things: the sample, the design, the groups, and whether anybody outside has done it again.

Miles: Sample.

Nora: Self selected and self reported, both. It's a questionnaire, so every figure in it is somebody's account of their own team. Recruitment ran through the authors' mailing lists and social media and whoever forwarded it on, so the people likeliest to be in it are the people already interested in the thing being measured. The book says that on the page, which is why it sits here as a description and not an accusation.

Miles: And what does it do to the numbers.

Nora: Two things. It can't tell you what the industry looks like, only what this pool looks like, so a sentence of the form "a quarter of teams are elite" was never available from data of this kind. And where the predictor and the outcome come out of one questionnaire filled in by one person on one afternoon, part of the correlation between them belongs to the person rather than the world. That has a name in survey research and the authors take it on. It limits how tightly you read the estimate. It isn't a reason to bin it.

Miles: Now the design.

Nora: Cross sectional, and the book states it flatly. Each respondent answered once, at one moment, so the data holds no before and no after. Then the statistics on top: latent variables, meaning something like culture that you can't measure directly gets measured through several questions at once, and structural equation modelling, which fits a whole diagram of relationships in one pass rather than one link at a time.

Miles: Plain words for what that can and can't do.

Nora: It can tell you which quantities move together, and how strongly, across a whole set at once while holding the others still. What it cannot do is tell you which one came first. A measurement taken at a single moment has no clock in it, so the arrows in those diagrams point the way the theory says, not the way the data says. Reverse any one and the data fits just as well.

Miles: So when the book says one thing drives another.

Nora: It's a hypothesis the numbers are consistent with. And I'd rather quote the authors than characterize them, because they've answered it in print. Replying to a reviewer, they wrote that they can use words like drives and predicts and impacts, but not causes. And that this is not a randomized controlled experiment, and one would be impossible to run in software engineering.

Miles: Which is a fair thing to write.

Nora: It's accurate, and more than most books here offer. It also means the word on the cover carries more than the design can lift. Both are true at once, and neither is a complaint about anybody.

Miles: And the groups.

Nora: Sturdiest part, and the most modest. Cluster analysis doesn't test a claim, it sorts. Hand it these four measures and it found three groups, four years running, and a peer reviewed version of that exists: the same team, an information systems conference in twenty twenty, seven thousand five hundred and twenty two respondents, hierarchical clustering, three profiles. So the grouping replicates inside the programme. What it isn't is a league table, and the research group now says so in writing: the measures belong to a service, the aim is your own team over time, and they're not for competing between teams.

Miles: And outside the programme.

Nora: Very little, and I went looking. A Swiss group reviewed every article citing this work, hunting for formal research on how you'd measure the four, and reported finding none in the peer reviewed literature. Their own study was one team of ten at a postal company, six of whom answered the survey. Another group instrumented a live system at a Zurich firm, twelve developers and thirty seven services over four weeks, and found something worth having: measured automatically, service by service, only about a quarter of the services tracked the team's overall figure closely.

Miles: So nobody's repeated it.

Nora: Not with an independent sample, and not forward in time. And the underlying data can't be released, because the people who filled it in were promised it wouldn't be. That last one isn't a flaw in the research. It is a reason no outsider can reanalyze it.

Miles: And the tier statement, please.

Nora: It's the most tangled one I've had to give. Practitioner literature, like everything on this shelf, and unlike anything on it, carrying a research programme with peer reviewed pieces and a published method you can argue with. What it isn't is what the title says. Four years of cross sectional questionnaires, recruited through the authors' own networks, read through a model whose arrows come from theory, by authors who say in print that they can't claim causation. So: the best measured book on this shelf, resting on the thinnest design that still counts as measurement. Write down both halves, and neither one is a verdict on a person.

Miles: Our half, and you told me this was the actual reason we're here.

Nora: Because of what those four leave out, and nobody set out to do us a favour. Not one of them asks when anybody was at a desk. Judge a team on how often it ships, how fast a change reaches a user and how quickly it recovers, and you've stopped measuring attendance. That's the most useful thing any measurement system in this series has done for a person who works differently, and it's a side effect.

Miles: Take the timing part first.

Nora: Coogan and McGowan, twenty seventeen, in a journal on attention deficit and hyperactivity disorders. A systematic review: sixty two studies, four thousand four hundred and sixty two people with ADHD. Consistent evidence of later chronotype, and of phase delay on physiological markers, including the evening rise of melatonin measured in dim light. Group findings from clinical samples. Not a test, and not a description of anybody in particular.

Miles: Plain words.

Nora: For a lot of people the internal clock runs late, and it isn't a preference and it isn't tiredness. Which makes a nine o'clock start a measurement decision rather than a fact about the work. It's a stand-in for output that costs some people two hours of their best thinking a day, and appears on no report anywhere as a cost.

Miles: And the alternative has evidence behind it?

Nora: Some, and I'll be careful with it. The Cochrane review, Joyce, Pabayo, Critchley and Bambra, twenty ten. Controlled studies of flexible working and health, twelve databases screened, ten studies clearing the bar, every one of them controlled before and after rather than randomized. The split in the results is the finding. Where the flexibility belonged to the worker, self scheduling of shifts, retirement tapered under the employee's control, they saw improvements: blood pressure and heart rate, tiredness, sleep, self rated health. Where it belonged to the employer, fixed term contracts, involuntary part time, the effects were equivocal or worse. The reviewers call their own evidence base partial and limited, so take the direction and not the size.

Miles: So control is the active ingredient.

Nora: On that evidence it looks like the active ingredient. And an outcome measure hands control over timing back without anybody negotiating for it, which is the pattern this whole run keeps producing. The adjustment that works is the one written into how the work gets counted, so nobody has to ask.

Miles: And the cost of the current arrangement.

Nora: Has been measured on the other side. Cage and Troxell-Whitman, twenty nineteen, Journal of Autism and Developmental Disorders. Two hundred and sixty two autistic adults, questionnaires on camouflaging behaviour, the settings they do it in, the reasons, and mental health. Work was one of the settings, and getting by in formal places like a job was one of the reasons. Camouflaging heavily across contexts went with poorer mental health, and so did switching it on in some places and off in others. Self report, measured once, so there's no direction in it.

Miles: And looking engaged all day is part of that.

Nora: It is a performance, it costs, and it is invisible to the people it's performed for. A team judged on what shipped this fortnight has far less of it to do.

Miles: Cautions.

Nora: Two. The first is the admission I've made every day this week. Nobody has tested flow measures with neurodivergent workers as the outcome. What I've done is read one body of work against another and join them up myself, and the join is mine rather than the literature's.

Miles: And the one after that.

Nora: Is the one that will bite. These four were built for a service and got aimed at people almost at once, and the research group's own guidance now says not to. Turn deployment frequency into an individual target and you've built a stopwatch with better branding, and handed it to whoever's likeliest to use it on the person whose output arrives unevenly. The measure protects that person only while it stays a property of the work.

Miles: Fair enough.

Miles: Sixty seconds, in this show's frame. Accelerate says four numbers describe how well a team delivers software, they come in two pairs, and in this data the pairs climb together rather than trading off. For anybody leading a brain that came off a different drawing, or living inside one, here's Monday. Measure the stream and not the seat: pick two or three outcome numbers and retire the ones that count presence, because once output is the measure, when somebody works stops being anybody's business. Keep those numbers on the service and never on a named person, and that isn't my rule, it's the research group's own. Treat your start time as a decision rather than a given, because the evidence on internal clocks says a fixed early hour is a tax some people pay and nobody records. And hand out the choosing along with the flexibility, since the only version that came through the controlled studies well was the one the worker controlled.

Miles: Bring the machines in.

Nora: The standing cautions about this whole class of tool are parked at the very end, past the music, in wording nobody touches, so I'll send you there and spend these minutes on what this book alone hands us.

Miles: Go on.

Nora: It hands us the instrument. Everybody's arguing about whether these tools help, mostly from feeling, and this book gives you four numbers that answer it in your own building. Better, they're outcome measures, so counting generated code can't flatter them. Nothing a model hands you touches any of the four until it's running in production and hasn't broken.

Miles: And somebody's already pointed it at the question.

Nora: The same programme has. Twenty twenty four report, nearly three thousand respondents, same self reported cross sectional design, same recruitment. A twenty five percent rise in their measure of AI adoption went with throughput down one and a half percent and delivery stability down seven point two, while documentation and code quality both went up.

Miles: So better parts, worse delivery.

Nora: On those figures, in that year, in that pool. Then twenty twenty five, nearly five thousand respondents, ninety percent using these tools at work against seventy six the year before, and throughput flipped positive while the instability stayed. Same design limits both times. So it's a moving picture with a wide margin round it rather than a verdict, and anybody quoting one of those years at you has picked a side.

Miles: And which of the four moves first.

Nora: The stability pair, and that's the book's own structure talking. Generation raises the rate at which change arrives and does nothing to the rate at which you recover. So change failure rate and time to restore are where a team finds out, and they're the two almost nobody instruments, because tempo is the flattering half.

Miles: Send us out on something.

Nora: The book's own move, which costs nothing and almost nobody makes. Before you decide whether these tools are helping, write your four numbers down. Then adopt, wait a quarter, write them down again. That's the method this whole book argues for, and it's available to any team with a calendar and the nerve to look.

Miles: That's Accelerate.

Miles: Next time: Good Strategy Bad Strategy, by Richard Rumelt. Five days on how work moves, and this book closes the run with four good numbers: your speed, your recovery, how often you break something. Not one of them can tell you whether the direction is right. A team can hit every mark on that list all quarter, deploying beautifully, straight into a wall. So tomorrow we start the harder job, which is deciding when you genuinely cannot know.

Nora: And Rumelt begins with a diagnosis, which after five days of measurement is a completely different instrument.

Miles: That was worth the wait, Nora. Thank you.

Nora: Same time tomorrow.

Miles: Okay. Next book.

Miles: Stoar Consulting and Coaching is behind Sixty Books. Logan Williamson is a Licensed Professional Counselor in Texas and Louisiana. You just got one book read closely, for people who lead teams and people who sit in them. That was not coaching and it was not counseling. And nothing in it was a statement about your life.

Nora: Every claim has a citation on the episode page, next to the transcript. Book thirty six tomorrow.

Miles: Two matters of record. Both voices came out of a machine. There is no Miles, there is no Nora, neither is licensed to practise anything, and none of the last twenty minutes was aimed at whoever's listening. AI sits in on the drafting of every script, and then Logan Williamson, a Licensed Professional Counselor in Texas and Louisiana, at Stoar Consulting and Coaching, edits it and signs it off.

Nora: And one note about the AI segment. The evidence that these tools help neurodivergent people at work is early. There are controlled trials now, but they measure things like conversation skill rather than how a job actually goes, and for work itself most of what exists is individual providers reporting success. So take it as promising and not proven. The tools also get rebuilt every few months, so whatever you set up today is built on something that moves.

Sources

Every claim in the episode, in the order it comes up, with where it comes from. 36 claims. Studies in psychology and medicine were checked on PubMed; everything else against the publisher, the paper or the original record.

See the sources

  1. The book draws on four years of survey data, over 23,000 survey responses, and over 2,000 unique organizations ranging from fewer than five employees to more than ten thousand.

    Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press. ISBN 9781942788331 (paperback), 9781942788355 (ebook). Published 27 March 2018, 288 pages.

    Publisher's page, IT Revolution, https://itrevolution.com/product/accelerate/ , giving the publication date of 27 March 2018, 288 pages, the ISBNs, and "four years of groundbreaking research to include data collected from the State of DevOps reports conducted with Puppet".

  2. The years pooled are 2014 through 2017, and the final year of that window had about 3,200 respondents.

    Forsgren, Humble & Kim (2018), as in item 1, which states the data come from the State of DevOps surveys of 2014 to 2017.

    2017 State of DevOps Report (Puppet and DORA), https://dora.dev/research/2017/2017-state-of-devops-report.pdf , which reports 3,200 participants in the 2017 survey and "more than 27,000 responses" collected over six years.

  3. The first large survey in the series went out in December 2013 and drew more than 9,200 respondents from 110 countries.

    2014 State of DevOps Report (Puppet Labs, IT Revolution and ThoughtWorks), https://dora.dev/research/2014/2014-state-of-devops-report.pdf , which reports more than 9,200 survey participants responding to a December 2013 survey, from 110 countries, and describes that as "more than twice as many people as our previous survey".

  4. Each year the survey was distributed by emailing the authors' own mailing lists and posting on social media, and asking recipients to share it with their networks. The book names this method: snowball sampling.

    Forsgren, Humble & Kim (2018), as in item 1: "To collect the data, each year we emailed invitations to our mailing lists and leveraged social media, including Twitter, LinkedIn, and Facebook... This is called snowball sampling, and we talk about why this was an appropriate data collection method for this research project in Chapter 15, 'The Data for the Project.'"

    2017 State of DevOps Report, as in item 2, which describes the same approach: the researchers "promoted the survey via email lists, online promotions, and social media, and also asked people to share the survey with their networks", and state that the resulting sample is "likely limited to organizations and teams that are familiar with DevOps".

  5. The surveys were rebuilt as research once Nicole Forsgren joined: constructs defined in advance, items written to measure one idea each, and reliability and validity assessed before results were interpreted.

    Forsgren, Humble & Kim (2018), Part II ("The Research"), chapters on the science behind the book, psychometrics and construct measurement, which describe defining latent constructs, writing multiple items per construct, and testing reliability, convergent validity and discriminant validity before analysis. The book's own account: "we call what we have measured a 'construct' (in this case, our construct would be 'Westrum organizational culture'), and we can then use this measure in further research."

    Nicole Forsgren, Wikipedia, https://en.wikipedia.org/wiki/Nicole_Forsgren , recording that she co-founded DORA with Humble and Kim toward the end of 2015, served as its CEO, and had previously been at Chef Software from 2014 to 2015.

  6. Nicole Forsgren holds a PhD in Management Information Systems and a master's in accounting, both from the University of Arizona, previously worked as a developer, systems administrator and performance engineer, and has held professorships at more than one university.

    Nicole Forsgren, "About", https://nicolefv.com/about , which states "PhD in Management Information Systems and Masters in Accounting from the University of Arizona" and describes prior work "as a developer, sysadmin, and performance engineer" plus service "in academia as a professor".

  7. Jez Humble read physics and philosophy at Balliol College, Oxford, took a master's in ethnomusicology in London, worked on code and infrastructure for about twenty years across three continents including a period with the United States federal government's digital services team, and lectured at the University of California, Berkeley. His first book, Continuous Delivery, appeared in 2010 and won a Jolt Award.

    Jez Humble, faculty page, UC Berkeley School of Information, https://www.ischool.berkeley.edu/people/jez-humble , giving "BA (Hons) Physics and Philosophy, Balliol College, Oxford University", "MMus Ethnomusicology, School of Oriental and African Studies, University of London", more than twenty years in software across three continents, work with the US federal government's 18F team, co-founding DORA, and lecturing at the School of Information.

    Continuous Delivery, "About", https://continuousdelivery.com/about/ , which lists him as "co-author of several books on software: Jolt Award winner Continuous Delivery, Lean Enterprise, The DevOps Handbook, and Shingo Publication Award winner Accelerate".

    Humble, J., & Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley. ISBN 9780321601919.

  8. Accelerate was published in March 2018 by the press Gene Kim founded, runs to 288 pages with roughly a third of it given to the research method, won the Shingo Publication Award, and carries a foreword by Martin Fowler.

    Publisher's page, IT Revolution, https://itrevolution.com/product/accelerate/ (as in item 1), for the date, page count and ISBNs.

    Accelerate (book), Wikipedia, https://en.wikipedia.org/wiki/Accelerate_(book) , recording the Shingo Publication Award (February 2020) and IT Revolution as publisher. Continuous Delivery, "About" (as in item 7), also describes Accelerate as a "Shingo Publication Award winner".

    Forsgren, Humble & Kim (2018), as in item 1, for the foreword by Martin Fowler and for Part II, "The Research", comprising chapters 12 to 15 plus the statistical appendices.

  9. The book's four measures of software delivery performance are deployment frequency, delivery lead time, time to restore service and change failure rate, grouped as two tempo measures and two stability measures, and defined as follows: how often the organization deploys code for the primary service the respondent works on; the time from code committed to code successfully running in production; how long it generally takes to restore service when a service incident occurs; and what percentage of changes to production either result in degraded service or subsequently require remediation.

    DORA, "DORA's software delivery performance metrics", https://dora.dev/guides/dora-metrics/ , the programme's current published guide to the same four measures.

  10. None of the four measures records hours worked, lines of code, story points, tickets closed, utilization or headcount.

    Forsgren, Humble & Kim (2018), as in item 1, chapter 2, which sets the four measures against earlier attempts to measure output and explicitly rejects lines of code and velocity as performance measures on the grounds that they reward volume and are local rather than global.

  11. The book reports no trade-off between tempo and stability: high performers did better on all four measures at once. It also reports that change approval by an external body, such as a change advisory board, was negatively correlated with both tempo and stability.

    Forsgren, Humble & Kim (2018), as in item 1. On the trade-off: "there is no tradeoff between improving performance and achieving higher levels of stability and quality. Rather, high performers do better at all of these measures," and "This huge increase in responsiveness does not come at a cost in stability, since these organizations find their updates cause failures at a fraction of the rate of their less-performing peers." On approval boards, the book poses the question "Do change management boards actually improve delivery performance?" and answers it in its own text: "(Spoiler alert: they do not; they are negatively correlated with tempo and stability.)"

  12. Cluster analysis was run on the four measures each year and returned three statistically distinct groups every year: high, medium and low performers. The word "elite" is not in the book; it first appears in the 2018 report, published after the book.

    Forsgren, Humble & Kim (2018), as in item 1: "we used a technique called cluster analysis. Cluster analysis is a foundational technique in statistical data analysis that attempts to group responses so that responses in the same group are more similar to each other than to responses in other groups," and the finding that "every year, there were significantly different categories of software delivery performance in the industry", named as high, medium and low.

    2017 State of DevOps Report, as in item 2, which states that cluster analysis identified three statistically distinct groups, high, medium and low performers, on deployment frequency, lead time for changes, mean time to restore and change failure rate.

    Accelerate State of DevOps 2018 (DORA), https://dora.dev/research/2018/dora-report/ and https://dora.dev/research/2018/dora-report/2018-dora-accelerate-state-of-devops-report.pdf , which introduces an elite tier above high performers and reports that elite performers "deploy code 46 times more frequently, have 2,555 times faster lead times for changes, 2,604 times faster time to restore service, and seven times lower change failure rates" than low performers.

  13. In the book's 2017 data, high performers reported 46 times more frequent code deployments, 440 times faster lead time from commit to deploy, 170 times faster mean time to recover, and a change failure rate five times lower than low performers.

    Forsgren, Humble & Kim (2018), as in item 1, reporting for 2017: deployment frequency "46 times more frequent", lead time "440 times faster", mean time to restore "170 times faster", and change failure rate "5 times lower" (changes one fifth as likely to fail).

  14. The same programme's 2016 figures for the same comparison were 200 times more frequent deployments, 2,555 times faster lead times, 24 times faster recovery and three times lower change failure rates.

    2016 State of DevOps Report (Puppet and DORA), https://dora.dev/research/2016/2016-state-of-devops-report.pdf , and the programme's own summary page, https://dora.dev/research/2016/ , reporting that high performers deploy code 200 times more frequently than low performers, with 2,555 times faster lead times, 24 times faster recovery and three times lower change failure rates.

  15. The book identifies twenty four capabilities in five categories, including version control for application and infrastructure configuration and test automation, and measures organizational culture with a typology drawn from a patient safety journal.

    Forsgren, Humble & Kim (2018), as in item 1, which identifies 24 key capabilities grouped as continuous delivery, architecture, product and process, lean management and monitoring, and cultural.

    Westrum, R. (2004). "A typology of organisational cultures." Quality & Safety in Health Care, 13(Suppl 2), ii22-ii27. PMID: 15576687. PMC: PMC1765804. DOI: 10.1136/qhc.13.suppl_2.ii22. A conceptual paper in a patient safety journal arguing that how information flows through an organization predicts its safety performance, and proposing a typology sorting organizations as pathological (power-oriented), bureaucratic (rule-oriented) or generative (performance-oriented). The author states in the paper itself that the relationship between culture and safety "requires more exploration before the connection can be considered definitive". It is a typology built from case studies and some systematic research rather than an empirical test, and the book adapts it into a measured survey construct.

  16. The book's outcome measures are organizational performance, measured by asking respondents to rate their own organization's relative performance on profitability, market share and productivity, and a set of workforce outcomes including burnout and deployment pain, which continuous delivery practices were associated with reducing.

    Forsgren, Humble & Kim (2018), as in item 1, which asked respondents to "rate their organization's relative performance across several dimensions: profitability, market share, and productivity", describes that scale as validated in prior research and "highly correlated to measures of return on investment (ROI)", and uses a separate validated scale for non-commercial settings covering quantity of goods and services, operating efficiency, customer satisfaction, quality of products or services and achievement of mission goals. It reports that continuous delivery practices are associated with reduced deployment pain and reduced team burnout.

    Corroborated, as to the same model and outcome set, by the programme's earliest peer-reviewed statement of it: Forsgren, N., & Humble, J. (2016). "The role of continuous delivery in IT and organizational performance." Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV. SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2681909 , whose abstract states that continuous delivery affects workforce outcomes including burnout and deployment pain and indirectly affects organizational performance through IT performance.

  17. The sample is self-selected and every measure in it is self-reported, and the book says so.

    2017 State of DevOps Report, as in item 2, for the authors' own statement that the sample is "likely limited to organizations and teams that are familiar with DevOps".

  18. Where the predictor and the outcome are both self-reported by the same person in the same questionnaire, part of the observed correlation can belong to the respondent rather than to the relationship being measured. The authors address this.

    The phenomenon is common method bias, or common method variance. Standard reference: Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). "Common method biases in behavioral research: a critical review of the literature and recommended remedies." Journal of Applied Psychology, 88(5), 879-903. PMID: 14516251. DOI: 10.1037/0021-9010.88.5.879. A methodological review of the sources of measurement error introduced when predictor and criterion are collected by the same method from the same source, and of the design and statistical remedies available.

    The authors' engagement with it is published: Humble, J. (2019). "Response to Keunwoo Lee's review of Accelerate", Medium, https://medium.com/@jezhumble/response-to-keunwoo-lees-review-of-accelerate-611ef75cad3 , in which the authors argue against a systematic halo effect on the grounds that "There are plenty of responses from organizations that have high software delivery performance but do badly at multiple given practices", note that "divergent understanding of data is not limited to psychometrics, it's a problem of measurement in general", and describe using specific manifest items rather than abstract terms as a mitigation.

  19. The study design is cross-sectional, and the book states this.

    Forsgren, Humble & Kim (2018), as in item 1: "To conduct the research found in this book (in addition to research we still actively conduct), we use cross-sectional studies."

  20. The analysis uses latent variables and structural equation modelling, fitted with partial least squares.

    Forsgren, Humble & Kim (2018), as in item 1, which describes latent constructs measured through multiple items and the use of structural equation modelling to fit the full model, and states that an arrow between boxes in its figures "signifies a predictive relationship".

    Forsgren & Humble (2016), as in item 18, whose keywords and method give partial least squares as the estimator for the same model, on data from 4,976 respondents.

    For what the method is and is not able to establish: Russo, D., & Stol, K. J. (2021). "PLS-SEM for software engineering research: An introduction and survey." ACM Computing Surveys, 54(4), Article 78, 38 pages. DOI: 10.1145/3447580. A methodological survey written for software engineering researchers: it synthesises the methodological literature on partial least squares structural equation modelling, explains how the technique handles unobservable constructs such as organizational culture and trust, and reviews published software engineering studies against established reporting guidelines.

  21. The authors state in print that they can use words like "drives", "predicts" and "impacts" but not "causes", and that their work is not a randomized controlled experiment, which they say would be impossible to run in software engineering.

    Humble, J. (2019). "Response to Keunwoo Lee's review of Accelerate", Medium, https://medium.com/@jezhumble/response-to-keunwoo-lees-review-of-accelerate-611ef75cad3 , in which the authors write that "we can use words like 'drives', 'predicts', and 'impacts', but not 'causes'", explain that what they term "inferential predictive" analysis corresponds to what the statistician whose taxonomy they cite calls "inferential", and state "we're not performing a randomized, controlled experiment (which would be impossible in software engineering)". The same piece states that the underlying data cannot be shared because of the consent terms respondents were given.

    The published review being answered: Lee, K. "A review of Accelerate: The Science of Lean Software and DevOps", https://keunwoo.com/notes/accelerate-devops/ , a blog review whose methodological criticisms concern selection effects, self-report, construct validity, the coinage "inferential predictive analysis" and the circularity of the clustered measures.

  22. A peer-reviewed version of the cluster analysis exists, published by the same team at an information systems conference in 2020, using 7,522 respondents and hierarchical clustering, and finding three performance profiles.

    Forsgren, N., Rothenberger, M. A., Humble, J., Thatcher, J. B., & Smith, D. (2020). "A Taxonomy of Software Delivery Performance Profiles: Investigating the Effects of DevOps Practices." AMCIS 2020 Proceedings, 8. AIS Electronic Library, https://aisel.aisnet.org/amcis2020/systems_analysis_design/systems_analysis_design/8/

  23. The research programme itself now publishes guidance that the four measures are meant for an application or service, that the aim is a team's own improvement over time, and that they are not for competing between teams.

    DORA, "DORA's software delivery performance metrics", https://dora.dev/guides/dora-metrics/ , which lists pitfalls including "Making disparate comparisons" ("These metrics are meant to be applied at the application or service level. Comparing metrics between vastly different applications ... can be misleading"), "Setting metrics as a goal" (with reference to Goodhart's law), "Having siloed ownership", and "Competing" ("the goal is to improve your team's performance over time, not to compete against other teams").

    Corroborated as to date: DevOps Research and Assessment, Wikipedia, https://en.wikipedia.org/wiki/DevOps_Research_and_Assessment , which records that in October 2023 the DORA team cautioned against using these metrics for team-by-team performance evaluation.

  24. A multivocal literature review searching for formal research on how the four key metrics are measured found none in the peer-reviewed literature. The authors' own case study was a single ten-person team at a postal company, six of whom completed the survey.

    Sallin, M., Kropp, M., Anslow, C., Quilty, J. W., & Meier, A. (2021). "Measuring Software Delivery Performance Using the Four Key Metrics of DevOps." In Agile Processes in Software Engineering and Extreme Programming (XP 2021), Lecture Notes in Business Information Processing, vol. 419, pp. 103-119. Springer, Cham. DOI: 10.1007/978-3-030-78098-2_7. Open copy: https://homepages.ecs.vuw.ac.nz/~craig/publications/xp2021-sallin.pdf

  25. An independent industrial study that measured the four automatically found that individual services within one team diverged from the team's aggregate figures: only about a quarter of the individual systems tracked the aggregate closely.

    Rüegger, J., Kropp, M., Graf, S., & Anslow, C. (2024). "Fully Automated DORA Metrics Measurement for Continuous Improvement." In Proceedings of the 2024 International Conference on Software and System Processes (ICSSP '24), pp. 1-10. ACM. DOI: 10.1145/3666015.3666020. Open copy: https://homepages.ecs.vuw.ac.nz/~craig/publications/icssp2024-ruegger.pdf

  26. The underlying survey data cannot be released, because respondents were given consent terms promising it would not be.

    Humble, J. (2019), as in item 23, which states that the data cannot be shared because of the user consent forms under which it was collected and the privacy commitments made to respondents.

  27. Nobody has repeated the study with an independent sample, and nobody has run the model forward in time.

    Corroborating the same conclusion from within the literature: Sallin et al. (2021), item 26, whose multivocal review is the published statement that formal research on these measures was absent from the peer-reviewed literature as of that date.

  28. CUT FOR RUNTIME. A 2025 study compared DevOps team formations using three of the four measures, on 104 respondents recruited by targeted outreach and snowball sampling, and did not test the Accelerate model.

    Korkmaz, H. E., & Aydin, M. N. (2025). "An empirical study on performance comparisons of different types of DevOps team formations." Frontiers in Computer Science, 7, 1554299. DOI: 10.3389/fcomp.2025.1554299

  29. CUT FOR RUNTIME. The earliest peer-reviewed form of the model was presented in 2016 on 4,976 respondents and fitted with partial least squares.

    Forsgren, N., & Humble, J. (2016). "The role of continuous delivery in IT and organizational performance." Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV. SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2681909

  30. A systematic review of circadian function and chronotype in ADHD, covering 62 studies and 4,462 patients with ADHD, found consistent evidence of later chronotype and of phase delay in circadian markers including dim light melatonin onset and sleep onset.

    Coogan, A. N., & McGowan, N. M. (2017). "A systematic review of circadian function, chronotype and chronotherapy in attention deficit hyperactivity disorder." Attention Deficit and Hyperactivity Disorders, 9(3), 129-147. PMID: 28064405. DOI: 10.1007/s12402-016-0214-5

  31. A Cochrane review of flexible working and employee health found ten eligible studies, all controlled before-and-after designs, in which flexibility controlled by the worker was associated with health improvements while flexibility dictated by the employer was associated with equivocal or negative effects.

    Joyce, K., Pabayo, R., Critchley, J. A., & Bambra, C. (2010). "Flexible working conditions and their effects on employee health and wellbeing." Cochrane Database of Systematic Reviews, 2010(2), CD008009. PMID: 20166100. PMC: PMC7175959. DOI: 10.1002/14651858.CD008009.pub2

  32. A study of 262 autistic adults found that camouflaging heavily across contexts, and switching between camouflaging and not, were both related to poorer mental health; work was among the contexts reported and getting by in formal settings such as work was among the reasons given.

    Cage, E., & Troxell-Whitman, Z. (2019). "Understanding the Reasons, Contexts and Costs of Camouflaging for Autistic Adults." Journal of Autism and Developmental Disorders, 49(5), 1899-1911. PMID: 30627892. PMC: PMC6483965. DOI: 10.1007/s10803-018-03878-x

  33. CUT FOR RUNTIME. In ten national surveys of the WHO World Mental Health Survey Initiative, an average of 3.5 percent of workers screened positive for adult ADHD, and screening positive was associated with 22.1 annual days of excess lost role performance compared with otherwise similar respondents.

    de Graaf, R., Kessler, R. C., Fayyad, J., ten Have, M., Alonso, J., Angermeyer, M., Borges, G., Demyttenaere, K., Gasquet, I., de Girolamo, G., Haro, J. M., Jin, R., Karam, E. G., Ormel, J., & Posada-Villa, J. (2008). "The prevalence and effects of adult attention-deficit/hyperactivity disorder (ADHD) on the performance of workers: results from the WHO World Mental Health Survey Initiative." Occupational and Environmental Medicine, 65(12), 835-842. PMID: 18505771. PMC: PMC2665789. DOI: 10.1136/oem.2007.038448

  34. CUT FOR RUNTIME. A meta-analysis of social jetlag and anxiety in adolescents, covering 12 studies and 235,526 participants, found a small association between greater weekday-to-weekend sleep timing discrepancy and more severe anxiety.

    Ravenhall, H., Bellato, A., Chellappa, S. L., & Garner, M. (2026). "'Out of sync and overlooked' - Relationship between social jetlag and anxiety in adolescents: Systematic review and meta-analysis." Sleep Medicine Reviews, 89, 102345. PMID: 42480409. DOI: 10.1016/j.smrv.2026.102345

  35. The 2024 report of the same programme, on nearly three thousand respondents recruited the same way, found that a 25 percent increase in its measure of AI adoption was associated with a 1.5 percent decrease in software delivery throughput and a 7.2 percent decrease in delivery stability, alongside increases in documentation quality and code quality.

    Accelerate State of DevOps Report 2024 (DORA, Google Cloud), programme page https://dora.dev/research/2024/ ; report infographic https://dora.dev/research/2024/2024-DORA-Report-Infographic.pdf , which states that a 25 percent increase in AI adoption is associated with a 7.5 percent increase in documentation quality, a 3.4 percent increase in code quality, a 3.1 percent increase in code review speed, a 1.3 percent increase in approval speed and a 1.8 percent decrease in code complexity, and that AI adoption "is positively correlated with increased flow, productivity, job satisfaction, code quality, and internal documentation quality. However, it is associated with a decrease in delivery stability."

    The two delivery figures are taken from two independent secondary summaries of the same report, which agree: RedMonk, "DORA Report 2024 - A Look at Throughput and Stability", https://redmonk.com/rstephens/2024/11/26/dora2024/ , giving throughput down 1.5 percent, delivery stability down 7.2 percent and time spent doing valuable work down 2.6 percent for a 25 percent increase in AI adoption, on a survey of roughly 3,000 people of whom 75.9 percent relied on AI for at least part of their job; and The New Stack, "DORA 2024: AI and Platform Engineering Fall Short", https://thenewstack.io/dora-2024-ai-and-platform-engineering-fall-short/ , giving the same 7.2 percent reduction in delivery stability and 1.5 percent decrease in throughput on a sample of nearly 3,000 technology workers. *The full report PDF could not be retrieved for direct verification, so the two throughput and stability figures rest on two agreeing secondary summaries rather than on the primary document, and that is recorded here rather than hidden.*

  36. The 2025 report of the same programme, on nearly five thousand respondents surveyed between 13 June and 21 July 2025, found 90 percent of respondents using AI at work against 76 percent the previous year, a shift to a positive association between AI adoption and delivery throughput, and a continuing association with increased delivery instability.

    State of AI-assisted Software Development 2025 (DORA, Google Cloud), https://services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf ; programme page https://dora.dev/dora-report-2025/

All Sixty Books episodes · Subscribe by RSS