Why Projects Fail and How to Avoid It: The Bias Behind Most Overruns

What “failure” means

People use the word in three different ways, which is why arguments about failure rates go nowhere.

• Constraint failure. The project finished late, over budget or with less scope than planned. This is the easiest to measure and the most common definition.
• Benefit failure. The project delivered what was asked, but the expected benefits did not appear.
• Stakeholder failure. The result was on time and on budget, but the people it was meant for were unhappy.

A project can pass one test and fail another. When you read a statistic about failure, check which one it means. Our guide to the iron triangle covers the first definition in detail.

What the data says

Much-quoted failure rates often come from industry surveys with unclear methods. Researchers at the University of Oxford have studied actual project outcomes. Their working paper on information and communication technology (ICT) projects analyzed a sample of 1,471 projects worth USD 241 billion in total. The paper reports:

The key point is the shape of the data. Most overruns are modest, but a minority are enormous, and the average hides them. A plan that allows 10 or 20 percent contingency covers the typical case and is useless against the rare disaster.

A related paper, Five Things You Should Know about Cost Overrun, reviews large transportation projects. It reports that 86 percent had cost overrun, averaging 28 percent in real terms, and that overruns are far more common than underruns. Its authors name human bias, psychological and political, as the root cause, with scope changes and complexity as intermediate causes.

Limits. The ICT paper is a working draft, and the authors say the sample may favor organizations with stronger project management, which would make real-world results worse, not better. The infrastructure data covers very large projects, and your project may differ. The lesson is not that every project overruns by 27 percent. It is that estimates tend to be too optimistic, and the error is not random.

Root causes versus symptoms

Most project advice works at the symptom level: communicate more, report more. That helps, but if the original estimate was an optimistic guess approved under pressure, no amount of reporting rescues it. The root-cause tools below address the estimate and the decision itself.

Eight failure patterns and their fixes

The patterns are our synthesis of common causes, tied to the root-cause model above.

For the complete set of day-to-day challenges and how to rank them, see the top 10 project management challenges.

Reference class forecasting: a worked example

Reference class forecasting is one of the methods the Oxford authors recommend for avoiding overrun. Instead of building an estimate from the details of your project alone, you look at how similar past projects actually turned out and use that distribution to adjust your plan.

This is a made-up example. A team estimates a software project at $500,000. They find ten similar projects from their company’s history. The cost overruns against original budgets were:

What to learn. The average (28.7 percent) is well above the median (17.5 percent) because of the two big overruns, which is the fat-tail pattern in miniature. If the organization wants an 80 percent chance of finishing within budget, it should approve about $650,000, not $500,000. If it cannot afford that, it should reduce scope or break the project into stages, and not hope the estimate holds. The worst case, a doubling, tells you why staged funding and early checkpoints matter.

You do not need ten perfect matches. Even a rough list from your own history, or from colleagues in your industry, beats a single optimistic number. Note that the overruns in this example are invented to show the method, not data from any real company.

The pre-mortem

Gary Klein described the project pre-mortem in a 2007 Harvard Business Review article. In the article’s summary, team members assume the project they are planning has just failed and then generate plausible reasons for its demise. The idea is that people with reservations can speak freely at the outset, so the project can be improved rather than autopsied.

A simple version, 45 minutes.

1. Set the scene (5 minutes). “It is a year from now. The project has failed badly.”
2. Write silently (10 minutes). Each person lists reasons for the failure. No discussion yet.
3. Share (10 minutes). Go round the room. One reason per person per turn, no debate.
4. Group and rank (10 minutes). Cluster the reasons and vote on the five most likely and damaging.
5. Act (10 minutes). For each, name a prevention step and an owner, and add them to the plan or the risk log.

Run it before the plan is approved, and again at major stage gates. It works best when senior people speak last, so they do not steer the list. The method does not guarantee success, but it is cheap and it brings out concerns that would otherwise stay private.

Early warning signs

Track a few numbers as well. Our guide to five useful project metrics shows how.

If your project is already in trouble

1. Say so. Tell the sponsor early, with facts and options.
2. Re-estimate from today. Use actual progress and a reference class, not the original plan.
3. Reset the goal. Decide what must stay, what can shrink, and what can wait.
4. Consider stopping. Money already spent is gone. The question is whether the remaining cost is worth the remaining benefit.
5. Learn from it. Hold a review and record the lessons for the next project.

Stopping a failing project early is not a failure of management. Continuing because you have already invested is a common mistake.

Common mistakes

• Blaming the team. Many failures are decided in the estimate and approval, before work begins.
• Planning for the average. The costly cases sit in the tail.
• Treating a plan as a promise. Plans are estimates; update them.
• Punishing bad news. People then stop delivering it.
• Skipping the lessons. Without records, you cannot build a reference class.
• Copying a checklist. Look at what actually went wrong in similar projects.

Learn more

If you want structured learning, online providers such as Coursera and Udemy list project management courses, some with modules on risk and estimation. We may earn a commission if you sign up, at no cost to you. We are not affiliated with any university. Check the syllabus before you pay. For the bigger picture, see what is project management and the benefits of project management.

Frequently asked questions

Why do projects fail?

Often because early estimates and approvals were too optimistic, then scope changes, complexity and poor communication made things worse. Oxford researchers name human bias as the root cause of cost overruns in large projects.

What percentage of projects fail?

It depends on the definition and the sample. In one Oxford study of 1,471 ICT projects, the average cost overrun was 27 percent, and about 17 percent became very large overruns. Surveys with different definitions give different figures.

What is the number one reason projects fail?

No single reason fits every project. Researchers point to bias in estimating and decision-making as a root cause, and to unclear requirements, scope change and weak sponsorship as common contributors.

What is reference class forecasting?

A method that adjusts an estimate using the actual results of similar past projects, to counter optimism.

What is a pre-mortem?

A meeting in which a team imagines the project has failed and lists the reasons, so risks can be addressed before work starts.

How do you prevent project failure?

Use realistic estimates based on outside data, define the goal and benefit clearly, manage requirements and change, escalate bad news early, and review at stage gates.

When should you cancel a project?

When the remaining cost is greater than the remaining benefit, and no realistic reset fixes it. Past spending should not decide the question.

What is the difference between project failure and project challenges?

Challenges are problems along the way. Failure is the outcome when they are not handled. See our guide to the top project management challenges for the common ones.