An interview with David James, Co-founder of J+D Forecasting. Originally published on 31st October 2022 on pharmaphorum.com
Having good forecast models is critical for effective and efficient decision-making. This will deliver accurate and reliable insights to help support business objectives and drive brand or company growth, highlight risk, and quantify opportunity. This article explores the key ingredients needed for developing good pharmaceutical forecast models.
It’s widely accepted that there is no such thing as 100% accuracy in long range forecasting. In fact, according to research, pharmaceutical forecasts can be out by as much as 40 percent (Nature reviews drug discovery). Often inaccuracies are driven by a poorly built model which lacks transparency and doesn’t follow best practice, combined with poor assumptions.
It is possible to design a model that both minimises these inaccuracies and identifies where and why weak assumptions exist. In the sections below, we draw on our extensive experience of building bespoke pharmaceutical forecasting models to share the key ingredients for creating the most successful models.
1. Know what good looks like when designing the model
Often inaccuracies creep in when teams are inexperienced in building models using sound pharmaceutical forecasting principles. A lack of understanding of the approach and thought process involved in selecting the right model design and methodology can result in fundamental errors in the resulting forecast. In addition, a model that does not align with the business decision it supports brings little value.
What to do: When building a forecast model, it’s necessary to select the right model structure and forecast methodology. This decision is not only about reflecting the structure of a disease that is being forecasted, but also about what data exists and precisely how much complexity is needed. Creating a sophisticated model for a complex disease can only work if the forecaster has access to the data and assumptions that make it work. Our approach is to create a strawman schematic of the required model design based on the disease and therapeutic structure along with a future event audit. Then map this against the available data. Where this data doesn’t exist, there are several options – change the model to reflect the lack of data, collect the data or model the variability this lack of data creates.
2. A good forecast will have no secrets
Forecasting exists to help organisations make important decisions around investments such as the launching a new drug. A poorly built model will not deliver transparency which means that vested interests will often drive assumptions and therefore deliver the outcomes a particular stakeholder is looking for rather than an accurate forecast.
It will never be possible to drive politics out of your forecast completely. However, by creating a model where assumptions and their impact is clear and transparent will help highlight overly or negatively ambitious input drivers.
What to do: Apply a trending and event based forecast methodology. This will allow for stakeholders to see where a market is tracking based on the historical trend or baseline. Then use events to model how the market will change in the future if, for example a new product comes to market. The event will clearly communicate assumptions such as time to peak, source of business or peak share. This approach will make the forecast clear and transparent to stakeholders and allow for them to challenge, or perhaps accept with full knowledge.
3. A model designed for everyone
With decades of experience, we have learned from not just our mistakes but those of others; many models we see are overly complex not due to their structure but due to poor design. They are organic beasts that that have grown as more and more information has been added. It is not uncommon for the team to review models with hundreds of lines of data and more tabs than can be counted. The result is a web of confusing numbers with no clear logic or guidance and this type of chaotic approach not only leads to error and a lack of trust in the numbers, but it becomes a one-user model where only the creator knows how it works.
What to do: A forecast model is a complex piece of software and like any other software should be built with users in mind. We have invested in model useability and apply a step-by-step approach to models that follows best practice principles. Each section is distinct and clearly explained, it will include charts that show the outputs for that section and the resulting final forecast to reduce unnecessary scrolling. Other simple suggestions are to limit the number of colours and font sizes. Additionally, avoid having too many rows on one tab and try to keep each data section within one screen view.
4. Flexibility built in
The model should be flexible in 2 ways. Firstly it should be simple for new data or assumptions to be entered when they become available. Secondly, it should allow for assumptions to be adjusted in order to pressure test the market or create independent scenarios that reflect a more positive or pessimistic future.
The benefit of a model that is easily updated is self-explanatory, however why the need for pressure testing or scenarios?
Pressure testing forecast assumptions is a key forecasting step. It can identify key market levers that drive the most change. It can also allow for communication of the impact of less than perfect assumptions or data so that the decision maker is fully cognisant of variance these might create.
Scenarios are key for planning, they allow for different future outcomes to be modelled and planned for, such as if a competitor does or doesn’t launch, or they can be used to evaluate returns of .possible investment levels.
What to do: Using the trending and event approach previously discussed is the first step. Then the key is to create varying versions of the model that are clearly labelled and filed. This functionality is built into all our models along with automated consolidation and visualisation to support clear communication.
5. Keep it simple
There is often a push for increased complexity within models, perhaps because varying stakeholders want to see the data in differing ways, or perhaps because complexity is confused with accuracy. But as any good forecaster knows, superfluous complexity is the enemy of a good forecast. At best it can make the model difficult to maintain and communicate. At worse it leads to guess upon guess being layered into the model that will undermine confidence in the final forecast.
What to do: To be successful, a forecast model must be as simple as possible whilst at the same time providing outputs that will support decision making. To determine where the balance should lie, it’s important to question the remit of the model. What functions of the business does the forecast model feed into? What outputs do these functions need to make accurate, informed decisions? By defining the remit, you can begin to identify whether features within the model are necessary additions or ‘nice-to-haves’ to keep it simple.
Why does it matter?
A model that contains these key ingredients:
- Ensures the greatest possible accuracy of the forecast
- Reduces effort and user fatigue, which increases compliance
- Is easy to maintain, flex and scale so it will serve you well for several years
- Weaves a narrative everyone can understand for optimum buy-in
Speed of buy-in is vital. When decision-makers try to agree on a forecast that doesn’t have a good model underpinning it, there tends to be pushback. Outputs are challenged, answers get rejected, and the whole forecast has to be adjusted and re-run over many days and weeks. It’s incredibly time-consuming and resource intensive. This happens because of bias, misalignment with business objectives, lack of transparency, and all the other reasons we have seen.
Final thoughts – should you go it alone?
If companies are to remain competitive, they will need models that remove innate prejudices, align across borders, flex for different scenarios, justify assumptions, clarify opportunity, and quantify risk.
It takes considerable knowledge and experience to design and build this type of model – data analysts who deeply understand the industry, market, data sources, model types, stakeholder expectations, and the compromises you need to make when delivering a forecast. Teams working with inherited, legacy models which are not fit for purpose will result in delivering poor outputs upon which to make important business decisions.
The key ingredients explored above provide the best foundation for your models, but forecasting demands a level of human judgement that few non-specialists possess.
Working with a consultant team of expert forecasters like those at J+D Forecasting enables you to discuss how a bespoke model that takes the unique requirements of your organisation and decision-makers into account can be built. For multiple model build independence, specialist software can be accessed along with training and support to help build in-house expertise and give you an advantage when it comes to achieving business objectives, staying competitive, and driving company growth.
