Q1 In the absence of key knowledge, explorers typically rely on statistical approaches to the uncertainty they are facing in portfolio management and exploration strategy design. What problems arise if this approach is either not used, or incorrectly applied?
Milkov (2017) notes that,in common with many corporate exploration programmes, Lundin’s Norwegian Continental Shelf portfolio drilled out in a five-year period from 2011 to 2015 both underestimated POS compared to that experienced by other explorers in the basin, and overestimated likely delivered volumes. His postulate is that, had the underlying base rate for the NCS been used as a weighting factor, these errors could have been mitigated, if not fully compensated-for.
Milkov argues that Lundin, in not correctly utilizing the regional data available to them, fell prey to the cognitive bias of “Base Rate Neglect”, which is a variant of the“Representativeness” heuristic wherein a background statistical likelihood is ignored in favour of immediately-available evidentiary data.
His evidence is drawn from overall success rates for NCS wells and a creaming curve drawn from the publicly-available Norwegian Petroleum Directorate (NPD) database.
Q2 In portfolio management, decisionmakers are looking for tools to “compare apples with apples”. Basin CreamingCurves represent the exploration history of a basin and are used as a proxy in YTFprognosis. What are potential pitfalls of this approach?
Milkov (2017) presents data from 25 wells drilled by Lundin Petroleum offshore Norway between 2011 and 2015 in the Norwegian North Sea (NNS 17 wells), Norwegian Sea (1 well) and the Barents Sea (7 wells). He compares the results of those 25 wells to Lundin’s Expected outcomes (POS and volumes), and the outcome to be expected from a model based on the expectations of a “base rate portfolio” of theoretical wells based on the five-year average of POS and volumes taken from the Norwegian Petroleum Directorate database for the Norwegian Continental Shelf for those years. The results were as follows:
Avg. POS VOLUME (MM BOE)
Lundin Predicted 0.27 989 (total risked gross recoverable)
Lundin Actual 0.57 540 (gross recoverable)
Base Rate Portfolio 0.52 650
This analysis clearly shows that Lundin underestimated the Probability of Success (POS), but overestimated the volumes found. The “Base Rate Calibrated” result, however, is much closer to the actual outcome.
There is, unfortunately, a problem in this seemingly tidy outcome, which demonstrates a potential flaw in the use of “gross” Creaming Curves, rather than curves reflecting the plays targeted.
The NNS and Barents Sea basins differ in a significant number of ways reflecting geology, geography,data availability, ease of access and economics. Separate creaming curves for the two basins were constructed from the NPD database using the method described by Milkov to exclude appraisal wells clearly demonstrate that the NNS is considerably more mature than the Barents Sea: it very much in “harvest” mode (Figure 1), whereas the Barents is probably in the transition from “Emerging” to “Core” as a number of new fairways have been opened up and exploited within the past decade (Figure 2):
Figure 1:
Figure 2:
Lundin’s performance in the two basins very much reflects this difference in maturity, as reflected in Figures 3 (Norwegian North Sea) and 4 (Barents):
Figure 3:
Figure 4:
Re-examining Lundin’s statistics in this light, a rather different picture emerges:
Avg.POS VOLUME
NNS Barents NNS Barents
Lundin Actual 0.17 0.57 117 424
Lundin Predicted 0.29 0.23 515 475
Clearly in the NNS during the period analyzed, Lundin were very prone to overestimating volumes (although the estimate is much closer to the “Base Rate” prediction) and, with a technical success rate of around 17%, also overestimated POS.
In contrast, in the Barents, they have a technical success rate of 57% in those same years, and came close to achieving their risked volumetric promise (albeit a little lumpily).In this case, Milkov is correct that they have underestimated POS, but incorrect in his statement of significant overestimate of volume.
Thus it would seem that “Base Rate Neglect” does not explain Lundin’s relatively poor performance in the North Sea basin, but their performance in the Barents Sea basin is rather closer to the behavior of the creaming curve.
Q3. What does this say about the validity of the Creaming Curve/Prior Results as a tool for statistical understanding/prediction of future success?
There is no reason to suppose that this example invalidates the use of the creaming curve as one tool in the toolkit for understanding and predicting the outcome of a portfolio. Milkov (2017 and 2015 and references therein) elegantly demonstrates, in fact, that statistical analyses, correctly undertaken, are an invaluable aid to analyzing data trends and correcting for the biases of all types that bedevil ever-optimistic explorers.
Q4. What is your view on the way industry deals with risk and uncertainty and how could we avoid/mitigate cognitive biases in basin comparisons?
Overall, I conclude that Milkov is correct in saying that we should be considering the base rates for success and volumes when establishing the parameters for prospects. Having said that, the tool must be deployed with an appreciation of the scale and context of the portfolio:
One must ask "Is the basin being explored represented correctly by the creaming curve chosen?" In this instance, the curve for the entire Norwegian Continental Shelf is not representative of the Barents Sea Basin, which is in a different phase of exploration than the more mature basins to the south.
Also, "Is the play being explored “typical”of those in the segment of the creaming curve analyzed?". In this instance, are the fairway(s) in Lundin’s portfolio comparable to those being drilled by their competitors? In the North Sea basin during the period in question, Lundin were probably chasing a risky long-distance migration play, hoping to emulate their success at the super-giant Johan Sverdrup field (discovered by them in 2010, just before Milkov’s analysis commences). This would compare unfavourably in both risk and anticipated volume with the near-field tilted fault-block targets of most other operators in the area in this time-period.
In fact, Milkov’s analysis highlights several potential problems with the use of basin-wide exploration statistics to condition a risked portfolio:
1. He has lumped where he should have split, taking information from two separate basins, and a number of individual fairways to arrive at his conclusions:
2. He may have used a statistical analysis with insufficient knowledge of the context (target fairway) of the portfolio.
Ideally, in this sort of analysis, that sort of data mixing should be avoided. At the very least, he should recognize the difference between the two basins (NNS and Barents Sea),which have fundamentally different geological controls and, possibly more seriously, are at very different points in their exploration history. This does not necessarily invalidate his conclusion, but by analyzing that data in a more granular way, he could have arrived at a more refined view.
It is my contention that Milkov himself has fallen prey to two different flavours of heuristic, that of Anchoring, wherein he has lumped a lot of evidence together which does not belong together, but which overall supports his case, and he has proven what he set out to prove (was anchored to) by selectively using the data available to him (Confirmation).
References:
Milkov, A.V., 2015: Risk tables for less biased and more consistent estimation of probability of geological success (PoS) for segments with conventional oil and gas prospective resources. Earth Science Reviews 150, 453 – 476
Milkov, A.V., 2017: Integrate instead of ignoring: Base rate neglect as a common fallacy of petroleum explorers. American Association of Petroleum Geologists Bulletin 101/12,1905 - 1916