Robustness and traceability in mangrove statistics

South-east Asia is the global centre of mangrove diversity (Ellison et al. Reference Ellison, Farnsworth and Merkt1999), and mangroves are receiving increased attention as a highly threatened ecosystem. Quantifying mangrove loss requires accurate historical statistics; however recent analysis revealed that trends can show varying degrees of loss, or even positive trends depending upon which particular data points are chosen (Ruiz-Luna et al. Reference Ruiz-Luna, Acosta-Velázquez and Berlanga-Robles2008). We conducted a literature search for mangrove cover estimates for Peninsular Malaysia, through Web of Science and Google Scholar. By plotting all published estimates, we found that overall the land cover analyses (and therefore change estimates) exhibited a substantial lack of agreement (Fig. 1).

Figure 1 Estimates of mangrove area in Peninsular Malaysia. ‘Date’ refers to the known or assumed date of the data. Filled symbol = year known, source not given. Open symbol = year assumed. Note that not one data point has a verifiable method associated with its collection (see text).

While the results suggest a negative temporal trend in Peninsular Malaysia's mangrove cover (Fig. 1), the high variation in the point distribution within and among years creates a wide prediction interval. The wide variability in estimates leads to high levels of uncertainty. A review of the literature and analysis of the documents providing the data revealed three principle causes of uncertainty:

1. (1) Lack of robust methodology: much of the data was published in the grey literature and government documents, and no single data point (Fig. 1) had an explicitly described methodology for its calculation. The volume of grey literature for tropical ecosystems may equal that of peer-reviewed material (Corlett Reference Corlett2011), but frequently lacks sufficient (or any) description of robust methodology.

2. (2) Lack of traceability: all data points are based on secondary data with highly limited traceability or accessibility, which is a generally acknowledged problem for grey literature (Corlett Reference Corlett2011). Indeed, two data points were taken from a newspaper article quoting a government report, which we were unable to locate. Interestingly, two studies were peer-reviewed, but used secondary data where the source was not mentioned (Jusoff & Taha Reference Jusoff and Taha2008) or the data were acquired via personal communication (Loneragan et al. Reference Loneragan, Ahmed Adnam, Connolly and Manson2005).

3. (3) Data assumptions: the studies we examined relied on implicit and explicit assumptions. For example, only 82.6% of Peninsular Malaysia's mangroves are controlled by the Forestry Department (data from Chong Reference Chong2006), and many land cover estimates were based on this area only. None of the data points identified this fact, indicating that significant error remains undisclosed. An explicit assumption in several of the data points taken from the Food and Agriculture Organization of the United Nations (FAO 2007) was that the publication date of the ‘original’ cover estimate was assumed to be equivalent to the year of analysis. Even when this error is articulated, researchers may still use statistics of land cover change based on an unverifiable temporal axis.

These three points raise serious concerns about estimates of mangrove cover, and hence derived change calculations. It is impossible to check original source validity, or calculate meaningful error estimates to help policy makers ascertain the level of confidence in the analyses. This inability to eliminate flawed or erroneous data points results in high levels of uncertainty. Even if forestry authorities wanted to craft ‘no net loss’ policies, they would be hard pressed to find robust data upon which to base verification. The greatest concern is, if accepted, unverifiable or false estimates can be propagated through the literature.

Propagation of untraceable results: the case of saltmarsh loss

The fourth barrier to improving information quality is the propagation of unverifiable information. Such a barrier can be seen in the saltmarsh literature of the UK, a country that has lost more than half of its saltmarsh cover since the Roman era (Airoldi & Beck Reference Airoldi and Beck2007). European Union (EU) member countries must specify habitat loss rates under the EU Habitats Directive (92/43/EEC), which dictates that there must be no net loss of habitat within designated areas, and which provides legislative enforcement measures to ensure adequate habitat compensation. Data underpinning the statutory habitat targets must therefore be accurate and verifiable.

The UK has implemented its legislative obligations in part through priority Habitat Action Plans. The most recent Action Plan reports a decline in coastal saltmarsh up to 2008, though is unable to give a quantitative loss estimate as ‘no comprehensive UK-wide assessment of trends is available’. Despite the lack of national assessment, successive saltmarsh Habitat Action Plans have used a national loss figure of 100 ha yr⁻¹ (see http://www.ukbap-reporting.org.uk). The current figure was first written in a report by UK Biodiversity Group (1999) in which no method of calculation or reference was given.

This example is noteworthy because the unverifiable figure has been propagated through the academic literature, various local government reports and even the Intergovernmental Panel on Climate Change (IPCC) Third Assessment (Table 1). The figure of 100 ha yr⁻¹ has been repeatedly quoted by multiple trustworthy sources, and has thus gained currency through ‘proof by assertion’. Propagation of unverifiable official government estimates has also been observed in the tropical peat swamp literature (Page et al. Reference Page, Rieley and Banks2010). Both the saltmarsh and the peat swamp examples concur with previous observations that potentially erroneous information from authoritative sources (such as the UK government) can quickly become established within the literature (Grainger Reference Grainger1996). Potentially erroneous figures become propagated and entrenched because it is easier for researchers to trust ‘best estimates’ from sources of perceived authority, rather than to follow the information trail and hunt for obscure government reports. Thus, even in a highly developed bureaucracy such as the UK, unverifiable figures of wetland loss increase uncertainty in an otherwise strong legislative conservation mechanism.

Table 1 Estimates of saltmarsh loss in the UK, ranked by year. Note that the UK Biodiversity Group (1999) is the basis for the saltmarsh habitat action plan. This table does not include the large number of local government reports using the figure of 100 ha yr⁻¹. IPCC = Intergovernmental Panel on Climate Change, JNCC = Joint Nature Conservation Committee, NGO = non-governmental organization.

While it is entirely possible that the published figures are accurate, their reliability can be questionable owing to the lack of confidence and transparency in their derivation. The consequent unenviable decision is to accept potentially flawed estimates as ‘the best available’ regardless of their accuracy, or to discard all previous effort to the detriment of conservation.

Improving the ‘evidence’ in evidence-based policy

Accurate techniques and information are required to develop and implement effective habitat policy (Walters et al. Reference Walters, Rönnbäck, Kovacs, Crona, Hussain, Badola, Primavera, Barbier and Dahdouh-Guebas2008). Unreliable figures of area and change incorporated into scientific assessments without proper verification may form the basis for policy measures that could prove inadequate at conserving, managing or restoring valuable ecological systems. The example of wetlands has shown that information uncertainty can be inherent in research from both developed and less-developed nations, and occurs at scales from hectares to hundreds of square kilometres. Two immediate needs are (1) improved transparency and confidence in historical analyses, and (2) greater rigour in methodology and publication going forward.

Transparency in data quality is crucial for the appropriate integration of historical scientific evidence in the policy-making process; the credibility of scientific advice (along with salience and legitimacy) is a key influence on the choices of decision-makers (Cash et al. Reference Cash, Clark, Alcock, Dickson, Eckley, Guston, Jäger and Mitchell2003). The uncertainties we describe must be better recognized by researchers and communicated to policy-makers, either as methodological or statistical caveats. Researchers collating secondary information from multiple sources (whatever the ecosystem) should critically appraise such data (Grainger Reference Grainger2008). Researchers must verify sources, question discrepancies in secondary data reporting, check references and clearly articulate the level of confidence in data quality. In the case of wetland assessments, most estimates have been devoid of such caveats (one exception is FAO 2007). By giving statements of confidence in the data quality, researchers can increase the credibility of proffered advice and contribute to a two-way science and policy deliberation approach; policy-makers can gauge the scientific certainty of change and scientists can contribute to a well-grounded policy debate. By promoting a two-way dialogue based on credible scientific evidence, the effectiveness of policy-making over threatened habitat conservation can be significantly improved (Cash et al. Reference Cash, Clark, Alcock, Dickson, Eckley, Guston, Jäger and Mitchell2003).

Future data analysis should be conducted with standardized and widely accepted techniques, and methods and interpretation should be independently assessed to ensure the accuracy of such a large task as quantifying land cover. Recent studies demonstrate improved methodological accountability for wetland monitoring at the regional and global scale by using remote sensing (Giri et al. Reference Giri, Zhu, Tieszen, Singh, Gillette and Kelmelis2008, Reference Giri, Ochieng, Tieszen, Zhu, Singh, Loveland, Masek and Duke2011; Spalding et al. Reference Spalding, Kainuma and Collins2010). Improved methodological rigour provides concurrently increased confidence in ecosystem trajectory analyses; this can be achieved by increasing investment in technology (particularly remote sensing infrastructure/access) and human resources. Increased collaboration between the scientific community and government/non-government agencies may increase the rate of peer-reviewed publication, thus enhancing the rigour and transparency of data collection and analysis. If decision-makers want credible evidence (Cash et al. Reference Cash, Clark, Alcock, Dickson, Eckley, Guston, Jäger and Mitchell2003) assimilated from multiple sources, then the evidence produced by their own researchers must be of sufficient quality to be accepted internationally. Professional advancement that considers peer-reviewed publications as an indicator of achievement could increase the quality and reputation of government-produced science, and increase data accessibility by bringing publications out of the grey literature (Corlett Reference Corlett2011). Although peer review is not without its own problems (Rissgård Reference Riisgård2003), it is a well-used mechanism that could increase the quality of science produced outside of academia. Reviewers and editors during the peer review process should recognize and confront instances where authors propagate potentially erroneous or untraceable estimates, in order to further increase credibility to decision-makers.

With these actions, accurate land cover change information can better provide policy with a strong quantitative foundation, and strong scientific inference for habitat loss and conservation programmes. Researchers may try to support policy with the best science currently available, though this may be further improved in order to contribute to management decisions that are better able to conserve valuable ecosystems. There can be no evidence-based policy without robust evidence.