The Arctic is a hotbed of climate change research and a vast and varied place, much of it remote and difficult to reach.
That may explain why nearly one-third of study citations focusing on Arctic research were based on data collected within 50 kilometers, about 31 miles, of just two locations: Toolik Lake in Alaska and Abisko in Sweden.
That’s the finding reported in a recent Nature Ecology & Evolution study by Dan Metcalfe, senior lecturer at Lund University in Sweden, and colleagues.
Metcalfe’s study is just one of an increasing number of studies in the rapidly evolving field of meta-research. This type of research essentially involves researching research itself, analyzing how and why data is collected, what practices and techniques are used, and how to improve these efforts.
[Editor’s note: A fascinating September 21 “Science under scrutiny” Science magazine cover story and “special section” reports more than 11,000 meta-analyses were published last year, compared with fewer than 1,000 meta-analyses published in 2000. Studying why and how science takes place is crucial to understanding the results, argues one of the authors in the Science series, which refers to “journalology” – using scientific methods to study publishing of research.]
Choosing where to conduct research
For example, research sites are selected for many different reasons. While it would be ideal to set up research stations in a representative pattern and sample uniformly throughout the entire Arctic, that optimum approach frequently isn’t possible for a number of reasons, ranging from geopolitics to costs. But the data that is collected and cited sometimes is used to make wide-ranging decisions, which is why these locations and citations are so important.
“The reason really we’re worried about only sampling at a few places is of course because we’re thinking maybe those places have a set of physical and environmental conditions that may or may not be representative, so we’re trying to test that a bit,” Metcalfe says.
Asked in a telephone interview if his findings of a limited geographical area studied cast doubts on the legitimacy of overall Arctic research findings, Metcalfe was firm in responding: “We’re not at all suggesting that individual papers are biased,” he said. “They’re all excellent studies, critically important, and they can be conducted only by a few institutions. If there are any issues, it’s in aggregations” in which the findings are extrapolated to wider areas where they may or may not apply. He subsequently elaborated by e-mail (see below).
In order to explore the topic, Metcalfe and his colleagues used “a comprehensive, geo-referenced database of primary field measurements in 1,840 published studies across the Arctic.” They found the least-cited sites were typically colder areas experiencing faster temperatures increases. They wrote that it is crucial to study those locations.
The research team chose to analyze citations because how frequently a study is cited demonstrates the level of influence that study has within the scientific community and beyond. The study authors wrote that citations “are an important proxy for the degree of influence that scientific studies, and the geographic locations at which studies took place, have exerted over the science, modelling, and policy communities.”
Metcalfe emphasizes that studies focusing on certain locations, like Toolik and Abisko, are extremely useful, but he says concerns can arise when the research is presumed representative of the entire Arctic. He notes that the phenomenon of concentrated research sites isn’t limited to the Arctic and exists also in other areas, such as the tropics, as it does in other broad areas of academic research.
[Editor’s note: Researchers refer to concerns about the “streetlight effect,” in which one looks for a lost object, for instance a set of missing keys or in this case data, where it’s easiest to find things … such as under a streetlight, because “that’s where the light is”.]
“What happens is that policy makers or scientists who want to do a review, for example, will put together all of the work on a certain process, say the effects of warming on carbon storage, something very important for climate change. And they’ll do a literature review of all the studies, and of course lo and behold what will happen, they may not realize that the vast majority of those studies that they get will be from a limited number of places,” Metcalfe says.
Unsurprisingly, the most cited Arctic locations are the ones that are comparatively easy to reach.
Sites chosen for ease of access, wealth of data
Toolik Lake, Alaska, and Abisko, Sweden, constituted 31 percent of citations and 13 percent of sampled sites in the Metcalfe study. Other sites with many citations include areas near Greenland’s Zackenberg field station and parts of Canada. The two locations with the most citations both have extensive infrastructure and, by Arctic standards, are relatively easy to access.
Toolik Field Station is located along Alaska’s Dalton Highway, a rough, remote, mostly unpaved road that is a 9- to 12-hour drive from the closest major city of Fairbanks, 370 miles away. The station has an extensive website detailing everything from kitchen procedures to the facility’s laundry availability (one load per researcher every other week). Sweden’s Abisko station is also quite accessible, located 100 kilometers, about 62 miles, away from two airports and accessible by train or car. This relative ease of access means researchers can get to and from these sites without needing to obtain additional funding to charter planes or helicopters.
In addition to accessibility, researchers often look for sites that provide housing and scientific infrastructure, such as the consistent electricity needed to run complex equipment, or even basic laptops.
Another reason many of these sites are so valuable is the wealth of scientific knowledge and data already amassed there, which can provide valuable insights into things like how the locations are changing over time.
“There’s a lot of synergy that’s available,” Metcalfe says. “You can go out there and make your measurements and then you can compare that to measurements that have already been collected at those sites, or you know there’s already weather data from those sites, so that’s very valuable.”
“Scientific inertia” is another reason Metcalfe notes in describing why particular sites may be so popular among researchers: As scientists write successful grant proposals for one site, they may return again and again, with their PhD and postdoctoral students also conducting research at these locations.
This wealth of scientific knowledge at these sites makes studying them particularly valuable. “These places that have this concentration of research are fantastic and they have contributed a fantastic amount to our knowledge,” Metcalfe says.
Nonetheless, with so much research at relatively few locations, a wide swath of the Arctic remains underrepresented.
Remote sites underrepresented
Metcalfe’s study found portions of Alaska outside of the major field stations were underrepresented, as well as several Russian regions (Yamal, Nenets, and Sakha), and parts of the Canada (namely, the “southernmost portion of the Canadian Arctic archipelago”).
“The current pattern of sampling locations captures mean Arctic conditions reasonably well, but does not capture more extreme conditions that are nevertheless widespread,” the study notes,
Some of the areas underrepresented in terms of citations, such as parts of Russia and Canada, do have research stations. “Even though there are … certainly some very well established fairly productive research stations in the Canadian Arctic islands … they’re very remote, they’re quite hard to access,” Metcalfe says.
One reason for the low number of citations representing Arctic Russia may be that Metcalfe’s study focused on papers written in English, which was one limitation of the study.
Solutions to diversify research citations
Given that citations are so important in terms of policy development and research, Metcalfe urges scientists to be meticulous in choosing what research to cite, and to carefully consider citing publications and data from less-cited locations that have a wealth of research and data.
He suggests that when scientists look for citations and evidence, they should “try and search a little bit further afield than we’re used to, because there may be perfect studies that can illustrate our point, but not from the usual suspects.”
While many under-cited areas are quite challenging to reach, others are relatively easy to access, so Metcalfe suggests researchers consider these sites. The Metcalfe report encourages scientists to explain to funders just why they need the additional funding to reach these particular locations and the value of doing research at these sites. He says he hopes his paper will help them make this point to funders to help increase the wealth of Arctic knowledge and make the knowledge base more representative of the widely varying conditions.
Author Dan Metcalfe’s further thoughts on his research paper
Does the paper cast doubt on climate change/existing science?
No. Scientists are skeptical, doubtful people by nature generally. We know that no one ever knows the entirety of anything. Do we understand ecological and climate systems in their entirety? No, but we do know that the vast majority of environmental research points to the same basic conclusion: that the climate is changing in a number of different ways, and this is causing a large number of meaningful impacts on ecosystems.
We are open to countervailing results, and watchful for things we might have missed. My paper is an effort in this spirit – to identify knowledge gaps where we need to improve. It is a (very distant) possibility that filling in these knowledge gaps will completely reverse our current consensus about climate change. But just as, maybe more, likely is that we will discover that we have in fact been underestimating the rate, scale, and impacts of climate change. Indeed, the data we have suggest that the latter scenario has occurred. Since most of the large productive research stations are located in relatively warmer portions of the Arctic, which are projected to warm relatively slowly, we may currently be underestimating the ecological impacts of rapid warming predicted across large areas of the colder, more remote Arctic.
Does the paper suggest that (a) existing research is biased? (b) that large stations are not important/useful?
No. (a) The biases mentioned in the paper do not refer to any of the individual studies, which have all been peer-reviewed and are likely to be accurate and useful contributions to their field. The bias we discuss emerges from aggregation of these individual findings, either formally in something like a meta-analysis, or informally in the formulation of synthesis papers, book chapters and reports. In all of these, the authors can only interpret from available studies, and if these studies come only from a subset of the environment about which they want to make an inference, then the conclusion may be biased. (b) Research at these large, productive research stations will and should continue. The reality is that there are many scientific questions that simply could not be addressed without the facilities and logistical support available at the larger research stations. These include multi-disciplinary, multi-factorial and long-term experiments which are particularly powerful in addressing important ecological questions under climate change, but which would be very difficult to install and maintain at simpler and more remote field sites. My hope for our paper is, first, that it might stimulate more awareness around each site about the extent to which their results might be mediated by the local environment, and in what respects their site is typical or atypical for the Arctic as a whole; and secondly, that it may catalyze greater efforts to include existing and planned research from often overlooked parts of the Arctic.
One other thing: Our study talks about the dangers of drawing inferences about the whole Arctic region based upon data from just a subset of that region. This is a relevant issue, for example, for major international reports (e.g., the IPCC) that seek to present consensus trends for entire regions. But of course there are many scientific questions and endeavors where up-scaling across the entire Arctic is not the purpose.
You can also reasonably argue that some regions are interesting and worthwhile of study precisely because they are not representative of the wider Arctic, maybe because they have an unusual vegetation type or climate. For example, Scandinavia is an unusually warm portion of the land surface above the Arctic circle. In one sense, you could argue that this lowers its research value because findings from this area may not be easily applicable to the rest of the Arctic, which is generally much colder. In another sense, this could enhance its research value because it potentially provides a window into how a much warmer Arctic might look and function.