Judith Curry (not a poet, by the way) on the 97% myth again http://judithcurry.com/2015/08/27/the-conceits-of-consensus/#more-19735:
The conceits of consensus
Posted on August 27, 2015 | 500 Comments
by Judith Curry
Critiques, the 3%, and is 47 the new 97?
For background, see my previous post The 97% feud.
Cook et al. critiques
At the heart of the consensus controversy is the paper by Cook et al. (2013), which inferred a 97% consensus by classifying abstracts from published papers.The study was based on a search of broad academic literature using casual English terms like “global warming”, which missed many climate science papers but included lots of non-climate-science papers that mentioned climate change – social science papers, surveys of the general public, surveys of cooking stove use, the economics of a carbon tax, and scientific papers from non-climate science fields that studied impacts and mitigation.
The Cook et al. paper has been refuted in the published literature in an article by Richard Tol: Quantifying the consensus on anthropogenic global warming in the literature: A re-analysis (behind paywall). Summary points from the abstract:
A trend in composition is mistaken for a trend in endorsement. Reported results are inconsistent and biased. The sample is not representative and contains many irrelevant papers. Overall, data quality is low. Cook׳s validation test shows that the data are invalid. Data disclosure is incomplete so that key results cannot be reproduced or tested.
Social psychologist Jose Duarte has a series of blog posts that document the ludicrousness of the selection and categorization of papers by Cook et al., including citation of specific articles that they categorized as supporting the climate change consensus:
Ignore climate consensus studies based on random people rating journal abstracts
Cooking stove use, housing associations, white males and the 97%
From this analysis, Duarte concludes: ignore climate consensus studies based on random people rating journal article abstracts. I find it difficult to disagree with him on this.
The 3%
So, does all this leave you wondering what the 3% of papers not included in the consensus had to say? Well, wonder no more. There is a new paper out, published by Cook and colleagues:
Learning from mistakes
Rasmus Benestad, Dana Nuccitelli, Stephan Lewandowski, Katherine Hayhoe, Hans Olav Hygen, Rob van Dorland, John Cook
Abstract. Among papers stating a position on anthropogenic global warming (AGW), 97 % endorse AGW. What is happening with the 2 % of papers that reject AGW? We examine a selection of papers rejecting AGW. An analytical tool has been developed to replicate and test the results and methods used in these studies; our replication reveals a number of methodological flaws, and a pattern of common mistakes emerges that is not visible when looking at single isolated cases. Thus, real-life scientific disputes in some cases can be resolved, and we can learn from mistakes. A common denominator seems to be missing contextual information or ignoring information that does not fit the conclusions, be it other relevant work or related geophysical data. In many cases, shortcomings are due to insufficient model evaluation, leading to results that are not universally valid but rather are an artifact of a particular experimental setup. Other typical weaknesses include false dichotomies, inappropriate statistical methods, or basing conclusions on misconceived or incomplete physics. We also argue that science is never settled and that both mainstream and contrarian papers must be subject to sustained scrutiny. The merit of replication is highlighted and we discuss how the quality of the scientific literature may benefit from replication.
Published in Theoretical and Applied Climatology [link to full paper].
A look at the Supplementary Material shows that they considered credible skeptical papers (38 in total) – by Humlum, Scafetta, Solheim and others.
The gist of their analysis is that the authors were ‘outsiders’, not fully steeped in consensus lore and not referencing their preferred papers.
RealClimate has an entertaining post on the paper, Let’s learn from mistakes, where we learn that this paper was rejected by five journals before being published by Theoretical and Applied Climatology. I guess the real lesson from this paper is that you can get any kind of twaddle published, if you keep trying and submit it to different journals.
A consensus on what, exactly?
The consensus inferred from the Cook et al. analysis is a vague one indeed; exactly what are these scientists agreeing on? The ‘97% of the world’s climate scientists agree that humans are causing climate change’ is a fairly meaningless statement unless the relative amount (%) of human caused climate change is specified. Roy Spencer’s 2013 Senate testimony included the following statement:
“It should also be noted that the fact that I believe at least some of recent warming is human-caused places me in the 97% of researchers recently claimed to support the global warming consensus (actually, it’s 97% of the published papers, Cook et al., 2013). The 97% statement is therefore rather innocuous, since it probably includes all of the global warming “skeptics” I know of who are actively working in the field. Skeptics generally are skeptical of the view that recent warming is all human-caused, and/or that it is of a sufficient magnitude to warrant immediate action given the cost of energy policies to the poor. They do not claim humans have no impact on climate whatsoever.
The only credible way to ascertain whether scientists support the consensus on climate change is through surveys of climate scientists. This point is eloquently made in another post by Joe Duarte: The climate science consensus is 78-84%. Now I don’t agree with Duarte’s conclusion on that, but he makes some very salient points:
Tips for being a good science consumer and science writer. When you see an estimate of the climate science consensus:
Make sure it’s a direct survey of climate scientists. Climate scientists have full speech faculties and reading comprehension. Anyone wishing to know their views can fruitfully ask them. Also, be alert to the inclusion of people outside of climate science.
Make sure that the researchers are actual, qualified professionals. You would think you could take this for granted in a study published in a peer-reviewed journal, but sadly this is simply not the case when it comes to climate consensus research. They’ll publish anything with high estimates.
Be wary of researchers who are political activists. Their conflicts of interest will be at least as strong as that of an oil company that had produced a consensus study – moral and ideological identity is incredibly powerful, and is often a larger concern than money.
In general, do not trust methods that rest on intermediaries or interpreters, like people reviewing the climate science literature. Thus far, such work has been dominated by untrained amateurs motivated by political agendas.
Be mindful of the exact questions asked. The wording of a survey is everything.
Be cautious about papers published in climate science journals, or really in any journal that is not a survey research journal. Our experience with the ERL fraud illustrated that climate science journals may not be able to properly review consensus studies, since the methods (surveys or subjective coding of text) are outside their domains of expertise. The risk of junk science is even greater if the journal is run by political interests and is motivated to publish inflated estimates. For example, I would advise strong skepticism of anything published by Environmental Research Letters on the consensus – they’re run by political people like Kammen.
Is 47 the new 97?
The key question is to what extent climate scientists agree with key consensus statement of the IPCC:
“It is extremely likely {95%+ certainty} that more than half of the observed increase in global average surface temperature from 1951 to 2010 was caused by the anthropogenic increase in greenhouse gas concentrations and other anthropogenic forcings together. ”
Several surveys of climate scientists have addressed using survey questions that more or less address the issue of whether humans are the dominant cause of recent warming (discussed in the previous post by Duarte and summarized in my post The 97% feud).
The survey that I like the best is:
Verheggan et al. (2014) Scientists view about attribution of climate change. Environmental Science & Technology [link]
Recently, a more detailed report on the survey was made available [link] . Fabius Maximus has a fascinating post New study undercuts key IPCC finding (the text below draws liberally from this post). This survey examines agreement with the keynote statement of the IPCC AR5:
“It is extremely likely {95%+ certainty} that more than half of the observed increase in global average surface temperature from 1951 to 2010 was caused by the anthropogenic increase in greenhouse gas concentrations and other anthropogenic forcings together. ”
The survey examines both facets of the attribution statement – how much warming is caused by humans, and what is the confidence in that assessment.
In response to the question: What fraction of global warming since the mid 20th century can be attributed to human induced increases in atmospheric greenhouse gas concentrations? A total of 1,222 of 1,868 (64% of respondents) agreed with AR5 that the answer was over 50%. Excluding the 164 (8.8%) “I don’t know” respondents, yields 72% agree with the IPCC.
Slide1
The second question is: “What confidence level would you ascribe to your estimate that the anthropogenic greenhouse gas warming is more than 50%?” Of the 1,222 respondents who said that the anthropogenic contribution was over 50%, 797 (65%) said it was 95%+ certain (which the IPCC defines as “virtually certain” or “extremely likely”).
Slide2The 797 respondents who are highly confident that more than 50% of the warming is human caused) are 43% of all 1,868 respondents (47% excluding the “don’t know” group). Hence this survey finds that slightly less than half of climate scientists surveyed agree with the AR5 keynote statement in terms of confidence in the attribution statement.
Whose opinion ‘counts’?
Surveys of actual climate scientists is a much better way to elicit the actual opinions of scientist on this issue. But surveys raise the issue as to exactly who are the experts on the issue of attribution of climate change? The Verheggan et al. study was criticized in a published comment by Duarte, in terms of the basis for selecting participants to respond to the survey:
“There is a deeper problem. Inclusion of mitigation and impacts papers – even from physical sciences or engineering – creates a structural bias that will inflate estimates of consensus, because these categories have no symmetric disconfirming counterparts. These researchers have simply imported a consensus in global warming. They then proceed to their area of expertise. [These papers] do not carry any data or epistemic information about climate change or its causes, and the authors are unlikely to be experts on the subject, since it is not their field.
Increased public interest in any topic will reliably draw scholars from various fields. However, their endorsement (or rejection) of human-caused warming does not represent knowledge or independent assessments. Their votes are not quanta of consnsensus, but simply artifacts of career choices, and the changing political climate. Their inclusion will artificially inflate sample sizes, and will likely bias the results.”
Roy Spencer also addresses this issue in his Senate testimony (cited above):
“(R)elatively few researchers in the world – probably not much more than a dozen – have researched how sensitive today’s climate system is based upon actual measurements. This is why popular surveys of climate scientists and their beliefs regarding global warming have little meaning: very few of them have actually worked on the details involved in determining exactly how much warming might result from anthropogenic greenhouse gas emissions.”
The number of real experts on the detection and attribution of climate change is small, only a fraction of the respondents to these surveys. I raised this same issue in the pre-Climate Etc. days in response to the Anderegg et al. paper, in a comment at Collide-a-Scape (referenced by Columbia Journalism Review):
The scientific litmus test for the paper is the AR4 statement: “anthropogenic greenhouse gases have been responsible for “most” of the “unequivocal” warming of the Earth’s average global temperature over the second half of the 20th century”.
The climate experts with credibility in evaluating this statement are those scientists that are active in the area of detection and attribution. “Climate” scientists whose research areas is ecosystems, carbon cycle, economics, etc speak with no more authority on this subject than say Freeman Dyson.
I define the 20th century detection and attribution field to include those that create datasets, climate dynamicists that interpret the variability, radiative forcing, climate modeling, sensitivity analysis, feedback analysis. With this definition, 75% of the names on the list disappear. If you further eliminate people that create datasets but don’t interpret the datasets, you have less than 20% of the original list.
Apart from Anderegg’s classification of the likes of Freeman Dyson as not a ‘climate expert’ (since he didn’t have 20 peer reviewed publications that they classed as ‘climate papers’), they also did not include solar – climate experts such as Syun Akasofu (since apparently Akasofu’s solar papers do not count as ‘climate’).
But perhaps the most important point is that of the scientists who are skeptical of the IPCC consensus, a disproportionately large number of these skeptical scientists are experts on climate change detection/attribution. Think Spencer, Christy, Lindzen, etc. etc.
Bottom line: inflating the numbers of ‘climate scientists’ in such surveys attempts to hide that there is a serious scientific debate about the detection and attribution of recent warming, and that scientists who are skeptical of the IPCC consensus conclusion are disproportionately expert in the area of climate change detection and attribution.
Conceits of consensus
And finally, a fascinating article The conceits of ‘consensus’ in Halakhic rhetoric. Read the whole thing, it is superb. A few choice excerpts:
The distinguishing characteristic of these appeals to consensus is that the legitimacy or rejection of an opinion is not determined by intrinsic, objective, qualifiable criteria or its merits, but by its adoption by certain people. The primary premise of such arguments is that unanimity or a plurality of agreement among a given collective is halakhically binding on the Jewish population and cannot be further contested or subject to review.
Just as the appeal to consensus stresses people over logic, subsequent debate will also focus on the merits of individuals and their worthiness to be included or excluded from the conversation. This situation runs the risk of the No True Scotsman fallacy whereby one excludes a contradictory opinion on the grounds that no one who could possibly hold such an opinion is worth consideration.
Debates over inclusion and exclusion for consensus are susceptible to social manipulations as well. Since these determinations imply a hierarchy or rank of some sort, attempts which disturb an existing order may be met with various forms of bullying or intimidation – either in terms of giving too much credit to one opinion or individual or not enough deference to another. Thus any consensus reached on this basis would not be not based out of genuine agreement, but fear of reprisals. The consensus of the collective may be similarly manipulated through implicit or overt marketing as a way to artificially besmirch or enhance someone’s reputation.
The next premise to consider is the correlation between consensus and correctness such that if most (or all) people believe something to be true, then by the value of its widespread acceptance and popularity, it must be correct. This is a well known logical fallacy known as argumentum ad populum, sometimes called the bandwagon fallacy. This should be familiar to anyone who has ever been admonished, “if all your friends would jump off a bridge would you follow?” It should also be obvious that at face value that Jews, especially Orthodox Jews, ought to reject this idea as a matter of principle.
Appeals to consensus are common and relatively simply to assert, but those who rely on consensus rarely if ever acknowledge, address, or defend, the assumptions inherent with the invoking of consensus as a source – if not the determinant – of practical Jewish law. As I will demonstrate, appeals to consensus are laden with problematic logical and halakhic assumptions such that while “consensus” may constitute one factor in determining a specific psak, it is not nearly the definitive halakhic criterion its proponents would like to believe.
) I'm done here. :bowing:
