Trump’s Survey Data Is (Not Surprising) Also Biased

In defending his proposals to end Muslim immigration, Donald Trump cites research that says 25 percent of American Muslims agree that violence against the U.S. is justified, and 51 percent want American Muslims to have the option of being governed by Islamic law.

The poll that generated this data, commissioned by the Center for Security Policy, is unreliable and filled with survey error — or to employ a more popular term, crap.

In my Reporting classes, I teach my students how to evaluate survey data.  This survey fails on two counts that I talk about in class — one mathematical and the other conceptual — that render it statistically worthless, except as a gathering place for flies.

First, the math part (and yes, we’re starting with the math because if I put this second you would change the channel before you got to it).

The survey uses what we call a non-probability sample, which means that you cannot generalize its results to the population as a whole.  To swap the negatives, with a probability sample (a random sample being one example), you can generalize the results, with caution, of course.

A typical survey that employs a random sample involves mail, telephone or direct interview of a pre-selected sample.  The CSP survey was an online survey on a web site, a voluntary survey, that anyone could click on and answer.  Not only is the sample flawed mathematically, but the pollsters cannot even verify who took the survey.  For all we know, CSP staff could have clicked on the survey themselves and answered it in a way that advanced its organization’s initiatives.

You used to see these all the time, often on the entry page of news or sports sites like CNN, FOX News and ESPN.  They would put some current events or favorite athlete question on the page and let you click on it.  But it always had a disclaimer that this was not a scientific survey.

You also see these now on Twitter, with its polling feature.  It too is all in fun.  The results are meaningless for getting society’s pulse.

I also warn my students to note who is conducting the survey, to see if they have an interest in the results.  For this reason, sorry, but surveys conducted by politicians are usually biased and self-serving, worthless beyond the politician’s interests.

Speaking of which, this survey of Muslims!  As this report from Foreign Policy points out, the Center for Security Policy has a political agenda that it was seeking to advance with the survey.

The CSP actively promotes an anti-Muslim message, and the survey seems slanted to be consistent with the agenda.  Its executive director, Frank Gaffney, has a reputation for making extreme, even outlandish comments about political opponents.

That immediately casts suspicions on the results, which can be expected to support CSP’s agenda.  According to this report from Georgetown, that bias shows up in survey wording and in how the results are interpreted.

By comparison, the most recent numbers available from Pew Research Center (whom Trump also cited, though vaguely), 86 percent of the Muslims surveyed said tactics such as suicide bombing were rarely or never justified.

As with so much of Trump’s message, the numbers are hard to believe, once you look at them closely.

Advertisements