Lies, damned lies, and slippery surveys

On the afternoon of Thursday 16th February, perhaps in light of a week that has been even rockier than usual, current US President Donald Trump held a controversial press conference. Whilst this was, in itself, newsworthy for a variety of reasons, there was an unexpected plot-twist. Trump followed up with sending out a mailshot to his Republican supporters…

Click to embiggen, but just in case you can’t read the image for some reason, as its preface, the email opened with the ongoing narrative that “the mainstream media” (this damning moniker seems to exclude pro-Trump agencies such as Fox News, incidentally) is carrying out hit jobs, attacks, deceptions, and so forth, specifically against Trump and the Republicans. As part of the resistance to this, recipients were encouraged to complete a “Mainstream Media Accountability Survey” (PDF for posterity).

Very quickly, that terribly biased, pesky mainstream media noted that this survey was, itself, rather biased. In fairness to both sides, claiming that a survey is biased is an easy win. Every survey and questionnaire contains bias right from the start – the goal of the survey, the topic choice, the time of asking, the person who is asked, the person doing the asking – all are the product of intentional choice and have an ability to alter the results, but the point is to limit and control for all of these factors as much as possible. More importantly, it’s an easy claim to make because it can be surprisingly difficult to pick out the exact features that are creating larger degrees of bias than we would consider acceptable. You might read the survey and get an intuitive sense that it isn’t playing fairly, but it’s helpful to be able to specifically identify the very methods that are being employed to push you one way or another. And that’s what I do in this post.

At the bottom you can find the twenty-five questions in brief. I could, of course, pick apart every single one. (No really. Every… Single… One.) But then this would turn into some sort of epic methodological screed – more than it already is – on the best practices for eliciting unbiased answers from respondents on emotive topics, and I would froth myself into an artery-warping rage. So, in the spirit of keeping this down to an easily digestible blog post, I’ll only have a go at the sampling and premise in general, and then five specifically linguistic points of contention.

Sampling demographic and context

As any good scholar who elicits data from participants will tell you, how you set up your survey and introduce it to your participants is as important as the survey itself. Let’s say you want to investigate the UK’s general attitude towards legalising fox hunting. Going to a drag hunt and canvassing the jolly people on horseback is not going to get you a particularly representative sample. No one will be very shocked if your ultimate conclusions from your data are that fox hunting should be legalised again.

This inherent skewing of your results will only get worse, too, if your opening salvo is something like, “My dad lost his job taking care of the fox hounds after hunting was criminalised and then he couldn’t find another job and drank himself to death. So, what do you think about this appalling business?” Even in the unlikely event that you find a hunter who is actually in favour of criminalisation, emotionally and sympathetically you’ve made it much harder for them to disagree with you. To do so might seem like an affront to your personal tragedy.

In a nutshell, to gather a proper set of responses you need to sample fairly not just from people who might support Idea A, but also people who might not support it, and from people who might have surprising and different ideas. Unfortunately in this case, by only emailing and addressing his survey at Republicans, Trump has dived deep into his echo chamber, and yelled “Isn’t the mainstream media terrible?!” It will be absolutely no surprise if he then hears a thousand little voices all echoing back at him, “Terrible! Terrible! Terrible!”

Question 1: it’s so unfair!

Let’s start with an easy one – the very first question:

1. Do you believe that the mainstream media has reported unfairly on our movement? [Y | N | No opinion | Other]

(See also Q11 and Q15 for the same unfairness.) In the grand scheme of things, this one is actually not awful, but it comes first, so we’ll start there. This is a gently leading question, as opposed to some of the howling maniacs holding guns to your head that come later, and the sneaky little issue here is the word unfairly. What’s the problem? Well, in lots of choices of antonymic pairs (e.g. hot/cold, big/small, etc.), there is a marked versus an unmarked option, e.g.

“How tall are you?” vs “How short are you?”

“How fast does it go?” vs “How slow does it go?”

“How many *buns are left?” vs “How few *buns are left?”

(*That’s cupcakes to my US readers…)

In the left-hand version of each question is the unmarked, and default choice. This is the one we expect, and it tends to be the one with greater positive valence (i.e. the nicer thing to ask), but despite this, it’s considerably more value neutral. “How tall are you?” doesn’t really suggest that you are, in fact, tall. It just asks for your height. Of course you could ask in such a way, staring up as if into the sun, shading your eyes, thereby stressing that you’re asking a human giraffe, but that’s an exception. We’re thinking here of the scenarios in which the doctor is taking notes about your health or your friend is checking whether you could borrow her clothing. Similarly “How fast does it go?” does not automatically suggest that something is fast, and “How many buns are left?” does not imply that many are left. There might be none, especially if I’ve been nearby. The point is, these left-hand forms are unmarked and are all much closer to neutral. Meanwhile the right-hand forms are marked. We notice them as unusual, and because they lack that closer-to-neutral valence, we start looking for explanations. “How few buns are left? That means there aren’t many! How short am I? You think I’m short?!” And so on.

Back to unfairly. The idea is now subtly planted in the reader’s brain that unfairness has occurred. It’s not world-ending by itself, but it doesn’t help in the acquisition of good-quality data. In a perfect world, of course, you wouldn’t even go about asking the same question with fairly because, to a much weaker extent, the same argument can be made. Instead, you might present a series of statements and ask the respondent to tick the one they agreed with, e.g.

Please rate how strongly you agree or disagree with the following statements:

  1. The mainstream media has reported very fairly on our movement
  2. The mainstream media has reported somewhat fairly on our movement
  3. The mainstream media has reported neither fairly nor unfairly on our movement
  4. The mainstream media has reported unfairly on our movement
  5. The mainstream media has reported very unfairly on our movement

In practice, whilst phrasing things in this way is less leading, it’s also looooooong, both to write, and to read, and for every degree of friction you add to the survey-taking process – more questions, more reading, more whatever – you lose an unspecified number of survey respondents. People simply don’t have the time or energy to wade through a thesis. If we extend the survey writers the benefit of the doubt, we might guess that they chose brevity over neutrality to ensure a greater number of respondents, but in practice the trade-off for increased quantity is usually a decrease in quality, and that undermines the entire point of the survey altogether.

Questions 5, 17, 21: they do the worst job!

Let’s pick another good one.

5. On which issues does the mainstream media do the worst job of representing Republicans? (Select as many that apply.) [Immigration, Economics, Pro-life values, Religion, Individual liberty, Conservatism, Foreign policy, Second Amendment rights]

What we have in this question is a lovely little structural presupposition. If we reassemble this as a statement by tacking the questioning element onto the end, what we now get is something like “The mainstream media does the worst job of representing Republicans… on which issues…?” As you can see, this presupposes (hence the technical term) a certain state of affairs – that the media does “the worst job” – and then simply asks for you to fill in a few more details about that assertion. Interestingly, this neat little trick doesn’t just occur once. Consider these two gems:

17. Do you believe that the media has been far too quick to spread false stories about our movement? [Y | N | No opinion | Other…]

21. Do you believe that the mainstream media has been too eager to jump to conclusions about rumored stories? [Y | N | No opinion | Other…]

Now think really carefully about what you’re actually saying if you tick yes. “I agree. The media has been far too quick to spread false stories. And, it is much too eager to jump to conclusions about rumored stories.” Aaaaand now what if you say no? “I disagree. The media has not been quick enough to spread false stories. Also, it is not eager enough to jump to conclusions about rumored stories.”

Er… Is that quite what you wanted to convey?

Of course you could choose “other” and write glib little responses in the free-text boxes, as others may have already done, but most people will simply tick based on a gut feeling of what the question seems to ask. Let me give you another example. If you quickly ask people, “Do you often break the law?” and press them to answer promptly, most will instinctively say “no”. This is because, under pressure, or when not paying attention, people largely hear, “Do you break the law?” and answer accordingly. In reality, their answer pertains specifically to the oftenness of their crime sprees. Read the question again. “Do you often break the law?” If yes, you’re a career criminal. If no, maybe you’re more of a hobbyist criminal. Either way, you agree to the assumption that you do in fact break the law. Likewise, our two questions above focus not on whether the underlying presupposition is true, i.e. that the media spreads false stories or jumps to conclusions. Instead they are asking about speed (too quick…) and attitude (too eager…) and the answer given pertains directly to that.

Question 12: what DO you know?

This next example also has shades of the previous section:

12. Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump’s temporary restriction executive order? [Y | N | No opinion | Other…]

My first instinct here is to wonder what on earth the point of this question is. If I were to assume deviousness on the part of the survey, I might conclude that its purpose is to demonstrate that a poll that favours Trump has been buried. Either way, no answer to it seems especially informative, unless we are intending to twist quite what that answer means. We could say that yes, we were aware of the poll (but this says nothing about agreeing with it) or no, we were not aware of it. More pressing, of course, is which poll does this mean anyway?? The respondent must guess or presume and of course some might be tempted to just tick no out of uncertainty.

In case you’re interested, a poll by IPSOS/Reuter on 31st January 2017 did in fact show that 48% of US citizens agreed with the travel ban, versus 41% who disagreed (see third question from the end) but I have taken the liberty of assuming that this is indeed the correct poll, and that the survey above is asking about that specific question in the IPSOS poll. Why might that be difficult to tell? Because that isn’t the only question in there, and most of the rest of that very same poll points in virtually the opposite direction. For example, the majority of respondents to that IPSOS poll agreed… (a) with the US taking in immigrants and refugees, (b) that singling out a group based on religion is a violation of American principles, (c) that all countries including the US should open borders to refugees, and (d) that the US borders should be open to those fleeing ISIS specifically. Similarly the majority agreed that it was not okay… (a) to welcome Christian but not Muslim refugees, or (b) to only allow in refugees from some conflicts but not others. As an additional side-note, several other polls (yes, that last one really is a Fox News poll, too) came to the opposite conclusion about the apparent popularity of Trump’s executive order. So in that context, how do we answer this question again…?

Question 14 & 15: are you confused yet?

In case the above became difficult to answer, don’t worry. Things get increasingly opaque as we move on. Consider these two:

14. Do you believe that contrary to what the media says, raising taxes does not create jobs? [Y | N | No opinion | Other…]

15. Do you believe that people of faith have been unfairly characterized by the media? [Y | N | No opinion | Other…]

I had to read Q14 about eight times and I’m still not sure how to answer it. If anyone can parse this for me, seriously, please, please, PLEASE give me an idiot’s guide to it. Maybe that’s just my brain, though. So what about Q15? Well, this is a different form of ambiguity and here we return to wondering quite what the purpose of the question is. In this politically sensitive climate, in which Christians and Muslims are being frequently juxtaposed as the goodies versus the baddies, what is meant by “people of faith”, exactly, and quite how will my answer be characterised later?

Think of it this way: 90% of people answer yes. Will this become the headline: “90% think Christians unfairly characterised by media!” Alternatively, only 10% answer yes. Will this instead become, “90% think Muslims fairly characterised by media!” More pragmatically, however, let’s imagine that the respondent thinks that one group has had a very easy ride in the media, whilst another has been horrifically and unfairly demonised. How does one capture this in a simple yes/no/other? Overall, this question is over-general in the group that it is asking about, and as a result of that, it is unfairly restrictive in its range of possible answers.

All the questions

A last issue that is more a cumulative effect than something that exists in a single feature, is the pervasive tone of the survey, including a fairly divisive “us versus them” positioning. The most obvious case of this is the use of inclusive pronouns, like “our” and “we” and “us” (see Q1, Q20, Q24, Q25), implicitly in contradistinction to groups such as the Democrats, the media, and so on. Why is this so bad? Because it’s appealing to solidarity and conformity. It’s asking, what do we think? What is our stance? As such, it encourages the respondent to think and identify as a member of this specific group, rather than as an individual who just also happens to be a member of a group.

Another, and perhaps more pressing issue is the use of emotive language that conjures up a host of wrongdoings, fears, and horrors. For instance, if we go question by question and pick out common themes and words, by the end of the survey we have been called upon to consider, amongst other things, “noisy”, attacking, unfair, biased, false, divisive, feud-creating, slur-ridden, rumor-based, non-fact-checked and non-diligent reporting (Q1, Q10, Q11, Q13, Q15, Q17, Q18, Q19, Q20, Q21, Q22, Q24), our trust in, and the truthfulness of the media (Q2, Q3, Q4, Q9), issues that receive the worst reporting (Q5), illegal immigration and radical Islamic terrorism (Q11, Q13), raising taxes and non-creation of jobs (Q14), and the Democrats obstructing Trump (Q22).

That’s a lot of fear and loathing to wade through, when you think about it, and each question primes us for the next.

Conclusion

In case you needed to even read this next sentence, as an exercise in gathering a useful overview of the views of the populace, this survey is a complete and utter fail. Even as an exercise in gathering an overview of Republican opinions on the matter, this is still a staggeringly bad effort. This will not reflect a diversity of opinion within the base, even if no one but Republicans answer it. That’s without dissecting every question in minute detail, as I’m sure some would wish I had. (Okay, not many, but maybe one or two.)

And yet, there is still one last very big fish left to fry. With occasional exceptions, throughout I’ve largely pretended that the entire purpose of this survey is to elicit useful data from which to actually learn something and then make informed decisions. It doesn’t take any degrees in linguistics, or indeed anything, to realise that this fairly obviously isn’t its point. As with so many polls and surveys, it is being used as a political weapon. Sadly, in this case, the effort seems rather strongly directed at a censorship of portions of the media, and we might infer that it is especially aimed at the non-supportive sections.

I don’t for one second suggest that the media is flawless. You need only go back a few months to find a post in which I rather bluntly expose a broadsheet journalist doing a less-than-stellar job. However, there’s a distinction between pushing for ever-higher journalistic standards, and shutting down all critical, negative, or challenging discussion. Whilst the survey itself could be seen as an insignificant blip in the overall novelty show of the Trump administration’s daily activities, what it symbolises is rather more troublesome, and for that reason, examples like this should not be too lightly waved aside.

Nope. That’s two thumbs down from me, Mr Trump.

All the survey questions in brief

  1. Do you believe that the mainstream media has reported unfairly on our movement? [Y/N/etc.]
  2. Do you trust MSNBC to report fairly on Trump’s presidency? [Y/N/etc.]
  3. Do you trust CNN to report fairly on Trump’s presidency? [Y/N/etc.]
  4. Do you trust Fox News to report fairly on Trump’s presidency? [Y/N/etc.]
  5. On which issues does the mainstream media do the worst job of representing Republicans? (Select as many that apply.) [List]
  6. Which television source do you primarily get your news from? [List]
  7. Do you use a source not listed above? [Free-text box]
  8. Which online source do you use the most? [Free-text box]
  9. Do you trust the mainstream media to tell the truth about the Republican Party’s positions and actions? [Y/N/etc.]
  10. Do you believe that the mainstream media does not do their due diligence fact-checking before publishing stories on the Trump administration? [Y/N/etc.]
  11. Do you believe that the media unfairly reported on President Trump’s executive order temporarily restricting people entering our country from nations compromised by radical Islamic terrorism? [Y/N/etc.]
  12. Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump’s temporary restriction executive order? [Y/N/etc.]
  13. Do you believe that political correctness has created biased news coverage on both illegal immigration and radical Islamic terrorism? [Y/N/etc.]
  14. Do you believe that contrary to what the media says, raising taxes does not create jobs? [Y/N/etc.]
  15. Do you believe that people of faith have been unfairly characterized by the media? [Y/N/etc.]
  16. Do you believe that the media wrongly attributes gun violence to Second Amendment rights? [Y/N/etc.]
  17. Do you believe that the media has been far too quick to spread false stories about our movement? [Y/N/etc.]
  18. Do you believe that the media uses slurs rather than facts to attack conservative stances on issues like border control, religious liberties, and ObamaCare? [Y/N/etc.]
  19. Do you believe that the media purposely tries to divide Republicans against each other in order to help elect Democrats? [Y/N/etc.]
  20. Do you believe that the media creates false feuds within our Party in order to make us seem divided? [Y/N/etc.]
  21. Do you believe that the mainstream media has been too eager to jump to conclusions about rumored stories? [Y/N/etc.]
  22. Do you believe that if Republicans were obstructing Obama like Democrats are doing to President Trump, the mainstream media would attack Republicans? [Y/N/etc.]
  23. Do you agree with the President’s decision to break with tradition by giving lesser known reporters and bloggers the chance to ask the White House Press Secretary questions? [Y/N/etc.]
  24. Do you agree with President Trump’s media strategy to cut through the media’s noise and deliver our message straight to the people? [Y/N/etc.]
  25. Do you believe that our Party should spend more time and resources holding the mainstream media accountable? [Y/N/etc.]