Lies, damned lies, and slippery surveys

On the afternoon of Thursday 16th February, perhaps in light of a week that has been even rockier than usual, current US President Donald Trump held a controversial press conference. Whilst this was, in itself, newsworthy for a variety of reasons, there was an unexpected plot-twist. Trump followed up with sending out a mailshot to his Republican supporters…

Click to embiggen, but just in case you can’t read the image for some reason, as its preface, the email opened with the ongoing narrative that “the mainstream media” (this damning moniker seems to exclude pro-Trump agencies such as Fox News, incidentally) is carrying out hit jobs, attacks, deceptions, and so forth, specifically against Trump and the Republicans. As part of the resistance to this, recipients were encouraged to complete a “Mainstream Media Accountability Survey” (PDF for posterity).

Very quickly, that terribly biased, pesky mainstream media noted that this survey was, itself, rather biased. In fairness to both sides, claiming that a survey is biased is an easy win. Every survey and questionnaire contains bias right from the start – the goal of the survey, the topic choice, the time of asking, the person who is asked, the person doing the asking – all are the product of intentional choice and have an ability to alter the results, but the point is to limit and control for all of these factors as much as possible. More importantly, it’s an easy claim to make because it can be surprisingly difficult to pick out the exact features that are creating larger degrees of bias than we would consider acceptable. You might read the survey and get an intuitive sense that it isn’t playing fairly, but it’s helpful to be able to specifically identify the very methods that are being employed to push you one way or another. And that’s what I do in this post. Continue reading