The Sorry State of Surveys

Do you think business news stories based on surveys are useful? Please answer "Yes" or "No."

"But," you may object, "what if I think, 'It depends?'" Ah-HA! When then, indeed?

I received this week a survey from one of my service providers. It asked how I felt about social media and different relevant services. My answer choices were, "Like it" and "Hate it." That's all.

I started to simply "abandon" the survey—the scientific term for quitting—and go on to something worth my time. But I want this company to do well. It is a good-sized employer in my area up against a heavyweight competitor based elsewhere. I have met a few people who work there and interviewed the CEO, who is a nice guy.

Their survey had an e-mail address. So first I wrote a paragraph trying to make clear I was not flaming them. Then I said, "I appreciate that you are trying to keep the survey short, but it takes little longer to select from five or more choices of a scientifically valid Likert scale than it does to choose from two choices that may not be true. These 'false choices' are well proven to raise abandonment rates, which skews your data. Also, when you force someone to treat their 51% Hate the same as another item they 100% Hate, those responses are highly inaccurate—by definition, they are off by as much as 49% from the person's true feelings." The company has not responded.

A few weeks back an e-mail request arrived for input on a story to be published by a large professional group. The story was based on a survey from a consulting firm which reported a number of conclusions about "employee" attitudes. I looked up the survey report and was appalled. As a member of the group, I wrote the requestor and suggested she run the survey by a university expert before basing a story on it. "The white paper on (the company's) site indicates they only surveyed 'business leaders.' By definition, we cannot draw accurate conclusions about employees without surveying employees. The samples are too small to represent the larger population of business leaders. The paper presents no information on methodology or the demographics of the sample." She wrote back and said she would get it reviewed.

The latter criticisms have been applicable to almost every report I've checked that came out on business blogs or sites. One I wanted to report on here in Teams Blog. But then I looked at the "Methodology" section. In a scientific journal, this would explain how the study was designed; how the people being tested (the "subjects") were chosen; how the survey was developed and tested to ensure, for example, that people answered it the same way each time; and how the results were collected and analyzed. This information allows other scientists to make sure the survey was done correctly and repeat it to see if they get the same results ("replicate the findings").

But reports from consulting firms offer limited details. Some leave out the methodology altogether. This one I had wanted to cover was nowhere near scientific standards, but better than most. The organization had a database of people interested in a particular profession, sent the survey to all of them who held certain titles, and got back thousands of responses. This method made it possible the answers would pretty well match the answers everyone holding those titles would give if you could survey everyone. In scientific terms, it would be a "representative sample."

Looking more carefully, however, I realized the database only included members of the organization. Next I looked at the demographics (the characteristics) of the subjects. About 70% of them came from large employers. As politicians hammer home all the time, correctly for once, most jobs are in small businesses. (The U.S. Census Bureau says only 48% of Americans work in large organizations). The survey's sample was probably representative of members of that profession who were members of the organization and worked for large companies.

The report did not say that, however. It acted as if the results were representative of everyone in the profession. It's like someone surveyed the Florida Marlins baseball team, then kept saying, "baseball players believe" rather than "Florida Marlins believe." The Marlins' answers might be very different from those of other Major Leaguers, not to mention those of minor leaguers and amateurs.

Now to another survey problem I will illustrate with an extreme example. Please answer on a 1 to 5 scale, from "Completely Disagree" to "Completely Agree," the following: "I like football and needlepoint."

I know there are people who like both, and some like neither, but I'm guessing most folks only like one or the other to some degree. So how would those folks accurately answer the question? Making things worse, do I mean viewing football and needlepoint, or doing them?

Conjunctions (and, but, or) in survey questions are a no-no according to most experts, yet I have seen a survey questionnaire created by organizational psychologists in a consulting firm that broke the rule repeatedly. The second problem above is less common, but it occurs in popular business assessments. If you cannot be reasonably sure what the respondent was responding to, or how they interpreted the question, your data is useless.

The point to this post is survey design is not for amateurs. I had the honor of studying the topic with a professor who wrote a widely used textbook on research methods. Yet if I were planning on investing time or money based on the results of a survey I created, I would pay an expert to review it. Your team deserves better than to make decisions on inaccurate data or poorly researched news stories about poorly done surveys. So yet again I warn you: Question your sources.

Action Item: Post this near your computer as a reminder, and next time you see an article or blog post about a survey put out by a company or professional group (instead of a university), look up the survey report online. In it, look for a "Methodology" section and scan for some of the problems I have mentioned. The more you find, the more careful you should be in using the results.