Boringly Reliable Polls – the Divisive but Critical Business of Consumer Polling
Results You Can Count On
Recent polls, including those conducted by Maru/Blue, have generated some lively debate among certain political leaders, who are naturally sensitive to the impact that positive – or negative – poll results can have on their reputation.
At Maru/Blue we take great care to treat our respondents with respect and continuously test the validity and reliability of our market communities by tracking the same results over time. The result? Communities that are consistent, accurate and reliable. Boringly reliable.
What People Like and Dislike About Doing Surveys
“Reliable” is not what you would call a sexy word. Nor does it evoke visions of speed and agility. Rather, it conjures up pictures of immovable mountains, a lighthouse weathering a storm and a dog named Old Faithful.
The word reliable has its origins in relier, Old French for “fasten” or “attach.” Something reliable can be counted on to be predictable and stable.
In the world of survey research, reliability is essential. You need to be certain that if you ask a question today, that you’d get the same answer if you asked it again tomorrow.
Year in Review
The success or failure of the insights industry is entirely dependant on the quality of our relationships with the people who complete our surveys. At Maru/Blue we are committed to treating our respondents like people—people willing to share their time, opinions and experiences with us. We recently conducted video interviews with members of our Maru Voice UK community, and asked them what they like and dislike about doing surveys. We also solicited suggestions for improving the survey experience. The results reveal a need to radically change how the industry conducts research and how it treats respondents.
Bad Respondent Experiences are Killing the Insights Industry
Happy New Year!
2018 was a momentous year for Maru/Blue in our first year of our new journey. We would like to say thank you for your support and partnership and to share an update about the great progress that we have made. This past year we:
Launched Maru/Blue, premium quality data services firm
Formerly part of Maru/Matchbox, Maru/Blue is focused on providing access to deeply engaged, well-curated respondents who feed-back actionable insights to businesses and brands. The quality of these responses subsequently enables clients to make better-informed decisions about their business strategies.
CMO’s Challenges: An Interview with PwC Consulting’s Pina Sciarra
I received a survey invite from a market research community I had recently joined. Clicking on it, I was sent to an opening page that asked me to enter my age. I was told I qualified for a survey and was sent to a large, well-known survey router service. The landing page told me the survey was 18 minutes long and to click the link to begin the survey. So, I did.
Then I was met with a screen that, after whirling a bit, said “Unfortunately, based on your responses to pre-qualification questions, it looks like your profile isn’t a great fit for the survey we had in mind for you. Hold on while we check to see if we have something better…” At least they let me down gently and promised me another survey.
Bad Sample and Misleading Results: A Cautionary Tale
It’s tough being a CMO. They have an enormous mandate, are required to stay on top of a rapidly changing toolkit and have very little time in which to make their mark. Are Canada’s marketers keeping up with the pace of change? That’s the question we set out to answer with a recent survey of more than 100 Canadian CMOs and senior-level marketing decision makers. Together, APEX Public Relations, ruckus digital and Maru/Blue launched CMO Lab™, a research-driven project to equip senior-level marketers with the insights they need to help their organizations succeed.
Interest in Cannabis Beverages—Higher in America: Feedback From our Cannabis Community
Sampling is not something people think much about. It gets taken for granted and is often treated like the making of sausages—people don’t necessarily want to know what goes into it. But people ignore the issue of sample quality at their peril, because bad sample can lead to deeply misleading results. And when those misleading results are made public, they undermine the reputation of the entire insights industry—as we’ll see in a cautionary tale from a recent election.
A Cautionary Tale Of How Badly Research Can Fail If We Don't Take Sample Seriously
Cannabis became legal in Canada a few weeks ago, making it the second country in the world (after Uruguay) to repeal the prohibition of marijuana. While the world watched with a mix of skepticism, interest, and humor, Canada got excited for a few hours, then carried on as usual. Sure, there were some long lines at cannabis shops, stock ran low and selection was limited. Legalization became water cooler talk around the country…but after the initial excitement, life continued as usual.
Interest in Cannabis Beverages—Higher in America: Feedback From our Cannabis Community
The Plea To Think Of Respondents As People Not Sample
“Something that is reliable is usually boring and we don’t think much of it. When the office building you are sitting in doesn’t suddenly become a splintered mass of glass, concrete and steel girders do we send a thank you note to the architects and engineers? Nope. We just take these things for granted and assume that we can count on the science behind them?” I wrote that, with my colleague Andrew Grenville, in our whitepaper (Still) Boringly Reliable. This paper provides evidence of the reliability of our market communities; Maru Springboard America and Maru Voice Canada.
Maru Springboard America Is The #1 Online Quality Sample Source, According To Nate Silver
As an industry, everything we do hinges on the information we get from people who answer our questions. But we tend to take them for granted, and think of them as just "sample." And we ask a lot of them. We ask them to give us their time, put up with our obscure and often boring questions, work their way through long grids, and plow through long, mobile-unfriendly surveys. And then after we're done taking from them, we say thanks and goodbye. We don't ask "was it as good for you as it was for me?"
Profiling People Leads To A Superior Survey Experience And Better Information
At Maru/Blue we take great pride in the quality of our market communities. We go to great lengths to ensure they are of the highest quality. We know they are representative and reliable, but it’s great when a respected researcher’s analysis proves it, again.
Nate Silver’s FiveThirtyEight website aggregates election polls and makes predictions based on those aggregations. As part of their process, they weight the polls based on their accuracy.
We Are Maru/Blue: 5 Declarations
Data shows that 9 out of 10 people who are being routed through sample exchanges as river sample fail to qualify for a survey they are screened for. So they get asked to qualify for another study—and get asked basics like age, gender, race and region over and over again. That’s a terrible respondent experience. And it is completely unnecessary.
A well-profiled community allows you to target the right person with the right survey.
What Keeps Researchers Up At Night? Sample Quality. And They Are Right to Be Worried.
Blue evokes the limitless possibilities of the open sky, the depths of the ocean, and the purity of a blue flame. This suits our sample and data services offer perfectly because Maru/Blue make it easy to connect with an almost limitless number of deeply profiled community members, whose consistency and quality is proven.
Our communities are representative of everyone. That’s why we’re the #1 online pollster in Nate Silver’s 538 pollster ratings. We reflect reality. No bias. No skew. Just the voice of the people—all the people.
Known Respondents Yield Quicker, Less Expensive Survey Results
Concerns about the quality of sample bubbled to the surface when a sample of 1,533 researchers from around the globe—both client-side and supplier—were asked an open-ended question about challenges that exist in the research industry today.
We’re glad to see that people are concerned about sample quality—they should be. We’ve been tracking this problem for years and it’s getting worse, not better.
The High Cost Of A Cheap Sample
If you were heading to the mall to get a new smartphone, would you go randomly from store to store asking whether they sell smartphones? Or would you want to know which store sells them and to go straight to that store?
Chances are you don’t want to a) waste time going from store to store, or b) look like a fool by asking the clerk at the women’s clothing store if they sell smartphones. You just want to get straight to the smartphone store, right?
“At least it will give us directional information. I mean, how wrong can it be?” I’ve heard those remarks and many like them bantered about when people rationalize using cheap sample sources.
The fact is, bad or “cheap” sample can give you information that is dead wrong. So wrong that the information it provides is directional—it’s just pointing in the wrong direction.