Quality in our Industry

A colleague recently completed a business-to-business survey circulated by a trade publication. Quality issues were clear right away. First, there was no effort to qualify the respondents: she was able to easily complete the full survey, even though our business was not in the target sample. The questions were not well designed: it was easy to see what kinds of answers they were looking for. There was no anonymity or confidentiality, so it appeared highly likely that respondents with the right profile would end up being sales targets.

In all, this seemed like a survey that would likely deliver low-quality results for the client. This matters in some cases more than others, since not all clients conduct research because they have real questions and want real answers. Some conduct surveys for other reasons – say, to attract some media coverage or generate sales leads. In those cases, quality matters much less.

Real questions, real answers

But when an organization is looking for research that will help them make strategic choices or guide significant marketing investments, quality is a critical concern. Quality information and analysis do tend to require more resources, but a high-quality research project is also much more likely to generate useful insights and deliver a strong return on investment.

The first step in designing high-quality research is understanding the question an organization wants to answer. Clients often engage Environics when they want to solve a problem that’s complex and ambiguous – something they’ve been struggling with for a while. In these cases, working together to carefully define the research question can be an important step in itself.

As consultants, our job is not to simply field a survey or conduct some focus groups. We work with clients to help them clarify what they need to know in order to meet their goals, and to fully understand their research options. Through this consultative engagement, we help clients reach good strategic conclusions regarding the design of the project – choosing the research tools and analytical approaches that will best meet their needs.

Threats to quality

The biggest challenge in quality is getting the right people to participate in your study. This challenge takes several forms:

• Online panels have a growing quality problem, with fraudulent respondents giving the answers they think researchers want in order to receive incentives. Respondents show up in samples repeatedly as panel companies cut corners to reduce costs.
• Members of some groups – notably Indigenous, racialized, and young participants (especially young men) – tend to be more difficult to reach than others. It’s not an impossible task, but it often takes some patience and targeted strategies.
• It’s not only demographic diversity that can be hard to capture. The psychographic diversity that exists in society – variations in values and attitudes – can also be elusive, as research participants give socially appropriate answers instead of sharing what they really think. Known as social desirability bias, this is not purely a sampling problem; questionnaire design also matters. Researchers have to be creative in developing items that don’t have an obviously “wrong” or inappropriate answer.

Fast, cheap, and good?

Overcoming the major threats to research quality typically requires some combination of time and money. In research as in other fields, buyers tend to want projects fast, cheap, and good – but in reality they can only have two out of three. Skipping on cost or time will almost inevitably compromise quality.

Why is time important? If a client wants a survey completed by 1,000 respondents by the end of the week, it’s achievable. But the people who complete the survey will have a few key qualities in common: they became aware of the survey, were available to complete it, and were eager enough that they got it done right away. This group is unlikely to fully represent the entire population of interest.

Suppose you’re a tourism company. If you insist on hitting your target in a week, you’ll miss out on people who were away on vacations that week. If you extend your field dates, you can reach them when they come back (and others before they leave) – reaching a higher-quality sample and getting more useful insights to act on.

The right fit

Not every high-quality research project has to be a major investment of time and money. There are lean tools that can be deployed quickly to answer targeted questions effectively. What matters is designing a research approach that’s capable of giving illuminating answers to the business questions at hand.

In research as in other areas, if the solution to a complex challenge seems too cheap to be true, it probably is. A higher-quality research initiative – with a sample that matches your target population, thoughtful questionnaire design, and rigorous analysis – will tend to demand not only more resources but more engagement from the client’s team. But investing in a thoughtful approach to every part of the process, from clarifying the question the research should answer to mapping out how exactly the organization will act on the insights, is likely to deliver a substantial return.

Related insights