How to improve data quality in research with Indigenous peoples

(And why quality is non-negotiable)

One of the questions we hear most often from clients is, “Which online research panel I can use to reach Indigenous peoples?”

Increasingly, organizations of all stripes are seeking to hear from Indigenous individuals. There are varied reasons for this, including desire to appeal to this growing market, to design relevant policies or programs, and/or to support reconciliation or DEI (diversity, equity and inclusion) efforts. While some organizations may know how to reach Indigenous leaders, subject matter experts, or representatives of the Indigenous business community, it can be more challenging to seek the attitudes and opinions of the broader Indigenous public in their respective communities.

The challenge is partly a matter of geography. Indigenous peoples comprise 5% of the Canadian population, and they’re by no means evenly distributed across the country. Per Statistics Canada census data, more than half live in cities; the rest live in smaller communities, on reserves, and elsewhere – including in remote communities.

But geography is not the only barrier to engaging Indigenous participants. Because of a long history of harmful or disrespectful research by non-Indigenous people, distrust of research is widespread and many are reluctant to participate.

There is also a notably small number of Indigenous people employed in public opinion and market research, as well as a limited number of Indigenous companies with experience providing guidance on quantitative research with Indigenous participants.

What does this mean for organizations who (rightly) want the voices of Indigenous participants to be reflected in their work?

Questioning online research panels

Although online panel research is a widely accepted methodology in Canada, it’s not safe to assume it is a way to gather high-quality data on the attitudes and opinions of Indigenous peoples. Because of the issues noted above – geographic distribution and research reluctance – panels often don’t include enough Indigenous participants to support large-scale quantitative research.

Moreover, some of the efforts researchers have made to engage Indigenous participants has likely distorted some samples through excessive incentives. We’ve observed survey responses from participants who are clearly not Indigenous, but who have developed the impression that they will be “rewarded” if they identify as such in the screening and demographic section of online surveys.

Panel fraud is a wider concern for the market research industry. But in this case it causes special harm, because it misrepresents the views of Indigenous peoples – exacerbating the very problem researchers are trying to overcome: that Indigenous voices are not well enough reflected across society. The risk of harm from poor data is so great – a direct threat to the aims of individual research projects and to reconciliation overall – that it outweighs possible benefit.

At a recent conference of the Canadian Research Insights Council, a professional standards body, a participant in a panel on data quality argued (we paraphrase) that if an online panel company promises they can get you 1,000 survey completions by Indigenous research participants, you shouldn’t believe them. Environics’ recent experience suggests panel companies are recognizing their limits when it comes to Indigenous participation and are revising their estimates downward to be more realistic.

Exploring better approaches

If you decide primary research with Indigenous people is truly necessary to your project (which is not always the case; we’ve written about that question here), you have a few options.

Use qualitative research instead. Since qualitative research requires fewer participants, it becomes possible to work with Indigenous service organizations to locate the right people for your study. Done properly, qualitative research is also a culturally appropriate approach that respects oral storytelling traditions. Participants share information through discussion and dialogue, instead of being required to check boxes on a scale that may or may not allow them to accurately express themselves. Perhaps most importantly, qualitative research gives researchers the opportunity to hear from Indigenous participants as people first, not as numbers or “completes.”
Consider mixing online research with telephone – or using only the latter. If your research really needs to be quantitative, the telephone might serve you better than the internet. Using the phone to find 5% of the population that’s not evenly distributed by geography has its challenges, but you can start by identifying areas of the country where Indigenous people are more likely to live and focusing dialing efforts accordingly. The cost of this methodology can be substantially higher, but it’s worth the investment to accurately reflect Indigenous views.
Use online research in the context of a survey of the broader population. If the Indigenous identity question is asked at the back of the survey, dishonest participants have less incentive to fraudulently identify as Indigenous. Keep in mind, however, that a survey designed for the general population doesn’t necessarily reflect the way Indigenous people look at the issue. Moreover, overly relying on the findings from a small sample set that treats Indigenous people as a monolith, and does not differentiate between First Nations, Inuit or Metis identity, on/off reserve, or urban, rural or remote residency, risks perpetuating stereotypes. So before moving to a full-scale quantitative survey, it’s wise to ensure that you have a sense of Indigenous perspective on the topic (and how it may vary across the population) by – again – starting with a qualitative phase.

We believe decisions about research approach need to focus on collecting data and stories that honestly reflect Indigenous perspectives and form an appropriate basis for business and policy decisions. By choosing unrealistic methodologies unsuited to the research goals, or underinvesting in data collection, we risk falling short on both counts. After Canada’s long history of silencing and ignoring Indigenous voices, researchers should reject methodologies that risk distorting those voices once again; better to acknowledge a gap in the data than to use a dubious approach and assume that it’s “close enough” to what Indigenous peoples really think.