Data Quality: How Important is It?
By Greg Heist, Chief Innovation Officer, Gongos, Inc.
Over the past several years, online communities have developed into powerful platforms for engaging customers in extended conversations. As more and more corporations embrace online communities, many market researchers are eager to pursue a more sophisticated set of research applications within them.
General Motors was among the first in the industry to take private online communities to this next level in order to substantiate the application of consumer insights. Broadening their scope, they needed online communities not only to act as a vehicle for interaction and observation, but also to carry statistical weight.
GM’s experience, and the experiences of other companies pursuing quantitative results, suggests that the industry still yearns for answers to significant questions about the quality of insights generated by online communities:
- Are business insights from online communities similar to those from online access panels?
- What types of consumers join online communities?
- What level of data quality do online communities provide?
- How does the online community experience contribute to data quality?
The answers to these questions are critical, since they point to the potential for online communities to represent a new research paradigm — one that combines the tools and statistical power of quantitative research with a highly interactive and engaging environment that fuels additional types of insights.
This article explores these crucial questions and examines how the implications of our findings will affect the future of online communities in marketing research.
Comparing Insights from Online Communities and Online Access Panels
For online communities to become the basis for a new paradigm in market research, it’s vital that they provide the same explanatory power and business insights as current approaches. In contemporary market research, that standard has been set by online access panels. While Internet panels are certainly not without their detractors, in a world where online access is widespread and surveys in other modes are increasingly difficult, online panels are the most relevant benchmark for quantitative research in any new research platform.
Because participants in online communities are recruited using methods similar to online access panels (or in some cases, recruited directly from them), it would be surprising to find that the two types of sample generate radically different results. To substantiate this intuition, we investigated three studies that were conducted using parallel samples from online communities and from online access panels.
As suspected, we found a very high degree of similarity between online community results and our benchmark – online panel results. The results from the side-by-side studies (unweighted studies of American adults) are as follows:
- Greeting Cards – In a study of seven holiday greeting concepts, a measurement of purchase intent using a constant-sum allocation task yielded identical rankings of the concepts in both samples. Across each, average interest for the concepts differed by no more than 2.5 points on a 100-point scale.
- Pet Care – In an importance rating of 61 evaluative statements concerning shopping for pet care items, in both samples the top three statements were in the same order, and the top 10 statements contained eight items in common. Over the entire list, rankings differed by an average of only two spots. A ranking of satisfaction with pet-care retailers was identical (considering the six retailers with at least 100 responses from each sample).
- Electric Vehicles – A study of electric vehicle concepts yielded very similar results about attitudes, concept evaluation, and manufacturer rankings (see sidebar A).
Further, it should be noted that the negligible differences between the samples would not have changed the nature of any business insights from the three studies.
This is not to say that participants in community surveys are in all ways identical to online panelists. Not surprisingly, members of an automotive-related community tend to have greater interest in, and expertise about, automotive issues (see sidebar A). Nevertheless, despite these differences, this analysis clearly shows that online communities and online panels provide equivalent business insights, and would produce the same business decisions.
Who Joins Online Communities?
Online communities and online panels share another characteristic – the potential for selection effects. Membership in an online panel or an online community does not happen at random – participants have chosen to join, and have chosen to stay involved. Therefore, some characteristic differences from the general population are to be expected in any sample obtained online.
In a sample from Consumer Village – a Gongos Research-managed community with over 12,000 members – we found that respondents spend more time online per week (21 hours) than the average online American (10 hours). Respondents from Consumer Village also engage in a more eclectic set of online activities – they are more likely to use online classifieds, buy in online auctions, and do their banking online.
Still, these observed differences should not be concerning for two reasons. First, members of online communities and the general population engage in the same types of online activities, albeit to varying degrees. However, unless these differences between online community members and the general population are correlated with responses to the questions of interest, they will not impact results. In such cases, conclusions can be considered projectable to the general population.
Likewise, for topics where online behaviors are found to be relevant, such as exposure to information found online, or opinions related to online privacy, knowing the nature of the differences also provides the power to mitigate them. It is possible to use national benchmarks for online behavior as a “safety net,” using weighting or sample stratification to balance results. This type of adjustment would be in addition to any stratification, quotas, or weighting implemented to balance sample demographics to known benchmarks.
Data Quality in Online Communities
We also find evidence that online communities provide high-quality data. In particular, Consumer Village has:
- High response rates. Over the past 12 months, the average response rate has been 33%, with 90% of studies having response rates between 22% and 41%.
- High respondent attentiveness. Inattentiveness varies with study type and length, but respondents from online communities are consistently more attentive than industry standards, as measured by “straight lining” and consistency with data quality traps.
- High completion rates. On average, of those who qualify for a study, 92% complete it. If we include those who are terminated because they fail to qualify, the completion rate rises to 97%.
- High Retention. Consumer Village retains an average of 88% of its members per quarter (where attrition is defined as non-participation in a six-month period).
- High Trust. At least 95% of Consumer Village respondents, and sometimes more, are willing to provide information about their household’s income.
Other communities, which structure incentives to promote participation over the life of the community, can be expected to have even higher response rates, completion rates, and retention rates. But even without this boost, data analysis can proceed without significant concerns about data quality.
Does the Community Experience Contribute to Data Quality?
There are some standard elements of online communities that facilitate data quality, like customization, visual appeal, and ease of use. But we believe that the key drivers of data quality are the positive and diverse experiences available to participants in online communities.
To understand how these opportunities impact respondent motivation, we asked a sample of Consumer Village members to rate the importance of various statements about participation in online research. Some of the usual suspects emerged:
- Nearly 90% considered “earning incentives or rewards for participation” to be important.
- More than three-fourths identified “influencing the decisions that companies make about products,” and “expressing my opinion” as important.
Yet there was also broad interest in the types of experiences provided by online communities:
- 47% considered it important to “interact with others about topics that interest me.”
- 39% considered it important to “belong to a community.”
A more in-depth dialogue with members in Consumer Village echoed these results, and indicated the importance of community interaction in producing a positive online experience (see sidebar B).
To measure community participation, we looked at the frequency with which respondents posted messages in community forums, either in response to a moderator-sponsored activity or on their own initiative. We found that for many members, their community experience tends to resemble that of an online panel, as they respond to survey invitations but decline to participate in discussion forums. But significant numbers take advantage of the interactions that the community has to offer: 19% post on average at least once every two weeks, and another 10% post on average at least once a month.
Further, these community-oriented behaviors are strongly associated with increased participation in quantitative studies.
- Among those who post on average once a month, participation in quantitative surveys is 29 percentage points higher than for the average member of Consumer Village.
- For the most frequent posters, those who engage in discussions on average at least once every two weeks, the increment is 36 percentage points.
This finding makes sense on both practical and motivational grounds. There is a direct effect of participation, as more frequent posters have greater exposure to surveys posted in the community, and therefore encounter more opportunities to participate.
But more importantly, there is an indirect effect of participation, based in the motivations of community members. Those who are inspired to share their opinion in one way (via discussion forums) will also be likely to express that opinion in other ways (quantitative surveys). And because communities tend to attract and retain those who are interested in expressing their opinions in an interactive community, those same individuals will respond at relatively high rates when invited to participate in quantitative studies.
High response rates would not be especially helpful if highly motivated respondents skewed a study’s results, because easier access to lower-quality data is not a winning combination. But as we’ve seen from our sample comparisons, online communities generate the same conclusions as online access panels. Online community members may express their opinions more frequently, more reliably, more avidly, and more vividly, but the opinions they express don’t differ significantly from their more reserved counterparts.
Implications for the Future
Our findings point to a number of exciting implications for the future. We have shown that consumers participating in an online community provide high quality data for quantitative research, with results that are highly similar to equivalent results from online access panels. Practitioners can be confident that an online research community can indeed generate the same statistical insights as the existing platform of the online access panel.
Yet online communities are far more than a technologically-driven reinvention of the market-research wheel. The power of online communities lies in their ability to usher in a new type of relationship with consumers:
- The highly interactive and collaborative community environment creates a different dynamic for marketing research. Beyond a linear transaction of data, communities offer a wide range of ways for consumers to share opinions, exchange ideas with others and learn about how others feel about various topics.
- The ability to post topics of their own enables community members to connect with other consumers, providing researchers with the ability to observe consumer behavior on a larger scale. By mining consumer-generated content for insights, researchers can identify innovation opportunities and gain fresh perspectives.
- The community model paves the way for corporations to conduct high-quality research while simultaneously reducing costs and increasing the speed with which decisions can be made using consumer input.
- Since communities generate real-time longitudinal insights that can be continuously integrated into the decision-making process, it can truly be said that communities “bring consumers to the executive table” within an organization.
- The informal and interactive nature of online communities provides an opportunity to integrate marketing and branding activities within the context of a true market research environment (see sidebar C). This emerging model of consumer engagement has tremendous potential to integrate and synchronize the marketing, branding and research functions of corporations in ways never considered possible.
It is clear from our analysis that online communities represent a powerful new paradigm in the field of marketing research. Online communities deliver statistical results that are equivalent to traditional online access panels, while simultaneously creating a rich new realm for interacting with, and learning from, consumers in relevant and dynamic ways.
It’s not surprising that over 90% of respondents in a recent automotive study agreed that “the cost of gasoline is rising at an alarming rate.” Similarly large majorities concurred that “we rely too much on foreign countries for our oil/petroleum needs,” and that “I am concerned with the current cost of fuel for my vehicle,” while less than 10% of respondents reported that “I do not think fuel-efficient vehicles are important.”
These were among 42 statements about fuel economy, environmental issues, and vehicle styling that were recently evaluated using two different samples: an automotive online community and an online access panel. Importance rankings were highly consistent across the two samples: the top eight statements in each sample were identical, and differed by no more than one place. Overall, half of the 42 statements were ranked identically or within one place, and on average, each statement’s ranking differed by only 2.5 places between the two samples.
When asked about purchase consideration for a plug-in hybrid vehicle on a 10-point scale, the average response differed by no more than 0.25 points between the two samples. And when asked about purchase consideration with respect to specific manufacturers, company rankings were identical in the two samples.
Community members are more likely to describe themselves as people who “like introducing new technologies to [their] friends” (42% vs. 27%) and whose friends “think of [them] as a good source of information when it comes to vehicles” (45% vs. 29%). But, as the strong similarities between samples suggests, these knowledgeable community members are equally helpful at providing useful insight into the vehicle-related attitudes and perceptions of the general public.
In talking to members of Consumer Village about membership in a research community, we found that they cite a number of elements of the experience that make it distinctive and enjoyable, including:
Truly voicing their opinions. Community members feel as though they are able to express themselves beyond answering multiple choice questions in a quantitative survey. This provides some community members with a sense of being able to truly “speak their mind” in a much more powerful way.
Seeing what others have to say. Community members appreciate the opportunity to see how others feel about a given topic. This provides community members with a unique opportunity to learn from hundreds—or thousands—of other consumers on a variety of topics.
Chatting with other community members. One of the unique aspects of a research community is that it provides members with a platform to create discussion topics – or even their own surveys – and interact with others who have similar interests. With many consumers discussing numerous topics at any one time, a community member can easily find a topic of interest.
A sense of “community” and personal connection. Members feel a more personal connection to the moderator and other community members than in a typical panel setting. Some consumers find real value in the sense of “belonging” to a community of like-minded individuals, and it keeps them returning to the community time after time.
Intrinsic rewards. Communities provide the opportunity for rewards beyond traditional monetary. Spotlighting outstanding community member contributions, creation of community-generated charitable contributions, getting feedback about the community population and the like all create a set of intangible benefits that have real value to some members.
Feeling “listened to.” The more “personal” feel of a community extends to a sense that someone is truly listening to the community—and doing something specific with what they’re learning from community members. In this respect, having heavily engaged moderators and clients who are willing to share and give back to the community generates a powerful dynamic that is unique to the community environment.
Project Driveway, a multi-year online community for the GM Fuel Cell Program, represents an innovative online community model that blends market research, viral marketing, public relations and in-field product testing. Representing a joint effort between GM’s Fuel Cell engineering team, Global Product Research and Corporate Marketing, members of Project Driveway are selected based upon their geographical location (living close to existing hydrogen refueling stations), various demographic and attitudinal factors and their interest and passion for “green” vehicle technologies.
Select members of Project Driveway are given the opportunity to be among the first consumers in the world to test drive GM’s fleet of hydrogen fuel cell vehicles for two or three months. Drivers participate in press and media events and provide ongoing detailed feedback to the community about their driving experience.
By integrating all of these disciplines into one community, Project Driveway highlights ways in which online communities can become even more valuable for meeting the needs of multiple disciplines within an organization.
As published in Alert Magazine.