Data Quality: How Important is It?


By Greg Heist, Chief Innovation Officer, Gongos, Inc. 

Over the past several years, online communities have developed into powerful platforms for engaging customers in extended conversations.  As more and more corporations embrace online communities, many market researchers are eager to pursue a more sophisticated set of research applications within them.

General Motors was among the first in the industry to take private online communities to this next level in order to substantiate the application of consumer insights. Broadening their scope, they needed online communities not only to act as a vehicle for interaction and observation, but also to carry statistical weight.

GM’s experience, and the experiences of other companies pursuing quantitative results, suggests that the industry still yearns for answers to significant questions about the quality of insights generated by online communities:

The answers to these questions are critical, since they point to the potential for online communities to represent a new research paradigm — one that combines the tools and statistical power of quantitative research with a highly interactive and engaging environment that fuels additional types of insights.

This article explores these crucial questions and examines how the implications of our findings will affect the future of online communities in marketing research.

Comparing Insights from Online Communities and Online Access Panels

For online communities to become the basis for a new paradigm in market research, it’s vital that they provide the same explanatory power and business insights as current approaches.  In contemporary market research, that standard has been set by online access panels.  While Internet panels are certainly not without their detractors, in a world where online access is widespread and surveys in other modes are increasingly difficult, online panels are the most relevant benchmark for quantitative research in any new research platform.

Because participants in online communities are recruited using methods similar to online access panels (or in some cases, recruited directly from them), it would be surprising to find that the two types of sample generate radically different results.  To substantiate this intuition, we investigated three studies that were conducted using parallel samples from online communities and from online access panels.

As suspected, we found a very high degree of similarity between online community results and our benchmark – online panel results.  The results from the side-by-side studies (unweighted studies of American adults) are as follows:

Further, it should be noted that the negligible differences between the samples would not have changed the nature of any business insights from the three studies.

This is not to say that participants in community surveys are in all ways identical to online panelists.  Not surprisingly, members of an automotive-related community tend to have greater interest in, and expertise about, automotive issues (see sidebar A).  Nevertheless, despite these differences, this analysis clearly shows that online communities and online panels provide equivalent business insights, and would produce the same business decisions.

Who Joins Online Communities?

Online communities and online panels share another characteristic – the potential for selection effects.  Membership in an online panel or an online community does not happen at random – participants have chosen to join, and have chosen to stay involved. Therefore, some characteristic differences from the general population are to be expected in any sample obtained online.

In a sample from Consumer Village – a Gongos Research-managed community with over 12,000 members – we found that respondents spend more time online per week (21 hours) than the average online American (10 hours).  Respondents from Consumer Village also engage in a more eclectic set of online activities – they are more likely to use online classifieds, buy in online auctions, and do their banking online.

Still, these observed differences should not be concerning for two reasons.  First, members of online communities and the general population engage in the same types of online activities, albeit to varying degrees.  However, unless these differences between online community members and the general population are correlated with responses to the questions of interest, they will not impact results.  In such cases, conclusions can be considered projectable to the general population.

Likewise, for topics where online behaviors are found to be relevant, such as exposure to information found online, or opinions related to online privacy, knowing the nature of the differences also provides the power to mitigate them.  It is possible to use national benchmarks for online behavior as a “safety net,” using weighting or sample stratification to balance results.  This type of adjustment would be in addition to any stratification, quotas, or weighting implemented to balance sample demographics to known benchmarks.

Data Quality in Online Communities

We also find evidence that online communities provide high-quality data.  In particular, Consumer Village has:

Other communities, which structure incentives to promote participation over the life of the community, can be expected to have even higher response rates, completion rates, and retention rates.  But even without this boost, data analysis can proceed without significant concerns about data quality.

Does the Community Experience Contribute to Data Quality?

There are some standard elements of online communities that facilitate data quality, like customization, visual appeal, and ease of use.  But we believe that the key drivers of data quality are the positive and diverse experiences available to participants in online communities.

To understand how these opportunities impact respondent motivation, we asked a sample of Consumer Village members to rate the importance of various statements about participation in online research.  Some of the usual suspects emerged:

Yet there was also broad interest in the types of experiences provided by online communities:

A more in-depth dialogue with members in Consumer Village echoed these results, and indicated the importance of community interaction in producing a positive online experience (see sidebar B).

To measure community participation, we looked at the frequency with which respondents posted messages in community forums, either in response to a moderator-sponsored activity or on their own initiative.  We found that for many members, their community experience tends to resemble that of an online panel, as they respond to survey invitations but decline to participate in discussion forums.  But significant numbers take advantage of the interactions that the community has to offer:  19% post on average at least once every two weeks, and another 10% post on average at least once a month.

Further, these community-oriented behaviors are strongly associated with increased participation in quantitative studies.

This finding makes sense on both practical and motivational grounds.  There is a direct effect of participation, as more frequent posters have greater exposure to surveys posted in the community, and therefore encounter more opportunities to participate.

But more importantly, there is an indirect effect of participation, based in the motivations of community members.  Those who are inspired to share their opinion in one way (via discussion forums) will also be likely to express that opinion in other ways (quantitative surveys).  And because communities tend to attract and retain those who are interested in expressing their opinions in an interactive community, those same individuals will respond at relatively high rates when invited to participate in quantitative studies.

High response rates would not be especially helpful if highly motivated respondents skewed a study’s results, because easier access to lower-quality data is not a winning combination.  But as we’ve seen from our sample comparisons, online communities generate the same conclusions as online access panels.  Online community members may express their opinions more frequently, more reliably, more avidly, and more vividly, but the opinions they express don’t differ significantly from their more reserved counterparts.

Implications for the Future

Our findings point to a number of exciting implications for the future.  We have shown that consumers participating in an online community provide high quality data for quantitative research, with results that are highly similar to equivalent results from online access panels.  Practitioners can be confident that an online research community can indeed generate the same statistical insights as the existing platform of the online access panel.

Yet online communities are far more than a technologically-driven reinvention of the market-research wheel.  The power of online communities lies in their ability to usher in a new type of relationship with consumers:

It is clear from our analysis that online communities represent a powerful new paradigm in the field of marketing research.  Online communities deliver statistical results that are equivalent to traditional online access panels, while simultaneously creating a rich new realm for interacting with, and learning from, consumers in relevant and dynamic ways.

It’s not surprising that over 90% of respondents in a recent automotive study agreed that “the cost of gasoline is rising at an alarming rate.”  Similarly large majorities concurred that “we rely too much on foreign countries for our oil/petroleum needs,” and that “I am concerned with the current cost of fuel for my vehicle,” while less than 10% of respondents reported that “I do not think fuel-efficient vehicles are important.”

These were among 42 statements about fuel economy, environmental issues, and vehicle styling that were recently evaluated using two different samples:  an automotive online community and an online access panel.  Importance rankings were highly consistent across the two samples: the top eight statements in each sample were identical, and differed by no more than one place.  Overall, half of the 42 statements were ranked identically or within one place, and on average, each statement’s ranking differed by only 2.5 places between the two samples.

When asked about purchase consideration for a plug-in hybrid vehicle on a 10-point scale, the average response differed by no more than 0.25 points between the two samples.  And when asked about purchase consideration with respect to specific manufacturers, company rankings were identical in the two samples.

Community members are more likely to describe themselves as people who “like introducing new technologies to [their] friends” (42% vs. 27%) and whose friends “think of [them] as a good source of information when it comes to vehicles” (45% vs. 29%).  But, as the strong similarities between samples suggests, these knowledgeable community members are equally helpful at providing useful insight into the vehicle-related attitudes and perceptions of the general public.

In talking to members of Consumer Village about membership in a research community, we found that they cite a number of elements of the experience that make it distinctive and enjoyable, including:

Truly voicing their opinions.  Community members feel as though they are able to express themselves beyond answering multiple choice questions in a quantitative survey.  This provides some community members with a sense of being able to truly “speak their mind” in a much more powerful way.

Seeing what others have to say.  Community members appreciate the opportunity to see how others feel about a given topic. This provides community members with a unique opportunity to learn from hundreds—or thousands—of other consumers on a variety of topics.

Chatting with other community members. One of the unique aspects of a research community is that it provides members with a platform to create discussion topics – or even their own surveys – and interact with others who have similar interests.  With many consumers discussing numerous topics at any one time, a community member can easily find a topic of interest.

A sense of “community” and personal connection.  Members feel a more personal connection to the moderator and other community members than in a typical panel setting.  Some consumers find real value in the sense of “belonging” to a community of like-minded individuals, and it keeps them returning to the community time after time.

Intrinsic rewards.  Communities provide the opportunity for rewards beyond traditional monetary.  Spotlighting outstanding community member contributions, creation of community-generated charitable contributions, getting feedback about the community population and the like all create a set of intangible benefits that have real value to some members.

Feeling “listened to.”  The more “personal” feel of a community extends to a sense that someone is truly listening to the community—and doing something specific with what they’re learning from community members.  In this respect, having heavily engaged moderators and clients who are willing to share and give back to the community generates a powerful dynamic that is unique to the community environment.

Project Driveway, a multi-year online community for the GM Fuel Cell Program, represents an innovative online community model that blends market research, viral marketing, public relations and in-field product testing.  Representing a joint effort between GM’s Fuel Cell engineering team, Global Product Research and Corporate Marketing, members of Project Driveway are selected based upon their geographical location (living close to existing hydrogen refueling stations), various demographic and attitudinal factors and their interest and passion for “green” vehicle technologies.

Select members of Project Driveway are given the opportunity to be among the first consumers in the world to test drive GM’s fleet of hydrogen fuel cell vehicles for two or three months.  Drivers participate in press and media events and provide ongoing detailed feedback to the community about their driving experience.

By integrating all of these disciplines into one community, Project Driveway highlights ways in which online communities can become even more valuable for meeting the needs of multiple disciplines within an organization.

As published in Alert Magazine.