A few months ago, in a paper that eventually appeared at WSDM 2012, John, Michael and I (Giorgos) presented a finding that attracted a fair bit of attention: using data we collected from Groupon and Yelp we observed that subsequent to Groupon daily deals Yelp merchant ratings, on average, declined. Our finding can be pretty much summarized in one figure.
The figure displays the running-average Yelp rating (reset on day zero) of about five thousand merchants who ran Groupon deals centered around their deal offer dates. You can immediately observe a sharp discontinuity at offset zero coinciding with the day the deals were offered. Our original plot, limited by the data available to us at the time, only went out to offset 180. With more data becoming available we plotted Yelp ratings further out in the future. A second dip around offset 180 initially puzzled us. Looking deeper into it we realized that a large fraction of the deals we were studying had a six-month expiry date suggesting consumers were rushing to use their purchased coupons in the last minute.
Naturally, these observations raised more questions than they could answer. Why were ratings declining? Was it truly something to do with Groupon? Both online (e.g., see Rocky Agrawal's blog post), and at various places where I presented our work people came up with various legitimate hypotheses that could explain the decline, some of them unrelated to Groupon.
Here's the list I compiled:
1. Intrinsic decline: Prior work (e.g., Godes and Silva) shows that review scores fall over time.
2. Critical reviewers: Groupon users are more critical than their peers.
3. Bad, unprepared, or discriminatory businesses: Merchants who feel compelled to offer a Groupon are in trouble, unprepared to handle the influx of Groupon customers, or discriminatory against Groupon bearing customers.
4. Experimentation: Groupon users are trying new things out when using coupons and hence are more prone to disappointment.
To the above we added a hypothesis of our own:
5. Artificial reviews: Reviews mentioning Groupon are less likely to be fake, hence, the observed decline is due, at least in part, to a previously inflated Yelp rating.
In a new paper to be presented at EC 2012, we consider the above hypotheses by building empirical models for our data. We find varying degrees of support for all of them except, interestingly, for the second one: Groupon users are not by nature that much more critical than their peers -- that is, unless they mention Groupon in their reviews. In fact, they are more moderate, providing fewer 1- and 5-star ratings. Furthermore, we find that they are more active on Yelp, they write better and longer reviews that are less likely to be filtered, and they have more Yelp friends. This suggests to us that, if anything, merchants running daily deal offers should be treating their Groupon customers with special care.
Even though the focus of this work is fairly specific, I think it has broader implications. First, it demonstrates the side-effects marketing efforts can have when considered as part of the greater online ecosystem; and, second, that in fact, these effects can be accurately measured and accounted for. Groupon, in a recent NPR program discussing our work, commented: "It's been documented that anytime an influx of customers visits a business their online reviews tend to see a decrease in quality. This effect holds true for online as well as traditional advertising such as print, broadcast, radio." I find their comment particularly insightful. The greater question to be answered now is not the isolated effectiveness of Groupon (and other daily deal providers) as a marketing mechanism but how it fares in comparison to other customer acquisition methods. I hope that our techniques can be applied in answering this question.
Thank you to everyone who's come to my talks, asked questions, and shared your thoughts on the subject. Hopefully, our new paper demonstrates that I was listening!
No comments:
Post a Comment