Reviews are vital to the success of a product or service. Good reviews encourage people to buy. Bad reviews discourage them from buying.
In the WordPress world, reviews can make or break a developer’s product. Review ratings are prominently displayed next to themes and plugins on WordPress.org. They give website owners good feedback on the performance of the theme or plugin in question from other fellow WordPress users.
Even products with high scores have negative reviews on WordPress.org.
What happens when developers give an incentive to users to leave a positive review?
Should they be punished for trying to give users a reason for leaving a review?
This was something that we found out this week when Dan Cameron from Sprout Apps was punished by WordPress.org for trying to “defraud their ratings system”.
In his blog post “What Now? No Way? Huh?” Dan explained his surprise and frustration about the reaction from WordPress.org and why he was unhappy about them removing reviews of his WordPress plugin Sprout Invoices.
Sprout Invoices Encouraging Reviews Through Discounts
During my recent trip to Vienna for WordCamp Europe I was lucky enough to meet BlogVault owner Akshat Choudhary. He stressed that I should join the Advanced WordPress group on Facebook.
Tens of thousands of WordPress users and developers participate in the group every day. It was in this Facebook group that someone posted about Dan’s situation with Sprout Invoices.
The response in the group to what happened was split. Some people said that Dan was treated very unfairly. Many others, including myself, feel that the response by WordPress.org was just and fair.
To get a full overview of what happened, please read Dan’s blog post “What Now? No Way? Huh?“.
For those who are short on time, here is a brief overview of what happened.
- The plugin Sprout Invoices included an offer that gave users who published a review a discount on upgrading
- WordPress.org discovered this and stated that “bribery” was not allowed and refer the plugin author Dan Cameron to the detailed plugin guidelines
- In response to incentivising reviews, WordPress.org removed all five star reviews of Sprout Invoices on WordPress.org since they were effectively spam
- Dan Cameron states that this response is highly unfair and says that there is nothing in the plugin guidelines that states that this is now allowed
Dan comes across as a nice guy who is just trying to work hard and promote his product, however his surprise reaction to all of this highlights how his view on this situation is clearly biased.
As someone who does not have a dog in the fight, I feel differently to Dan about WordPress.org’s response.
Why I Feel Those Who Incentivise Reviews Should Be Punished
Dan states in his blog post that what he did was not bribery and that he was simply compensating users for the time they took to publish reviews.
It is a curious way to look at the situation.
By that same rational I should be allowed to compensate website owners who link to this blog as it takes them time to link here. As I am sure you are aware, I am not allowed to do that.
If I had to ask people to link to this blog in exchange for $10, Google would apply a penalty on this blog and I would lose all my search engine traffic. Bloggers who ran competitions to see who could generate the most incoming links rightfully got penalties by Google for doing so.
When you take a step back and look at the situation, you understand that “Compensating users for their time” is just another way of saying “I will pay you for publishing a review”. Call it compensation, call it bribery, call it encouragement, call it marketing. You can call it anything you want but any way you look at it you are manipulating the ratings system in order to push your own product ahead of the competition.
Sprout Invoices gave a discount to those who published reviews.
Dan argues that since no specific rule against was stated in the guidelines that he broke no rules. I think that this is a poor argument as WordPress.org cannot be expected to state a rule for every situation and eventuality that may occur.
The rules do in fact state that it is not permitted. Guideline number 9 states:
The plugin must not do anything illegal, or be morally offensive. That’s subjective, we know. Still, if we don’t like it for any reason, it’s gone. This includes spam, for whatever definition of spam we want to use. This includes (but is not restricted to) the following examples:
- Keyword stuffing or SEO scamming in the readme.
- Compensating or blackmailing users for reviews.
- Creating sockpuppet accounts to generate fake reviews.
- Taking other developers’ plugins and presenting it as original work.
This guideline may have been added after the discount offer was inserted into the Sprout Invoices WordPress plugin, but I believe that all WordPress plugin and theme developers should be expected to have some common sense and realise that incentivising reviews is not allowed. You would have to be deluded to not realise this gives you an unfair advantage in the marketplace.
In order to get a discount on the premium version of Sprout Invoices, users had to first publish the review and then send Dan Cameron the link.
This setup guarantees five star reviews as users who did not like the plugin would not be encouraged to leave a review and those who did would only publish five star reviews to ensure they got a discount.
So when Dan says he wanted to compensate users for their time, I think what he really means is that he wanted to compensate users who published five star reviews. There was absolutely no incentive to leave a review of his plugin if you were not going to publish an overwhelmingly positive review.
If Dan truly wanted to compensate users for their time then he would have displayed a box in the admin area of his plugin that said “Please review our plugin for $25. Negative reviews will be paid too” (Note: I’m obviously not expecting any developer to be crazy enough to pay for negative reviews!).
Sprout Invoices had all previous five star reviews removed.
Was WordPress right to remove all five star ratings?
The response from WordPress may seem a little harsh as Sprout Invoices had undoubtedly a lot of genuine five star reviews too; though I do not see what else WordPress could have done as Sprout Invoices had been incentivising reviews for years. It is impossible to know which reviews were genuine and which were not.
In many ways WordPress were forced into a corner about this. If they had only removed 50% of the reviews then many of the remaining reviews would have been fake. This would have showed to others that you can manipulate the ratings system and still benefit from doing so even if you get caught.
Manipulating the ratings system is something that simply cannot be allowed.
As soon as any website with a ratings system suffers from spam submissions and ratings that were not genuine, the ratings system becomes ineffective and ultimately pointless. Ratings are useful for WordPress users but if they are to remain on WordPress.org there has to be rules put in place to ensure that the vast majority of reviews and ratings are real (I say “vast majority” because no reviews system is impregnable from fake reviews).
What Dan Cameron did was wrong.
He manipulated the ratings system in order to increase reviews and ratings. This leads to more traffic and more downloads. This in turn leads to more people upgrading his product and generating more money for him.
The problem with him doing this is that when he manipulated the ratings system he did so at the cost of WordPress users and of fellow WordPress plugin developers. He gained an unfair advantage over competitors by incentivising reviews and that cannot be allowed. He also mislead WordPress users by artificially inflating the ranking of his product.
I have no doubt that other WordPress plugin and theme developers have used a similar promotions to encourage reviews. If Sprout Invoices is being singled out and others are not, that is unfair; however I still believe that WordPress’s response was right as they cannot allow reviews to be manipulated in any shape or form.
I take absolutely no pleasure in saying that as I know that this decision has hurt Dan’s business. Moving forward, I hope he can develop his product further and gain positive reviews in an organic way.
What Say You?
The response from the Advanced WordPress Facebook group highlighted how divided the WordPress community is about this topic.
How do you feel about it?
Do you feel that those who incentivise reviews should be penalised?
I would love to hear your thoughts on this issue so please let me know in the comments area below.
Thanks for reading.
Kevin
I think Sprout Invoices was treated unfairly. Looking at the guidelines on the WordPress.org site, #9 only states this:
“The plugin must not do anything illegal, or be morally offensive. That’s subjective, we know. Still, if we don’t like it for any reason, it’s gone. This includes spam, for whatever definition of spam we want to use.”
https://wordpress.org/plugins/about/guidelines/
If you think about it, affiliate links are nothing more than paying people to link to your site. You are compensating them for driving traffic to your site, there is no need for them to actually use/enjoy your product/services, they get paid as long as people click on the link they provide.
S.I. never requested the review be favorable nor did they even validate that you had indeed left the review. You asked, they gave the discount.
My opinion would have been to first let him know that this type of behavior is not something they want to see happen on their reviews. Secondly, as per his suggestion, was to remove the reviews of those whom he offered the discount to. Or have them add to the review that it was a compensated review.
At the end of the day, for all of the vaunted “open source flower power attitudes” – the tail end of their statement says it best “for whatever definition of spam we want to use”. WP is not a democracy, it’s run and controlled by people and they do as they see fit.
Hi Bill.
I appreciate you taking the time to comment.
I think this is something we have a completely different view on. It was the same in the Advanced WordPress Facebook group.
When a theme company pays an affiliate for generating a sale, they are not hurting the integrity of the ratings system. The affiliation occurs between the the company and the affiliate. No one else is involved (well, except the customer).
I insert affiliate links in my blog posts, but whether I do or not makes no difference to my ranking in the search engines or those who compete for the same keywords and key phrases.
In Dan’s the problem is that the ratings system was manipulated. Would it be fair for a WordPress theme company to pay their customers $25 to leave a positive review of their themes on WordPress.org? I do not think it would be.
If a plugin asks you to publish the review and then email with verification of that review, then you are asking for the review to be favourable. No one is going to leave a one star review and then email the developer and ask for a discount. If Dan was open about the whole setup I am sure we would find that 100% of all the reviews generated through this scheme were five star reviews.
The problem is that WordPress has no way of determining which reviews were genuine and which were not.
I do agree with you that WordPress is not a democracy. Those in control have the power to do what they want and we have seen many times that they are willing to abuse that power (a quick look at the number of Automattic plugins on the high traffic featured plugins page confirms this).
I still feel that is very unfair on other plugin developers to incentivise reviews. Reviews have to be genuine.
Put yourself on the other side of the fence.
How would you feel if you found that your main competitor had generated tens of thousands of additional downloads because they were paying people for positive reviews? This move could have cost your own business hundreds of thousands of dollars as a result of lower rankings, less downloads, and less upgrades to the premium version of your plugin.
Dan’s decision to do this directly increased his income by allowing his plugin to get higher rankings on WordPress.org etc. He is not innocent in this but perhaps he was naive to think that nothing would happen if he got caught.
🙂
Kevin
Kevin, how can you possibly justify your stance on this when you explicitly encourage blog owners to charge for reviews (http://www.kevinmuldoon.com/blogs-should-charge-reviews/)?
From what I can see, your assumption is that the developer in question is acting unethically without any knowledge of that fact. Do you know for a fact that they wouldn’t have actually followed through with a discount for a 4, 3, 2, or 1 star review or are you just assuming that?
If you don’t know that, then how is what they did any different from what you do with your blog reviews, and what you have recommended that others do?
Secondly, how can you then justify using guest posting to get links, which is effectively the same as paying a blog to link to you (because it lowers the amount they spend on content). Per your services page, a 1,000 word guest blog post would cost $500, which certainly seems like you are paying someone for a link, even if not with currency.
Most of your comment seems to be about comparing my position to completely unrelated issues. This situation has nothing to do with paid blog reviews or guest posting.
With regards to the plugin ratings issue. Do I know for a fact they hadn’t published negative reviews? The fact that the plugin has no 0, 1, 2, 3 or 4 star reviews, indicates how no one was willing to leave a negative review.
The other points you raised have nothing to do with this, but OK I’ll bite.
I do believe that blog owners should charge for reviews. I do not expect bloggers to spend four hours writing a review in exchange for a $5 commission when the minimum payment threshold is $100. That’s insane. I would be bankrupt in a matter of months if I did that.
For the record, I do not justify guest posting. In fact, there are many articles by me telling bloggers not to waste their time guest blogging as they should be paid for their work. The only real reason to guest post is to raise your profile and bring yourself to the attention of a new audience.
Many other companies continue to use guest posts in their marketing strategies. That’s their call, not mine.
You are saying that my services page states that I charge $500 for a guest post. That is incredibly misleading.
I do not guest post myself. I have made that clear.
If you actually took the time to read that page you would see that I charge that high rate for companies that do not want to display my bio on their website with me as the author. It is clearly marked as ghost writing for other companies. For example, if a WordPress backup company wanted me to submit a guest post to a WordPress blog but they wanted their own name as the author, I would charge the higher rate so that their company would have their brand exposed, not mine.
It should also be made clear that guest posting does not mean the article is going to be full of links. In fact, when I did guest post several years ago, my articles rarely linked to my own websites. The only link to my blog was in the bio area and that was frequently linked using the dofollow attribute.
For the record, I never charge that amount for writing. The whole point that high rate is on my services page is to stress the fact that I want my bio displayed on the articles I publish online.
I really don’t see need for any debate here. Those who think Dan’s penalty was unfair should have a rethink.
What we need is the understanding of the review system on WordPress. It’s actually for users to genuinely state their level of satisfaction after having an experience with the product.
For people to cast a vote before having to buy the product is fully MISLEADING. Giving 5 star to a product you haven’t completely tested can never be justified.
If WordPress opens up for such an action, trust me, it’s soon going to be bank of garbage where products of very low value will get thousands of 5 star reviews.
The only factor that should create a review must be your level of satisfaction. No incentives!
Just to clarify Enstine, those voting had fully tested the WordPress plugin. They were reviewing the free version in order to get a discount on the pro version. 🙂
I never read the WordPress.org review guidelines, but if you asked me if compensating someone for a review was allowed, I would say “No, that is not allowed. That is scummy behavior.”
I think the word scummy is a bit too much. All the developer was doing was trying to promote their product. Their intentions were perhaps selfish when you consider the marketplace as a whole, but I do not believe his intentions were bad. 🙂
Yes, that was a poor word choice. My point was that it seemed obvious to me that incentivizing reviews would not be allowed. I image that if a survey was taken before this happened and people were asked if it was allowed, that a large majority would say it was not. Perhaps that is an assumption on my part.
Yes I agree. It is difficult to feel too much sympathy for someone when they were using such tactics to get an edge.
As a longtime designer of sites using WordPress, but admittedly not a technical person, I rely heavily on plugins based on both the reviews and the support response. I read the support first because if there are numerous posts with no response or the responses are very old I stay away from the plugin. If there are great reviews but the support shows a lot of issues then I am suspicious about the reviews.
The point with plugins is their compatibility with themes and other plugins. I suspect that many website designers are as fed up as I am when the theme developer blames the plugins and the plugin developers blame the theme. How does that solve anything? WordPress.org would do better to delete any plugin that has not been updated for a year or more, or on which support is lacking. It would be even better if they had some sort of certification process to validate plugins.
Good advice there Gary. I do the same. When I see lots of threads about errors and no one is responding, I quickly see that the plugin is not supported well.
Plugin conflicts is a difficult thing.
Some developers always want to blame others for problems being caused by their own plugins and themes. However, this does occur sometimes because many users go directly to the wrong people. I was at WordCamp recently and spoke to the guys at CommerceGurus who advise that many customers frequently come to them with issues which are being caused by plugins they have no control over. It is a big drain on their time.
I do not believe older plugins should be deleted if they have not been updated in a year. There are a lot of basic plugins that add small important functionality to a website and they do not need constantly updated. Removing them would hurt the community. 🙂
Totally agreed with : @Enstine Muki
The users must drive by their satisfaction or dissatisfaction to review any product.
Kevin,
The point of my post is to say that everything has value, and in this case you are arguing that someone cannot offer something of value to a customer (a free/discounted upgrade) while getting something of value in return (a review).
Your evidence of a plugin not having lower star reviews might seem strange, but it doesn’t mean that people are afraid to publish a review that isn’t five stars. Maybe the plugin is really good (admittedly there should be a diverse range of reviews on any plugin though)? If you hated the plugin, you would leave a low review and stop using the plugin, so you’re not eliminating negative or lower reviews by utilizing this strategy.
The main point I was trying to make is that you have no proof that he is actually not going to give you an upgrade if you left a four or three star review (or even a one star). Your argument would be FAR more compelling if you had actually LEFT a three star review and were denied the upgrade.
The rest of my post, which you deride as meaningless and irrelevant, is simply to point out the fact that you often write about providing something of value to someone else in exchange for another item of value coming back to you. In a paid review, you are getting money in exchange for writing a post (writing content which you value at the amount I calculated from your services page). You are, in effect, participating in the same value exchange that you condemn so quickly in this post.
If I were so bold, I could argue that being paid to write a review post incentivizes you to post a positive review, similarly to how you point out that customers are incentivized to write a good review in exchange for an upgrade.
Would you say that you engage in unethical behavior by taking money in exchange for writing a paid review post that is overly positive? If that isn’t the case (and I don’t believe that it is), how can you assume that a customer will set aside their ethics in order to leave an unethical review?
I appreciate the thoughtful response to my comment, and really enjoyed hearing your point of view.
Hi Henry.
Thanks for dropping by and commenting again.
I realise that this strategy did not eliminate poor reviews, but it did generate many five star reviews. It may not have had any reviews at all if the developer didn’t use this strategy. That doesn’t mean the plugin is not good. It’s simply a reflection of the fact that very small percentage of users leave reviews on WordPress.org.
Someone who didn’t like the plugin is unlikely to upgrade. They are more likely to just try another plugin and not leave any plugin. I am sure that those who upgraded to the premium version of the plugin would have left five stars if they were to leave a review. The point is that they wouldn’t have left a review if there was not an incentive to do so. This kind of incentivisation leads to skewed results in the marketplace.
I disagree with your analogy about my writing services as paying for a review does not guarantee a positive review. In fact, I have had some very unhappy customers who have paid me for a review and then threatened to take me to court because I wrote a negative review and refused to remove it from my blog . Check out this review as an example of this.
As I have already said, paying for a review does not guarantee a positive review and I have made a lot of people unhappy by pointing out that their service needs improved etc and doesn’t offer value for money.
I will say, however, that many bloggers will happily give positive reviews to any product or service if it pays a big enough commission. I’m not like that. If I was, I’d be making a lot more money, however I would never do that. Firstly, because my reputation would be hurt, and secondly, because I want to be true to my readers and give them an honest review of everything I cover on this blog. The irony of paid reviews is that bloggers have only had to turn to generating income in this way because of so many people using ad blockers. Ad Blockers have killed a huge revenue stream for many bloggers which is why we need to charge companies directly for our time etc. That’s a story for another day 🙂
The key issue is that this kind of marketing technique hurts the marketplace.
Consider this situation which is perhaps a little extreme, but is still possible.
There are two WordPress plugins. One is, in most people’s opinion, the best solution on the market. The other isn’t that great. If this kind of marketing continues to be allowed, we could see a situation where the lesser plugin is higher ranked. This in turn generates more downloads and more income for the developer, but it also means that WordPress users aren’t seeing the better plugin promoted as much (obviously, whether a plugin is better or worse can be subjective, but you get my point).
I don’t think that the developer did anything malicious and I certainly don’t think what he did was the same as what black hat SEO companies do, however what he did gave him an unfair advantage over other WordPress plugins. That is why I feel WordPress.org did the right thing here.
Kevin
I think yes, Those who incentive reviews should be penalized.