The New Rating System

I think that it is sad to have people within the Biznik community routinely give articles "0" ratings for what appears to be sheer personal amusement.

I too am interested in when this new change in ranking articles will take place. I have not submitted an article in the past because of the obvious ranking sabotage that was taking place and I might consider writing something should this new system solve that problem.

The idea of rating anything online, for lack of a better word, is bad.

I unfortunately haven't seen any rating system that cannot have a train driven through it, over it, or simply into it. (I do however realize that it is part of my profession to examine common holes in different community systems)

The easiest way to prevent massive abuse, is to simply only count people in your network who have rated it. Hopefully if they are in your network, you can trust them a little more to apply some basic steps to rating an article.

As far as topical articles go- I also see there to be a spread between being technical accurate, useful to the community, and useful to you. There are several categories of articles that I rate at 1 or 2 (some SEO and marketing ones especially) that I count as being detrimental to the reader. When I see an article that has old information it gets a "1" in my rating. Anything old in my industry can really hurt a business (and is thusly detrimental on the Biznik scale) when I read it.

However, if I didn't know a thing about SEO- I would probably rate some of them as 9 or 10 for being well written with information that I was not aware was simply wrong or inaccurate.

This type of problem and trend is simply horribly complex, especially with a smaller community doing the voting.

Barry, your comments are insightful, as usual. You bring up several good points. Someone who has exceptional expertise in an area may not feel that a rating of, say 9 or 10, is justified. While someone with less experience will see the same article as "perfectly helpful." I personally have seen a number of very good articles that were not helpful to me, but that certainly would be to some of our other members. In those situations, I just don't rate it. Not because I didn't like it, but because I didn't want to skew the ratings.

Throw in the whole SEO thing that goes on in Barry's field, which seems to change by the minute anyway, and you could very well have a "detrimental" element.

I agree with Joe. I am leaning toward the "thumbs up, thumbs down" system. It just seems more fair.

How about a categorical rating? Journals and magazines have a rating for articles they believe worthy of publication. Each category receives 1-3 points for meeting the categorical criteria, 1 being lowest and 3 being highest.

For example:

CATEGORY POINTS

Exhibited knowledge of subject Addressed core issue in topic Meaty Clear thesis Body supported thesis Conclusion brought application

Floyd

We haven't fully settled on what the new rating system will look like, so I love seeing everyone's thoughts on the matter here. If you've got a strong opinion on this subject, post it here and we'll be sure to see it.

Re: expected delivery date - I don't have one yet. It'll be at least a couple of weeks, while we ramp up our new development team.

My remarks from an earlier email exchange with another member dated January 30, 2008:

"I'm with you on the rating system. It's flawed. I try to read and rate all the articles but refuse to rate the articles I don't really find very useful or original. I don't think downrating is fair without detailed and specific critique or discussion, which this system doesn't allow.

Totally original ideas are rare and are great to encounter. I'm always appreciative of old ideas reframed or presented in new ways that happen to click for me.

The current rating system is too subjective and too subject to all the odd twists that come from a total lack of standards for judging. The system is also subject to the popularity bias. I believe that the system will end up killing the original concept by discouraging members from sharing their ideas for fear of poor ratings.

I like the idea of broad, positive categories of response such as in the Yelp! system or the categories you suggested. If the article shows positive responses by broad category, as in your "helpful", "entertaining", "cutting edge", it keeps the tone positive and encouraging. If it also allows comments so people can explain their feelings about the articles and the writers and others can carry on conversations about the thoughts presented, it becomes a powerful tool to encourage discussion and evolution of the ideas while allowing disagreement. This would be a much more honest and open way to get the work accomplished that I think the founders had in mind when they set this up."

and from the next day...

"It might be a good idea to just leave the material already published where it is, as it is, and start a new program. The change over could be explained as "It isn't working as we wanted it to work, so, we are reforming it in a more honest and helpful format".

I would like to be able to rate in a positive way or not rate at all. And I would like to be able to engage the writer to support, expand on or challenge the ideas offered up.

This would still be dangerous to egos. People with positive responses could check a box and comment, if they wanted to. People with negative responses could not check a box but would have to have the guts to put their reactions out there in words attributed to themselves; and they would have to deal with other people's reactions to their comments.

If the number of people looking at the article is posted with the article, any discrepancy between number of views and the number of ratings would also be telling, but in a less brutal way than a numerical downrating. People who submit numerous articles that get high viewing numbers with no or low comment numbers would get the message that people aren't really appreciating their offerings. I think this would be good feedback.

I think an approach like I'm suggesting would be more open and honest and would be more likely to expose biases and favoritism."

I still feel the same way.

Thanks for the feedback everyone. I think based on what you're saying, the simple "I like it" approach is going to be the way to go.

My experience with rating systems has been that they work better if people can see who has voted (and which way) -- and yet I know there are also vociferous adherents of "secret ballot" systems.

In a more "sophisticated" system, each user might be able to assign their own "weights" to arrive at "scoring" -- such that I might weigh "legal expertise" differently than "artistic expertise". Somesthing like this would probably only be feasible to keep track of in a small group (such as "my network").

So there could be 3 levels of "scoring":

1. biznik-wide

2. my network

3. according to my weightings (and perhaps there could be 2 or 3 alternative weightings)

Perhaps levels #2 and #3 need not be calculated for every page-load, but rather with a mouse-over or a click (in order to reduce the computational load).

:) nmw

That's an interesting idea, Norbert. Sounds a little complicated for this community though, which is decidedly non-techie.

"How helpful?" is relative to readers' preference and understanding of the topic. Rating based on it is subjective. And it does provide some guidance to the writer if no one abuses it.

I noticed there were more viewers than voters. I wonder if more people vote, would a few abusive ratings make any difference?

Simple "I like it" might work. We can give it a try.

I am the mystery emailer John referred to above.

Glad this system will be changing as it has a lot of problems. Here are some more suggestions:

People should be able to change their rating they give an article at any time.

• I've heard of a few cases from other bizniks where they voted very low for something because they weren't yet ready for that information or didn't understand it. Days later, after giving an article a dismal grade, they realized the author was right indeed. At that point the vote had been cast and could not be changed. For articles that are more challenging, cutting edge, or hard to take at first, this is very important because they will often be rated down.

• I've also had the experience of giving an author too much credit and then finding out that their article was giving bad information or just out of date. In that case I would have liked to lower the rating, but couldn't.

Add texture and interest to rating system.

I much prefer a system that does not rate numerically or simply thumbs up or thumbs down. I think a system with 3 descriptors, such as "useful" "insightful" "fun to read" would serve bizniks much better. Voters could toggle any or all or none of these. The article page would show a total of each of these.

I am not sure what criteria people are using when they vote on articles. This is one of the big problems with a numerical system--it's rather subjective and personal. My 7 may be your 10 and we both felt the same about the article. But this problem does not disappear with a thumb's up approach.

• Frankly, it does not ever disappear completely, but by adding some layers to how we vote, and making them meaningful layers, like the 3 descriptors I am proposing, I think we'll get a better idea before opening an article whether we want to read it or not. I would be much more interested in reading an article that got a bunch of "useful" or "cutting edge" votes than one that got a 9.1.

• This would help when readers are looking for an article to read. There are sometimes when you really just want something useful and don't care about how it's written.

• On the other hand, I know bizniks who have some really funny or insightful ideas but have been afraid to post an article because the current rating system does not allow readers to qualify their vote (as in, "this was a fun read and gave me some good insight but wasn't necessarily full of links and tips).

• This would also make the articles feature more robust, adding both texture to the ratings and also making it more engaging. A simple thumbs up isn't that engaging. But if I get to read an article thinking "Is this helpful, cutting edge, well written?" then it gives me a tool to read with.

• This could also be something tallied up on authors' profiles, much like they do on Yelp. For instance, there could be small icons on the profiles that let you know this person wrote xx articles and readers thought they were: 14 useful, 27 insightful, 88 well written.

• This would encourage authors to write more interesting and well written articles instead of simply churning out an article every week just to keep their photo on the top of the Learn page.

(By the way, if you go to the Yelp site, note that I am not referring to the 5-star rating system for the things being reviewed, like restaurants. I am referring to the votes by readers of the actual reviews. If you look there, find someone who has reviewed a lot and then click on one of their reviews. You have to be logged on to see the toggles for the words they use: "useful, funny, cool."

I am so glad there is consensus about the current system. It's a travesty to see someone who took the time to write an article who has to live with a 3.5 by their smiling face.

That is not good for biznik as it:

• Discourages the low rated article author from writing more articles.

• Puts fear into the minds of people who have not written articles. I've actually had people tell me "I don't want to be humiliated on the Learn section of biznik!" when I was encouraging them to post an article.

• Discourages authors from sending their clients, friends, colleagues to biznik out of fear they will see their low rating.

Positive rating is the way to go, but it will still be perfectly clear which articles are more useful, funny, cool, etc. if we use a system that allows us to tally them up in categories relevant to bizniks.

I trust you will look into this and really consider this. Dan and Lara, this is some of the feedback I meant to offer you months ago--but as you can see, had a lot to say it and no time to write you back then.

Good luck coming to a better solution. It's something you have become quite skilled at, the constant improvement of things biznik. Thanks!

Karrie

What good thinking people have been doing on this. The more I think about it, the more complex it gets...

I think that no matter what rating system evolves, it is ALWAYS going to be subjective! As a "7" means different things to different readers, so does "useful," "fun" and a million other adjectives. And, what would a rating of "cool" really mean?

Here's a radical idea: I understand the reasoning behind the ratings, but I almost wish (sigh) that we could just go to comments.

What I most like about the "Learn" section is the interchange of ideas among readers and the discussions the articles provoke. I get as much, if not more, from that than I do reading the article alone. And I have seen too many 4s and 5s given that do not really make sense to me.

I agree with Karrie that the present system may be discouraging some bizniks from submitting an article.

One thing I don't agree with is that we should be able to go back and change our ratings. I think that might open another can of worms and leave the system open to manipulation.

No matter what system is selected, you can't please us all. I have faith in Dan and Lara that they will weigh the comments and do what they truly feel is best for the community.

Thanks again, Joe, for introducing this topic.

Yes, no system is perfect (hence my nod to Danny Bronski with the "Frankly" comment). But if we can get away from numbers and use words that actually mean something then we might really do a service for readers looking for a specific quality in an article.

Seems there are articles with lots of links to resources and others that are sort of touchy-feely and still others based on the expertise and experience of the author. Some people like the new-age type articles that validate and offer mantras, while others would prefer an article full of data or information, or something else. Maybe this is where we get the descriptors from: what KIND of article is it--just a thought.

Maybe it would work to have a thumbs up or down in addition if we did it this way, if the words themselves did not judge the article (as I originally posted) but instead described it more.

I firmly believe voters should be able to change their vote. Remember, with the system I propose or even with a thumb's-up system, the vote would no longer feed into an average. It would just add or remove one point for that article. That is a very easy change that would not have a huge impact (like averages do) on the overall score.

Judy, maybe you could explain how this would open the system to manipulation. Maybe I don't understand. Great to have your feedback on this as I've given it some thought but of course haven't thought down every avenue.

I hope a lot of folks will weigh in on this and share your experiences with the rating system (as authors and readers) that could add more insight to this topic. Thanks Joe, for both bringing up the topic and for your previous post. It really helps to hear what people have experienced.

Ah, now with your added explanation, I get it, Karrie. I thought you were wanting to allow people to move up and down in getting an article to another score in the averaging process. I've seen too much of what look like "scoring wars," I guess. It would then take away the score you had given previously and substitute your new score?

I really like your idea about "what KIND of article is it?" There are some vastly different kinds of articles submitted.

And, Joe. I read your blog post and I loved it. You are wondering if a short, "one point" article has a place in the article section? Don't know if readers would rate it lower because it's shorter, but, on the other hand, it was very useful. This is a case of "ratings inhibition?" : )

Wow, there have been so many great points on this important topic, it's tough to add anything original, but I do want to make the following general points:

• 1) I think the benefits of Biznik's articles features (SEO and other exposure) should skew heavily to reward good content from certain members (namely, those who invest in Biznik through paid membership and who also invest in the community that makes up Biznik). There is no reason a random person should be permitted to get a PR windfall just by signing up for Biznik as a non-paid member and submitting an article while not contributing anything else to the community; that totally undermines the idea of community and that turns Biznik into nothing more but another garbage dump for content that already litters the landscape of the internet.
• 2) Barry is absolutely right - online ratings suck by nature, and will always be imperfect and subject to manipulation.
• 3) Karrie's point of adding texture to the rating system is absolutely critical to improving it, in my opinion

• 4) Negative feedback should be limited to the comments. People abuse the ability to provide negative feedback, especially when there are no checks in place (transparency, earning the right, etc.).

• Something to explore is to make transparent positive feedback from your network, i.e. person X, who is in my network and who I greatly respect, rated article A positively as fun and article B positively as useful, and thus I will be more likely to check out article A when I am the mood for fun, light reading and article B if I want to learn about something more in depth.

Anyway, I have probably benefited from the "popularity bias" mentioned above, as my articles are very highly rated, but regardless I still can't wait to do away with the numerical rating system. Any system that will make Biznik a stronger community gets me very excited.

To numerical ratings, I say...off with it's head!

Yes, Danny...we don't want to be another part of the littered landscape. Here here. There are so many bad bad articles floating around and I do fear biznik might rapidly become a MAGNET for them.

And a RESOUNDING YES that paying members should not only get the most SEO from good articles but also greater visibility on the Learn section and homepage. I am starting to feel like people are "using" biznik and reaping the rewards of their SEO and visibility here but not giving much back in some cases (measured in paying membership and in actual community building).

Judy, glad we are on the same page about being able to change a vote. Yes, with a toggle you can turn on or off this will not lead to multiple votes or manipulating averages.

Glad to hear people like the textured voting idea. I'll be interested to see how this develops.

I'd like to hear a brainstorm from bizniks about what descriptors we could use. Here are some ideas:

useful, cool, funny (what Yelp uses so we should avoid copying them, but a good place to start)

helpful, fun to read, cutting edge

good links!, good insight!, well written!

(What word could we use for the more new-age type articles that seem to be in abundance--the ones telling you how to live and giving you guidance that is more on the softer side? Some people want to spot those to avoid them and others want to only read articles like that.)

With so many articles on the site now, we also need to offer some suggestions on the current categories that articles are grouped in. Some, like Marketing, are so full that it's daunting to open that listing. If you reply with a brainstorm for some of these article categories, let's be sure to distinguish them from the toggle descriptors for voting we've been discussing here, so we don't get confused.

Thanks Danny, for weighing in with new thoughts and some great insight.

I've been silent on this subject until now, but I have to give Danny a BIG THUMBS UP or a "10" or whatever it ends up being for his POINT #1.

I'm actually getting a little steamed about the idea of people who haven't invested in a paying membership getting any kind of benefit boost by having such high visibility.

This isn't participation; it's exploitation.

Articles from non-paying members should get the same play as their profiles get on the Members page, listed after paying members and never on the home page.

My $24 per month.

John

It's inflation, Joe. $0.02 doesn't go very far anymore.

John

Why are people getting something for nothing?

I have watched this thread with great interest, and agree with most of the comments above. I feel compelled to chime in on this last issue, though. Not all non-paying members are "mooches" who contribute nothing. Some are business people, or soon-to-be solopreneurs, for whom $24 a month is simply unaffordable at the moment. Making a $24 payment every month does not always translate into adding value to this community either.

Caveat - I am not saying Biznik isn't worth it (obviously it is, otherwise I wouldn't be a Supporting Member), nor am I saying paying members shouldn't get added benefits - they should, and they do. I just object to the characterization that those who don't pay, don't contribute. Money is not the only value members can contribute to this community.

stepping off my soapbox :)

Valerie, good points and I just want to note that both myself and Danny included "paying members" and "community builders" in our comments. What so many people seem upset about (myself included) are the people who do neither (pay or build) and yet they are on the homepage nearly every week as well as dominating the Learn section.