The New Rating System

Oh, yeah, if I read an article and I don't rate it. Their probably wasn't anything worth commenting on in it. Art T.

As a guy who's got two out of the 5 most read articles on biznik (and who regularly sees his articles start off with "0" ratings) I'm all about the rating change.

Honestly, I tend to see low ratings as a good thing because they show that somebody feels passionately about what I wrote.

But yea, those early 0s are annoying.

Wow- I started to write a reply, and then realized I was only at the bottom of page 1. I can't believe I missed this discussion when it started.

I am astounded by the thoroughness and generosity of the suggestions.

The sheer interest and complexity of replies I think supports how much a rating system would be useful- I fully support a ratings system of some type.

I also like the multiple categories/tags/descriptors what-you-may-call it.

I think it IS useful to use numbers to rate it, if we are clear with the different categories of how we rate it. The Wisdom of Crowds by James Surowiecki shows how large numbers of people make better decisions than an individual with specialty training, and that in situations like this one, especially, those aggregate numbers tend to balance out problems.

I would be against the "thumbs up/thumbs down" because it's too simplistic- it's not useful to other readers, it's not satisfying to the author in terms of getting real feedback on what's going on, and it's certainly not expressive enough for the person rating it.

But, simply adding one or two more sliders to the current rating system, and giving it an individual category and aggregate score, would be really, really helpful, I believe, without getting too complex either in terms of development/programming time, or in terms of making the rating system hard to navigate or understand.

I believe it's okay to be discerning, and for people to give and get feedback from each other. Low ratings hurt the ego, but can be incredibly helpful for someone who is trying to get their start writing articles.

If truth doesn't have compassion, it's only a partial truth, and if compassion doesn't have truth, it's denial, not compassion.

One point I keep coming back to with all the discussion about rating systems... Any rating system, whether using numbers or descriptors, is much too subjective and subject to a universe of interpretations.

The only way out of this dilemma, IMHO, is to require that ratings be backed up with explanations for the ratings (i.e. comments).

A rating without an explanation can be so open to interpretation that it is useless. A rating with an explanation can also be misinterpreted; but, at least, it gives a view of what was on the mind of the rater and gives someplace to start with a response to the article and the rating by readers.

Joe said:

"So, the upside of your argument = more responsible, more positive feedback."

In my first response I noted that it is very important that feedback be transparent -- such that anyone (or at least any member) can tell who voted which way. If all votes are carried out by phantoms, they aren't as meaningful as if someone is willing to stand behind their own opinions. There is a certain fanaticism supporting "secret ballot" voting -- but I strongly doubt that secret ballot voting would lead to reliable (or to use Joe's term: "responsible") results.

Yes, agreed that mandatory comments are neither "user friendly" nor "easy" to execute.

But having the system keep track of who is voting for what is both user friendly and relatively simple -- and would also be very informative to boot.

Wow! If psychic ownership is one measure of "community" we've got it.

This is truly a "hot button" and has been widely debated! I know that my suggestion to drop the rating system was frowned upon, and I humbly accept that.

There are lots of interesting suggestions here, and I can only wonder what the community as a whole thinks about this. Unfortunately, some members aren't comfortable joining in on this or maybe they are overwhelmed.

With all these pros and cons, Dan and Lara have a challenging task ahead of them. One thing I know for sure, they will always keep in mind what is best for this community and what fits in with their overall vision of Biznik. Good luck to them and after only two posts in this thread, I bow out gracefully.

I just posted my first article. I decided that I would not care about the rating, but more importantly the reading. The things I share in the article are the same things I have shared in Events on Video I hosted.

So it is always nice when someone compliments you, but if an article needs to be rated, it's at best only a snapshot of the moment. As others have admitted, opinions can change over time.

So will I write another article? Don't really know. I have to post my next video event and see who signs up.

But I really appreciated the discussion. There is no one answer to this issue. But this is mine.

It's great to see this much passion around the topic of business networking!

Lara and I had cocktails with Karrie Kohlhaas and her sister, Kimberly Kohlhaas, who is an active Biznik member in town from Austin, last Friday, and we spent most of the time talking about ratings, and how they can be improved.

We'll be weighing all of this carefully as we prepare to make changes. And it's likely that supporting members will be invited to participate in a beta version before we launch it to the rest of the community. Stay tuned.

I notice that after my "massive" brain dump of several days ago, that I overlooked to follow up on some of Karrie's questions. First and foremost: No I have not seen anything like the level of sophistication I describe here. I was quite involved in a similar community rating scheme developed with input from the community, but the archives from that project have been taken down from the Internet. To cut to the chase, I think this would be a "pioneering" scenario.

I have also run the the ideas through my "database" perspective -- and I must say that it is indeed very complex -- especially if some members are permitted to have "personal" tags (so I would perhaps limit this quite strongly).

I would say that the crux of the "tagging" system (I am going to stick with this term -- the descriptors actually function quite similarly to "tags"... indeed, so similarly, that it might be useful to think of the data architecture in similar terms [but that need not worry us here]): let's say the "community wide" vocabulary contained, say, 20 - 40 terms. For "information retrieval" (aka "search"), this would be the number of "types" the search engine would have to check, tally up into a score, etc. -- resulting in a rating such as "very funny" or "quite insightful" or "super helpful" (in these examples, the terms would be "funny", "insightful", "helpful"). IMHO, there is no reason why the same principle might not equally apply to such descriptors as "marketing", "accounting", "teamwork" etc. (but it might nonetheless be useful to differentiate between articles that are funny / helpful and articles that are about how to be funny / helpful.

Here are some more concrete suggestions -- I would suggest three "levels":

0: Basic membership

allows members to choose e.g. 4-6 terms to tag from palette of community-wide terms

1: Active membership

allows members to choose e.g. 6-8 terms to tag from a palette community-wide terms (terms may be "freely" assigned until the community-wide vocabulary is "filled")

2: Supporting membership

allows members to choose e.g. 6-8 terms to tag from a palette community-wide terms plus 1 freely chosen term

The availability of freely chosen terms means that the language will predominantly be shaped by active+supporting members, but every member will still be "free" to apply his/her chosen terms as he/she feels is most appropriate.

Some might find it "quirky" that it might be "necessary" to have 2 levels of vocabulary (community-wide vs. selected terms). What is really going on here is that this helps members to focus on what they feel is most important from their specific expertise (or "point of view"). So whereas I personally may not be an expert at deciding whether something is "world-changing" or not (and perhaps I might not care, either), but I may instead look for whether something has the attribute "promotes social cohesion" -- and if that is something I feel strongly about, then I might choose that as an attribute to "rate" things. So the point of 2 levels is to allow what might be referred to as "linguistic relativism" -- like Eskimos have many words for "snow" vs. Germans have many words for "beer".

Well, I guess that all sounds quite abstract (and I guess it is abstract).

One simple approach to "get the show on the road" in a simple / straightforward manner would be to simply start out allowing every user to use up to 4 tags and the community-wide vocabulary would "fill up" at say 40 terms. Then, at that point, only the 40 most popular terms would be maintained (and if someone had selected a term outside of this vocabulary, it would then be set to "undefined", "nondescript" or something like that).

This simpler approach would perhaps be a way to "kickstart" such an approach without introducing alot of complexity. Later, this could be refined to add such "bells and whistles" as providing methods for allowing the language to "evolve".

OMG -- I didn't want to write this much! :O

Sorry again -- I will have to practice being more succinct (if at all possible?)...

:) nmw

I do not want to take the lid off of the pot which seems to have come to a simmer from a fierce boil. That gets messy and I hate messes.

I would like to add that I read Joe's article and it was good. It was informative. It was basically helpful, generally helpful, very helpful, greatly helpful, extraordinarily helpful and OFF THE CHARTS helpful....depending on who you are reading the article.

The first person that rated this article gave Joe a 5 "basically helpful". This might say to all of the other Bizniks who awoke this morning "well looks like someone wrote a borning useless article..."

I can't help but feel that instead of nailing Joe with a 5 that is just so middle of the raod, average, try again fella, that perhaps the person who chose to rate him could have passed on this article.

Finally, I would like to add one last thing before I go get my pop tart and coffee....I have a lot of article ideas floating around in my noggin but there is no way I would consider publishing them in this forum until the rating system works itself out. I would not handle a 5 as well as Joe. I bruise too easily...

Except that, by the time I'm reading this, 14 minutes later, Joe has an 8.3 rating going. That's because ratings tap into the wisdom of crowds, and even though one person who rated him a 5 might have gotten it wrong, after enough people rate something, it tends to find it's correct level.

I agree completely, though, that the current rating system isn't the best. And that's why we're changing it.

I'll make this one short. :)

Today I updated my Facebook status to read:

>> Norbert thinks digg.com should allow its users to recategorize news that was submitted in the "wrong" category (according to popular vote). <<

Do others see how adding descriptors/tags to the thumbs up/down would be similar to my proposal for digg.com? Why not create a better system than digg.com? For more inspiration, here's a quote from Marrianne.COM:

Our deepest fear is not that we are inadequate. Our deepest fear is that we are powerful beyond measure. It is our light, not our darkness that most frightens us. We ask ourselves, Who am I to be brilliant, gorgeous, talented, fabulous? Actually, who are you not to be? You are a child of God. Your playing small does not serve the world. There is nothing enlightened about shrinking so that other people won't feel insecure around you. We are all meant to shine, as children do. We were born to make manifest the glory of God that is within us. It's not just in some of us; it's in everyone. And as we let our own light shine, we unconsciously give other people permission to do the same. As we are liberated from our own fear, our presence automatically liberates others.

Look for this new rating system sometime this fall.

Wow, can't believe I missed this discussion! Arggghhhh.

I was very, very hesitant to publish articles after my first couple went from being above 8's to below 6's.

And Joe had to do some strong convincing to get me to publish the latest Google Analytics article, because I, like Elizabeth, am affected a bit by the seemingly unsupported low-ball ratings some articles get.

I'd like to suggest that the new rating system be one that works in this way:

1. Read article
2. Be asked a simple yes or no question: Was this article helpful?
3. Select a radio button: Yes or No
4. The system keeps count of each "Yes" and displays it: e.g., "59 Bizniks found this article helpful."

It's easy, still effective and if no one finds the article helpful, the writer will get the hint.

Tia,

Yes, sorry you missed this conversation. The idea you outline succinctly above is similar to the ratings concept I discussed several times in this thread. You can go back and read those and see what you think.

Would be curious to hear what you think about this idea, Tia. And I am sorry to hear that you had a bad experience with the current ratings system. There are lots of problems with numerical ratings, one being that people rate on different scales, even using the same numbers. Some people rarely give a 10 and a 7 would be a compliment from that person. But others give everything a 10.

There are a slew or reasons why I don't like the numerical system. It can really act to alienate someone from this community too and that's not something we want. I do like the multiple radio button tallies because that way the person gets some feedback on what people did and did not like about the article.

Glad you are part of this community, Tia. Whatever ratings your articles get, please don't get discouraged. It's so subjective, these numerical rating systems!

This is very much the approach we have in mind, but with a twist I hadn't thought of (making it an answer to a question.) Thanks Tia.

Essentially, this is a "numbers" game. If you've got the entire Internet on your homepage several times a day, then you have such large numbers that you could create a far more sophisticated system than if you have "only" 1% of the Internet or "only".01% of the Internet (I think digg.com boasts something 30MM uniques per month?)

So if you've got a homepage with extremely wide reach and a box to enter 1 or 2 words, then you ought to be doing something fantastic. If instead, you only do something mediocre (of even something inferior) then you will ultimately fail (for example, I believe it is not at all far-fetched that Google will soon implode due to engaging in "self-promotion" [blogspot,youtube,knol,etc] rather than providing at least "passable" results [as it is today, Google only provides passable results for "navigational" searches -- i.e, people who do not yet understand that they do not need to type "shoes" into Google to get to shoes.com]).

Oops -- went off on a tangent again... :O

If you only have a relatively small group visiting a page (to rate it) then you are perhaps not reaching "the law of large numbers" scale -- and so this is why you should choose your statistical method with a view to what can nonetheless provide meaningful data.

A Yes/No question is one thing, but asking a question may channel the responses into one particular pigeon hole. For example: is a humorous (or satirical) rant useful or not?

I can't remember what I wrote last, but I would (continue) to argue for a three-state logic: "approve", "disapprove" and "other" ("all other cases") plus a write-in box limited to one word (no spaces or hyphens) where people could write in "useful", "funny", "understanding", "knowledgeable", "informative" (and so on [note that this, although it seems like a quite "open" or "free" response is nonetheless focused on "predicating" the contribution that has been made]).

:) nmw

Interesting thoughts. Who thought that a rating system would generate so many different interesting opinions and differences of approach?

I don't necessarily think this is the best idea, but fuel for the fire: What if you could only rate AFTER you left a comment --encouraging a bit more justification for the rating (or thumbs up), good dialogue, and accountability.