The New Rating System

Valerie: Let me clarify my last remark..I was not trying to suggest that all non-paying members are mooches or do not contribute as much as a paying member. I agree that money is not the only measure of how a person contributes to a community. I believe that this thread was as of late talking about articles and perhaps I should have been more specific.

What I was saying is I believe that there are members of Biznik who simply publish articles, that is all that they do and as a result they have earned badges and front page presence. These folks typically do not contribute to discussions, they do not attend events, they do not add value to the community that I see....they just publish. Quite frankly they use our Biznik community as an advertising forum for themselves and their work. To me, that exposure is worth $24/month.

I wish I could say it as well as Karrie. Those are my sentiments also.

Elizabeth, if I had been in this thread, I couldn't have said it better. Thumbs up ; )

And I should've included this in my last comment - I do agree that those non-paying members who essentially "game" the system for higher exposure should not have better rankings and visibility than paying members automatically. Community participation (attending events, hosting events, participating in BizTalk, etc.) should be more heavily weighted in the rankings than publishing articles - in my opinion.

Love the discussion - thanks all!

Great points everyone. Karrie, thanks for jumping into this conversation with your points. The timing is perfect - our team is ramping up to address this very point right now, so hearing from the community on this issue is exactly what we need right now.

I have to vehemently disagree that non-paying members are in any way mooches or getting something for nothing. Anyone who takes the time to write an article is making an extremely valuable contribution to the entire community.

We built Biznik to be an open network, and we are deeply committed to that philosophy of openness. That philosophy is partly responsible for why we are now ranked #1 for the term "business networking" on Google (outranking LinkedIn). The paying members will get more - more visibility, more tools, more benefits. Limiting members ability to contribute value to Biznik is not something we're interested in doing. We look for ways to give paying members even more.

Keep the ideas coming - and know that we are listing and will weigh heavily what you are all saying here as we make the important decisions about reworking the rating system.

When my colleagues and I give presentations that include evaluations, we ask that anyone who rates average or below give ideas of how it could be improved. We ask they include their names so we can contact them to learn more if we choose to.

Maybe the voters could reveal themselves and if they have suggestions that would help, contact the author directly.

We've also found that some people are simply incapable of giving a 10. or receiving a 10. i.e. nothing is ever good enough. Numbers (and people) can be funny that way.

Criticism without vision isn't worth much and can keep a lot of talent from being expressed. Caring feedback toward a vision is fuel for growth.

Alignment is key. Hopefully only the people who really want specific information would even open up the article.

I appreciate the care in everyone's comments and the overall intention to make this something of value.

Dan,

I'm not certain but I don't believe that anyone was suggesting that posting articles is not a contribution and a way of participating, especially when they express original ideas, present old ideas well or are otherwise valuable to members.

The issue for me, as a member posting no articles to date, is how the articles and their authors are promoted within the Biznik system.

This isn't an all or nothing deal. My belief is that the same formula that places members in the system, based on all the variables that you measure ($ only being one of them), should be applied to article contributions.

Maybe the formula needs tweaking. Maybe this whole issue will become a moot point when the new rating system is unveiled.

I know that a rating system that encourages conversation rather than ranking will be much more valuable to me.

What if, a writer could opt out of having his/her article or tip rated?

What if, members had to earn the right to rate articles (I dunno how exactly--participation, longevity, cuteness)?

What if, when we rate articles and tips, our overall average rating given is displayed on our profile?

I love this conversation. I'm a little sorry I'm just catching it now. Heh.

I agree with the general sentiment that rankings should combine descriptors such as "fun" or "useful" with the thumbs up or down. That, to me, would help in my decision to read an article or not.

As is, I barely go to the Learn section these days, mostly because I feel like there's so much duplicated content and no way for me to evaluate at a glance what's worth reading.

My brainstorm for some possible descriptors (just ideas, not all to be used):

• entertaining
• resourceful
• accurate
• profound
• life-changing (why not?!)
• paradigm-shifting
• thought-provoking
• supportive
• exactly what I needed to know

Others? Thoughts on mine? I feel like a couple might be good fits in the "new-agey" category that Karrie refers to, since I'm one of the posters of such material. :)

I love "paradigm-shifting," Rachel. Great solution for what I was calling "cutting edge.” I like this a lot because even a useful article might not be paradigm shifting so this would indeed set those apart from the rest. I did not initially think of this for the “new age” type of articles but it could apply to new thoughts on that level or a whole new paradigm for looking at business plans or SEO or pricing, etc. I think this could work across all of them, so it depends on if this is used in conjunction with thumbs up or if it replaces thumbs up votes.

I like "resourceful" but only if people understand it means that the article was full of good links or helpful resources (not that the author was resourceful as that is not really helpful information in this case.)

"Entertaining" could work as a stand-in for "funny" or "fun to read."

I propose some criteria for the descriptors:

• Clear / consistent meaning to multiple users
• Truly helpful for reader browsing articles
• All three words should be distinct and not ambiguously overlap
• Descriptors should not apply to all articles. (We haven’t done anything here if all the articles warrant all of these descriptors.)

Glad to hear your thoughts on this Rachel. I hope more biznik brains will participate in this brainstorm.

I totally agree with your criteria, Karrie, esp. the first (clear meaning) and last (descriptors should not apply to everything).

Let's avoid "interesting" and "synergy" at all costs. :)

Both Karrie and Pamela present very good ideas (and I have been an adamant proponent of such systems, too). Before I elaborate, let me add that it is really important to be able to link (point) to what is being rated -- and in some cases this may even be not a text, but rather a person (for example: when the person says "I'm sorry" and/or acknowledges a mistake). In addition, comments to an article should also be "rateable". And the article itself and the discussion ensuing from that article should be rated separately -- and it's primarily the discussion that should allow revisions of ratings (for the obvious reason that a discussion is open-ended).

Now to the particular case of "descriptors" or "tags" which could be used to make a qualitative (rather than a quantitative) statement. I would suggest that each member of the community be allowed to have a certain number (I like 8, because I like the fact that I have 8 fingers ;). And these 8 labels could then be applied as checkboxes to anything being rated. Anything that is rated would have descriptors listed in the order in which such discriptors have been assigned (in proportion to the total number of decriptors). If a person assigns 2 descriptors, each descriptor gets 1/2 wieght; for 3 descriptors, 1/3 weight (an so on).

Non-Paying members are not allowed to assign new descriptors. Then for paying members, there might be a limited ability to add new descriptors (say 3 or 4), and then at a higher level of membership all 8 descriptors could be created/modified at the whim of the member. When a certain number / proportion of members has applied a descriptor, then that descriptor can become a "community" descriptor, and then it is no longer the "personal" descriptor of a paying member. The number of community-wide descriptors should be limited to a couple dozen or at most a couple hundred terms (since otherwise the descriptor selection process becomes too difficult). If this certain limit is reached, the communty should vote which descriptors are superfluous and should be lopped off (perhaps a more sophisticated system would allow paying members to choose from a larger vocabulary of descriptors/terms than non paying members, but this may indeed be "overkill").

Note that in this system each person would be limited in the number of descriptors they are allowed to apply in general to anything (maybe paying members could be allowed to use 1 or 2 more descriptors in their "personal vocabulary").

In general, however: less is more. If I have to choose from 100 descriptors, I would probably not choose at all -- but with just a handful of options then it's easy.

If someone were to remove a descriptor type from their palette of descriptors, then all of the things that descriptor has been applied to in the past would lose this information (and in that case the "weight" would also be lost -- for this purpose the user would be forced to have 1 of their palette of descriptors set as "nondescript" and the weight of the previous descriptions would be transferred to this "nondescript" tag). Hopefully that would be sufficient to discourage people from assigning descriptors haphazardly.

It is important to note that this is something that the community needs to "control" (it is a referred to as a "controlled vocabulary" for good reasons! ;). Otherwise, such descriptors would become similar to the "uncontrolled" alphabet soup you might find at a place like del.icio.us or something like that.

Sorry if this has been a little technical -- but it is actually very important point in having a well-functioning community.

Kudos to the managers of the site for engaging in such an open discussion about such critical issues!

:) nmw

Uh, Norbert, are you the guy who designed the rules for the new "Good to Go" High-Occupancy Toll lanes on Interstate 167?

LOL -- no, I'm an information scientist (and designing such systems is what information scientists have thought about for many many decades) -- I guess Melvil Dewey might be considered to be one of the "first"! ;D

( BTW: I wanted to add a ps [but was too late]: perhaps with a better rating system, it will no longer be necessary to have "requirements" [e.g. length for articles] -- whereas a poor article that might meet certain criteria might then squander in some old dust-bin, a very good article that happens to not fulfill certain criteria may nonetheless be promoted by being highly rated. :)

Holy moly Norbert, you are one thorough mind! Wow. The first time I read through your comment I thought it was WAY too much for our humble little land of biznik, but after absorbing it a bit, you are really onto something--It would be great if comments on articles could be rated, yes. Not sure this is going to happen, but that would be so helpful, as some threads (low cough) go on for a long time and it's hard to know what is worthwhile to read—especially when people write such LONG comments. I don’t use emoticons but you can see my smile.

And the notion of the virtual community producing a vocabulary is fascinating to me as an anthropologist.

I also think your breakdown of the descriptors and palette and all of it was great and is probably the wave of the future, but again, not sure it's going to happen here, or at least right away. We may need an interim solution and then grow into something like you are talking about. I really don't know. Dan's the man on this one.

I also want to clarify, the type of system you propose is not the same as what I was proposing. It sounds much more complex (and possibly better but also possibly too much for our little biznik team of elves to create, implement and manage--not sure you know how small--albeit powerful--the team is but last I checked it was 2-3 people max running this whole show).

I may have used the wrong language by naming those voting categories "descriptors" as it sounds from your post that that term may already be co-opted by you information scientists and my using it then becomes ambiguous.

The articles are already set up with "tags" by the author. Authors of articles are allowed to make a long list of tags that help people searching for the keywords.

What I am referring to is a way for readers to distinguish from the masses of articles which are more suitable for them to read based on limited, pre-set criteria/descriptors/words/voting categories. For each of these pre-set words, there would be a tally for each article. This would hopefully help readers decide if they wanted to open that article. It's more information than an article title and a tally of thumbs up but clearly not as much information as what you propose.

Additionally, on the author's profile there could be a total in each of the three categories for all articles written.

For the writers, this would encourage people to write to these categories and hopefully we would get more funny, useful and mind blowing articles.

Which brings up a good point: whatever terms are pre-selected will shape the incentive system in the Learn section. Currently the incentive is to write a "8-10 article" (expert status) and there is an incentive to write many articles (weekly visibility). So, the words we choose will encourage people to write articles that will get them votes in the pre-set categories.

Yikes, the more I write after reading Norbert's comment I cannot help but feel that my big idea shrinks in comparison to his. What he is suggesting would allow a much more robust ranking system.

I also like that the Norbert Solution suggests giving preference to paying members who have more voting options. However, I do not like that if a voter’s descriptor is changed that it would delete that vote for all those articles in which that descriptor was used to vote—this could hurt the article author who wrote a great article.

Let’s remember the point of this:

• To help readers sort through the barrage of articles and make easier choices about what to spend time reading.
• To create a new system that could be implemented quickly and easily to replace the existing system without a lot of confusion.
• To increase number of high caliber articles.
• To encourage bizniks to write articles who have something to say but have been, under the current system, intimidated to write.
• To provide more texture and meaning to the system.

I just wonder if Norbert’s Solution provides too much texture. I agree, Norbert “The number of community-wide descriptors should be limited… (since otherwise the descriptor selection process becomes too difficult).” Right! How about 3-6 preset descriptors to start? Then, as things grow, maybe this whole system could be evaluated in a year. How about:

• Useful
• Therapeutic
• Well written
• Informative/Resources (meaning links, book titles, stores, etc.)
• Funny
• Mind blowing/paradigm shifting

Maybe I need to see The Norbert Solution in action somewhere online and try it out before I vote on it. Can you direct us to somewhere this system is being used on the web?

Also, Norbert, I encourage you to contact Dan and Lara as it sounds like you have the expertise to help solve this conundrum!

Too much texture can be just as damning as not enough. Sometimes we need "freedom from choice"

Here we need a system that:

• 1) stops rewarding freeloaders at the expense of members
• 2) doesn't stop rewarding good content
• 3) encourages broader participation from members (as opposed to freeloaders)
• 4) uses a few but not too many ways (three is ideal, eight is way too many) to allow people to sift through the content simply so that people will actually read more articles and benefit more from the wealth of perspectives
• 5) doesn't punish unorthodox good content

I may be missing something, but if this comment was too long, just remember the "freedom from choice" part

One thing authors need to take into consideration about their lower than normal ratings is that when I rate an article, I take into consideration if I learned something new, if it is generally useful to others, is it self promoting or only promoting one's own industry, and is the article readable, uses proper grammar, spelling, and punctuation.

Too often, I am subjected to a tidal wave of type. A quarter of the time, I can't figure out what the author is saying (too much technical language.)

Frequently, there are no blank paragraph breaks after 3 or less sentences to give my eyes a rest. Sometimes, I sense a lack of organization. There is no use of bold face or italics or other character formatting to emphasize certain topics in a sentence or paragraph.

It is rare to see someone who has figured out how to use HTML within the Biznik site (although, the site could make this easier for us to use.)

When I can't easily read what people have written, even if it is a nine and very helpful, I subtract a point for not being readable and give it a lower number accordingly.

I do the same if I feel that the article is a little self centered, is poorly written, or if the author makes statements that are not correct and that they should know in their industry.

I encourage others to do the same and for authors to get on board and make their posts readable. You do want us to read them, right?

In reference to Danny's post:

I think 6 could work as 3 may be too limited. 8 - 100 seems a little unmanageable! But Norbert's post really got me thinking.

Danny, no matter what system is implemented, there will be people who post articles without paying or getting involved with the community. I am beginning to see there is no way around that. Call them "freeloaders" or call them part of the "open network vision" but either way, it's bound to continue.

It seems that if we could come up with words that will give readers enough information to see which articles are highly valued and for what reasons that would go far in pushing the less valuable or spammy type of articles to the bottom.

That's what I'd like to see.

Also, we need a way to sort through the redundant material on here!

Maybe this should be a new thread, but one idea I have is launch a new kind of article--a Super-Article, if you will--that actually consolidate the best information from all articles already written on a topic--like "networking" for instance. Damn there are a lot of articles on how to network and I am not going to read them all. But someone could and they could write a BEST OF biznik super article on them all, with links as they reference other articles that had golden nuggets to include in the Best of article.

In fact, this could be a whole new aspect to the Learn section. A BEST OF for each specific topic. Articles would have to be changeable as new articles are added on that topic. I think this could be very cool.

And only paying members (who have Expert status ?) can author one of these--another perk and another way to encourage membership upgrade. Or, maybe these super articles must be authored by more than one person--encouraging community building and better writing!

I don't want to dilute our conversation about ratings but I think this is relevant here.

In fact, "Best of" articles could factor into the article rating system. If your article is quoted or mentioned or linked to in the Best of article for that topic, you get some extra icon next to your article, like a blue ribbon. Do we already have a blue ribbon? This way, even an article that wasn't great over all but had a key idea or resource in it could still get some recognition for it's golden nugget, even if it didn't get great over all votes for being mind blowing. And a single article could get multiple blue ribbons if referenced in multiple "Best of" articles.

Okay guys, I have gotten entrenched in this conversation! I better go now. Looking forward to seeing where this goes. What a very cool collaboration.

Still want to hear from more and new commenters about the 6 voting words I used (afraid to call them descriptors for clarity sake).

Reply to Keith's post:

Good points! I think a lot of the things you mentioned are covered in the six voting words I suggested. See four posts up, second bulleted list.

If you had six options like those, you could get your message across that an article was:

useful, and offered resources but was not well written. That's what I like about this system. It helps the readers find articles worth reading (according to what they value...humor, finding resources, usefulness, etc.)..

AND it also gives more feedback to the author about what they need to improve. Someone who gets a lot of "useful" but 0 "well written" will start to look at their writing, hopefully.

And conversely, someone who gets a lot of "well written" and "funny" but few other comments will start to see that though they write well, their articles could be more purposeful, or not, depending on what they are shooting for.

Seems if someone is NOT getting any votes on their articles under the "well written category" they might get the message to proofread or get a tutor.

Seriously signing off now...

Karrie said:

"However, I do not like that if a voter’s descriptor is changed that it would delete that vote for all those articles in which that descriptor was used to vote—this could hurt the article author who wrote a great article."

Yes, I agree -- but actually the vote wouldn't be "lost" -- just the descriptor.

I also think that the numbers I suggested above may make it sound more complex than it actually is. I expect that I personally wouldn't need many descriptors (over time, I have found that "insightful", "interesting" and "funny" are most useful -- perhaps "well-written" might also be a good one to start off with).

Let me give a concrete example. All the paying members start off by voting for something and start assigning such descriptors as "funny" or "well-written". If say 1% of the membership uses the term "funny" (or maybe simply start off with 10 people or so) then that term becomes available as "community-wide" term to rate content with.

Note that in order for such a system to be useful, these terms need to stay more/less fixed over time. If a person could say something is "helpful" today, but next year "helpful" is dropped and "useful" is used -- does that turn what used to be helpful into something which is now useful? No, not necessarily -- and this is why when a descriptor is dropped the "qualitative value" should be transferred to nondescript.

This system is still far from perfect -- I may find something very funny but not at all helpful. But implementing such a system would involve something like 3-state logics (positive, negative and undefined) -- such that I could assign "funny" +1, "helpful" -1, and everything else 0 -- now that is complex.

To keep it simple, I agree that only "positive values" should be considered -- and that would also avoid any argumentation about "negative" sentiments.

So let's say that each member could press a button that would open a box saying something like "I appreciate this because" and then 8 checkboxes appear. The user could focus on one aspect and therefore send a clear and distinct message, or the user could distribute their appreciation over several (or even all aspects).

Then how would such ratings be presented on the site?

You could say "most appreciated" (in general -- simple tally of all the people supporting), or "funniest" or "most useful". I think, however, that the precise method chosen to present the information is rather secondary -- the primary issue is to first decide which information to collect.

:)

I may be opening up a can of worms here, and not sure if anyone had addressed this, but why not consider doing away with the rating system.

I feel that the "learn" area is just for that, not to boost your ratings. Articles are very subjective and, of course, some will find them useful, others not. I look for articles that draw me to them by subject, not ratings. And for some readers, they may see a poor rating and pass it up because of that -- where otherwise it might have been very helpful to them

Through the comments everyone could give their own particular "rating".

What's the benefit to the writer: You are sharing your knowledge/expertise. You still get the "plug".

I hope Dan isn't staying awake at nights considering all the options that have been listed here. Maybe mine seems too simple, but honestly I joined Biznik to learn, network and build relationships. Not to obsess over ratings. : )

Bob,

seeing as I have just written so much, I will take it that I am being addressed when you mention "obsession" -- and it is indeed probably true that I am obsessed with "separating the wheat from the chaff" (insofar as that is my profession). And I do not expect everybody to have this obsession -- others might be "obsessed" with art or writing or marketing or sales (I guess everyone focuses on what they're interested in).

The question for people with a limited amount of time is: If there are 100 articles about a topic (e.g. "successful marketing") is there any way to help a person interested in that topic to decide which one to read first?

Of course if there are only 1 or 2 articles on a rather particular topic, then it may indeed be much more effective to "search out" the detailed topic (and for that I would agree that it may be good to look to other methods of finding the most appropriate information [such as -- for the above example: "which articles do successful marketers recommend?"])

I am amazed at the complexity of the some of the solutions being proposed here! It seems that a selection of "descriptors" might help readers evaluate whether an article is worth their time. But, really, isn't it all still "subjective?" (What's "funny" or "useful" to me may not be to you?) And "well-written?" As a professional editor ("writing is my life!)—well, even that can be interpreted differently by different people.

If a rating system is made too complicated, will busy people even bother to rate the articles? Just thinking out loud.

Yes, I agree with you, too, Judy! ;D

Indeed, if you recall the suggestion I made about a week ago it in fact had much more to do with personal preferences -- namely enabling users to find things via the help of people whose opinions they personally value.

Indeed: There are many ways to find "relevant" information -- and it is wrong to think that one way is the best for everyone every time.

I would use the term "invested" rather than "obsessed" because there is a purpose to this dialogue and it could make a big difference to the community.

A poor system, or too simplistic a system or no system at all does not serve biznik, though I like the sentiment that it's all about learning. And Bob, this is less about authors getting "credit" and more about readers finding appropriate, high caliber articles with value as the number of articles on the site grows.

Yes, Judy, it's all subjective and imperfect, but patterns will show up in the numbers as people vote. An article that gets 430 useful votes and just 2 funny votes does tell you something. While the reverse would also be true.

Judy, busy people will vote if the voting system A. makes sense, B. is viewed as valuable and C. is easy to toggle the words and move on. If bizniks take the time to comment this much on biztalk, we've surely got the 5 seconds to click on a button that says "useful" and help the next person separate the wheat from the chaff.

Great discussion. I'll be out most of the day, but interested to see what happens here.