RFC - Review Shakeup
-
- vvvvvvv
- 2578 posts since 24 Oct, 2000 from skelmersdale, west lancs, uk
Good suggestions, Ben
Member 12, Studio One v6.5, VPS Avenger, Kontakt 7, Spitfire, Dune, Arturia, Sonible, Baby Audio, CableGuys, Nektar Panorama P1, Vaporizer 2 to test out
-
- GRRRRRRR!
- 15965 posts since 14 Jun, 2001 from Somewhere else, on principle
The problem would be the same - 90% of people would give anything they review 5 out of 5 stars.bobaudio wrote:Maybe its my short attention span (I didn't even read all of the posts in this thread), but I like a 0 to 5 star summary like MANY MANY websites use. If I want detail, I'll read the comments. Basically a system like CNET (pros and cons) and Amazon (Stars and Reviews).
NOVAkILL : Asus RoG Flow Z13, Core i9, 16GB RAM, Win11 | EVO 16 | Studio One | bx_oberhausen, GR-8, JP6K, Union, Hexeract, Olga, TRK-01, SEM, BA-1, Thorn, Prestige, Spire, Legend-HZ, ANA-2, VG Iron 2 | Uno Pro, Rocket.
-
- KVRist
- 34 posts since 2 Oct, 2004
Wouldn't that give away the Big Secret vst?DevonB wrote:Seriously, one more category that would be nice is
Most Underrated VST/DX?
Lot of us have favorites that NO ONE talkes about, and it might get some more chatter around some of the more obsure plugins.
Devon
-
Voidoid Surrealist Voidoid Surrealist https://www.kvraudio.com/forum/memberlist.php?mode=viewprofile&u=41079
- KVRAF
- 4048 posts since 18 Sep, 2004 from Places far less tedious than this blue trainwreck...
I also like the idea of being able to rate the pro's and con's. I think that would be WAY more informative.
However, I'm going to be really disappointed if you end up backing down and leave out the "f**k me!" catagory
However, I'm going to be really disappointed if you end up backing down and leave out the "f**k me!" catagory
-
- KVRAF
- 3441 posts since 15 Mar, 2003
I think the rating numbers can be usefull for the newbees.
When I was starting out the number of available plugs was bewildering.The ratings way be somewhat arbitrary, but with enough reviews they tend to let the cream rise to the top.
The ratings allow a new user to get hold of some of the better plugs without having to spend forever reading reviews and downloading things.
When I was starting out the number of available plugs was bewildering.The ratings way be somewhat arbitrary, but with enough reviews they tend to let the cream rise to the top.
The ratings allow a new user to get hold of some of the better plugs without having to spend forever reading reviews and downloading things.
-
- KVRist
- 461 posts since 12 Jan, 2003 from Kyoto
Clearly the current ratings/rankings system is not perfect, but I'd hate to see the baby thrown out with the bath water. Nobody has to look at the ratings/rankings if they don't want to, but for some they're very valuable.
My main complaint with the ratings/rankings is that, as others have mentioned, they are a reflection of the time in which they were written. Synth X might have earned a deserved 9.5 when it was released in 2001 because it was a genuinely groundbreaking. But by 2004, there might be five other synths that do the same thing (maybe better or cheaper), which will get lower scores because they reflect the technology of 2004 -- while Synth X's score is probably still pretty close to 9.5 due to the weight of old ratings, and the fact that people are more likely to review new products. Synth X may have a greater claim to "greatness" than the others, but most people are probably looking for a more equal comparison between products. This also has the effect of influencing future reviews (Synth X got 9s? Synth Z is at least as good -- I'd better give it 10s! -- very soon there's nowhere to go).
There are probably many ways to reduce this problem -- monthly reviews have been mentioned. Another way would be to expire old ratings (say, those more than a year or 18 months old) or to only consider the most recent (x) ratings. Of course, the reviews & their original ratings would still be available to readers; they just wouldn't be used in calculating the "KvRers' favourites" list.
Of course, this currently is probably not tenable because there simply aren't enough people reviewing/rating stuff. Maybe this is partly because people don't think the current system is worthwhile, and partly because people are too lazy to use the current system.
One problem with ratings in general is, of course, the people who try out a copy of a product for a few minutes & judge it prematurely. Or in extreme cases, people who hold some personal grudge against the product/developer, or are shills for the company, etc. I don't think this is a big problem at KvR, largely because the system forces you to really consider your opinion -- you must write a review (not just enter scores); you must enter scores in every criterion, even if some are not important to you (or difficult to gauge). But I don't think that this would ever be a serious problem at KvR -- this isn't amazon.com; it's a niche and people don't come here on a whim. It's also a smaller community, and while you may not recognize every name signed to a review, I really think there's more of a sense of accountability for one's words. Again, there's a defense against crazy ratings -- disregard the top & bottom 10% of scores in calculating the favourites lists.
So I would suggest:
- any criterion that a score can be applied to should be optional -- there should be a choice for "not applicable" or "I prefer not to review this area" or whatever.
- you should be able to submit just scores, without a review (I'm sure many will disagree).
- the ratings could be more meaningful if accompanied by a line where people could explain their rating (EG instead of just seeing "value for money: 7", we might see "value for money: 7 / It's not cheap but its price is what I would expect for this feature set"). Yes, some people do this kind of breakdown in their reviews already.
Anyway, that was pretty verbose for a couple of modest and likely unimpressive suggestions. Sorry about that.
By the way, does anyone else think that a ranking for "most under-rated" is a bit paradoxical? I mean, if Synth Y tops that list, is it really underrated?
My main complaint with the ratings/rankings is that, as others have mentioned, they are a reflection of the time in which they were written. Synth X might have earned a deserved 9.5 when it was released in 2001 because it was a genuinely groundbreaking. But by 2004, there might be five other synths that do the same thing (maybe better or cheaper), which will get lower scores because they reflect the technology of 2004 -- while Synth X's score is probably still pretty close to 9.5 due to the weight of old ratings, and the fact that people are more likely to review new products. Synth X may have a greater claim to "greatness" than the others, but most people are probably looking for a more equal comparison between products. This also has the effect of influencing future reviews (Synth X got 9s? Synth Z is at least as good -- I'd better give it 10s! -- very soon there's nowhere to go).
There are probably many ways to reduce this problem -- monthly reviews have been mentioned. Another way would be to expire old ratings (say, those more than a year or 18 months old) or to only consider the most recent (x) ratings. Of course, the reviews & their original ratings would still be available to readers; they just wouldn't be used in calculating the "KvRers' favourites" list.
Of course, this currently is probably not tenable because there simply aren't enough people reviewing/rating stuff. Maybe this is partly because people don't think the current system is worthwhile, and partly because people are too lazy to use the current system.
One problem with ratings in general is, of course, the people who try out a copy of a product for a few minutes & judge it prematurely. Or in extreme cases, people who hold some personal grudge against the product/developer, or are shills for the company, etc. I don't think this is a big problem at KvR, largely because the system forces you to really consider your opinion -- you must write a review (not just enter scores); you must enter scores in every criterion, even if some are not important to you (or difficult to gauge). But I don't think that this would ever be a serious problem at KvR -- this isn't amazon.com; it's a niche and people don't come here on a whim. It's also a smaller community, and while you may not recognize every name signed to a review, I really think there's more of a sense of accountability for one's words. Again, there's a defense against crazy ratings -- disregard the top & bottom 10% of scores in calculating the favourites lists.
So I would suggest:
- any criterion that a score can be applied to should be optional -- there should be a choice for "not applicable" or "I prefer not to review this area" or whatever.
- you should be able to submit just scores, without a review (I'm sure many will disagree).
- the ratings could be more meaningful if accompanied by a line where people could explain their rating (EG instead of just seeing "value for money: 7", we might see "value for money: 7 / It's not cheap but its price is what I would expect for this feature set"). Yes, some people do this kind of breakdown in their reviews already.
Anyway, that was pretty verbose for a couple of modest and likely unimpressive suggestions. Sorry about that.
By the way, does anyone else think that a ranking for "most under-rated" is a bit paradoxical? I mean, if Synth Y tops that list, is it really underrated?
-
- KVRist
- 72 posts since 22 Oct, 2002 from Rose of the Shires, UK
My concern would be that peeps wouldn't feel inclined to post any reviews if the criteria is made too long.
And yes, the current rating system does allow users to vent their frustration out on the product which is often a reflection of their inability to use them properly, but they are still a valuable factor when used alongside the program review.
Do you really believe the current system is so far out of synch and can be improved? I am not so sure.
Thanks for asking though, appreciate the consultation
And yes, the current rating system does allow users to vent their frustration out on the product which is often a reflection of their inability to use them properly, but they are still a valuable factor when used alongside the program review.
Do you really believe the current system is so far out of synch and can be improved? I am not so sure.
Thanks for asking though, appreciate the consultation
Music is the food of the soul!
-
- KVRAF
- 6496 posts since 26 Nov, 2004 from Frederick, MD
I agree that the current rating system is a bit flawed; however, I would not like to see ratings go away entirely. They make a good quick way of judging people's reactions. And wouldn't doing away with them limit our ability to sort instruments/effects/etc. by user opinion?
My proposal would be to move more toward the rating scales used in a lot of industry surveys. Keep the categories, because they are very well chosen, but instead of having a 1-10 rating for each you do something like:
This Product Is Very Stable
Not Applicable
Disagree Strongly
Disagree Somewhat
Neutral
Agree Somewhat
Agree Very Strongly
In calculating an overall rating, then, you ignore the rating of "Not Applicable" thus the rating is not skewed for low ratings on something like Support when the user hasn't needed any support.
I've also often thought that there should not be an equal weight placed on the categories. Though Documentation and Support are important, to most users Stability and Sound Quality are going to be much more important. I'd way rather spend extra time learning a poorly documented product that sounds fantastic than waste time on a brilliantly documented piece of crap.
As far as the character count required in reviews goes, I'm not sure 2,000 is the right number. I wouldn't mind seeing longer reviews, but in publishing terms 2,000 characters equals 1.6 pages of print per review (250 words per page at 5 characters per word). Do you know what the average review is right now? In browsing through the reviews myself I haven't really been bothered by the size of them. They're generally of a much higher quality than music reviews on amazon.com, for example. Duh, hard to be worse than that.
Something else that might be a little helpful in gauging how much faith to put in any one review is how long the reviewer has used the program (1-3 days, 1 week, 2-3 weeks, 1 month, etc.). Initial impressions might be favorable or unfavorable, but often the test of time is what counts.
My proposal would be to move more toward the rating scales used in a lot of industry surveys. Keep the categories, because they are very well chosen, but instead of having a 1-10 rating for each you do something like:
This Product Is Very Stable
Not Applicable
Disagree Strongly
Disagree Somewhat
Neutral
Agree Somewhat
Agree Very Strongly
In calculating an overall rating, then, you ignore the rating of "Not Applicable" thus the rating is not skewed for low ratings on something like Support when the user hasn't needed any support.
I've also often thought that there should not be an equal weight placed on the categories. Though Documentation and Support are important, to most users Stability and Sound Quality are going to be much more important. I'd way rather spend extra time learning a poorly documented product that sounds fantastic than waste time on a brilliantly documented piece of crap.
As far as the character count required in reviews goes, I'm not sure 2,000 is the right number. I wouldn't mind seeing longer reviews, but in publishing terms 2,000 characters equals 1.6 pages of print per review (250 words per page at 5 characters per word). Do you know what the average review is right now? In browsing through the reviews myself I haven't really been bothered by the size of them. They're generally of a much higher quality than music reviews on amazon.com, for example. Duh, hard to be worse than that.
Something else that might be a little helpful in gauging how much faith to put in any one review is how long the reviewer has used the program (1-3 days, 1 week, 2-3 weeks, 1 month, etc.). Initial impressions might be favorable or unfavorable, but often the test of time is what counts.
-
- KVRist
- 72 posts since 22 Oct, 2002 from Rose of the Shires, UK
emdot_ambients makes some good points. I recently did a Vsti review and found a 2000 word max. word count difficult to achieve whilst putting over a balanced and decent enough view of the product.
And I totally agree with the emdot_ambients idea about using a different form of scoring; the one he suggests is used extensively in industry and can be weighted accordingly.
And I totally agree with the emdot_ambients idea about using a different form of scoring; the one he suggests is used extensively in industry and can be weighted accordingly.
Music is the food of the soul!
-
- KVRAF
- 6496 posts since 26 Nov, 2004 from Frederick, MD
Character count. Character count. Not word count!
Actually I'll take back what I said about the character count. My last post had 1,646 characters not including spaces, and just over 2,000 with spaces. So, that doesn't seem too long for a review. If the reviewer makes any effort at all to address why they rate a product as they do, in each of the categories, plus add a touch of what's cool and what's not cool about the product, then 2,000 characters should be a doddle.
Actually I'll take back what I said about the character count. My last post had 1,646 characters not including spaces, and just over 2,000 with spaces. So, that doesn't seem too long for a review. If the reviewer makes any effort at all to address why they rate a product as they do, in each of the categories, plus add a touch of what's cool and what's not cool about the product, then 2,000 characters should be a doddle.
-
- KVRist
- 72 posts since 22 Oct, 2002 from Rose of the Shires, UK
Sorry, I did mean 2000 characters...thats why I struggled to do justice to a Vsti in my last review. Maybe 2000 characters inc. spaces is okay for an effect but not for an instrument?
Music is the food of the soul!
-
- Mr KVR
- Topic Starter
- 1431 posts since 23 Oct, 2000 from UK
- KVR Audio
Its actually a 1000 minimum and 5000 maximum at the moment, I'm not averse to upping that to 10,000.
Anyway, hopefully these changes will be implemented before Christmas.
Anyway, hopefully these changes will be implemented before Christmas.
-
- KVRAF
- 8388 posts since 11 Apr, 2003 from back on the hillside again - but now with a garden!
of the few reviews I've done, I've struggled to keep within the 5000 limit. I write them in note pad saying everything I want to, then paste it in to the box. Pretty much each time, I have had to go through and take out things.
Somtimes a whole paragraph has ended up being a rather lame sentence, or lost altogther to stay within the scope. I have found this hard, since sometimes there are a few points which I feel are all worth making, but I have to make them in a rather terse fashion.
I don't know what others experience is, but a 10,000 limit may suit better...
DSP
Somtimes a whole paragraph has ended up being a rather lame sentence, or lost altogther to stay within the scope. I have found this hard, since sometimes there are a few points which I feel are all worth making, but I have to make them in a rather terse fashion.
I don't know what others experience is, but a 10,000 limit may suit better...
DSP
-
- KVRer
- 5 posts since 27 Nov, 2004 from Windsor CT - usa
i know i just got here nstuff as far as becoming a member, ive got alot of good stuff from here in the past , im an old fart 33 www.acidplanet.com/ninjatron, check my stuff out , Im an Ex Liquid Sky member , i was formaly known as Megatron 1996, anyways i wanted to say i love this site , my eyes were wide open when i saw what this was . Sooo many people dont even know where to look for effects -you should run this site on some major search engines so more people can find this place , ive done lots of searches for effects / software and never seen this before . - never heard of half thease effects, well , im working an another explosive track , it will be up at acidplanet in a few days -im gonna deticate it to this forum / site whatever ,ill keep in touch , Ben im me ninjatron2@Hotm or Aim take it slow
