Reviews?!! Are they?!!

[Robmobius]

I'd replace the 1-10 scale with some descriptive phrases:

1. Worst plugin possible, wouldn't even consider touching with a ten foot pole
2. Had significant trouble (bugs/crashes/aliasing/...), plugin has potential but is currently unusable for more serious use in a project
3. Many issues to complain about, barely usable
4. Has some issues, but works reasonably well
5. Average, works well with no major complaints
6. Great, has some features that are above average
7. Excellent, many features are above average and includes additional features not often found in plugins
8. Brilliant, far above average, absolutely stable, not a single complaint
9. Near perfect, top of its class, hard pressed to think of anything that could be improved
10. Perfect, so amazing it leaves the user at a loss for words, it would be impossible to improve upon this epic masterpiece.

Keep in mind that 10 should never be used, it is the same as a zero score which would be "not even a plugin, attempted to load mscomctl.dll as a VST plugin and was grossly disappointed."

So, equivalent strengths:
1 = 9
2 = 8
3 = 7
4 = 6
5

This is a really good idea!

[EnGee]

The idea to categorically rate plugins makes a bit of sense, but it might be difficult to come up with categories which apply to every possible type of plugin.

For example it would be difficult to rate the subjective audio quality of a plugin oscilloscope.

If such a feature were to be desired it would make most sense for each category to be optional.

Once this consideration is taken into account it is more difficult to argue that we should need such a system as opposed to simply allowing good detailed reviews to be written by hand (with bbcode perhaps?) by the reviewers.

It can be also one form for all, but the field must be explained, for example, "Built Quality" for Hardware might be the same as "Stability" for software, "Sound" for synths can be the same as "Audio engine" for hosts ..etc. So, with some flexibility it can be done IMO.

[aciddose]

It could be done, but by whom? I don't agree the extra effort to creating and maintaining such a system is worth it.

From Ben's perspective I suspect it would not be even close to worth it, as the investment required to change the format of the database and all the associated code is much larger than what may be apparent to someone unfamiliar with such systems.

There is nothing to prevent reviewers from using the existing text fields to enter this data themselves. Additional bbcode-like commands could be added to produce graphs or other images and formatting and existing factors such as the 1-10 rating can likely be changed with minimal effort to display strings without actually changing any of the data in the database or any code other than that used to display the data to the user.

I'm not arguing that it is a bad idea, just that to actually have such a system is a lot more complex than you may realize. My arguments are purely technical.

[EnGee]

I'm not arguing that it is a bad idea, just that to actually have such a system is a lot more complex than you may realize. My arguments are purely technical.

If KVR can't afford such a change in the code, then at least it can afford putting some rules (a guide maybe) and clean all the reviews (or is that also more complex than I realize?).

[aciddose]

That seems like a huge amount of work. Yeah, I'd think considering the raw number of reviews on the site and number of plugin entries in the database it may be considerably more effort than all the programming work to change the format of the database.

My point is really just to say that maybe there are other ways you could come up with to get the same results you're looking for?

I think a guide is a good idea also, a wiki already exists although the issue is just who exactly will write such a guide? What rules would be put into effect? How would they be enforced?

[EnGee]

There is a guide in almost every forum or subforum listing in general the rules of posting. Who wrote them?

The only enforcement can be (in case of not changing anything that costs) by KVR members themselves like in the market place. If 'we' see someone that bumped his post in less than 48 hours, one or more of the members write something to the poster reminding him/her of the rules. I know what you will say, that brings me to...

The rules should be clear, easy to understand and not flexible (to avoid clever playing with words ...etc). The shorter and more concise the better to memorise.
There are already comments in the reviews section but they are not belong to a specific review! I don't know how you solve that without code. I don't see the point of not organised comments.

[Slaapstadseun]

Here is a KVR review of Reaper:

"Brilliant piece of software... Learn the reaper way of doing things and chances are you wont miss much from any other DAW. All the tools you need to make a pro mix are there, and the modulation routing makes it possible to create advanced Automations from audio signals... Vocal riding, side chaining, etc.

Highly underrated by many... And loads of free tutorials online to help you get started."

There ya go - a DAW reviewed in just 3 sentences.

[el-bo (formerly ebow)]

This is a really good idea!

it absolutely is, and it's something i have been building up to tackling myself (you did it better )

as to the idea of it being too much work to implement ?? i'd like to think that the reviews are not there just as filler for the side-bar, and that ben would be interested in them becoming a more useful tool than they currently are

if they aren't to be improved, then maybe just get rid of them. currently, they are no use at all. like the op, i don't believe in the concept of 10/10. there is always room for improvements, refinements etc

the scale idea is solid, and should be adhered to, on pain of death

[EnGee]

Here is a KVR review of Reaper:

"Brilliant piece of software... Learn the reaper way of doing things and chances are you wont miss much from any other DAW. All the tools you need to make a pro mix are there, and the modulation routing makes it possible to create advanced Automations from audio signals... Vocal riding, side chaining, etc.

Highly underrated by many... And loads of free tutorials online to help you get started."

There ya go - a DAW reviewed in just 3 sentences.

This is a nice "Conclusion" but it is not really a review. A review as I understand it (and I'm not a native speaker), is like an essay more or less. There is introduction, paragraphs with single idea or subject, then a conclusion.

Ok, I don't want to complicate things, but few sentences is just a "chit chat" for me and for some others. Writing what the pros, what the cons, the reviewer opinion, some facts ..etc. All that is good information. I mean who said a review is a simple thing to do?

On the other hand, it can continue like now with few words and sentences reflecting the 'user' opinion and experience, but I thought the forum topics are for such needs!

Actually, if I want to read about a product, lets say a synth, I don't read the KVR reviews! I read the topics in KVR because they give details about the real life usage of the synth. However, ten years ago, I always have read the reviews in KVR because they give so much details that I don't have to pay for Computer Music or Sound on Sound and hunting this synth's review! IMO, ten years ago, the reviewers knew what a 'review' means

[Doug1978]

In truth, it would be far better to do away with the scoring system of numbers (or alternatively alphabet grades) altogether.


As this is a leading music website, reviewers should instead create a jingle or other interpretative musical 'perfume', to sum up their feelings about a product.
Something like a KVR scratch and sniff, if you will.


Anyone visiting that comment in the future could then also leave their own musical 'marking', to show exactly what they thought about it.

[fluffy_little_something]

Basically there is a relative rating and an absolute rating. Relative basically meaning what you get for the price, and absolute meaning the verdict if you did not know the price.
I remember writing a few reviews in the past, one was of a cheap SE synth. My relative verdict would have been 9, the absolute only 4 or 5.

[sjm]

The idea to categorically rate plugins makes a bit of sense, but it might be difficult to come up with categories which apply to every possible type of plugin.

For example it would be difficult to rate the subjective audio quality of a plugin oscilloscope.

If such a feature were to be desired it would make most sense for each category to be optional.

Yeah, the KVR reviews used to be like this and then suffered from exactly this problem: people didn't know how to rate certain areas and unfortunately you couldn't opt out. How do you rate customer support if you've never used it? How do you rate the sound quality of a plugin that doesn't make a sound (e.g. a MIDI plugin, or Zen, which I once reviewed)?

Some people would vote 10 or whatever their average rating was so as not to drag the ratings down. Some people went for 5 or 6 to make it neutral. I guess some may have gone for the lowest rating. At any rate, it ended up skewing the ratings because the average was taken as the final score.

Rating a plugin over multiple categories does partly explain why the older reviews are not all 10/10 scores, so there's an obvious benefit to a more gradiated marking system.

Unfortunately the whole internet is suffering from bipolar review ratings - metacritic etc. are no different. Most scores are either 1 or 10, which cannot be true. Not everything can be the best/worst thing evah. As a result, I tend to skip the extreme reviews on sites like metacritic, and read the more realistic ones to get a truer picture. At least at KVR I understand why plugins generally get high ratings - users are more likely to review plugins they like than ones they tried out, didn't click with and never used again.

[Sendy]

Regarding giving number ratings to products, I always find that a very open-ended thing.

Scale from 1 to 10. Ok, we all know that 1 is worthless and 10 is perfect (whatever that is? It's a bit like infinity, you can always think of a higher number, just as you can always code a more impressive plugin - it all depends on expectations and as plugins have been getting better at an alarming rate, that might explain the skew towards 10).

But the points between 1 and 10, how are they divided up? Some people think about it linearly, some logarithmically. Some put 5 as "average", some 7. Some factor in the general cultural trend of higher marks (only using the upper 50% of the spectrum), others try and combat this. But if you choose to do this, you're misrepresenting every plugin you review (which will probably be good ones you want to give props to), as you're throwing in a "normal bias" rating into a sea of "high end bias" ratings. When you throw in your mark are you not placing it AMONG the other marks, relative to them, rather than in isolation? One could argue the notion that "a mark of 10 should never be used until the perfect plugin is finally released at the end of history" has it's problems, as it is somewhat exponential. Do we really want a curve that shoots off to infinity, or is it more sensible to think that there ARE a few plugins that fall into the top 10%?

One option is just to accept a logarithmic scale as normal and be done with it. 6 as average, 7 good, 8 excellent, 9 amazing and 10 for (apparent) near perfection. 5 and below for panning... Though in 2 years plugins will be better and computers will be faster, so no matter what scale we're using, we will run out of room and have nowhere to go.

My thing with reviews is that it should tell us not just what the plugin can do, or rattle off specs and features like a manual, but rather the experience of using the product should be described subjectively - What is using this plugin like? How does it make you feel? The emotional side of the equation. How did you respond to the sounds and workflow? How did it assist you as a musician/player/composer/arranger/soundmurderer?

At the end of the day, slapping a number on a review is a formality. The words are what's important, and getting the ratio of clarity to wordcount is an artform in it's self.

Maybe we should start reviewing the reviews?

[aciddose]

On a log scale 6 wouldn't be average, it would be 3...4. Why not just use the scale I propose?

The only advantage to skewing the distribution is to use more of the values to provide higher resolution.

A log scale would provide more "above average" values by sacrificing "below average" values.

You'd have only:

1. Unusable
2. Many issues
3. Sub-par
4. Average
5. Above-average
6. Great
7. Excellent
8. Brilliant
9. Near-perfect
10. Perfect

(If you can't see the difference from what I posted earlier, I'm not surprised. It is hardly a difference. Using the original table would make far more sense than a skewed table with 4 rather than 5.)

Ideally the distribution of values should be normal using my system. If it were shown that the majority of ratings were +7, we could algorithmically deskew the ratings and set those all to 5 by warping/adjusting of the distribution to make it look more gaussian.

Ultimately I think the numbers are the problem. The solution is to provide descriptive phrases rather than single adjectives or numbers/letters. People very stupidly think it makes sense to use an exponential distribution, all they've done is provide very little resolution surrounding "average" while placing all the resolution at the bottom end of the scale.

I really don't care that much if a plugin is "slightly unusable" or "completely unusable", I just want to know if the user rated it "unusable" or "usable". We don't need several graduations of "total shit" to know something is in fact total shit.

Exponential distributions for ratings are +5 total shit.

[aciddose]

Maybe we should start reviewing the reviews?

That isn't such a bad idea. A rating system for the reviewers would allow us to moderate ourselves.

The consequence can be an echo-chamber where "unpopular" ratings or views are suppressed.

There are plenty of examples of such moderation systems on the internet.