Monday, May 16, 2011

The narrow range of scoring wine

I love representing the Stars and Stripes abroad.
The Concours Mondial bills itself as the world's leading wine competition, beginning with the name. It has developed the most sophisticated system of judging in the world*, a system that always makes me think deeply about issues regarding the rating of wine.

* The Concours uses statistical analysis to try to track and respond to individual tasters' quirks. It's worthy of its own post.

The biggest this year, for me, was the narrow range of scoring. Everybody knows that American wine ratings have become compressed into an 85-to-100 point scale -- and American wine ratings are now the world standard, like it or not. The Concours system shows the foolishness of that, while encouraging some Parker haters into using an even more compressed range.

I also got to thinking about the standards we apply to ratings, and how easy it would be for a powerful critic like Parker's assumed successor Antonio Galloni to change the game of wine by changing the categories he respects. I'm going to address these topics in a separate post later this week.

Click on the photo for a closeup to see the categories.
At the Concours, while each of the 50 wines we taste each day are supposedly rated on the 100-point scale, in fact the ratings are divided into 10 categories (11 for bubblies), with a series of boxes for us to check. In theory, we're supposed to consider many different aspects of the wine: its intensity, "genuineness" and finish, for example.

I may write transgressively, but in games I'm a rule follower, and I enjoy checking the boxes where I believe the wine belongs. This leads to lower scores than I might give the wine at home, because I'm reluctant to give the highest possible score in most categories (visual categories are the exception). A wine with the second-highest score in the smell and taste categories, and perfect scores in sight, would get an 89, and that would be a wine I would enjoy drinking: good in every way.

Fittingly, an 89 score, if shared by the other judges, would earn the wine a gold medal. Isn't that accurate? Some years ago Thomas Matthews, executive editor of Wine Spectator, told me he enjoyed drinking 89 point wines with dinner. You won't find a man more knowledgeable and thoughtful on the difference between 89 and 90 point wines that is so crucial to their marketing.

I really liked some 85 point wines.
Yet it's impossible to convince most Americans that 89 point wines are worthy. Robert Parker long ago responded to this by adding about 4 points to each of his reviews; his 93 pointers are what used to be 89s. I have and will continue to criticize him for grade inflation (What's a 97+, Bob? Is it a 98 or not?), but it's a defensible way to tell consumers, "Mmm. Yummy wine."

Now, a provincial person might say, "That's why the 100 point scale doesn't work." You'd be wrong.

The Concours brings together judges from around the globe; 50 countries I believe. The lingua franca is French, which many of us don't speak. You can meet the leading wine critics from South Africa, Poland, Brazil, Chile, etc. My panel had a retail buyer from France, a winemaker from Luxembourg, a wine critic from Belgium and a bulk wine buyer from Spain, and the way we relate to wine was very different. (The winemaker would praise boring, unflawed wines and hated even a whiff of brett; the bulk wine buyer was very generous to wines the rest of us dismissed.)

How do these citizens of Babel discuss the merits of a wine? In shorthand, by saying, "It's an 87 for me." All of us. Most of these people don't use the 100-point scale in writing about wine, but for quick communication, the point rating is universal. I don't really know what a French guy means by un bon vin (apparently that means "meh"), but I know what he means by 84.

Moreover, of all the judges I spoke with -- and I spoke with dozens -- I'm the only one who fills in the boxes and doesn't worry about how the points add up. All the others, many of them Parker haters, decide that the wine is an 88 for them and then figure out how to make the boxes add up to that.

I didn't even keep track of all the ratings I gave. I was cautioned twice by the head of my jury on our first group of wines -- Muscadets-Sèvre et Maine -- that my ratings were too high, something you don't expect to hear at a competition. (At some competitions, directors walk through the room saying, "Think gold." Competitions make money from wine entry fees, and more golds this year should mean more entries next year.)

I don't care what he said: I like Muscadet, and I wasn't giving them scores in the 90s anyway (I checked). A little later that same day he told me my ratings on some red wines that I hated were very low; he didn't say "too low," but he said, "That is your style." Then I noticed that other judges on my panel, who were keeping track, were scoring EVERY wine between 81 and 89 points; an even narrower range than Wine Spectator. I think this is a natural consequence of thinking holistically about a rating, instead of about its components.

My highest rating out of 150 wines was probably 92, and I blame the wines themselves for the fact that I wasn't more generous, as the Muscadets turned out to be the best category we got (we had some awful flights, including erratic white wines from Crete and Carmeneres from Chile, which taught me that "smells like Brussels sprouts" doesn't translate into Spanish or French) But my lowest score for a non-flawed wine was probably about 69; that would be a wine that had no visual flaws, was average in three categories and below-average on the rest.

When was the last time you saw anybody rate a wine a 69? And yet, isn't that reflective of the wine experience? It's not a complete failure, but it's deficient. You could drink it if you had to for some reason, but you wouldn't enjoy it.

It made me want to come home and rate wines in a wider range that reflects the way we really perceive them. An adequate wine, drinkable but dull, should be a 75, not an 85.

Yet rating scales exist as a form of communication, and as long as a 75 means "Oh my God did you see what they rated that wine? The winery must not have paid for advertising in the magazine," I can't really use it.

But maybe we should reclaim the 70s and 80s for wine ratings. To do so, we have to acknowledge that high-80s wines are really good, really likable wines. B students, if you will. And that we don't need an A every single time we open a bottle. And that even a C is not a disaster (though I wouldn't order one if I knew a B was available.)

How we start this, I'm not sure. Maybe it begins with consumers telling their local shop that they're perfectly happy with 85 point wines. And maybe THAT begins with consumers accepting the concept for themselves.

Could you eat a great lunch every single day? A 3-course meal, with fancy flatware and impeccable service? I couldn't; after a few days I'd crave a shawarma sandwich or carnitas tacos or a banh mi. Wine is the same. Sometimes goodness is more appealing than greatness.

Feel free to rate this post in the comments on the expanded 100-point scale. And come back later this week for a discussion of what the Concours' categories are, and what maybe they should be.


Mike Dunne said...

Intriguing post, Blake, and I look forward to reading more of how this competition is run. Was the 50wines a day the maximum each judge was expected to evaluate? If so, it seems a goal worthy of emulation by other competitions, given that palates flag when they are exposed to more than 50 wines a day.

W. Blake Gray said...

Mike: Yes, the 50 wine limit is one of the best things about this competition. 50 Cabs, for example, would still be challenging, but I've never had a day of only red wines.

Anonymous said...

Blake, instead of consumers telling their wine shops that they're "perfectly happy with 85 point wines", why don't consumers tell their wine shops to get a clue and stock wines that are both appealing and diverse. Why the number? I feel like you're right on the cusp. In your posts, you make good, logical points (no pun), and then your conclusion is that you want to be able to rate a wine 75 points instead of just 85 points. It's dizzying. Obviously I know you defend the point system, and obviously you know I oppose it. If you won't join of your own volition, I just hope we can make the change in the industry that will render the point system irrelevant. You are helping by educating consumers through this blog. Please, keep it up.
Danielle Betras, Hedges Family Estate

W. Blake Gray said...

Danielle: You'd think I'd come back from Europe believing the 100-point scale isn't necessary. But it has become the international standard in communication about wine, like it or not. And that makes sense, because it translates into every language and is instantly comprehensible.

The problem isn't the scale, and it's not even the critics using it. It's the wine distribution system choosing an extremely small number of critics.

Jancis Robinson could double the great services she's already given the wine world if she would start using the 100-point scale tomorrow.

1WineDude said...

Reading this made my head ache, bro.

I mean, this concentration on math and numbers is boring and sucks some of the joy out of wine. I'm not saying it's wrong, or that there's a better way to execute the competition. But the joy-vacuum is on and is making a loud noise right now...

I guess the human need to categorize, compete, compare and dissect into portable spreadsheet format is strong indeed. :)

Kurt Burris said...

Blake: Great article and please follow up with more information on how they run the competition and evaluate the judges. Do you know if they would allow the judging forms to be used in an educational/staff training format?

Anonymous said...

The 100 point system is clearly an American system stylized for the impressiveness of numbers. It is a consistency that runs through so many areas. In US sports, soccer is considered boring because any sport that can have a 0-0 result must be dull and uninteresting. If the American Football system of scoring was adopted into soccer, with 6 points for every goal which would automatically be followed by a penalty kick that would be worth and extra point, and suddenly soccer scores could be 34-22 interest would grow instantly. If nascar races were forced to have cars that reached a maximum speed limit of 55 miles per hour no one would be interested even if the challenges crafted into the sport made the results as unpredictable as they are now. But 200 miles an! That is sexy and exciting!!!!

That is how the judging of wine has evolved. The evaluation of wine is not about quality, character, quantitative description or varietal styling which would offer some frame of reference by which to make an educated decision as to whether one might like the wine. But 95 points!!!!! Wow!!! This wine must be something special. A gold medal!!!! Spectacular!!!! It is all about impressiveness and trying to push your adrenaline button. If a wine were to be rated a 4.......who would care. Even if you tell them it is a 4 on a scale where 5 is one cares because it is just a measly.......4

W. Blake Gray said...

Joe: Take two acetaminophen and email me in the morning.

Kurt: I will have some more on the system in a later post, but I'm not going to get deeply into the statistics of it with Joe's head in such a fragile state; it took a 30-minute presentation just to explain it to us. You can probably find more on the Concours Mondial website.

Anon: You're absolutely right, a 0-0 tie is the most exciting result imaginable, as would be auto races with a maximum speed of 55 mph. How can sports fans not understand this?
You must be a ton of fun to hang out with.

pamela said...

What kind of nut plays with the boxes to get it to represent a fictitious number? If there are rules, follow them--Let the Xs fall where they may!! and let someone else add up the points.

Anonymous said...

The rating system equals safety, that's why it works. Granted, any other rating system would probably work just as well, but we appear to be stuck with the 100 point scale. If you know little to nothing about wine you just grab the nearest wine that Whatshisname gave 90 points and you're good to go. There has to be something wrong with YOU if you take it home and hate it.

I figure the wine rating system will eventually become a 150 or even 200 point scale anyways. Someone is going to rate a wine 101 or 102 points because they think it's so amazing and then all bets are off.

Anonymous said...

Either the wines in the competition were average or the judges were afraid to stand up for what was above and below average. A judging like this is supposed to provide an average based on the number of judges. You'd always hope the cream would rise, but like art, food, etc., it's a question of taste. One persons favorite is another's worst and as it should be. No judge is perfect, but when you find one who you can agree with most of the time I'd say that's a plus. It could be any scale as long as you're talking the same language.

Anonymous said...

Video game ratings have the same problem, the mark for an "average" video game is 70/100 which strikes me as ridiculous.

An average wine should have an average mark, 50/100 should be a common score in wine reviews.

W. Blake Gray said...

Anon: I wasn't aware of that standard for video game reviews. Bully for them!

You may be right statistically, but the easy understanding from school that 70=C is something I could live with if we could get there with wine.

Wilfred Wong said...

Hey Blake, As you know, I judged this event with you last year and I found it frustrating because there is a noticeable uneveness to the scores given by the various judges on my panel. I, too, filled out all the boxes, but the system was trying. The event has it merits and from my point of view I am not sure how well the medals translate into how great or now not so great the wines are. I enjoyed meeting wine judges from all over the world. If I recall the US had five members. I do agree with you that the range of scores is pretty narrow.
Cheers, Wilfred

Anonymous said...

interesting scoresheet, could you post a blank copy? i was thinking if you kept the categories, not sure about genuiness, but take the points off and let the judges score the wines while the tasters taste you might remove some stress.

Patrick said...

Nice Post, Blake. You really hit on a problem. As a possible solution, I like the CellarTracker scale; when I think in those terms, I give a far lower percentage of wines 90+ points than Parker does.

W. Blake Gray said...

Wilfred: I am your fan. I remember that you could not only rate each wine, but take tasting notes AND photos of each. It's all I can do to keep my concentration just to rate them.

Only 3 Americans this year, you should have come.

What I like about the system is the concept of it; the way that it tries to address essentially all of the complaints about wine judging. My experience with the wines that have gotten medals out of my panels has been that they've been very good wines, and I think that's the key to giving medals in the first place. There's a competition I have also judged at that I won't name that awards courtesy silvers and even golds to wines I thought were undrinkable.

Anon: They wouldn't let us keep any blank scoresheets. The one I have is the "calibration wine" they give us at the beginning of each day; a wine we all taste together and briefly discuss.

Patrick: CellarTracker does seem to give more scores in the 80s than other organizations. But I have personal knowledge of wine PR reps going on CellarTracker and giving high scores and glowing descriptions, and there's no way to tell who's who.

michelle said...

I haven't commented here before. I was linked here because of a blog award you were nominated for (congrats!). I really like you're view on the rating system. I agree that it's become inflated. Someone else referred to video games but I would liken it to Olympic scoring with gymnastics ( or was it ice skating), which just recently had to change the scoring system because of a similar type deal.

BikrDave said...

I don't think I'll ever grow tired of thoughtful posts that challenge the 100 point system, or at least challenge us to think about it. So this morning finds me chewing (sipping?) on the "what to do about it?" problem. And, alas, I don't think the answer is to tell your local retailer to carry more 85 point wines...seems to me that, in the current "gotta have 90" environment, that's a quick and easy step to lower sales.

In my 'hood (Venice, CA, for those keeping score at home), I have two local retailers that I probably frequent more than others. One of them has a ton of inventory, very good prices, and - inevitably - someone behind the register who will be shoving a bottle in my face and quoting Parker scores. The other is much smaller, definitely not trying to compete on price (but nevertheless has plenty of wines in the $10 - $25 range), and nary a Parker score or "shelf talker" in sight.

Wanna guess which one I try to send my friends to?

Anyway, the quick - and maybe too easy - lesson I draw with not enough coffee in me is this: if you're a retailer with limited shelf space, a loyal customer base, and a knowledgeable staff, don't talk scores at all. Sure, people will come in with WS Top 100 lists and RP scores, and maybe you'll even have some of the wines they want...but once you start to use scores as a selling point, I think you lose any hope of trying to convince a broad based of buyers that 85 (or even 89) is ok.

Maybe when I have a wine store of my own, the sign above the door will read, "Abandon hope, all ye Parker devotees who enter here."


- dave

W. Blake Gray said...

BikrDave: While I completely agree with you on your choice of wine store, it's important to divorce the 100-point scale from Robert Parker. He may have invented it, but he's not the only one using it. In a few years he'll be retired but the scale will live on.

Donn Rutkoff said...

Ten boxes? That is hard to swallow. I think 5 factors is all it takes for commercial judging. You can add more if doing a chemical analysis, but what is the point of that for a commercial consumer audience?

I like the +2, 0, -2, method, but it is not sexy enough, I guess.

I wonder, of the 500 million cases or gallons consumed in the US, what % are actually bought based on the score, not on price and simple repeat buying? Nobody cares what score KJ or Toasted Head Chard gets, or Menage a Trois red, or Beringer Founders Estate Cab. These are the biggest selling wines in the US. Scores sell magazines and newsletters. They don't sell wine, on an annuity year in year out basis.

W. Blake Gray said...

Donn: Scores are a great way for wines people have never heard of to get attention. I could list many more wineries than you have that have been completely established by scores, and not just the Screaming Eagles of the world. Same with gold medals, although in the US those are much less effective. When you talk about Toasted Head buyers, you're talking about a different class of consumers -- a large and important class, but not one worth pursuing for small wineries.

Donn Rutkoff said...

Hi Blake. Yes, a big score can get a new winery on the map. Did the winery make wine for love itself, or to get the score? How long will the winery stay on the map? Many newcomers to Napa with enough $ come, buy in, make a few vintages,and then, go. We both know that there are many fine wines with 85 points or no points at all. Commercial success comes from hard work and work in the distribution and sales, and not by making a hi point wine.

I like the +2 to -2 score system, it is only 5 choices, rates negative on chemical flaw, and I think it serves the consumer better. But, as I said, it ain't sexy, it does not sell newletters or magazines.

I wonder, do the hi point brands actually make a profit or not?

I would like to see someone do a time series study, looking at past 10 years of 95+ WS or WA Calif. wines, and see how many brands were selling well 5 years after the 1st hi score. I bet Andy Beckstoffer could tell all of us the history of high price, high point, failures. But he is not prone to such scribbling or such, at least, not maybe til he retires and decides to write a book.

W. Blake Gray said...

Donn: I have in my notebook the following, scribbled when I was bored at a seminar: "Good blog post -- who lost the most money ever on a winery?" My guess would be somebody in Napa who probably actually did get some good scores along the way. The reason I haven't written the post is I can't figure out how to research it.

Bob Henry said...

"Who lost the most money ever on a winery?"

No need to guess.

Consider Blake's 2010 blog titled "Kenzo Estate: Money well squandered"


It is now 2017.

I don't see any traction in the retail marketplace for these wines. Their all-star team of David Abreu and Heidi Petersen Barrett hasn't delivered the ratings or the clamor from collectors to be on the mailing list.

Given winery founder Kenzo Tsujimoto's personal fortune, he is the Charles Foster Kane of wine.

I can't wait for him to exclaim:

"You're right, I did lose a million dollars last year. I expect to lose a million dollars this year. I expect to lose a million dollars next year. You know, Mr. Thatcher, at the rate of a million dollars a year, I'll have to close this place in ... 60 years."