Sunday, May 6, 2012

Thoughts on wine judging: Are 3 opinions better than 5?

Since I'm not an athlete, this is my chance to hoist the Stars and Stripes
I'm flying home today from Portugal, where I just spent three days judging about 150 wines at Concours Mondial, Europe's biggest wine competition. I have to brag that I was a Jury President this year, the only one from the USA.

In past years I just worried about my own opinion of the wines. I enjoy discussing wines with other judges, but this year, because I collected the scoresheets, I really noticed how divergent the opinions of even wine professionals are.

My panel was also something of a tower of Babel, with five different first languages and no language fully shared by even three of us. That made it harder for us to influence each other, so our opinions were more pure.

It's not news that two people can taste the same wine and think differently about it. But when it comes to awarding medals, a pattern quickly emerged to the detriment of wineries that entered wanting one.


Two of us had similar palates. Two others had similar palates to each other. And the fifth judge was very different from any of us.

If group A liked a wine, that could be two votes for a gold, but then group B might not like it, and vice versa. On all of the rare occasions when all four of us liked a wine, the fifth judge did not. So I don't believe we awarded a single gold medal out of 150 wines, although the Concours uses statistical analysis to eliminate outliers and may be able to fix that.

All of us thought there were some gold medal worthy wines -- we just didn't agree on what they were.

If there were 3 judges, it would be easier for two who agreed to discuss the wine with the third judge and come to a consensus. But with 5, it's really hard.

Maybe the Concours wants that. Wine competitions lose respect if they give too many medals. One reason I love judging at it is that we taste so many wines we might never otherwise encounter. This year I enjoyed Rieslings from Bulgaria and Slovakia. In past years I loved a Brazilian Cabernet Franc and Viognier and discovered the high overall quality of Luxembourg Riesling. But realistically, if you taste a flight of 15 wines from an emerging wine country, you can't expect to give eight golds.

The Concours had 7,050 wines entered last year and gave only 703 gold and grand gold medals combined, along with 1,329 silvers. Giving 10% gold and 19% silver keeps the medals meaningful. So there's a different side to my argument.

However, there are winery owners in Rioja and Vinho Verde who won't get gold medals I think they deserved because the 5 of us couldn't agree, and even though being president of a jury with four Europeans felt like being a NATO general, I didn't feel it was my place to impose my will.

I would like it if you would address comments to El Presidente.

***

Last year I got in a Twitter discussion with Bruce Schoenfeld and Jamie Goode about which is worse: bad California Cabernet, or bad Bordeaux. Jamie asserted that we don't get the worst Bordeaux in the States and that its bottom end is lower.

We had two flights of Bordeaux and on the last one, we thought the wines were Pinot Noir from a country with bad hygiene.

Memo to Jamie Goode: You're right.

***

A guy from Quebec sat next to me at breakfast on the first day and said he'd just been to California, driving up from Santa Barbara ("did you know the wineries in Sideways really exist?") through the Santa Cruz Mountains to Napa. He liked Bonny Doon and said he had a Pinot Noir from Marin County that he liked.

He also said, "I had a California Cabernet, it was so oaky and bad, I couldn't drink it."

Me: "Which one?"

Him: "I don't remember."

The punchline is that he made this trip to write a book about California wine for Quebec readers, and he is already finished.

Memo to Quebec: Nous avons plus d'un Cabernet en Californie. (We have more than one.)

Follow me on Twitter: @wblakegray and like The Gray Report on Facebook.


9 comments:

Mike Dunne said...

Not a single gold medal out of 150 wines while the rest of the judges awarded gold at a 10-percent clip suggests the language differences played a significantly negative role in your deliberations, that your panel was exceptionally unlucky in the classes you were assigned, or that the panel preferences of your panel were so divergent and obstinate that it wouldn't matter whether you had three, five or seven judges chiming in.

Ray Krause said...

El Presidente,
After over 25 years a judge for the Orange County Fair Wine Competition in California, I do have some observations. The OCFWC is not a concensus judging. The five jurists per panel are asked to score each/all wines and also recommend (or not recommend) a medal. The scores and recommendations are then weighted/averaged by an objective external awards committee. It's a clean system wherein strong opinions do not drown others during the tasting.
Problematic to all wine competitions is the spectre of personal bias which is not at all the same thing as personal experience or skill. If the apple pies or roses at their respective competitions were to be judged as we do wines, Mom's pie and Grandma's roses would always be the marque....whether they were actually entered or not. Judges bring with them the illusion of the perfect wine ("Chenin Blanc cannot ever be of gold medal quality because Chardonnay is the best"). I argue that a wine competition is supposed to be about those who have gone to the time and expense of entering the competition and not some phantom product which has, more than likely, morphed over time. If there are three wines (or pies or roses) in a given class and none are flawed then there must be a Gold, a Silver and a Bronze winner no matter the judges' biased stytle preference or convoluted taste memory. Consequently, it is not reasonable, fair nor accurate to have a class where any medals are given(no flaws)and have no gold medal or blue ribbon awarded. Absent personal ego, the best wine, pie or rose on the table that day is, by definition, the Blue or the Gold by default.

SUAMW said...

The answer to your titular question is a resounding "No".
That's because you're looking at opinions, not empirical findings determined with similarly trained investigators using the same criteria and rubric to categorize findings and observations.
This is not physically impossible, it's just that if you tried to take all the self-anointed wine "experts" and try to get them to agree and adhere to an SOP, everyone's *opinions* would get in the way and you could not gather enough people to make up a panel.
Which leads me to the titular question in Jo Diaz's blog post today and my answer is: "use it for target practice".

Francly Speaking said...

El Presidente - you are a star athlete.
I have judged both wine competitions and beer competitions - tough duty.
You can hold that flag up high

Anonymous said...

Can you make any recommendations on how to get ahold of these Luxembourger Rieslings and Brazilian Cab Francs you liked so much? It seems pretty hard to find this sort of off-the-wall, far-away wine even with importers who bring in unique stuff. Are there any you can recommend who specialize in the truly hard to find?

Charlie Olken said...

El Pres Blake--

I don't see the benefits of three versus five when it comes to medals. Either the panel is in sync or it is not. I have sat on panels of two and panels of five. Nothing helps is the panelists can't talk to each other, and by that, I don't mean language differences but palate and expectation differences.

The note about the guy from Quebec is both instructive and hilarious. I love it when folks from other countries come here for brief visits and write books about California wine. I like it even better they go home, write learned articles in Decanter and get the bad vintages as good and vice versa because they think the only good CA wines are the ones structured like the masses of cheap European wines they are used to tasting.

This is serious business, this wine judging that we do, and while I often find myself in disagreement with your commenter, SUAMW, I do think he is on the right track when he says that it would not matter how many tasters you put on the panel if they start with radically different or unknowing viewpoints.

W. Blake Gray said...

Mike: We were a little unlucky in classes of wine, I think. Our best group of all was some non-appellation wines from Spain, and that should tell you something.

Ray: That's not the way the Concours works. I wrote about it in more detail last year: http://blog.wblakegray.com/2011/05/narrow-range-of-scoring-wine.html

The upshot is that we probably DID award some gold medals, but it wasn't clear to me by looking at the scoresheets, as there was no consensus.

Anon: There's just no reason for a Luxembourg winery to sell its wine here, or for an importer to buy it. That's a wealthy, thirsty market and they drink all their wines themselves.

You would buy them, and I would, but there aren't enough customers like you and me.

Anonymous said...

Yeah... I figured that'd be your answer about Luxembourg. I guess I'll just have to go over there and smuggle some back. I was reading some statistics about their wine import/export and it looks like they don't really import any New World wines to speak of either.

Probably not much interest in importing Brazilian wine either then.

Lisa said...

El Jefe (I like the Springsteenian element),

I have a Slovak acquaintance who also likes only Rieslings. Do you recommend any? How are they on the sweetness scale?