Wednesday, February 16, 2011

Watson's Unfair Buzzing Advantage on Jeopardy

By now you've probably heard about the exhibition Jeopardy! match between the IBM-designed machine Watson and the two most heralded players of the show's history.

After watching the constant frustration on Ken Jennings' face, I was confident that all the players knew most of the answers. It was just a matter of who buzzed in first, with Watson doing so most of the time.

Check out this description of the Jeopardy! buzzer, from Jennings' Web site:
If you watch Jeopardy! casually, it's easy to assume that the player doing most of the answering is the one who knew the most answers, but that's not necessarily true. All three contestants, after all, passed the same very hard test to be there. Most of the contestants can answer most of the questions. But Jeopardy! victory goes not to the biggest brain—it goes to the smoothest thumb. Timing on the tricky Jeopardy! buzzer is often what separates the winner from the, well, non-winners, and the Jeopardy! buzzer is a cruel mistress.

Here's how it works: the buzzers don't get activated until Alex is finished reading each question. If you buzz in too early, the system actually locks you out for a fifth of a second or so. But if you're too late, the player next to you is going to get in first. Somewhere between too early and too late is a very narrow sweet spot, like swinging a tennis racket or a baseball bat. No, that's not right. The Jeopardy! buzzer, she is like a woman. No, that's not it either. All I know is, the more I thought about the timing, the less I could nail it. When I could somehow just Zen out and not think about what I was doing, I would do okay.
Watson won primarily because it had first dibs on every question it pleased. I'm much less impressed by this victory because it involved a machine "hitting a button" much more precisely than any human ever could.

Yes, it's still impressive that the machine can perform so well on Jeopardy!-style questions, but the lopsided final score shouldn't imply that Watson is vastly superior to its human counterparts.


Adam Gurri said...

By that logic every Jeopardy winner--including Jennings himself--is less impressive, because they just happened to have superior buzzer-timing skills.

Jennings himself has said he didn't think of this advantage as being unfair; it's just one thing that machines have always been better at than humans.

But the buzzer advantage wouldn't matter if Watson couldn't answer the questions, just as Jennings' quickness with the buzzer during his streak wouldn't have kept him in the game if he didn't know the answers.

I do think that a more useful measure of Watson's abilities as compared to humans would be to have everyone give their answers to all the questions and compare the accuracy rate, rather than doing Jeopardy. But that wouldn't make for fun television, now would it?

Greg Finley said...

I could totally beat Watson if it were run off a standard laptop, instead of that mammoth server center they showed on TV.

It wouldn't make for good television, but I'd be interested to see the results of your proposal nonetheless.

Anonymous said...

Since Watson cannot be penalized for buzzing too soon, they should do the same for the humans.

John W. Wall said...

It's actually *not* good television. I won't even watch again tonight because of the buzzer disadvantage. That and the IBM infomercials that take up half of each game make the whole dog-and-pony show ludicrous.

Steve said...

Adam G you are mistaken. It may be true that Jennings' past winning is less impressive due to his button pressing skills, but no human could ever be a match for a computer programmed for "instant reflexes". If you read the article, the gist of it is that all three competitors know most of the answers, but Watson will win because he is programmed to hit the button perfectly. And I would even wager that on the questions he doesn't answer first, he was 'programmed' to not answer instantly - just to make the game appear more fair.