作者: Kurt Wagner
时间: 2013年04月28日 来源:
财富中文网美国统计学家内特?希尔是个数学天才，长于利用大数据进行预测。去年美国总统大选期间，他非常准确的预测了美国50个州的投票胜负。但他认为，大数据也不是万能的，有些领域的预测成功率就很低，比如地震，比如股市。
统计学家内特?希尔是个数学天才，却并非因此而出名。他的成名，是因为知道怎样把数学天才运用到真实世界。由于非常准确地预测了美国总统大选的结果，希尔成为全美国最有名的数据达人。他在去年11月份的美国总统大选期间，准确地预言了50个州的投票胜负。2008年，他也猜中了50个州中的49个。同时希尔还把他的大数据分析法应用到了体育【比如美国的大学篮球联赛（March Madness）、职业棒球大联盟等（Major League Baseball）】、赌博（今年夏天他将第三次参加世界扑克系列赛）、甚至是约会。希尔曾经给一个叫Baseball Prospectus的棒球网站写过文章，现在他扩大了涉足的领域。他既是作家，又是政治专家，而且还在《纽约时报》（New York Times）网站上开了自己的博客“
FiveThirtyEight”。
本周四，希尔作为主讲嘉宾在Lithium Technologies公司的年度LiNC大会上做了有关数据分析的演讲。《财富》杂志（Fortune）对他进行了专访，请他谈了谈大数据分析的局限性、大数据分析在股市中的角色、以及它如何应用到约会中的，甚至还请他预测了2016年的美国总统大选。这次专访的文字记录节选如下：
财富：我相信一直都会有人找你，想让你帮他们赌赢美国大学体育总会（NCAA）“疯狂三月”的比赛。
内特?希尔：我没有按自己的计算结果来下注，因为我觉得其他人也可能按我的选择下注。如果我按照自己的计算结果买，我已经赢了二等奖了。
或许你明年可以收一小笔版权费？
绝对的。或者我们可以先抛出一个假的计算公式，然后晚一点再抛出一个真的。哎呀，上一个里有编码错误！（笑）
你一开始是用统计学来研究和预测棒球比赛胜负，后来为什么又转向政治了？
回溯往事的时候，说你当初为什么做了某些事比较容易，但说出来的不一定就是当初推动你往那个方向走的合理动机。不过我认为，当初的部分理由是，我当时为棒球网站Baseball Prospectus工作了五年——从2003年到2008年，这期间我发现棒球行业取得了长足的进步。那个时代刚开始的时候，和电影《点球成金》（Moneyball ，由一本小说改编成的电影）里描写的时代非常像，当时统计学家和传统人士之间的矛盾很紧张。人们担心会有一堆宅男冲出来抢走他们的饭碗。现在情况完全反了过来。事情并不是像你雇了一个统计学家，然后偷偷把他藏在某个地方。而是每支球队——几乎是每支球队，当然也有例外——在它的组织内部的各个级别上都有人懂数据分析。
我看到统计分析方法在短短几年的时间里进步得很快。而政治报道玩的就是语言艺术。我发现无论是关于政治的新闻报道本身，还是从政治家们嘴里说出来的话，有很多都是在胡扯。所以当时我觉得时机已经成熟了，可以把某些非常基本的分析工具用在关于选举的新闻报道上。
Statistician Nate Silver isn't famous because he's a mathematical genius. (Although, he is.) Silver's well-known because he knows how to apply his craft to the real world. The country's most popular data cruncher is known for his spot-on election predictions -- he accurately called the winner in all 50 states of November's presidential election; in 2008, he went 49 for 50 -- but Silver's big data analytics have also translated to the worlds of sports (March Madness, Major League Baseball), gambling (Silver will play in his third World series of Poker event this summer), and even dating. Silver once wrote for the baseball website Baseball Prospectus but has since expanded his offerings; he is now a published author, a political pundit, and the creator of his very own New York Times blog,
FiveThirtyEight.
Silver was in San Francisco Thursday to talk analytics as the keynote speaker at Lithium Technologies' annual LiNC Conference. Fortune sat down with him to talk about big data's limitations, its role in the stock market, how it applies to dating, and even his predictions for the 2016 presidential election. A lightly edited transcript follows.
Fortune: I'm sure you get people coming up to you all the time to discuss how you helped them win their NCAA March Madness pool.
Nate Silver: I went against my bracket in my own pool because I thought other people would be using it. I would have gotten second place if I had taken my own advice.
Maybe take a small royalty fee next year?
Absolutely. Or we need to put out a fake bracket [first], and then put out a real one [later]. Oops, there was a coding error! [Laughs]
You started out using stats to better understand and predict success in baseball -- why did you move towards politics?
Of course it's easy to say in retrospect why you did certain things instead of what rational motivations were pushing you in that direction in real time, but I think part was that I was involved working for Baseball Prospectus for about five years -- 2003 to 2008 -- and you saw a great amount of progress in the baseball industry during that time. The start of that era was the era described in [the book-turned movie] Moneyball where you really had a lot of tension between stat-heads and traditionalists. People were terrified that nerds would come over and take their jobs. And really now that's been totally reversed, where it's not just that you have some stat-head that you've hired and have locked into a closet somewhere, but that every team -- almost every team, there are some exceptions -- understands analytics at different levels of the organization.
But seeing how quickly that progressed in a span of just a few years, and how behind politics coverage seemed to be where it's all about the narrative -- there's a lot of bullshit basically both in the news coverage of politics and from politicians themselves -- so it seemed like it was ripe to apply some very basic analytics tools to the coverage of elections.
预测大选的时候，把你的个人政治理念从工作中抛开会不会很困难？
无论我们干哪一个行业，都很难保持客观。没有人能左右现实，我们多多少少有些厌世的观点。不过我认为在体育上的训练对我是有帮助的，比如我虽然可以像小时候一样做底特律猛虎队（Detroit Tigers）的粉丝，但是我仍然认为洛杉矶天使队（Los Angeles Angels）的麦克?特劳特才应该当选为去年的最有价值球员。不过我认为政治有一点不同，这个行业里的很多人不光有自己的观点，且而还习惯于左右大众的观点。他们习惯性地认为，他们可以创造他们自己的现实。这就是为什么我认为有时候正确理解政治语言有困难。
有些人会想，如果我编出一个事实，或是编造一个民调数据，问题就解决了。而政治媒体圈里虽然有好人，但是也有人非常听话，而且乐于把政客在拉票活动上说的鬼话传播出来。我认为这就是问题所在。跟体育相比，人们在政治问题上不习惯检查一下现实。
那么你是怎样筛选信息，挑出那些“鬼话”的？
重点是忽略政治人物说的话，坚持使用能公开获得的数据。记录显示，大多数政治观察家一般爱把政治人物的一次失态或一场辩论看得太重了——当然总有例外，不过大体上民意调查数据还是提供了一个较为可靠的标准。至于老百姓，他们有自己的生活，也不总是消费政治新闻。他们衡量事物的方式非常复杂，比如他们会考虑经济问题，或者政府是不是让我们卷入了一场愚蠢的战争，又或者政府是不是出了什么大丑闻。这些因素才能帮助我们解释最终是谁赢得了大选，而不是政治评论家们关注的那些劲爆花边。
现在的数据比以前多了。你在选择数据的时候，怎样确定哪些数据才能正确回答你的问题？
其中一点是，你需要一个系统，而不是一次性的做法。我们在2008年设计了一个模型，在2012年进行了升级，我们用它来对每次民意调查进行分析。如果有些民调机构以往的信用很好，它在系统中就会占有更大的权重。并不是说其它民调就会被忽视。不是说我们只盯着一份民调，然后伸出手指说：“这份民调很重要，那份不重要。”基本上所有的难题和所有的决策过程都来自设计模型的过程。根据理论、实际和以往的经验，怎样设计一系列好的规则来处理这些信息？这个问题最重要，然后坚持这些标准。我们在每年6月推出这个模型后，就不会再更改了，除非模型里有bug，幸运的是到现在还没有发现。我们的基本原则始终不变，然后你再在这个规矩方圆里分析数据。
Is it hard to keep your own political beliefs separate from your work predicting elections?
It's always hard for us to be objective in any walk of life. None of us has a monopoly on reality, we all have rather jaded points of view. I do think the sports training helps though, where I can be a Detroit Tigers fan as I am [and was] growing up, I still thought Mike Trout [Los Angeles Angels] should have won the MVP award last year. What I think differentiates politics a bit is that you have an industry full of people who not only have views but are [also] used to manipulating public opinion. They're used to thinking they can create their own reality. That's why I think you have such trouble on the uptake there.
People think that, well, if I can spin a fact a certain way or spin polls a certain way, [the problem] goes away. When you have a political press where some people are very good, but some other people are very compliant and happy to pass along spin from the campaigns, I think that's the issue. People aren't used to getting a reality check in politics as much as in sports.
So how are you able to sift through that information then to pick out the BS?
The idea is to ignore what the politicians say and stick with publically available data. The record shows that in general, most political observers tend to overrate the importance of a gaffe or a debate -- there are always exceptions -- but in general the polls provide a pretty reliable benchmark. And the public, who have real lives and are not constantly consuming political news, are [sometimes] weighing things in a very sophisticated way where they're looking at things like the economy or are we involved in any stupid wars or major scandals from the administration. Those are the things that explain a lot about who wins the elections and not so much the petty stuff that the political pundits can focus on.
There is more data now than ever before. How are you able to determine which information to pull in order to properly answer your question?
Part of it is that you do need -- as Vegas might say -- you do need a system instead of an ad hoc way of doing it. So we have a model that we designed in 2008 that was updated for 2012 that was designed to account for every single poll. Some polls, if they're from a pollster that has a better track record, get more weight in the system. It doesn't mean that others are ignored. So it's not like we're just looking at a poll and sticking our fingers up in the air and saying, "Oh that poll is important, and that poll's not." Basically all the hard work and all the decision-making process comes from designing this model before the fact. Based on theory and practice and past experience, what are a good set of rules for processing this information? And then sticking to that. We don't make any alterations to the model once we launch it in June every year, unless there's a bug, which fortunately there hasn't been. But the principles are always the same, and then you have a disciplined way to analyze data in that context.
有什么问题是数据和分析不能回答的吗？
这都存在于一定的范围内。要知道，相对于我们的潜力，我们做得有多好，与某件事物在本质上有多大的可预测性，二者是有区别的。以棒球为例，虽然分析师已经研究棒球很久了，但是即便是最优秀的棒球队，胜率也只有三分之二。就算是最优秀的击球手，也只有40%的机会上垒。所以在某种意义上，它在本质上仍然是不可预测的，但是我们有了比较好的方法来衡量和了解我们所知道和不知道的事情。
在很多领域，数据分析还没有广泛应用。比如我在我的《
信号与声音：为什么很多地震预测失败了，但有些预测说中了》一书中谈到了地震的预报。千百年来人们一直在尝试预报地震，我们了解了一些现象——比如加州的地震要比新泽西州多，但是在某一时刻及时、精确地预报一场地震的能力可以说毫无进展。甚至就连经济也是如此，一旦我们试图做出长期的经济预测，我们大多数都会做得比较差。
是否有行业已经在关注大数据分析可能带来的影响？
有时并一不定是非常热门的行业。比如零售企业有大量的每个消费者的交易记录，也有大量的供应链管理方面的数据，所以在制定库存优化战略、定价优化战略以及供应链应急管理战略时都会用到大数据分析。并不是非常抢眼的东西，但这些人有非常好的数据储备，通常是高质量的数据，因此可以做出更好的决策。我相信有些企业已经这样做了，因此它会带来一些前所未有的效率。
另外还有其它案例，比如你可以看看电视行业是如何让人们花钱的。我认为广告行业定位顾客的手段变得更先进了。讽刺的是，这种效率从某种程度上也给媒体公司带来了坏处。广告业有一句老话：“你只有一半的广告预算花对了，但你不知道是哪一半。”现在人们可能知道那是哪一半了，所以他们只会花这一半。
人们能否通过数据或者分析法精确预测股市？
股市是一场竞赛，你在和其他股民进行竞争。所以问题来了：是否股市的某些交易者要比其他人更厉害？我认为答案可能是“是的”。我不是一个纯粹的股民，不过我玩扑克很久了。我认为玩扑克跟炒股在很多方面是相通的，你知道有些人越到长期越得心应手，而且更擅长应对不确定因素等等。不过股市里还有很多不稳定因素和很多运气成分，一个市场周期可以长达几个月或几年。很多不正当的刺激因素可能会影响股市。所以尽管我认为有些很好的交易者在短期甚至五到十年内都可以顺风顺水，但最终很大程度还是由运气决定的，所以很复杂。
Are there any questions out there that can't be answered using data and analytics?
So I think it all exists along a spectrum. It's important to know, too, that there's a difference between how good we are relative to our potential and how intrinsically predictable something might be. So for example if you look at baseball where analytics have come an awful long way, well it's still the case that the best baseball teams only win two-thirds of their games. The best hitters only get on base about 40% of the time. So it's still intrinsically unpredictable in a sense, but we have a good way of measuring and knowing what we know and what we don't know.
But there are a lot of fields where analytics have not come very far. I discuss earthquake forecasting in my book [
The Signal and the Noise: Why So Many Predictions Fail -- but Some Don't] for instance, where people have been trying for centuries. We know something -- there are more earthquakes here in California than in New Jersey -- but the ability to anticipate a particular earthquake with any precision at a particular moment in time has not gone very well at all. Even economics -- when we try to do long-term economic forecasting, it has been pretty poor for the most part.
Are there any industries out there that are overlooking the possible impact of big data analytics?
It's sometimes industries that aren't very sexy necessarily, so big retail businesses for example have tons of records on every consumer transaction they have [completed]. They have a ton of data on supply chain management, so things like that in terms of optimal inventory strategies or optimal pricing strategies or robust strategies for disruption in the supply chain. Not super-sexy stuff, but those people have really good data sets, often high quality data, and can make better decisions as a result. I'm sure that some are doing it already, and there are some efficiencies there that you weren't seeing before.
There are also cases like if you look at how people consume television, for example. I think the advertising industry is something that's gotten more sophisticated in terms of targeting customers. The irony is that efficiency has become harmful in some ways for media companies, because what was the old adage? "Half your advertising budget is well spent, but you don't know which half." Now people might know which half, so they're only spending half as much.
Can people use data or analytics to accurately predict the stock market in any way?
The problem is the stock market is this whole contest where you're competing against other creators. So the question is: Are there some traders that are better than others? I think the answer is probably, Yes. I'm not a pure markets guy, I played poker long enough which I think has parallel skills to trading in many respects where you know that some people are better over the long-term and better at accounting for uncertainty and so forth. But there's a lot of volatility and a lot of luck where a market cycle can last for months or years. There are a lot of perverse incentives that get in the way. So while I think there are some good traders, in the near term, even over a period of five or 10 years, it will mostly be dictated by luck, so it's tricky.
你有没有把统计模型运用到约会上？
几年前，为了给《纽约时报杂志》（New York Times Magazine）写一篇文章，对相亲网站OkCupid做了一个小分析。我们想知道在一个星期的七天里，哪天晚上最适合约会或者说“约炮”。OkCubid收集了一些出去约会的人以及他们使用移动应用情况的状态报告。我们研究了那些想发展长期关系的人和那些仅仅想“一夜风流”的人的比率。结果我们明显地发现，星期三晚上想搞一夜情的人的比率最高。
2016年谁有可能从大选中胜出？
在这个问题上，我可能让人失望了：我同意传统的看法。如果希拉里参选了，她很难不获得民主党的提名。共和党则很难避免一场混战，他们有一些出色的候选人，也有一些糟糕的候选人，不过现在还没人能垄断共和党，所以要打一场选战才能决出胜负。最后的大选当然取决于初选的结果，不过大家应该注意，如果希拉里?克林顿赢了初选，她将是一个很好的候选人，而且她的支持率也比较高。不过对于任何一个政党来说，连续三次赢得总统宝座都是很难的。如果到2016年美国经济很差，或者奥巴马的支持率只有38%左右，那么对于民主党候选人来说，处境将会很艰难，哪怕对于希拉里也是一样。不过现在就预言2016年有点早，确切地说是太早了。（财富中文网）
译者：朴成奎
Have you ever applied your model to dating?
I did one little analysis with OkCupid for a New York Times Magazine piece a couple of years ago where we were trying to figure out the best night of the week to go out and get laid basically. So OkCupid collected the data of some status reports of people who were out and about using their mobile application. And we looked at the ratio of people who wanted a long-term relationship vs. people who wanted just something for [that night]. On Wednesday apparently you get the highest ratio of people who are just looking for something quick and dirty.
2016 presidential election: Who should we be looking out for?
So I guess I'm disappointingly in line with the conventional wisdom here. If Hillary [Clinton] runs, it's hard to see her not winning the Democratic nomination. On the GOP side I don't think there's any way to avoid having a big messy primary, they have some good candidates and some bad candidates. But no one has a monopoly on that party right now, so they're going to have to fight it out. And then the general election of course depends on who wins the primaries, but people should be a little wary because look, Hillary Clinton might be a very good candidate if she wins [the primary], and she'd be a slight favorite, but still it's hard for any party to win the White House three terms in a row. If you have a poor economy by 2016, or if Obama's approval ratings are 38% or something like that, then it's hard for a Democratic processor to prevail even if it is Hillary. Trying to make predictions at this point is a little early. Actually, it's a lot early.
