Innovation Culture 创新方法论

Presentation on theme: "Innovation Culture 创新方法论"— Presentation transcript:

1 Innovation Culture 创新方法论
The Networked Economy (17) : Information Management, Strategy, and Innovation 网络经济: 信息管理, 战略, 和创新 Innovation Culture 创新方法论

2 Three ingredients for innovation 创新三要素
Methodology 方法 Statistics 统计学 Data Analysis 数据探测 Machine Learning 机器学习 Data 数据 Computer Science 计算机科学 Database Research 数据库研究 Was better in previ To take advantage of this… Relevant research Data/Technology Amazon.com 50M Customers 15M Products Multiple touchpoints Methodology/ PROCESS Why Stats: QUANTIFYING UNCERTAINTY -- might want to insert MBA sheet HIGH NOISE (there is alwas EXTERNAL stuff as src of noise) Decision always made in uncertain environment HIGH DIMENSIONALITY Characterization Prediction and Diagnosis E.g. Probabilistic Models Domain E.g., Psychology, Behavioral Economics What actions can be taken? Pricing, promotions Need for COLLABORATION Domain expertise 知识领域 Behavioral Economics 行为经济学 Marketing 市场营销学 Finance 金融学 plus: Culture

3 Q: Understanding cross-channel shopping behavior 读解跨渠道购物行为
Drive online shoppers to traditional channels through trackable means 通过跟踪手段鼓励网上购物者到传统渠道消费 E.g., giving them codes to redeem 如：抵换折扣 Drive offline shoppers to website 鼓励店铺消费者上网购物 E.g., Banana Republic giving offers with offline purchase to first online purchase 如： Banana Republic对其线下销售提供首次网上销售的机会 Survey shoppers 调查购物者 Survey employees 与员工交谈 Merge online and off-line transaction databases 合并网上和网下交易数据库 Collect additional data 收集其他数据 Video 录像 RFIDs… 电子标签… Doubleclick

4 Approaches towards understanding customers
Stated preferences 消费者宣称的主观偏好 Surveys 调研 Online 网上 Offline 网下 Revealed preferences 实际研究显示的客观偏好 Analyze clicks 分析点击 Methodologies 研究方法 Server data 网络服务器数据 Observe the store 观察消费者行踪 Browser data (e.g., toolbar) 浏览器数据(如：工具条） Follow customer across sites 观察网民的跨网浏览 Gain insights by combining surveys with click analysis 通过对网上调查的点击情况进行分析能够获得启发

5 Analyze clicks 分析点击 Definitions 定义 Visit 访问 Session 停留 Purchase 购买
Visit begins with first click on the site 始于网站上的第一次点击 To avoid counting redirects, require minimum of 5 seconds until next click 至少等5秒再点击，以区别自动重定向 Visit ends after inactivity on the site of a specified time (e.g.,30 minutes) 已终止网站活动一段时间（15到30分钟），访问即结束 Session 停留 Within a site, session and visit used interchangeably 在同一网站内，访问和停留是交替进行的 Across sites (i.e., browser - or proxy data), session denotes a continuous string of visits 跨网站（通过浏览器或代理数据） ，停留显示了一系列持续的访问 Includes competitor sites and others 包括访问竞争对手网站等 Purchase 购买 A visit that contains a order confirmation page from the site 网络访问中最后出现订购确认页面

6 Browser collected data 浏览器收集的数据
Advantages 优点 Advantage of browser-side data 主要优点 Behavior across sites 跨网站的用户行为 Of course, also behavior within site (for a smaller subset of server log) 在某特定网站上的行为 Caching of web pages no problem (compared to site data and proxy data) 网页缓存 Disadvantages 缺点 User needs to download software 需要网民下载软件 Privacy issues 隐私问题 MediaMetrix provides URL’s only MediaMetrix仅提供URL数据 No purchase data 而非购买数据 No site characteristics 无其它网站特征

7 An early study of online customer behavior Johnson et al
An early study of online customer behavior Johnson et al. (1999): The Data 约翰逊等就网络消费行为所作的早期研究：数据 MediaMetrix Panel from July 1997 through Feb. 99. (June 98 for CD, Books) MediaMetrix调查小组，从1997年7月到1999年2月（1998年6月对CD、图书的调查） Total of 19,465 panelists, capture every URL. 总计19465个成员，抓取每个URL There was still little shopping June ’97-Feb. ’ 年6月-99年2月，购物行为仍然很少

8 Research Questions 对一些问题的研究
Characterize paths through website 刻画访问网站的路径特征 “Modeling Online Browsing and Path Analysis Using Clickstream Data” 通过利用点击流数据来为在线浏览和路径 分析建立模型 by Alan L. Montgomery, Shibo Li, Kannan Srinivasan, and John C. Liechty. Marketing Science (2004). Predict intention and modality of the visit 预测访问者的访问意图和模式 “Seize the Occasion” 抓住机会 by Horacio D. Rozanski, Gerry Bollman, and Martin Lipman. Strategy and Business (2001). Understand and influence conversion 理解并影响转变率 “A Model of Web Site Browsing Behavior Estimated on Clickstream Data” 在点击流数据的评估中被认为是网站浏览 行为的模范 by Randolph E. Bucklin and Catarina Sismeiro, J of Marketing Research 40 (2003). “Dynamic Conversion Behavior at E- Commerce Sites” 电子商务网站中活跃的转变行为 by Wendy W. Moe and Peter S. Fader. Management Science (2004). Compute and apply customer network value 测定并利用消费者网络价值 “Mining the Network Value of Customers” 挖掘消费者的网络价值 by Pedro Domingos and Matt Richardson, KDD ACM Press.

9 加入购物车

10 Also want to make recommendations! 添加推荐!
去收银台 等一下！再加16.01美元就能…… 加入购物车 戴维斯的专辑 Revealed preferences vs stated preferences隐藏偏爱与明显偏爱 Leverage users, not manual labor杠杆使用者而非体力劳动 Very clean data! 很明显的数据 Not manual editing!!! Belief: automatic

11 Also want to make recommendations! 添加推荐!
购买戴维斯专辑的消费者 也买了另两张专辑 Complements (buy in addition to) 互补（额外购买） Customers who bought X also bought Y 买了X的消费者会再买Y Substitutes (buy instead of) 替代（选择购买） Customers who shopped for X also shopped for Y (based on clicks) Z买了X的消费者也会再 Revealed preferences vs stated preferences隐藏偏爱与明显偏爱 Leverage users, not manual labor杠杆使用者而非体力劳动 Very clean data! 很明显的数据 Not manual editing!!! Belief: automatic 想买戴维斯专辑的消费者 同时浏览了其它同类专辑

12 KEEP THIS AS FOCUSING STORY: WHAT DOES xxx do for us?

13 Which one is better? No such thing without agreeing on objective 

14 Result: Right vs Left 对比结果：左还是右
Metrics　 衡量标准 Conversion rate: Percentage of visits placing an order 转化率：下订单的浏览者所占的比例 Order size: Number of additional (from the second page) items put into cart 订单大小：（从第二页起）新购商品数量 Result　 结果 “Your Shopping Cart” on right is about 1% better than on left “Your Shopping Cart”置于右侧比置于左侧的效果提高1% Some details　 细节 Total revenues: GMS (in treatment over control) BUT: Interaction All customers 所有消费者 Existing customers 现有消费者 Cart-adds from 2nd page: 从第二页起新购商品数量： +0.6% Cart-adds from 2nd page: 从第二页起新购商品数量： +0.8% Wishlist-adds: 选择礼物清单： +1.4% DVD Cart-adds: 新购DVD： DVD (USD): +1.1% +1.0%

15 Amazon Shopping Cart Recommendations 亚马逊的购物车推荐
Situation and idea Add an item to your shopping cart at a website 在网站上把某种产品加入你的购物车 Most sites show the cart 大多数网站都有购物车 At Amazon, Greg Linden had the idea of showing recommendations based on cart items 在亚马逊，格雷格提出了根据购物车显 示推荐 Evaluation 评价 Pro: cross-sell more items 优点：多买产品 Con: distract people from checking out – VP asked to stop work on this idea 缺点：使人们在结帐时受到干扰- VP 提 出停止这个想法 As with many new things, hard to decide 正如许多新鲜事物，很难判断 A/B test was run A/B 实验还在继续 Feature is live on the site 这项特点在网站上很流行 ASW TO INNOVATIO from Greg Linden’s Blog:

16 Discussion 讨论 Strength and weaknesses of such experiments? 此类试验的优缺点?
Examples　 举例 Search Inside the Book　 图书内容全文搜索 Sponsored Links 广告链接 Pricing, promotions, …　　 定价、促销…… Short-term hit vs sustainable long-term effect 短期冲击还是长期持续效果 The science behind it Generalize from individual experiments to create scientific insights 总结试验结果，树立科学观点 Design experiments appropriately 精心设计试验 Consequence: meaning = use

17 Controlled Experiments 核对实验
Many names for same concept 一些概念的名称 A/B tests A/B 实验 Treatment and control groups 处理和控制团队 Controlled experiments 核对实验 Randomized Experimental Design 随机实验设计

18 Controlled Experiments 核对实验
Main idea 主要思路 Randomly split traffic between two versions 在两个版本之间随机分流 Treatment: New idea(s) 处理：新想法 Control: Current live version 控制：目前的现存版本 Collect metrics of interest 搜集兴趣度量 Analyze 分析 Statistical tests 统计实验 Data mining 数据挖掘

19 How old is this methodology? 方法论的历史
First known controlled experiment in the 1700s 核对实验初次被人们熟知是在18世纪 British captain noticed lack of scurvy in Mediterranean ships 英国的船长发现在地中海的船只上存在坏血病（因缺乏维生素C所致） Had half the sailors eat limes (treatment), half did not (control) 一半的船员吃柠檬（治疗），一半的不吃（控制） Experiment was so successful, British sailors are still called limeys 实验非常成功，至今英国的海员还被称为limeys（英国佬） Note: Success despite no understanding of vitamin C deficiency 注意：尽管对缺乏维生素C不了解，但是成功了

20 Advantages of Controlled Experiments 核对实验的优点
Controlled experiments test for causal relationships, not simply correlations 核对实验是关于因果关系的实验，而非简单的相关。 They insulate external factors 它们与外部因素无关 They are a standard requirement for drug approval by US Federal Drug Administration (FDA ) 它们被美国联邦药品管理局作为药品审核的标准要求 But like most great things, there are problems and it’s important to recognize them… 但是正如大多数事情一样，它们也存在问题，并且认清这些问题很重要

21 Issues with Controlled Experiments 核对实验的相关问题
If you don‘t know where you are going, any road will take you there 如果你不知道要往哪里去，任何路都可以带你去。 —Lewis Carroll 李维斯卡尔 Organization has to agree on key metric(s) to improve 组织不得不在一些主要的度量准则上取得一致来提高 While it may seem obvious that we need to know if we’re improving, it’s not easy to get clear agreement 因而很显而易见，我们需要了解我们是否正在提高，得到清晰的一致意见并不容易 If nothing else, bringing this question to the surface is a great benefit to the org! 如果没有其他的，那么把这个问题提到台面上来对于组织有很大的益处

22 Issues with Controlled Experiments 核对实验的问题
Quantitative metrics, not always explanations of “why” 定量的度量原则，不能总是用来解释“为什么” For example, we may know that lemons work against scurvy, but not why; it may take a while to understand vitamin C deficiency 例如：我们可能知道柠檬可以治疗坏血病，但是并不知道为什么；可能需要一些时间才能 了解释维生素C缺乏 Data Mining may help identify segments where difference is large, leading to better understanding 数据挖掘可以帮助识别出不同之处，进一步增加了解 Usability studies also useful at explaining 可用性研究在解释方面也非常有用

23 Issues with Controlled Experiments 控制实验的问题
Primacy effect 首要的影响 Changing navigation in a website may degrade the customer experience (temporarily), even if the new navigation is better 网站导航的改变将会减少顾客的使用（暂时的），即使新的导航比较好 Evaluation may need to focus on new users, or run for a long period 评价应该注重于新用户，或者是长期的运营 Multiple experiments 多项实验 Even though the methodology shields an experiment from other changes, statistical variance increases making it harder to get significant results 即使这种方法使得实验免于改变，统计的变化还是使得得到更好的结果变的艰难 It is useful to avoid multiple changes to the same “area.” QA also becomes harder when tests interact 对于同一区域避免多项变化比较有用，但是当实验时互动的时候，也会变得艰难 Good example of primacy: Office 2007 vs. Office 2003

24 Issues with Controlled Experiments 控制实验的问题
Consistency/contamination 一致性与缺陷 On the web, assignment is usually cookie-based, but people may use multiple computers, erase cookies, etc. Typically a small issue 在网络上，任务通常是基于cookie的，但是人们可以使用多个计算机，删除cookie等等。 通常这只是小问题 Launch events / media announcements sometimes preclude controlled experiments 开展事件/媒体公告有时可以排除控制实验 The journalists need to be shown the “new” version 记者需要看到新的版本 Good example of primacy: Office 2007 vs. Office 2003

25 Issues with Controlled Experiments 核对实验的问题
Short-term vs. Long-term 短期与长期 Hard to assess long term effects, such as customer abandonment 很难评估长期影响，例如客户流失 Example: if you optimize for ads for click-through revenues, you might plaster the site with ads. Long-term concerns should be part of metric (e.g., revenue per pixels of real estate on the window) 例如：如果你们为了点击收入而优化广告，你可能要用广告来重新建设你的网站。长期的 考虑应该是规测的一部分。（例如：窗口上每个象素的收入）

26 蓝色长裤 搜索： 广告链接 MIGHT WANT TO ADD POST a9 pictire Points to make
0) a9 launched mid-April 2004 1) after having used m/c learning, we can then get to the essence directly 2) purpose of goldbox (+ the 10 hits) 3) book 广告链接

27 Long-term effects 长期影响
Displaying Google’s “sponsored links” within Amazon.com’s site? 在亚马逊网站上显示Google的“友情链接”？ Initial effect positive… 初始效果是积极的… … but how to model whether this helps or hurts the company in the long run? …但是怎样模拟这给公司带来的长期影响是好是坏？ Offer free shipping 提供免费送货 Competitors quickly adopt, how to model the overall game? 竞争对手很快就会跟进，怎样模拟整个博弈过程？ Pricing, Promotions… 定价，促销… Title: What can go wrong? Add: Competitive changes (nonstationarity), drift of norms

28 Lesson : A/B Experiments 第一课： A/B试验
(1) Formulate relevant questions 设计问题 (2) “Ask” your customers: 让你的客户回答问题 Decide on set of metrics 确定衡量标准 Create a set of actions 设计行动 Run A/B test 开始A/B试验 Learn from the experiment 总结试验结果 Act 行动

29 Action: Invest USD 10M to improve customer satisfaction 行动：投资1000万提高客户满意度
Base decision on analysis of behavioral data 将决策建立在对行为数据的分析基础上 Quantify 数量化 Model 模型 Act 行动 Consider 考虑 Increase selection? 增加选择？ Increase availability? 保证供给？ Reduce clutter on web site? 减少网站的杂乱性？ Improve product search algorithms? 改进产品搜索算法？ DM to drive Strategic decision Hypothesis discovery 所以，如大家所见， 自从我们停用投诉表以后，用户满意度得到了显著的提升

30 Survey 调查问卷 “…. Please let us know what you plan to do at Amazon.com today” “……请告诉我今天你计划在亚马逊网站上 做什么” Free responses, hand-coded into non- exclusive categories 自愿回复，将回复进行分类编码，归入 一些非互斥的类型 Multiple assignments possible 可能回复多种可能 Average: 1.2 categories per response 平均：每个回复涵盖1.2个种类 Number of responses: 回复数量：1023 Response rate: 3.1% 回复率：3.1% Date: February 11-12, 日期：2003年二月11-12日 Say: self-selection Turns out that respondents are more likely to be good customers / they care REDO THIS Drop THANK YOU, instead translation Point of entry Note: Bias towards “shopping”

31 Why do people visit? 为何访问网站
35% Research: Response indicates having a target or topic for which customer wants to gather information in depth 搜索：抱着某种目标或主题，想深入收集一些信息（如“寻找关于智利的书籍”） 31% Browse: Looks at items casually at a more general level than Research 浏览：比搜索要低一个层次，更随意地浏览一些信息（如“看看音乐/书籍方面的信息”，而没有特指） 16% Buy: Response indicates intent to buy in this visit (“buy a book”) 购买：访问网站是想购买（如“买一本书”） 10% Complain: Complains about some feature of the site 抱怨：抱怨对于网站有些方面的不满 9% Post-buy: Mentions checking order status, other account activities 购后：查询订单状态，以及其它的一些帐户活动 7% Community: Mentions usage of some feature of community, like reviews 交流：使用网站的一些社区功能，如论坛 Price: Mentions getting information about price for specific items (does not include “looking for deals”  Browse) 询价：查询一些特定商品的价格信息（不包括“找找看看有没有要买的东西”——这属于浏览） Part of EXPLICITLY COLLECTED Note: A survey is a qualitative instrument primarily used for exploration. When extracting percentages, large systematic biases are possible due to the self-selection of respondents. Unaided CHECK RECOGNIZED VS UNRECOGNIZED

32 Why do people visit? 人们为什么访问网站？
2.9% Goldbox: Mentions Goldbox activity 百宝箱活动：进行百宝箱活动 2.2% Gift: Indicates looking for a gift for someone else 礼物：为其他人找一些礼物 2.0% Sell: Mentions a selling activity 出售：从事一些卖出活动 1.4% Personalization: Mentions personalization feature (e.g., recommendations) 个性化的回复：一些个性化的东西（如提建议） 0.8% Used: Mentions finding or transacting a used item 二手交易：求购或者出售一些二手东西

33 Stated vs revealed preferences 自认的偏好与实际的偏好
Obtain insights by combining individual survey response with click analysis: 结合个人调查问卷的回复与点击情况分析，得出一些结论 Look at those who ended up buying something: 最终购物的网民中 Only about one-half of those making a purchase indicated that they wanted to buy something in this visit 只有1/2事先称计划购物 Look at those who said they wanted to buy something: 最初想来购物的网民中 Only about one-third of those indicating intent to buy ended up making a purchase in that visit 只有1/3最终完成购物  Starting with a purchase and looking back in time, when did the customer first look at the item 开始一项购买且及时回顾，什么时候消费者首次看中这项或类似产品的？ Of those who did not indicate intend to buy, but actually bought: 没有显示他们有购买意图但事实上却购买了： 48% indicated browse 48%浏览中显示 42% indicated research 42%调研中显示 Influence / help the customers finding things whether or not they are …. Impact on satisfaction Are they impulse buyers? 他们是冲动消费者吗？ How clear are respondents about their intentions? 关于他们意图的回复清楚吗？ IS EXAMPLE OF THE THREE CIRCLES Q: How many buy without initially stating an intent to buy? 有多少最初没有明确购买意图的购买者？ Q: How many who stated they want to buy actually buy? 有多少人说他们想要购买并且也付诸行动了？ 

34 Customer satisfaction 客户满意度
Relate satisfaction rating to: 把满意度评价与下列两项联系起来： Visit intent (from survey) 访问意图（来源于调查问卷） Actions (from clickstream) 行动（从其点击行为看） Intermediate goal 中期目标 Predict satisfaction 预测满意度 Ultimate goal 最终目标 Increase satisfaction 提高满意度 Number of responses: 回复的数量：6117 Dates: March 15–24, 日期：2003年3月15-24日

35 Outcome of the analysis 分析结果
Created a9.com for web search and product search 诞生了a9.com，致力于网络和产品研究的附属机构

36 对于最终购买的物品，该顾客第一次访问该物品 的详细页面发生在多久以前？（视购买的东西不 同而不同）
How long ago did a customer first look at the detail page of an item eventually purchased? (i.e., conditioning on purchase) 对于最终购买的物品，该顾客第一次访问该物品 的详细页面发生在多久以前？（视购买的东西不 同而不同） 20% of items bought today were looked at before Analyze behavior of purchased items How long does it take a customer to buy an item after they first looked at it? Distribution of time between ordering an item and the first look at its detail page 在购买和访问相关详细信息之间相隔的周数 购买日

37 How long do customers take between adding an item to cart and checking out?
在访问与订购之间的时间 5% of items are purchased after having been in the cart for more than one week 5%的商品从放入购物车到最后购买之间的时间长于一周 90.5% of ordered items were in cart for less than 24 hours before purchase 所订购的商品中90.5%是在购买前24小时之内放入购物车中的 (6)-37 天数

38 What makes great groups Great Groups? 是什么使得一个优秀的团队优秀？
Source来源: Warren Bennis, USC,

39 A shared dream 共同的梦想 At the heart of every Great Group is a shared dream. 每一个伟大的团队的核心是一个共同的梦想。 All Great Groups believe that they are on a mission from God, that they could change the world, make a dent in the universe. They are obsessed with their work. It becomes not a job but a fervent quest. That belief is what brings the necessary cohesion and energy to their work. 每一个伟大的团队都坚信他们有着上帝赋予的改变世界的使命。他们痴迷于他们的工作， 工作已经不是所谓的工作而是一种热烈的寻求。 这种信仰赋予他们工作的凝聚力和能量。 E.g., Google wants to make the world’s information accessible 例如： Google（谷歌）的使命：让全世界所有的信息都可以访问 Source来源: Warren Bennis, USC,

40 There is a price to be part of the group 高额的代价是团队的一部分
Members pay a personal price. 团队成员付出个人代价 Membership in a Great Group isn’t a day job; it is a night and day job. Divorces, affairs, and other severe emotional fallout are typical, especially when a project ends. So groups strike a Faustian bargain for the intensity and energy that they generate. 一个优秀团队的成员不仅是白天工作，而是白天晚上都工作。离婚，婚外情和其它严重的 感情波动，特别是当一个项目结束的时候。所以，团队在与其产生的凝聚力和能量之间讨 价还价。

41 Curators, not creators 监护人，而非创造者
Great Groups make strong leaders. 优秀的团队造就强有力的领导者。 On one hand, Great Groups all nonhierarchical, open, and very egalitarian. Yet they all have strong leaders. That’s the paradox of group leadership. You cannot have a great leader without a Great Group -- and vice versa. In an important way, these groups made the leaders great. The leaders were connoisseurs of talent, more like curators than creators. 一方面，优秀的团队是不分等级，公开、平等的。，然而他们都有非常强的领导者。这是 团队领导的矛盾。没有优秀的团队就没有优秀的领导者，反过来，优秀的团队使得领导者 更加优秀。领导者们是人才的专家，更像是监护人而非创造者。

42 Abandon egos for the dream 为了梦想放弃自我
Great groups manage conflict by abandoning individual egos to the pursuit of the dream. 团队成员在追求梦想过程中通过放弃自我来解决冲突。 At a critical point in the Manhattan Project a great chemist who later served as Dwight Eisenhower‘s chief scientific advisor threatened to quit because he couldn’t get along with a colleague. Project leader Robert Oppenheimer simply said, “George, how can you leave this project? The free world hangs in the balance.” So conflict, even with these diverse people, is resolved by reminding people of the mission. 曼哈顿项目的一个关键点上一名卓越的化学家，后来在德怀特 艾森豪威尔公司任首席科学 顾问，因为不能和一名同事融洽相处而面临辞职。该项目领导人Robert Oppenheimer 仅 仅说到：“乔治，你怎么能离开这个项目呢？” 因此，冲突特别是和不同的人发生的，最 终通过提醒他团队的使命而解决。 E.g., Amazon, Google et al focus on measurement. 亚马逊，谷歌等等，专注于测量。

43 Recruiting 聘用 Great Groups are the product of meticulous recruiting. 优秀的团队是精心聘用的产物。 Cherry-picking the right talent for a group means knowing what you need and being able to spot it in others. It also means understanding the chemistry of a group. Candidates are often grilled, almost hazed, by other members of the group and its leader. You see the same thing in great coaches. They can place the right people in the right role. And get the right constellations and configurations within the group. 精心聘用适合的人才意味着明白你所需要的类型，并能在其他人中发现他们。这也意味着 对团队组成的了解。应聘者通常被考核，甚至经受一些被其它团队成员和领导者制造的挫 折。你会在优秀的领导者身上发现相同的东西。他们把合适的人放在合适的位置。使整个 团队人才济济，结构合理。 Hiring practice at Amazon, Microsoft et al. And firing practice (“top grading”). 亚马逊，微软等的聘用原则：雇用最优秀的人

44 Age 年龄 Great Groups are usually young. 优秀的团队通常比较年轻。
The average age of the physicists at Los Alamos was about 25. Oppenheimer -- “the old man” -- was in his 30s. But Great Groups are also young in their spirit, ethos, and culture. Most important, because they’re young and naive, group members don’t know what‘s supposed to be impossible, which gives them the ability to do the impossible.” Great Groups don’t lack the experience of possibilities. Los Alamos 的物理学者平均年龄是25岁，其中，奥本海默是最年长的，三十多岁。优秀的 团队在精神，气质和文化方面也同样年轻。最重要的是，因为团队成员的年轻率真，他们 不知道什么是不可能的事情，从而使他们有能力去做很多不可能的事情。优秀的团队并不 缺乏可能的经验。 Average age of employees at Google in 2007: 28 years. 谷歌：2007年员工的平均年龄：28岁

45 Deliver 成果 Real artists ship. 真正的艺术家 Yes! 是的
Steve Jobs constantly reminded his band of Apple renegades that their work meant nothing unless they brought a great product to market. In the end, Great Groups have to produce a tangible outcome external to themselves. Most dissolve after the product is delivered; but without something to show for their efforts, the most talented assemblage becomes little more than a social club or a therapy group. 史蒂夫 乔布经常提醒他的团队（苹果叛徒乐队）除非为市场创造优秀的产品否则他们的工 作一文不值。最终，优秀的团队必须有切实的成果。大多数团队在产品推出后解散，他们 的努力有时并不会显现。最有才华的团队比一个社交俱乐部或治疗团队要大的多。 Yes! 是的