答案句子选择 孙亚伟.

Presentation on theme: "答案句子选择 孙亚伟."— Presentation transcript:

1 答案句子选择 孙亚伟

2 目录 问题 评测 方法 Tree Match Word Alignment Deep Learning 挑战 小结

3 问题 背景 问题 定义：给一问句和候选句子列表，找出正确句子(答案+证据)
Q:Who won the best actor Oscar in 1973? S1:Jack Lemmon won the Academy Award for Best Actor for Save the Tiger (1973). S2:Oscar winner Kevin Spacey said that Jack Lemmon is remembered as always making time for other people.

4 评测 数据集QASent 指标 𝑀𝑅𝑅= 1 |𝑄| 𝑖=1 |𝑄| 1 𝑟𝑎𝑛𝑘 𝑖 𝑀𝐴𝑃= 𝑞=1 |𝑄| 𝐴𝑣𝑒𝑃(𝑞) |𝑄|
𝑀𝑅𝑅= 1 |𝑄| 𝑖=1 |𝑄| 1 𝑟𝑎𝑛𝑘 𝑖 𝑀𝐴𝑃= 𝑞=1 |𝑄| 𝐴𝑣𝑒𝑃(𝑞) |𝑄| QASent 问句 来源 TREC 8-13 QA track 特性 人工编辑过； 每一问句均有正确答案； 句子 参加TREC的系统检索返回的句子 问句与句子有共同的非停用词 Class QASent Location 37 (16%) Human 65 (29%) Numeric 70 (31%) Abbreviation 2 (1%) Entity Description 16 (7%) QASent # of questions 227 # of sentences 8478 # of answers 928 Average length of questions 9.59 Average length of sentences 28.85

5 方法汇总 相关工作 MAP MRR Vasin Punyakanok (2004) 0.419 0.494 Hang Cui (2005)
0.427 0.526 Mengqiu Wang(2007) 0.603 0.685 Michael Heilman (2010) 0.609 0.692 Mengqiu Wang (2010) 0.595 0.695 Xuchen Yao (2013) 0.631 0.748 Aliaksei Severyn (2013) 0.678 0.736 Eyal Shnarch (2013) 0.686 0.754 Wen-tau Yih (2013) 0.709 0.770 Lei Yu (2014) 0.711 0.785 Di Wang (2015) 0.713 0.791 Minwei Feng (2015) 0.800 Aliaksei Severyn (2015) 0.746 0.808 Kateryna Tymoshenko (2015) 0.726 0.813 Zhiguo Wang (2015) 0.820 Ming Tan (2015) 0.728 0.832 Cicero dos Santos (2016) 0.753 0.851 Zhiguo Wang (2016) 0.771 0.845

6 方法归类 方法归类 方法抽象 相关工作 Tree Match Vasin Punyakanok (2004) Hang Cui (2005)
Mengqiu Wang(2007) Michael Heilman (2010) Mengqiu Wang (2010) Xuchen Yao (2013) Aliaksei Severyn (2013) Kateryna Tymoshenko (2015) Word Alignment Eyal Shnarch (2013) Wen-tau Yih (2013) Deep Learning Lei Yu (2014) Di Wang (2015) Minwei Feng (2015) Aliaksei Severyn (2015) Zhiguo Wang (2015) Ming Tan (2015) Cicero dos Santos (2016) Zhiguo Wang (2016)

7 Tree Match Vasin Punyakanok (2004) 思想：计算问句依存树与句子依存树编辑距离。两棵树越像，越像正确句子
Hang Cui (2005)： 思想：选择正确句子时候，不仅考虑问句与句子之间的lexical matching，还考虑问句中的关系与句子中关系的匹配程度，即Tree Path Match。两棵树关系路径越像，越像正确句子 方法：基于统计模型的fuzzy relation matching方法 Mengqiu Wang(2007) 思想：把句子和问句之间的相关性，看成是两棵依存树的语法转换（即answer (Semantic + Syntactic) transformation question） 方法：采用概率的准同步上下文无关文法 Michael Heilman (2010) 思想：计算问句依存树与句子依存树编辑距离后，考虑树编辑序列的特征，而不是总体长度 方法：首先计算源树到目标树编辑序列；然后从该序列提33个语法特征；最后转化为分类问题 Mengqiu Wang (2010) 思想：学习句子间的对齐结构(Structured Latent Variables)，用于计算句子间的相关性 方法：提出一概率模型(Tree-edit CRF model) Xuchen Yao (2013) 思想： “TED特征+词汇语义特征”相结合方法来计算问句与句子之间的相关性 方法：特征= TED序列特征 + 词汇语义特征， 模型=逻辑回归 Kateryna Tymoshenko (2015) 思想：计算问句与句子相关性时，考虑了字符串中相似性以及字符串中概念实体的类属关系 方法：把问句和句子分别转换成树结构；利用外部资源给两棵树加关联边；利用Tree-Kernel自动抽取特征，转化为L2R问题。

8 Tree Match——Vasin Punyakanok (2004)
假设：正确句子依存树与问句依存树编辑距离(TED)最小，即“两棵树越像” 方法： 用树匹配近似算法来计算问句依存树与句子依存的编辑距离 Text Dependency Tree Tree Edit Distance(TED)Distance Score 实验：效果胜过词袋方法 补充： 有些句子局部能回答问句，不必分析全句 Lexical Semantic，需考虑更多语义关系

9 Tree Match——Hang Cui (2005)
假设：正确句子与问句，不仅词汇匹配，而且关系匹配 方法：基于aligned words在依存树上的路径，来算问句与句子之间的关系匹配。具体步骤： 从问句依存树和句子依存树上，抽取所有满足一定约束的路径； 借用机器翻译中对齐模型思想，来计算问句与句子关系匹配程度： 上式中 𝑃 𝑡 ( 𝑅𝑒𝑙 𝑖 S | 𝑅𝑒𝑙 𝐴 𝑖 Q )：采用最大期望和互信息来估计 实验：比词密度方法效果好 补充：

10 Tree Match——Michael Heilman (2010)
假设： TED过程，应考虑树编辑序列的特征，而不是总体长度 方法： 树转换树： 拓展树编辑操作：增加复杂操作，如重排序、移动 搜索最小编辑序列：启发式贪心搜索+Tree Kernel方法 特征表示：依据编辑操作，总结33个语法特征 分类器：逻辑回归算法 实验：在三个任务(RTE、Paraphrases、QA)，效果提升 树转换树 特征表示 分类器 Premise: Pierce built the home for his daughter off Rossville Blvd, as he lives nearby. Hypothesis: Pierce lives near Rossville Blvd.

11 Word Alignment——Eyal Shnarch (2013)
假设：句子级推理可由词汇级推理结果推出。 方法：提出了一个基于Markovian的句子级概率推理模型。该模型分两层： Term-Level： Text与Hypothesis词汇对齐 Sentence-Level：依据Term-level结果，运用Markovian（考虑邻近的词汇因素）来估算从Text推导Hypothesis的概率 实验：在两个任务(RTE、QA)，效果提升 补充：Lexical inference rules资源（如harmfuldangerous） 𝑥 𝑖 ∈ 0,1 ，1代表 ℎ 𝑖 inferred from Text; 𝑦 𝑖 ∈ 0,1 ：推理决策变量， 𝑦 𝑛 ：最终结果

12 Word Alignment——Wen-tau Yih (2013)
假设：正确句子与问句之间有一种潜在结构 方法：利用丰富的词汇语义关系和潜在结构，算出问句与句子之间的匹配程度，具体分两部分： 词汇语义关系：同义/反义；上下位；词语义相似 计算问句与句子匹配（两种模型）： Bag-of-Words Model Learning Latent Structures 实验：Learning Latent Structures效果好过Bag-of-Words Model；丰富的词汇语义关系有必要 𝑞: What is the fastest car in the world? 𝑠: The Jaguar XJ220 is the dearest, fastest and most sought after car on the planet. 𝒉 𝑓 𝑞,𝑠 = 𝜃 𝑇 Φ(ℎ)

13 Deep Learning——Di Wang (2015)
假设：决定正确句子因素match the same words ； words meaning 方法：结合关键词匹配和堆双向长短词语记忆模型（BLSTM，考虑单词序列上下文），来计算问句和句子之间相关性 特征：BLSTM相关性 首先把问句和句子中的words借用word2vec转换为向量表示； 把这些向量先后送入双向的BLSTM模型； 最后模型输出它们的相关性结果 特征： BM25关键词匹配 模型： Gradient boosted regression tree (GBDT) 实验：三层BLSTM+BM25方案效果最好 补充：该方法没用句法分析和外在语义资源 What sport does Jennifer Capriati play? Positive Sentence: “Capriati, 19, who has not played competitive tennis since November 1994, has been given a wild card to take part in the Paris tournament which starts on February 13.” Negative Sentence: “Capriati also was playing in the U.S. Open semifinals in ’91, one year before Davenport won the junior title on those same courts.”

14 思考 方法归类 方法抽象 思考 Tree Match Word Alignment Deep Learning 如何表示树结构？
如何度量两棵树相似？ 从两棵树相似程度，如何映射为问句与正确句子相关程度？ 在树匹配时候，借用知识库资源给树节点赋予语义知识？ 两棵树相似就代表是正确句子？ Word Alignment 问句与句子对齐就代表是正确句子？ 怎么挖掘词汇之间潜在关系？ 如何利用知识库资源？ 怎么利用词汇对齐推导句子之间的相关性？ 两句间“最佳”对齐结构是什么？ Deep Learning 句子级向量怎么表示？ 怎么学习词汇语义矩阵？ 怎么由词表示句？ 两个句子向量的相似程度就代表是正确句子？

15 简单抽象模型 Q： S： [Q]: Who received the will rogers award ?
答案类型 Focus/实体 谓词 约束限定 共指/匹配 Type Match 转述/推理 Match S： 实体 谓词 答案 约束限定 冗余部分 [Q]: Who received the will rogers award ? [A]: kudos , frank -- frank sinatra , that is , who wednesday night was honored with the will rogers award at the beverly hills diamond jubilee gala .

16 挑战 谓词方面 问句 正确句子 wife married the author of by constructed built end
late received honored came up with dubbed associated with related die killed started founded released spread top official or ceo executive director name at birth real name revenue sales original name a.k.a . affiliation member members performers [Q]: what is the name of durst 's group ? [A]: limp bizkit lead singer fred durst did a lot before he hit the big time . [Q]: what is florence nightingale famous for ? [A]: the newly named `` corner shop '' where asprey has traded since <num> occupies what were once nine Georgian houses ,one of which was the last london home of nursing pioneer florence nightingale . [A]: in <num> , the founder of modern nursing , florence nightingale , was born in florence , italy .

17 挑战 答案类型方面 特殊类问题方面(包含Non-factoid)
how far is yaroslavl from moscow yaroslavl , <num> miles northeast of moscow How often….occurs once every <num> years Why is XXX famous How is cataract treated ?  he had surgery for cataracts . 类型 答案 style of music rap artists industry chemical ethnic background jewish kind of business the clothing retailer profession ophthalmologist

18 小结 问题：答案句子选择 评测：数据集QASent 方法： Tree Match Word Alignment Deep Learning
挑战 谓词方面 答案类型方面 特殊类问题求解方面

19 参考文献 Vasin Punyakanok, Dan Roth, and Wen-Tau Yih. Mapping dependencies trees: An application to question answering. In Proceedings of the 8th International Symposium on Artificial Intelligence and Mathematics, Fort Lauderdale, FL, USA Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and Tat-Seng Chua. Question answering passage retrieval using dependency relations. In ACM-SIGIR 2005. Heilman, Michael and Smith, Noah A Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions. In NAACL-HLT 2010. Shnarch, Eyal, Ido Dagan, and Jacob Goldberger. A Probabilistic Lexical Model for Ranking Textual Inferences //Proceedings of the First Joint Conference on Lexical and Computational Semantics 2012. Yih, Wen-tau and Chang, Ming-Wei and Meek, Christopher and Pastusiak, Andrzej Question Answering Using Enhanced Lexical Semantic Models. In ACL 2013. Di Wang and Eric Nyberg A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering. In ACL 2015. More…

20 致谢 欢迎老师和同学提问！

21 示例 Q: Which was the first movie that James Dean was in?
S: James Dean, who began as an actor on TV dramas, didn’t make his screen debut until 1951’s “Fixed Bayonet.” Q: What was the GE building in rockefeller plaza called before ? S: Known as the RCA Building until 1988 , it is most famous for housing the headquarters of the television network NBC . Q: how long was I love lucy on the air ? S: The black-and-white series originally ran from October 15, 1951, to May 6, 1957, on the Columbia Broadcasting System ( CBS ) Q: In what film is Gordon Gekko the main character? S: He received a best actor Oscar in 1987 for his role as Gordon Gekko in “Wall Street”. Q: What is the name of Durst’s group? S: Limp Bizkit lead singer Fred Durst did a lot before he hit the big time.