Self-Attention huitr 2019.03.16.

Presentation on theme: "Self-Attention huitr 2019.03.16."— Presentation transcript:

1 Self-Attention huitr

2

3 Motivation 普通CNN堆叠卷积层不能够很好地捕捉long range dependency
提出non-local operation，对于特征图中每个像素点，都用其他所有像素点的变换结果做加权求和，归一化后作为该像素点的新特征 Self-attention，用同一张图片中的其他像素点来增强当前像素点

4 Formulation 𝑖:𝑖𝑛𝑑𝑒𝑥 𝑜𝑓 𝑜𝑢𝑡𝑝𝑢𝑡 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 X:𝑖𝑛𝑝𝑢𝑡 𝑓𝑒𝑎𝑡𝑢𝑟𝑒
𝑗:𝑖𝑛𝑑𝑒𝑥 𝑡ℎ𝑎𝑡 𝑒𝑛𝑢𝑚𝑒𝑟𝑎𝑡𝑒 𝑎𝑙𝑙 𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛𝑠 𝑓:𝑐𝑜𝑚𝑝𝑢𝑡𝑒 𝑝𝑎𝑖𝑟𝑤𝑖𝑠𝑒 𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛𝑠ℎ𝑖𝑝 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑖 𝑎𝑛𝑑 𝑗 𝑔:𝑐𝑜𝑚𝑝𝑢𝑡𝑒𝑠 𝑎 𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 X 𝐶 X :𝑁𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛

5 Instantiation 𝑐ℎ𝑜𝑖𝑐𝑒 𝑜𝑓 𝑓: Embedded dot product version

6

7 Core idea first gather key features from the entire space into a compact set then distribute them to each location adaptively

8 Method

9 Computational graph

10 Comparison Chen Y , Rohrbach M , Yan Z , et al. Graph-Based Global Reasoning Networks[J]

11 Experiments 5 extra A2-blocks at Res3 and Res4
6.5 GFLOPs and 33.0 M parameters

12 Experiments Chen Y , Rohrbach M , Yan Z , et al. Graph-Based Global Reasoning Networks[J]

13

14 Method

15 A Generic Formulation of Self-Attention
X: feature maps as a matrix of size s × c 𝐾 𝑋 : key function 𝑄 𝑋 : query function 𝑉 𝑋 : value function Implemented as linear layers S=𝑋𝐾 𝑋𝑄 𝑇𝑋𝑉 X: feature maps as a matrix of size s × c K, Q:c × 𝑏 matrice 𝑉:𝑐 × 𝑐 matrix

16 X: feature maps as a matrix of size s × c
K, Q:c × 𝑏 matrice 𝑉:𝑐 × 𝑐 matrix Left Associativity S= 𝑋𝐾 𝑋𝑄 𝑇 𝑋𝑉 𝑋𝐾 𝑋𝑄 𝑇 : 𝑠 × 𝑏 ∗𝑏 × 𝑠=𝑠 × 𝑠, 可以看成所有spatial location之间的相似度，即Non-local的思想 𝑋𝐾 𝑋𝑄 𝑇 𝑋𝑉: 𝑠 × 𝑠 ∗𝑠 × 𝑐=𝑠 × 𝑐 Right Associativity S=𝑋𝐾[ 𝑋𝑄 𝑇𝑋𝑉] [ 𝑋𝑄 𝑇𝑋𝑉]: b × 𝑠 ∗𝑠 × 𝑐=𝑏 × 𝑐, 可以看成b个c维的Global Descriptor，即Double Attention的思想 𝑋𝐾[ 𝑋𝑄 𝑇𝑋𝑉]: 𝑠 × 𝑏 ∗𝑏 × 𝑐=𝑠 × 𝑐

17 X: feature maps as a matrix of size s × c K, Q:c × 𝑏 matrice
Left Associativity S= 𝑋𝐾 𝑋𝑄 𝑇 𝑋𝑉 𝑋𝐾 𝑋𝑄 𝑇 : 𝑠 × 𝑏 ∗𝑏 × 𝑠=𝑠 × 𝑠, 可以看成所有spatial location之间的相似度，即Non-local的思想 𝑋𝐾 𝑋𝑄 𝑇 𝑋𝑉: 𝑠 × 𝑠 ∗𝑠 × 𝑐=𝑠 × 𝑐 Complexity: 𝑠 × 𝑏 × 𝑠+𝑠 × 𝑠 × 𝑐= 𝑠 2 (𝑏+𝑐) Right Associativity S=𝑋𝐾[ 𝑋𝑄 𝑇𝑋𝑉] [ 𝑋𝑄 𝑇𝑋𝑉]: b × 𝑠 ∗𝑠 × 𝑐=𝑏 × 𝑐, 可以看成b个c维的Global Descriptor，即Double Attention的思想 𝑋𝐾[ 𝑋𝑄 𝑇𝑋𝑉]: 𝑠 × 𝑏 ∗𝑏 × 𝑐=𝑠 × 𝑐 Complexity: 𝑏 × 𝑠 × 𝑐+𝑠 × 𝑏 × 𝑐=𝑠2𝑏𝑐

18

19 Framework

20 Experiments

21

22 Comparison with Non-local

23 Criss-cross attention module
H x W x C2 (H+W-1) x H x W H x W x C1 H x W x C2 H x W x C1

24 Criss-cross attention module
𝑸 𝒖 : C2 x 1 H x W x C2 (H+W-1) x H x W H x W x C1 H x W x C2 H x W x C1

25 Criss-cross attention module
𝑸 𝒖 : C2 x 1 H x W x C2 (H+W-1) x H x W H x W x C1 H x W x C2 𝛀 𝒖 : (H+W-1) x C2 H x W x C1

26 Criss-cross attention module
𝑸 𝒖 : C2 x 1 H x W x C2 (H+W-1) x H x W H x W x C1 H x W x C1 H x W x C2 𝛀 𝒖 : (H+W-1) x C2 H x W x C1 H x W x C1 𝚽 𝒖 : (H+W-1) x C1

27 Why 2 loops

28 Experiments

29 评价 直接使用Non-local（或者包装一下再使用），相对创新度不是很高，但是精度刷上去或许也可以
分析并降低Non-local的复杂度，从另一个方向理解计算图，比较有insight