[2305.07027] EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

EfficientViT: Memory Efficient Vision Transformer with

Cascaded Group Attention

Xinyu Liu 1,∗ , Houwen Peng 2 , Ningxin Zheng 2 , Yuqing Yang 2 , Han Hu 2 , Yixuan Yuan 1
1 The Chinese University of Hong Kong, 2 Microsoft Research

Abstract

Vision transformers have shown great success due to their high model capabilities. However, their remarkable performance is accompanied by heavy computation costs, which makes them unsuitable for real-time applications. In this paper, we propose a family of high-speed vision transformers named EfficientViT. We find that the speed of existing transformer models is commonly bounded by memory inefficient operations, especially the tensor reshaping and element-wise functions in MHSA. Therefore, we design a new building block with a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN layers, which improves memory efficiency while enhancing channel communication. Moreover, we discover that the attention maps share high similarities across heads, leading to computational redundancy. To address this, we present a cascaded group attention module feeding attention heads with different splits of the full feature, which not only saves computation cost but also improves attention diversity. Comprehensive experiments demonstrate EfficientViT outperforms existing efficient models, striking a good trade-off between speed and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by 1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while running 5.8×/3.7× faster on the GPU/CPU, and 7.4× faster when converted to ONNX format. Code and models are available at here .

† † ∗ Work done when Xinyu was an intern of Microsoft Research.

1

Introduction

Figure 1 : Speed and accuracy comparisons between EfficientViT (Ours) and other efficient CNN and ViT models tested on an Nvidia V100 GPU with ImageNet-1K dataset [ 17 ] .

Vision Transformers (ViTs) have taken computer vision domain by storm due to their high model capabilities and superior performance [ 18 , 69 , 44 ] . However, the constantly improved accuracy comes at the cost of increasing model sizes and computation overhead. For example, SwinV2 [ 43 ] uses 3.0B parameters, while V-MoE [ 62 ] taking 14.7B parameters, to achieve state-of-the-art performance on ImageNet  [ 17 ] . Such large model sizes and the accompanying heavy computational costs make these models unsuitable for applications with real-time requirements [ 86 , 40 , 78 ] .

There are several recent works designing light and efficient vision transformer models [ 81 , 9 , 50 , 19 , 56 , 49 , 79 , 29 ] . Unfortunately, most of these methods aim to reduce model parameters or Flops, which are indirect metrics for speed and do not reflect the actual inference throughput of models. For example, MobileViT-XS [ 50 ] using 700M Flops runs much slower than DeiT-T [ 69 ] with 1,220M Flops on an Nvidia V100 GPU. Although these methods have achieved good performance with fewer Flops or parameters, many of them do not show significant wall-clock speedup against standard isomorphic or hierarchical transformers, e.g. , DeiT [ 69 ] and Swin [ 44 ] , and have not gained wide adoption.

To address this issue, in this paper, we explore how to go faster with vision transformers, seeking to find principles for designing efficient transformer architectures. Based on the prevailing vision transformers DeiT [ 69 ] and Swin [ 44 ] , we systematically analyze three main factors that affect model inference speed, including memory access, computation redundancy, and parameter usage. In particular, we find that the speed of transformer models is commonly memory-bound. In other words, memory accessing delay prohibits the full utilization of the computing power in GPU/CPUs [ 21 , 32 , 72 ] , leading to a critically negative impact on the runtime speed of transformers [ 31 , 15 ] . The most memory-inefficient operations are the frequent tensor reshaping and element-wise functions in multi-head self-attention (MHSA). We observe that through an appropriate adjustment of the ratio between MHSA and FFN (feed-forward network) layers, the memory access time can be reduced significantly without compromising the performance. Moreover, we find that some attention heads tend to learn similar linear projections, resulting in redundancy in attention maps. The analysis shows that explicitly decomposing the computation of each head by feeding them with diverse features can mitigate this issue while improving computation efficiency. In addition, the parameter allocation in different modules is often overlooked by existing lightweight models, as they mainly follow the configurations in standard transformer models [ 69 , 44 ] . To improve parameter efficiency, we use structured pruning [ 45 ] to identify the most important network components, and summarize empirical guidance of parameter reallocation for model acceleration.

Based upon the analysis and findings, we propose a new family of memory efficient transformer models named EfficientViT. Specifically, we design a new block with a sandwich layout to build up the model. The sandwich layout block applies a single memory-bound MHSA layer between FFN layers. It reduces the time cost caused by memory-bound operations in MHSA, and applies more FFN layers to allow communication between different channels, which is more memory efficient. Then, we propose a new cascaded group attention (CGA) module to improve computation efficiency. The core idea is to enhance the diversity of the features fed into the attention heads. In contrast to prior self-attention using the same feature for all heads, CGA feeds each head with different input splits and cascades the output features across heads. This module not only reduces the computation redundancy in multi-head attention, but also elevates model capacity by increasing network depth. Last but not least, we redistribute parameters through expanding the channel width of critical network components such as value projections, while shrinking the ones with lower importance like hidden dimensions in FFNs. This reallocation finally promotes model parameter efficiency.

Experiments demonstrate that our models achieve clear improvements over existing efficient CNN and ViT models in terms of both speed and accuracy, as shown in Fig.  1 . For instance, our EfficientViT-M5 gets 77.1% top-1 accuracy on ImageNet with throughput of 10,621 images/s on an Nvidia V100 GPU and 56.8 images/s on an Intel Xeon E5-2690 v4 CPU @ 2.60GHz, outperforming MobileNetV3-Large [ 26 ] by 1.9% in accuracy, 40.4% in GPU inference speed, and 45.2% in CPU speed. Moreover, EfficientViT-M2 gets 70.8% accuracy, surpassing MobileViT-XXS [ 50 ] by 1.8%, while running 5.8 × \times /3.7 × \times faster on the GPU/CPU, and 7.4 × \times faster when converted to ONNX [ 3 ] format. When deployed on the mobile chipset, i.e. , Apple A13 Bionic chip in iPhone 11, EfficientViT-M2 model runs 2.3 × \times faster than MobileViT-XXS [ 50 ] using the CoreML [ 1 ] .

In summary, the contributions of this work are two-fold:

•
We present a systematic analysis on the factors that affect the inference speed of vision transformers, deriving a set of guidelines for efficient model design.

•
We design a new family of vision transformer models, which strike a good trade-off between efficiency and accuracy. The models also demonstrate good transfer ability on a variety of downstream tasks.

Figure 2 : Runtime profiling on two standard vision transformers Swin-T and DeiT-T. Red text denotes memory-bound operations, i.e. , the time taken by the operation is mainly determined by memory accesses, while time spent in computation is much smaller.

2

Going Faster with Vision Transformers

In this section, we explore how to improve the efficiency of vision transformers from three perspectives: memory access, computation redundancy, and parameter usage. We seek to identify the underlying speed bottlenecks through empirical studies, and summarize useful design guidelines.

2.1

Memory Efficiency

Memory access overhead is a critical factor affecting model speed [ 15 , 31 , 65 , 28 ] . Many operators in transformer [ 71 ] , such as frequent reshaping, element-wise addition, and normalization are memory inefficient, requiring time-consuming access across different memory units, as shown in Fig. 2 . Although there are some methods proposed to address this issue by simplifying the computation of standard softmax self-attention, e.g. , sparse attention [ 34 , 75 , 61 , 57 ] and low-rank approximation [ 11 , 74 , 51 ] , they often come at the cost of accuracy degradation and limited acceleration.

In this work, we turn to save memory access cost by reducing memory-inefficient layers. Recent studies reveal that memory-inefficient operations are mainly located in MHSA rather than FFN layers [ 31 , 33 ] . However, most existing ViTs [ 18 , 69 , 44 ] use an equivalent number of these two layers, which may not achieve the optimal efficiency. We thereby explore the optimal allocation of MHSA and FFN layers in small models with fast inference. Specifically, we scale down Swin-T [ 44 ] and DeiT-T [ 69 ] to several small subnetworks with 1.25 × \times and 1.5 × \times higher inference throughput, and compare the performance of subnetworks with different proportions of MHSA layers. As shown in Fig. 3 , subnetworks with 20%-40% MHSA layers tend to get better accuracy. Such ratios are much smaller than the typical ViTs that adopt 50% MHSA layers. Furthermore, we measure the time consumption on memory-bound operations to compare memory access efficiency, including reshaping, element-wise addition, copying, and normalization. Memory-bound operations is reduced to 44.26% of the total runtime in Swin-T-1.25 × \times that has 20% MHSA layers. The observation also generalizes to DeiT and smaller models with 1.5 × \times speed-up. It is demonstrated that reducing MHSA layer utilization ratio appropriately can enhance memory efficiency while improving model performance.

Figure 3 : The accuracy of downscaled baseline models with different MHSA layer proportions, where the dots on each line represent subnetworks with similar throughput. Left: Swin-T as the baseline. Right: DeiT-T as the baseline. The 1.25 × \times /1.5 × \times denote accelerating the baseline models by 1.25/1.5 times, respectively.

2.2

Computation Efficiency

MHSA embeds the input sequence into multiple subspaces (heads) and computes attention maps separately, which has been proven effective in improving performance [ 71 , 18 , 69 ] . However, attention maps are computationally expensive, and studies have shown that a number of them are not of vital importance [ 52 , 73 ] . To save computation cost, we explore how to reduce redundant attention in small ViT models. We train width downscaled Swin-T [ 44 ] and DeiT-T [ 69 ] models with 1.25 × \times inference speed-up, and measure the maximum cosine similarity of each head and the remaining heads within each block. From Fig. 4 , we observe there exists high similarities between attention heads, especially in the last blocks. The phenomenon suggests that many heads learn similar projections of the same full feature and incur computation redundancy. To explicitly encourage the heads to learn different patterns, we apply an intuitive solution by feeding each head with only a split of the full feature, which is similar to the idea of group convolution in [ 87 , 10 ] . We train the variants of downscaled models with the modified MHSA, and also compute the attention similarities in Fig. 4 . It is shown that using different channel-wise splits of the feature in different heads, instead of using the same full feature for all heads as MHSA, could effectively mitigate attention computation redundancy.

Figure 4 : The average maximum cosine similarity of each head in different blocks. Left: downscaled Swin-T models. Right: downscaled DeiT-T models. Blue lines denote Swin-T-1.25 × \times /DeiT-T-1.25 × \times model, while darkblue lines denote the variants that feed each head with only a split of the full feature.

Figure 5 : The ratio of the channels to the input embeddings before and after pruning Swin-T. Baseline accuracy: 79.1%; pruned accuracy: 76.5%. Results for DeiT-T are given in the supplementary.

2.3

Parameter Efficiency

Typical ViTs mainly inherit the design strategies from NLP transformer [ 71 ] , e.g. , using an equivalent width for Q 𝑄 Q , K 𝐾 K , V 𝑉 V projections, increasing heads over stages, and setting the expansion ratio to 4 in FFN. For lightweight models, the configurations of these components need to be carefully re-designed [ 7 , 39 , 8 ] . Inspired by [ 45 , 82 ] , we adopt Taylor structured pruning [ 53 ] to automatically find the important components in Swin-T and DeiT-T, and explore the underlying principles of parameter allocation. The pruning method removes unimportant channels under a certain resource constraint and keeps the most critical ones to best preserve the accuracy. It uses the multiplication of gradient and weight as channel importance, which approximates the loss fluctuation when removing channels [ 38 ] .

The ratio between the remaining output channels to the input channels is plotted in Fig. 5 , and the original ratios in the unpruned model are also given for reference. It is observed that: 1) The first two stages preserve more dimensions, while the last stage keeps much less; 2) The Q 𝑄 Q , K 𝐾 K and FFN dimensions are largely trimmed, whereas the dimension of V 𝑉 V is almost preserved and diminishes only at the last few blocks. These phenomena show that 1 ) the typical channel configuration, that doubles the channel after each stage [ 44 ] or use equivalent channels for all blocks [ 69 ] , may produce substantial redundancy in last few blocks; 2) The redundancy in Q 𝑄 Q , K 𝐾 K is much larger than V 𝑉 V when they have the same dimensions. V 𝑉 V prefers a relative large channels, being close to the input embedding dimension.

Figure 6 : Overview of EfficientViT. (a) Architecture of EfficientViT; (b) Sandwich Layout block; (c) Cascaded Group Attention.

3

Efficient Vision Transformer

Based upon the above analysis, in this section, we propose a new hierarchical model with fast inference named EfficientViT. The architecture overview is shown in Fig. 6 .

3.1

EfficientViT Building Blocks

We propose a new efficient building block for vision transformer, as shown in Fig. 6 (b). It is composed of a memory-efficient sandwich layout, a cascaded group attention module, and a parameter reallocation strategy, which focus on improving model efficiency in terms of memory, computation, and parameter, respectively.

Sandwich Layout. To build up a memory-efficient block, we propose a sandwich layout that employs less memory-bound self-attention layers and more memory-efficient FFN layers for channel communication. Specifically, it applies a single self-attention layer Φ i A subscript superscript Φ A 𝑖 \Phi^{\rm A}{i} for spatial mixing, which is sandwiched between FFN layers Φ i F subscript superscript Φ F 𝑖 \Phi^{\rm F}{i} . The computation can be formulated as:

   X   i  +  1    =     ∏  𝒩      Φ  i  F   ​   (    Φ  i  A   ​   (     ∏  𝒩      Φ  i  F   ​   (   X  i   )     )    )      ,        subscript  𝑋  𝑖  1      superscript  product  𝒩       superscript   subscript  Φ  𝑖   F       superscript   subscript  Φ  𝑖   A     superscript  product  𝒩       superscript   subscript  Φ  𝑖   F    subscript  𝑋  𝑖          \displaystyle X_{i+1}=\prod^{\mathcal{N}}\Phi_{i}^{\rm F}(\Phi_{i}^{\rm A}(\prod^{\mathcal{N}}\Phi_{i}^{\rm F}(X_{i}))),    

(1)

where X i subscript 𝑋 𝑖 {X}{i} is the full input feature for the i 𝑖 i -th block. The block transforms X i subscript 𝑋 𝑖 {X}{i} into X i + 1 subscript 𝑋 𝑖 1 {X}_{i+1} with 𝒩 𝒩 \mathcal{N} FFNs before and after the single self-attention layer. This design reduces the memory time consumption caused by self-attention layers in the model, and applies more FFN layers to allow communication between different feature channels efficiently. We also apply an extra token interaction layer before each FFN using a depthwise convolution (DWConv) [ 27 ] . It introduces inductive bias of the local structural information to enhance model capability [ 14 ] .

Cascaded Group Attention. Attention head redundancy is a severe issue in MHSA, which causes computation inefficiency. Inspired by group convolutions in efficient CNNs [ 37 , 10 , 87 , 64 ] , we propose a new attention module named cascaded group attention (CGA) for vision transformers. It feeds each head with different splits of the full features, thus explicitly decomposing the attention computation across heads. Formally, this attention can be formulated as:

  X  ~    i  ​  j      subscript   ~  𝑋      𝑖  𝑗     \displaystyle\vspace{-9pt}{\widetilde{X}}_{ij}    
    =   A  ​  t  ​  t  ​  n  ​   (    X   i  ​  j    ​   W   i  ​  j   Q    ,    X   i  ​  j    ​   W   i  ​  j   K    ,    X   i  ​  j    ​   W   i  ​  j   V    )     ,       absent     𝐴  𝑡  𝑡  𝑛       subscript  𝑋     𝑖  𝑗     subscript   superscript  𝑊  Q      𝑖  𝑗         subscript  𝑋     𝑖  𝑗     subscript   superscript  𝑊  K      𝑖  𝑗         subscript  𝑋     𝑖  𝑗     subscript   superscript  𝑊  V      𝑖  𝑗         \displaystyle=Attn(X_{ij}{W}^{\rm Q}_{ij},X_{ij}{W}^{\rm K}_{ij},X_{ij}{W}^{\rm V}_{ij}),    

(2)

  X  ~    i  +  1      subscript   ~  𝑋   𝑖  1     \displaystyle{\widetilde{X}}_{i+1}    
    =   C  ​  o  ​  n  ​  c  ​  a  ​  t  ​    [    X  ~    i  ​  j    ]     j  =  1   :  h    ​   W  i  P     ,       absent     𝐶  𝑜  𝑛  𝑐  𝑎  𝑡   subscript   delimited-[]   subscript   ~  𝑋      𝑖  𝑗      :     𝑗  1   ℎ     subscript   superscript  𝑊  P   𝑖      \displaystyle=Concat[{\widetilde{X}}_{ij}]_{j=1:h}{W}^{\rm P}_{i},    

where the j 𝑗 j -th head computes the self-attention over X i ​ j subscript 𝑋 𝑖 𝑗 {{X}}{ij} , which is the j 𝑗 j -th split of the input feature X i subscript 𝑋 𝑖 {{X}}{i} , i.e. , X i subscript 𝑋 𝑖 {{X}}{i} = = [ X i ​ 1 , X i ​ 2 , … , X i ​ h ] subscript 𝑋 𝑖 1 subscript 𝑋 𝑖 2 … subscript 𝑋 𝑖 ℎ [{{X}}{i1},{{X}}{i2},\ldots,{{X}}{ih}] and 1 1 1 ≤ \leq j 𝑗 j ≤ \leq h ℎ h . h ℎ h is the total number of heads, W i ​ j Q subscript superscript 𝑊 Q 𝑖 𝑗 {W}^{\rm Q}{ij} , W i ​ j K subscript superscript 𝑊 K 𝑖 𝑗 {W}^{\rm K}{ij} , and W i ​ j V subscript superscript 𝑊 V 𝑖 𝑗 {W}^{\rm V}{ij} are projection layers mapping the input feature split into different subspaces, and W i P subscript superscript 𝑊 P 𝑖 {W}^{\rm P}{i} is a linear layer that projects the concatenated output features back to the dimension consistent with the input.

Although using feature splits instead of the full features for each head is more efficient and saves computation overhead, we continue to improve its capacity, by encouraging the Q 𝑄 Q , K 𝐾 K , V 𝑉 V layers to learn projections on features with richer information. We compute the attention map of each head in a cascaded manner, as illustrated in Fig. 6 (c), which adds the output of each head to the subsequent head to refine the feature representations progressively:

    X   i  ​  j      ′    =    X   i  ​  j    +    X  ~    i  ​   (   j  −  1   )       ,   1  <  j  ≤  h    ,     formulae-sequence      subscript   superscript  𝑋   ′       𝑖  𝑗    subscript  𝑋     𝑖  𝑗     subscript   ~  𝑋      𝑖     𝑗  1             1  𝑗        ℎ      {X}^{{}^{\prime}}_{ij}=X_{ij}+{\widetilde{X}}_{i(j-1)},\quad 1 

Table 1 : Architecture details of EfficientViT model variants.

Model { C 1 subscript 𝐶 1 C_{1} , C 2 subscript 𝐶 2 C_{2} , C 3 subscript 𝐶 3 C_{3} } { L 1 subscript 𝐿 1 L_{1} , L 2 subscript 𝐿 2 L_{2} , L 3 subscript 𝐿 3 L_{3} } { H 1 subscript 𝐻 1 H_{1} , H 2 subscript 𝐻 2 H_{2} , H 3 subscript 𝐻 3 H_{3} }

EfficientViT-M0 {64, 128, 192} {1, 2, 3} {4, 4, 4}

EfficientViT-M1 {128, 144, 192} {1, 2, 3} {2, 3, 3}

EfficientViT-M2 {128, 192, 224} {1, 2, 3} {4, 3, 2}

EfficientViT-M3 {128, 240, 320} {1, 2, 3} {4, 3, 4}

EfficientViT-M4 {128, 256, 384} {1, 2, 3} {4, 4, 4}

EfficientViT-M5 {192, 288, 384} {1, 3, 4} {3, 3, 4}

Table 2: EfficientViT image classification performance on ImageNet-1K [ 17 ] with comparisons to state-of-the-art efficient CNN and ViT models trained without extra data. Throughput is tested on Nvidia V100 for GPU and Intel Xeon E5-2690 v4 @ 2.60 GHz processor for CPU and ONNX, where larger throughput means faster inference speed. ↑ ↑ \uparrow : finetune with higher resolution.

Model
Top-1
Top-5
Throughput (images/s)
Flops
Params
Input
Epochs

(%)
(%)
GPU
CPU
ONNX
(M)
(M)

EfficientViT-M0
63.2
85.4
27644
228.4
340.1
79
2.3
224
300

MobileNetV3-Small [ 26 ]

67.4

19738
156.5
231.7
57
2.5
224
600

EfficientViT-M1
68.4
88.7
20093
126.9
215.9
167
3.0
224
300

Mobile-Former-52M [ 9 ]

68.7

3141
32.8
21.5
52
3.5
224
450

MobileViT-XXS [ 50 ]

69.0

4456
29.4
41.7
410
1.3
256
300

ShuffleNetV2 1.0 × \times [ 48 ]

69.4
88.9
13301
106.7
177.0
146
2.3
224
300

MobileViTV2-0.5 [ 51 ]

70.2

5142
34.4
44.9
466
1.4
256
300

EfficientViT-M2
70.8
90.2
18218
121.2
158.7
201
4.2
224
300

MobileOne-S0 [ 70 ]

71.4

11320
67.4
128.6
274
2.1
224
300

MobileNetV2 1.0 × \times [ 63 ]

72.0
91.0
6534
32.5
80.4
300
3.4
224
300

EfficientViT-M3
73.4
91.4
16644
96.4
120.8
263
6.9
224
300

GhostNet 1.0 × \times [ 23 ]

73.9
91.4
7382
57.3
77.0
141
5.2
224
300

NASNet-A-Mobile [ 89 ]

74.1

2623
19.8
25.5
564
5.3
224
300

EfficientViT-M4
74.3
91.8
15914
88.5
108.6
299
8.8
224
300

EdgeViT-XXS [ 56 ]

74.4

3638
28.2
29.6
556
4.1
224
300

MobileViT-XS [ 50 ]

74.7

3344
11.1
20.5
986
2.3
256
300

ShuffleNetV2 2.0 × \times [ 48 ]

74.9
92.4
6962
37.9
52.3
591
7.4
224
300

MobileNetV3-Large [ 26 ]

75.2

7560
39.1
70.5
217
5.4
224
600

MobileViTV2-0.75 [ 51 ]

75.6

3350
16.0
22.7
1030
2.9
256
300

MobileOne-S1 [ 70 ]

75.9

6663
30.7
51.1
825
4.8
224
300

GLiT-Tiny [ 5 ]

76.4

3516
17.5
15.7
1333
7.3
224
300

EfficientNet-B0 [ 67 ]

77.1
93.3
4532
30.2
29.5
390
5.3
224
350

EfficientViT-M5
77.1
93.4
10621
56.8
62.5
522
12.4
224
300

EfficientViT-M4 ↑ ↑ \uparrow 384
79.8
95.0
3986
15.8
22.6
1486
12.4
384
330

EfficientViT-M5 ↑ ↑ \uparrow 512
80.8
95.5
2313
8.3
10.5
2670
12.4
512
360

Such a cascaded design enjoys two advantages. First, feeding each head with different feature splits could improve the diversity of attention maps, as validated in Sec. 2.2 . Similar to group convolutions [ 87 , 10 ] , the cascaded group attention could save the Flops and parameters by h ℎ h × \times , since the input and output channels in the Q 𝑄 Q K 𝐾 K V 𝑉 V layers are reduced by h ℎ h × \times . Second, cascading the attention heads allows for an increase of network depth, thus further elevating the model capacity without introducing any extra parameters. It only incurs minor latency overhead since the attention map computation in each head uses smaller Q 𝑄 Q K 𝐾 K channel dimensions.

Parameter Reallocation. To improve parameter efficiency, we reallocate the parameters in the network by expanding the channel width of critical modules while shrinking the unimportant ones. Specifically, based on the Taylor importance analysis in Sec. 2.3 , we set small channel dimensions for Q 𝑄 Q and K 𝐾 K projections in each head for all stages. For the V 𝑉 V projection, we allow it to have the same dimension as the input embedding. The expansion ratio in FFN is also reduced from 4 to 2 due to its parameter redundancy. With the proposed reallocation strategy, the important modules have larger number of channels to learn representations in a high dimensional space, which prevent the loss of feature information. Meanwhile, the redundant parameters in unimportant modules are removed to speed up inference and enhance the model efficiency.

3.2

EfficientViT Network Architectures

The overall architecture of our EfficientViT is presented in Fig. 6 (a). Concretely, we introduce overlapping patch embedding [ 80 , 20 ] to embed 16 × \times 16 patches into tokens with C 1 subscript 𝐶 1 C_{1} dimension, which enhances the model capacity in low-level visual representation learning. The architecture contains three stages. Each stage stacks the proposed EfficientViT building blocks and the number of tokens is reduced by 4 × \times at each subsampling layer (2 × \times subsampling of the resolution). To achieve efficient subsampling, we propose an EfficientViT subsample block which also has the sandwich layout, except that the self-attention layer is replaced by an inverted residual block to reduce the information loss during subsampling [ 63 , 26 ] . It is worth noting that we adopt BatchNorm (BN) [ 30 ] instead of LayerNorm (LN) [ 2 ] throughout the model, as BN can be folded into the preceding convolution or linear layers, which is a runtime advantage over LN. We also use ReLU [ 54 ] as the activation function, as the commonly used GELU [ 25 ] or HardSwish [ 26 ] are much slower, and sometimes not well-supported by certain inference deployment platforms [ 3 , 1 ] .

We build our model family with six different width and depth scales, and set different number of heads for each stage. We use fewer blocks in early stages than late stages similar to MobileNetV3 [ 26 ] and LeViT [ 20 ] , since that the processing on early stages with larger resolutions is more time consuming. We increase the width over stages with a small factor ( ≤ \leq 2) to alleviate redundancy in later stages, as analyzed in Sec. 2.3 . The architecture details of our model family are presented in Tab. 1 . C i subscript 𝐶 𝑖 C_{i} , L i subscript 𝐿 𝑖 L_{i} , and H i subscript 𝐻 𝑖 H_{i} refer to the width, depth, and number of heads in the i 𝑖 i -th stage.

4

Experiments

4.1

Implementation Details

We conduct image classification experiments on ImageNet-1K [ 17 ] . The models are built with PyTorch 1.11.0 [ 59 ] and Timm 0.5.4 [ 77 ] , and trained from scratch for 300 epochs on 8 Nvidia V100 GPUs using AdamW [ 46 ] optimizer and cosine learning rate scheduler. We set the total batchsize as 2,048. The input images are resized and randomly cropped into 224 × \times 224. The initial learning rate is 1 × \times 10 − 3 superscript 10 3 10^{-3} with weight decay of 2.5 × \times 10 − 2 superscript 10 2 10^{-2} . We use the same data augmentation as [ 69 ] , including Mixup [ 85 ] , auto-augmentation [ 13 ] , and random erasing [ 88 ] . In addition, we provide throughput evaluation on different hardware. For GPU, we measure the throughput on an Nvidia V100, with the maximum power-of-two batchsize that fits in memory following [ 69 , 20 ] . For CPU and ONNX, we measure the runtime on an Intel Xeon E5-2690 v4 @ 2.60 GHz processor, with batchsize 16 and run the model in a single thread following [ 20 ] . We also test the transferability of EfficientViT on downstream tasks. For the experiments on downstream image classification, we finetune the models for 300 epochs following [ 86 ] , using AdamW [ 46 ] with batchsize 256, learning rate 1 × 10 − 3 absent superscript 10 3 \times 10^{-3} and weight-decay 1 × 10 − 8 absent superscript 10 8 \times 10^{-8} . We use RetinaNet [ 41 ] for object detection on COCO [ 42 ] , and train the models for 12 epochs (1 × \times schedule) with the same settings as [ 44 ] on mmdetection [ 6 ] . For instance segmentation, please refer to the supplementary.

4.2

Results on ImageNet

We compare EfficientViT with prevailing efficient CNN and ViT models on ImageNet [ 17 ] , and report the results in Tab. 2 and Fig. 1 . The results show that, in most cases, our EfficientViT achieves the best accuracy and speed trade-off across different evaluation settings.

Comparisons with efficient CNNs. We first compare EfficientViT with vanilla CNN models, such as MobileNets [ 26 , 63 ] and EfficientNet [ 67 ] . Specifically, compared to MobileNetV2 1.0 × \times [ 63 ] , EfficientViT-M3 obtains 1.4% better top-1 accuracy, while running at 2.5 × \times and 3.0 × \times faster speed on V100 GPU and Intel CPU, respectively. Compared to the state-of-the-art MobileNetV3-Large [ 26 ] , EfficientViT-M5 achieves 1.9% higher accuracy yet runs much faster, e.g. , 40.5% faster on the V100 GPU and 45.2% faster on the Intel CPU but is 11.5% slower as ONNX models. This may because reshaping is slower in ONNX implementation, which is inevitable in computing self-attention. Moreover, EfficientViT-M5 achieves comparable accuracy with the searched model EfficientNet-B0 [ 67 ] , while runs 2.3 × \times /1.9 × \times faster on the V100 GPU/Intel CPU, and 2.1 × \times faster as ONNX models. Although our model uses more parameters, it reduces memory-inefficient operations that affect the inference speed and achieves higher throughput.

Comparisons with efficient ViTs. We also compare our models with recent efficient vision transformers [ 9 , 50 , 51 , 56 , 5 ] in Tab. 2 . In particular, when getting similar performance on ImageNet-1K [ 17 ] , our EfficientViT-M4 runs 4.4 × \times and 3.0 × \times faster than the recent EdgeViT-XXS [ 56 ] on the tested CPU and GPU devices. Even converted to ONNX runtime format, our model still gets 3.7 × \times higher speed. Compared to the state-of-the-art MobileViTV2-0.5 [ 51 ] , our EfficientViT-M2 achieves slightly better performance with higher throughput, e.g. , 3.4 × \times and 3.5 × \times higher throughput tested on the GPU and CPU devices, respectively. Furthermore, we compare with tiny variants of state-of-the-art large ViTs in Tab. 3 . PoolFormer-12S [ 83 ] has comparable accuracy with EfficientViT-M5 yet runs 3.0 × \times slower on the V100 GPU. Compared to Swin-T [ 44 ] , EfficientViT-M5 is 4.1% inferior in accuracy yet is 12.3 × \times faster on the Intel CPU, demonstrating the efficiency of the proposed design. In addition, we present the speed evaluation and comparison on mobile chipsets in the supplementary material.

Finetune with higher resolutions. Recent works on ViTs have demonstrated that finetuning with higher resolutions can further improve the capacity of the models. We also finetune our largest model EfficientViT-M5 to higher resolutions. EfficientViT-M5 ↑ ↑ \uparrow 384 reaches 79.8% top-1 accuracy with throughput of 3,986 images/s on the V100 GPU, and EfficientViT-M5 ↑ ↑ \uparrow 512 further improves the top-1 accuracy to 80.8%, demonstrating the efficiency on processing images with larger resolutions and the good model capacity.

Table 3 : Comparison with the tiny variants of state-of-the-art large-scale ViTs on ImageNet-1K [ 17 ] .

Model
Top-1 Throughput (imgs/s) Flops Params

(%) GPU CPU ONNX (G) (M)

PVTV2-B0 [ 76 ]

70.5 3507 12.7 18.5 0.6 1.4

T2T-ViT-7 [ 84 ]

71.7 1156 22.5 16.1 1.1 4.3

DeiT-T [ 69 ]

72.2 4631 26.0 25.1 1.3 5.9

PoolFormer-12S [ 83 ]

77.2 3534 10.4 14.6 1.9 12.0

EffFormer-L1 [ 40 ]

79.2 4465 12.9 21.2 1.3 12.3

Swin-T [ 44 ]

81.2
1393 4.6 6.4 4.5 29.0

EfficientViT-M5
77.1
10621
56.8
62.5
0.5
12.4

Table 4 : Results of EfficientViT and other efficient models on downstream image classification datasets.

Model

Throughput

ImageNet

CIFAR10

CIFAR100

Cars

Flowers

Pets

MobileNetV1 [ 27 ]

8543 70.6 96.1 82.3 91.4
96.7 89.9

MobileNetV2 [ 63 ]

6534 72.9 95.7 80.8 91.0 96.6 90.5

MobileNetV3 [ 26 ]

7560 75.2 97.6 85.5 91.2 97.0 90.1

NASNet-A-M [ 89 ]

2623 74.1 96.8 83.9 88.5 96.8 89.4

ViT-S/16 [ 18 ]

2135 81.4 97.6 85.7

86.4 90.4

EfficientViT-M5
10621
77.1
98.0
86.4
89.7
97.1
92.0

4.3

Transfer Learning Results

To further evaluate the transfer ability, we apply EfficientViT on various downstream tasks.

Downstream Image Classification. We transfer EfficientViT to downstream image classification datasets to test its generalization ability: 1) CIFAR-10 and CIFAR-100 [ 36 ] ; 2) fine-grained classification: Flowers [ 55 ] , Stanford Cars [ 35 ] , and Oxford-IIIT Pets [ 58 ] . We report the results in Tab. 4 . Compared to existing efficient models [ 27 , 63 , 26 , 89 , 18 ] , our EfficientViT-M5 achieves comparable or slightly better accuracy across all datasets with much higher throughput. An exception lies in Cars, where our model is slightly inferior in accuracy. This may because the subtle differences between classes lie more in local details thus is more feasible to be captured with convolution.

Object Detection. We compare EfficientViT-M4 with efficient models [ 63 , 26 , 22 , 66 , 12 , 68 ] on the COCO [ 42 ] object detection task, and present the results in Tab. 5 . Specifically, EfficientViT-M4 surpasses MobileNetV2 [ 63 ] by 4.4% AP with comparable Flops. Compared to the searched method SPOS [ 22 ] , our EfficientViT-M4 uses 18.1% fewer Flops while achieving 2.0% higher AP, demonstrating its capacity and generalization ability in different vision tasks.

4.4

Ablation Study

In this section, we ablate important design elements in the proposed EfficientViT on ImageNet-1K [ 17 ] . All models are trained for 100 epochs to magnify the differences and reduce training time [ 20 ] . Tab. 6 reports the results.

Table 5 : EfficientViT object detection performance on COCO val2017 [ 42 ] with comparisons to other efficient models.

Model
RetinaNet 1 × \times

Flops Params

AP   AP   {\rm AP}    
 AP  50     subscript  AP  50    {\rm AP}_{50}    
 AP  75     subscript  AP  75    {\rm AP}_{75}    
 AP  s     subscript  AP  𝑠    {\rm AP}_{s}    
 AP  m     subscript  AP  𝑚    {\rm AP}_{m}    
 AP  l     subscript  AP  𝑙    {\rm AP}_{l}    

(M) (M)

MobileNetV2 [ 63 ]

28.3 46.7 29.3 14.8 30.7 38.1 300 3.4

MobileNetV3 [ 26 ]

29.9 49.3 30.8 14.9 33.3 41.1 217 5.4

SPOS [ 22 ]

30.7 49.8 32.2 15.4 33.9 41.6 365 4.3

MNASNet-A2 [ 66 ]

30.5 50.2 32.0 16.6 34.1 41.1 340 4.8

FairNAS-C [ 12 ]

31.2 50.8 32.7 16.3 34.4 42.3 325 5.6

MixNet-M [ 68 ]

31.3 51.7 32.4 17.0 35.0 41.9 360 5.0

EfficientViT-M4
32.7
52.2
34.1
17.6
35.3
46.0
299
8.8

Table 6 : Ablation for EfficientViT-M4 on ImageNet-1K [ 17 ] dataset. Top-1 accuracy, GPU and ONNX throughput are reported.

Ablation
Top-1 (%)
Throughput (imgs/s)

 GPU ONNX

1
EfficientViT-M4
71.3
 15914
108.6

2 Sandwich → Swin [ 44 ]

68.3  15804 114.5

3

𝒩 𝒩 \mathcal{N} = 1 → 2 70.2  14977 112.3

4

𝒩 𝒩 \mathcal{N} = 1 → 3 65.7  15856 139.7

5 CGA → MHSA [ 18 ]

70.2  16243 102.2

6 Cascade → None 69.8  16411 111.0

7 QKV allocation → None 69.9  15132 103.1

8 FFN ratio 2 → 4 69.8  15310 112.4

9 DWConv → None 69.9  16325 110.4

10 BN → LN [ 2 ]

70.4  15463 103.6

11 ReLU → HSwish [ 26 ]

72.2  15887 87.5

Impact of the sandwich layout block. We first present an ablation study to verify the efficiency of the proposed sandwich layout design, by replacing the sandwich layout block with the original Swin block [ 44 ] . The depth is adjusted to {2, 2, 3} to guarantee similar throughput with EfficientViT-M4 for a fair comparison. The top-1 accuracy degrades by 3.0% at a similar speed, verifying that applying more FFNs instead of memory-bound MHSA is more effective for small models. Furthermore, to analyze the impact of the number of FFNs 𝒩 𝒩 \mathcal{N} before and after self-attention , we change the number from 1 to 2 and 3. The number of blocks is reduced accordingly to maintain similar throughput. As presented in Tab. 6 (#3 and #4), further increasing the number of FFNs is not effective due to the lack of long-range spatial relation and 𝒩 𝒩 \mathcal{N} =1 achieves the best efficiency.

Impact of the cascaded group attention. We have proposed CGA to improve the computation efficiency of MHSA. As shown in Tab. 6 (#5 and #6), replacing CGA with MHSA decreases the accuracy by 1.1% and ONNX speed by 5.9%, suggesting that addressing head redundancy improves the model efficiency. For the model without the cascade operation, its performance is comparable with MHSA but worse than CGA, demonstrating the efficacy of enhancing the feature representations of each head.

Impact of the parameter reallocation. Our EfficientViT-M4 yields 1.4%/1.5% higher top-1 accuracy, 4.9%/3.8% higher GPU throughput than the models without Q 𝑄 Q K 𝐾 K V 𝑉 V channel dimension reallocation or FFN ratio reduction, respectively, indicating the effectiveness of parameter reallocation (#1 vs. #7, #8). Moreover, we study the choices of Q 𝑄 Q K 𝐾 K dimension in each head and the ratio of V 𝑉 V dimension to the input embedding in Fig. 7 . It is shown that the performance is improved gradually as Q 𝑄 Q K 𝐾 K dimension increases from 4 to 16, while further increasing it gives inferior performance. Besides, the performance improves from 70.3% to 71.3% when increasing the ratio between V 𝑉 V dimension and input embedding from 0.4 to 1.0. When further enlarging the ratio to 1.2, it only gets 0.1% improvements. Therefore, setting the channels of V 𝑉 V close to the input embedding achieves the best parameter efficiency, which meets our analysis in Sec. 2.3 and design strategy.

Impact of other components. We ablate the impact of using DWConv for token interaction, the normalization layer, and the activation function, as presented in Tab. 6 (#9, #10, and #11). With DWConv, the accuracy improves by 1.4% with a minor latency overhead, demonstrating the effectiveness of introducing local structural information. Replacing BN with LN decreases accuracy by 0.9% and GPU speed by 2.9%. Using HardSwish instead of ReLU improves accuracy by 0.9% but leads to a large drop of 20.0% ONNX speed. The activation functions are element-wise operations that occupy a considerable amount of processing time on GPU/CPU [ 48 , 15 , 72 ] , thus utilizing ReLU instead of more complicated activation functions is of better efficiency.

Table 7 : Performance comparison on ImageNet-1K [ 17 ] and ImageNet-ReaL [ 4 ] . Results with † † {\dagger} are trained with 1,000 epochs and knowledge distillation following LeViT [ 20 ] .

Model
ImageNet (%) Throughput (imgs/s) Flops Params

Top-1 Top-1 †

ReaL †

GPU CPU ONNX (M) (M)

LeViT-128S [ 20 ]

73.6 76.6 82.6 14457 82.3 80.9 305 7.8

EfficientViT-M4
74.3
77.1
83.6
15914
88.5
108.6
299
8.8

Results of 1,000 training epochs and distillation. Tab. 7 shows the results with 1,000 training epochs and knowledge distillation using RegNetY-16GF [ 60 ] as the teacher model following [ 20 ] on ImageNet-1K [ 17 ] and ImageNet-ReaL [ 4 ] . Compared to LeViT-128S [ 20 ] , EfficientViT-M4 surpasses it by 0.5% on ImageNet-1K and 1.0% on ImageNet-ReaL, respectively. For the inference speed, our model has 34.2% higher throughput on ONNX and also shows superiority on other settings. The results demonstrate that the strong capability and generalization ability of EfficientViT can be further explored with longer training schedules.

5

Related Work

Efficient CNNs. With the demand of deploying CNNs on resource-constrained scenarios, efficient CNNs have been intensively studied in literature [ 63 , 26 , 48 , 67 , 87 , 23 , 24 ] . Xception [ 10 ] proposes an architecture built with depthwise separable convolutions. MobileNetV2 [ 63 ] builds an inverted residual structure which expands the input to a higher dimension. MobileNetV3 [ 26 ] and EfficientNet [ 67 ] resort to neural architecture search techniques to design compact models. To boost the actual speed on hardware, ShuffleNetV2 [ 48 ] introduces channel split and shuffle operations to improve the information communication among channel groups. However, the spatial locality of convolutional kernels hampers CNN models from capturing long-range dependencies, thus limiting their model capacity.

Efficient ViTs. ViT and its variants [ 18 , 69 , 44 , 76 ] have achieved success on various vision tasks. Despite the superior performance, most of them are inferior to typical CNNs in inference speed. Some efficient transformers have been proposed recently and they fall into two camps: 1) efficient self-attention; and 2) efficient architecture design. Efficient self-attention methods reduce the cost of softmax attention via sparse attention [ 34 , 75 , 61 , 57 ] or low-rank approximation [ 11 , 74 , 51 ] . However, they suffer from performance degradation with negligible or moderate inference acceleration over softmax attention [ 71 ] . Another line of work combines ViTs with lightweight CNNs to build efficient architectures [ 81 , 50 , 9 , 49 , 47 ] . LVT [ 81 ] proposes enhanced attention mechanisms with dilated convolution to improve the model performance and efficiency. Mobile-Former [ 9 ] designs a parallel CNN-transformer block to encode both local features and global interaction. However, most of them target at minimizing Flops and parameters [ 16 ] , which could have low correlations with actual inference latency [ 70 ] and still inferior to efficient CNNs in speed. Different from them, we explore models with fast inference by directly optimizing their throughput on different hardware and deployment settings, and design a family of hierarchical models with a good trade-off between speed and accuracy.

Figure 7 : Ablation on the Q 𝑄 Q K 𝐾 K dimension of each head and the ratio of V 𝑉 V dimension to the input embedding.

6

Conclusion

In this paper, we have presented a systematic analysis on the factors that affect the inference speed of vision transformers, and proposed a new family of fast vision transformers with memory-efficient operations and cascaded group attention, named EfficientViT. Extensive experiments have demonstrated the efficacy and high speed of EfficientViT, and also show its superiority on various downstream benchmarks.

Limitations . One limitation of EfficientViT is that, despite its high inference speed, the model size is slightly larger compared to state-of-the-art efficient CNN [ 26 ] due to the extra FFNs in the introduced sandwich layout. Besides, our models are designed manually based on the derived guidelines on building efficient vision transformers. In future work, we are interested in reducing the model size and incorporating automatic search techniques to further enhance the model capacity and efficiency.

Acknowledgement. Prof. Yuan was partially supported by Hong Kong Research Grants Council (RGC) General Research Fund 11211221, and Innovation and Technology Commission-Innovation and Technology Fund ITS/100/20.

References

[1]

Apple.

Coremltools: Use coremltools to convert machine learning models from third-party libraries to the core ml format, 2021.

[2]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.

Layer normalization.

arXiv , 2016.

[3]

Junjie Bai, Fang Lu, Ke Zhang, et al.

Onnx: Open neural network exchange.

https://github.com/onnx/onnx , 2019.

[4]

Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord.

Are we done with imagenet?

arXiv preprint arXiv:2006.07159 , 2020.

[5]

Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang.

Glit: Neural architecture search for global and local image transformer.

In ICCV , 2021.

[6]

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.

Mmdetection: Open mmlab detection toolbox and benchmark.

arXiv preprint arXiv:1906.07155 , 2019.

[7]

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling.

Autoformer: Searching transformers for visual recognition.

In ICCV , 2021.

[8]

Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou.

Auto-scaling vision transformers without training.

In ICLR , 2021.

[9]

Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu.

Mobile-former: Bridging mobilenet and transformer.

In CVPR , 2022.

[10]

François Chollet.

Xception: Deep learning with depthwise separable convolutions.

In CVPR , 2017.

[11]

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.

Rethinking attention with performers.

In ICLR , 2021.

[12]

Xiangxiang Chu, Bo Zhang, and Ruijun Xu.

Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search.

In ICCV , pages 12239–12248, 2021.

[13]

Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.

Autoaugment: Learning augmentation strategies from data.

In CVPR , pages 113–123, 2019.

[14]

Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan.

Coatnet: Marrying convolution and attention for all data sizes.

NeurIPS , 34:3965–3977, 2021.

[15]

Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

Flashattention: Fast and memory-efficient exact attention with io-awareness.

arXiv preprint arXiv:2205.14135 , 2022.

[16]

Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay.

The efficiency misnomer.

In ICLR , 2022.

[17]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.

Imagenet: A large-scale hierarchical image database.

In CVPR , 2009.

[18]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

An image is worth 16x16 words: Transformers for image recognition at scale.

ICLR , 2021.

[19]

Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra.

NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training.

In ICLR , 2022.

[20]

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.

Levit: a vision transformer in convnet’s clothing for faster inference.

In ICCV , 2021.

[21]

Jiaqi Gu, Hanqing Zhu, Chenghao Feng, Mingjie Liu, Zixuan Jiang, Ray T Chen, and David Z Pan.

Towards memory-efficient neural networks via multi-level in situ generation.

In ICCV , pages 5229–5238, 2021.

[22]

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun.

Single path one-shot neural architecture search with uniform sampling.

In ECCV , pages 544–560. Springer, 2020.

[23]

Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu.

Ghostnet: More features from cheap operations.

In CVPR , pages 1580–1589, 2020.

[24]

Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chunjing Xu, Enhua Wu, and Qi Tian.

Ghostnets on heterogeneous devices via cheap operations.

Int. J. Comput. Vis. , 130(4):1050–1069, 2022.

[25]

Dan Hendrycks and Kevin Gimpel.

Gaussian error linear units (gelus).

arXiv , 2016.

[26]

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.

Searching for mobilenetv3.

In ICCV , 2019.

[27]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.

Mobilenets: Efficient convolutional neural networks for mobile vision applications.

arXiv preprint arXiv:1704.04861 , 2017.

[28]

Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.

Transformer quality in linear time.

In ICML , pages 9099–9117. PMLR, 2022.

[29]

Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu.

Lightvit: Towards light-weight convolution-free vision transformers.

arXiv preprint arXiv:2207.05557 , 2022.

[30]

Sergey Ioffe and Christian Szegedy.

Batch normalization: Accelerating deep network training by reducing internal covariate shift.

In ICML , 2015.

[31]

Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler.

Data movement is all you need: A case study on optimizing transformers.

MLSys , 3:711–732, 2021.

[32]

Congfeng Jiang, Yitao Qiu, Weisong Shi, Zhefeng Ge, Jiwei Wang, Shenglei Chen, Christophe Cerin, Zujie Ren, Guoyao Xu, and Jiangbin Lin.

Characterizing co-located workloads in alibaba cloud datacenters.

IEEE Trans. on Cloud Comput. , 2020.

[33]

Sheng-Chun Kao, Suvinay Subramanian, Gaurav Agrawal, and Tushar Krishna.

An optimized dataflow for mitigating attention performance bottlenecks.

arXiv preprint arXiv:2107.06419 , 2021.

[34]

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.

Reformer: The efficient transformer.

In ICLR , 2020.

[35]

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.

3d object representations for fine-grained categorization.

In ICCVW , pages 554–561, 2013.

[36]

Alex Krizhevsky, Geoffrey Hinton, et al.

Learning multiple layers of features from tiny images.

[37]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.

Imagenet classification with deep convolutional neural networks.

Commun. ACM , 60(6):84–90, 2017.

[38]

Yann LeCun, John Denker, and Sara Solla.

Optimal brain damage.

NeurIPS , 2, 1989.

[39]

Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang.

Ds-net++: Dynamic weight slicing for efficient inference in cnns and vision transformers.

IEEE Trans. Pattern Anal. Mach. Intell. , 2022.

[40]

Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren.

Efficientformer: Vision transformers at mobilenet speed.

In NeurIPS , 2022.

[41]

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.

Focal loss for dense object detection.

In ICCV , pages 2980–2988, 2017.

[42]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.

Microsoft coco: Common objects in context.

In ECCV , 2014.

[43]

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.

Swin transformer v2: Scaling up capacity and resolution.

In CVPR , 2022.

[44]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.

Swin transformer: Hierarchical vision transformer using shifted windows.

In ICCV , 2021.

[45]

Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell.

Rethinking the value of network pruning.

In ICLR , 2018.

[46]

Ilya Loshchilov and Frank Hutter.

Decoupled weight decay regularization.

In ICLR , 2018.

[47]

Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, and Rongrong Ji.

Towards lightweight transformer via group-wise transformation for vision-and-language tasks.

IEEE Trans. Image Process. , 31:3386–3398, 2022.

[48]

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.

Shufflenet v2: Practical guidelines for efficient cnn architecture design.

In ECCV , pages 116–131, 2018.

[49]

Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammad Anwer, and Fahad Shahbaz Khan.

Edgenext: efficiently amalgamated cnn-transformer architecture for mobile vision applications.

In ECCVW , 2022.

[50]

Sachin Mehta and Mohammad Rastegari.

Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer.

In ICLR , 2021.

[51]

Sachin Mehta and Mohammad Rastegari.

Separable self-attention for mobile vision transformers.

arXiv preprint arXiv:2206.02680 , 2022.

[52]

Paul Michel, Omer Levy, and Graham Neubig.

Are sixteen heads really better than one?

NeurIPS , 32, 2019.

[53]

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz.

Importance estimation for neural network pruning.

In CVPR , pages 11264–11272, 2019.

[54]

Vinod Nair and Geoffrey E Hinton.

Rectified linear units improve restricted boltzmann machines.

In ICML , 2010.

[55]

M-E Nilsback and Andrew Zisserman.

A visual vocabulary for flower classification.

In CVPR , 2006.

[56]

Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez.

Edgevits: Competing light-weight cnns on mobile devices with vision transformers.

In ECCV , 2022.

[57]

Zizheng Pan, Jianfei Cai, and Bohan Zhuang.

Fast vision transformers with hilo attention.

NeurIPS , 2022.

[58]

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.

Cats and dogs.

In CVPR , pages 3498–3505, 2012.

[59]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.

Pytorch: An imperative style, high-performance deep learning library.

NeurIPS , 2019.

[60]

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.

Designing network design spaces.

In CVPR , pages 10428–10436, 2020.

[61]

Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai.

Combiner: Full attention transformer with sparse computation cost.

NeurIPS , 34:22470–22482, 2021.

[62]

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby.

Scaling vision with sparse mixture of experts.

NeurIPS , 34:8583–8595, 2021.

[63]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.

Mobilenetv2: Inverted residuals and linear bottlenecks.

In CVPR , pages 4510–4520, 2018.

[64]

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.

Rethinking the inception architecture for computer vision.

In CVPR , pages 2818–2826, 2016.

[65]

Hamid Tabani, Ajay Balasubramaniam, Shabbir Marzban, Elahe Arani, and Bahram Zonooz.

Improving the efficiency of transformers for resource-constrained devices.

In DSD , pages 449–456, 2021.

[66]

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.

Mnasnet: Platform-aware neural architecture search for mobile.

In CVPR , pages 2820–2828, 2019.

[67]

Mingxing Tan and Quoc Le.

Efficientnet: Rethinking model scaling for convolutional neural networks.

In ICML , 2019.

[68]

Mingxing Tan and Quoc V Le.

Mixconv: Mixed depthwise convolutional kernels.

In BMVC , 2019.

[69]

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.

Training data-efficient image transformers & distillation through attention.

In ICML . PMLR, 2021.

[70]

Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan.

An improved one millisecond mobile backbone.

arXiv preprint arXiv:2206.04040 , 2022.

[71]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

Attention is all you need.

In NeurIPS , 2017.

[72]

Anand Venkat, Tharindu Rusira, Raj Barik, Mary Hall, and Leonard Truong.

Swirl: High-performance many-core cpu code generation for deep neural networks.

Int. J. High Perform. Comput. Appl. , 33(6):1275–1289, 2019.

[73]

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.

In ACL , pages 5797–5808, 2019.

[74]

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.

Linformer: Self-attention with linear complexity.

arXiv preprint arXiv:2006.04768 , 2020.

[75]

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.

Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.

In ICCV , 2021.

[76]

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.

Pvtv2: Improved baselines with pyramid vision transformer.

Comput. Vis. Media , 2022.

[77]

Ross Wightman.

Pytorch image models, 2019.

[78]

Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.

Tinyvit: Fast pretraining distillation for small vision transformers.

In ECCV , 2022.

[79]

Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo.

Pale transformer: A general vision transformer backbone with pale-shaped attention.

In AAAI , 2022.

[80]

Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick.

Early convolutions help transformers see better.

NeurIPS , 2021.

[81]

Chenglin Yang, Yilin Wang, Jianming Zhang, He Zhang, Zijun Wei, Zhe Lin, and Alan Yuille.

Lite vision transformer with enhanced self-attention.

In CVPR , pages 11998–12008, 2022.

[82]

Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz.

Nvit: Vision transformer compression and parameter redistribution.

arXiv preprint arXiv:2110.04869 , 2021.

[83]

Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.

Metaformer is actually what you need for vision.

In CVPR , pages 10819–10829, 2022.

[84]

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.

Tokens-to-token vit: Training vision transformers from scratch on imagenet.

ICCV , 2021.

[85]

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.

mixup: Beyond empirical risk minimization.

In ICLR , 2018.

[86]

Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.

Minivit: Compressing vision transformers with weight multiplexing.

In CVPR , 2022.

[87]

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.

Shufflenet: An extremely efficient convolutional neural network for mobile devices.

In CVPR , 2018.

[88]

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.

Random erasing data augmentation.

In AAAI , 2020.

[89]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.

Learning transferable architectures for scalable image recognition.

In CVPR , pages 8697–8710, 2018.

◄

 Feeling lucky? 
  
 Conversion report 
 Report an issue 
 View original on arXiv  ►