Which Transformer Blocks Should You Keep?

Removing whole blocks reduces the parameter count and FLOPs of a pretrained transformer without requiring sparse kernels. We exhaustively evaluate all 2¹²−1 = 4095 nonempty block subsets of a 12-block vision foundation model (DINOv3 ViT-S/16) after three-epoch end-to-end finetuning on ImageNet-1k. Under this protocol, top-1 accuracy varies by as much as 28.6 percentage points among subsets of the same block budget.

A common baseline, prefix truncation, simply retains the first k blocks. Its accuracy falls below the mean of all subsets at every pruned depth k ∈ {1,…,11}, so a uniformly selected subset has higher expected accuracy. The value of a block is also budget-dependent: early and late blocks exchange importance as the budget increases, while the final block remains beneficial throughout. Modest local search on the complete landscape approaches the best observed subsets and outperforms random search. We release the complete measured landscape as a benchmark for budget-conditional structured on-device compression.

Subset choice matters at a fixed parameter count

Every subset with k blocks has the same deployed size, 0.68M + 1.77M·k parameters, and the same block-level FLOPs. Yet the spread between the best and worst k-block model grows from 7.3 points at k=2 to 28.6 at k=6 before shrinking to 2.4 at k=11. The best 4-block network (52.09%, 7.8M parameters) beats the worst 7-block network (50.36%, 13.1M) with 41% fewer parameters.

Truncation is a poor default

Keeping the first k blocks is the obvious static rule, and early-exit methods implicitly use it. In this model it ranks below the subset mean at every pruned depth. At least 94% of subsets beat it for k ∈ {7,8,9}, and every other subset beats it at k ∈ {10,11}.

The best subsets mix early and late blocks, and they are not nested

For every k from 3 to 11 the best observed subset keeps blocks 3, 4 and 12 and drops block 1. At intermediate depths it adds a run of later blocks (7–8 at k=5, 7–11 at k=8). In 4 of the 11 steps from k to k+1, the best subset swaps a block out, so no procedure that only adds or only removes blocks recovers the best subset at every budget.

Block importance is budget-dependent, not intrinsic

At k=4, subsets containing block 3 average 45.96% against 42.52% for those without it, so m₃(4) = +3.45. At k=8 the same contrast is −0.61. A ranking computed at one budget need not transfer to another, which is a problem for methods that score blocks once on the full model and reuse that ranking. Keeping block 12 alone is not enough either: the first k−1 blocks plus block 12 beat truncation at every budget but still sit below the subset mean for every k ≥ 5.

Deletion costs interact, so additive scoring fails

Deleting a single block and finetuning the remaining 11 costs between 0.66 points (block 1) and 3.10 points (block 12). The cost generally grows with depth, the opposite of LLM pruning practice, which tends to drop deeper blocks. If those costs added up, deleting blocks i and j would cost d_i+d_j. Instead every one of the 66 pairs costs more than the sum, and the resulting additive predictor ranks subsets backwards at tight budgets.

Replay a search on the measured landscape

Because every subset is measured, a search strategy can be replayed by lookup, with no training. Each lookup is charged against a budget B; in a new setting each one would cost a 3-epoch ImageNet finetune (36–80 GPU-minutes on an RTX 8000 for k = 3 to 9). Local search starts from a random subset, tries swapping one kept block for one dropped block in random order, moves on the first improvement, and restarts when stuck. In the paper's 300-trial replay it comes within 0.18 points of the best observed subset at B=50 and 0.02 at B=100; uniform sampling leaves 0.79 and 0.50. Truncation and the additive predictor fall short by 8.5 and 15.4 points.

What the map cost, and what a search costs

The 4,095 three-epoch runs took 4,034 GPU-hours on NVIDIA RTX 8000s, one GPU per candidate. A search with B=50 at k = 3, 6 or 9 would cost roughly 30, 48 or 66 GPU-hours. Random-search candidates train independently; local search can run restarts or neighbour evaluations in parallel.

Is three epochs enough? Does the backbone need finetuning?

For 160 subsets the authors also finetuned for 30 epochs. Their 3-epoch and 30-epoch accuracies agree closely in rank (Spearman ρ = 0.99), which supports the short protocol for screening. Freezing the backbone is not an option: removing a single block with a frozen backbone costs between 7.6 and 74.9 points more than finetuning the same subset.

Citation