#weight tying

Retrofitting Recurrent Depth into a Pretrained Language Model:⭐7

Recurrent depth retrofit boosts pretrained models' iterative reasoning, beats scratchpads, extends beyond learned depth, at two parameter budgets.