Sakana AI's PC-ALM: A Layer-Local Alternative to Backpropagation That Trains 1,000-Layer Networks
Backpropagation is a global algorithm: a forward pass, then a backward pass, then a weight update, each stage locked behind the previous one. Brains have no known mechanism for that kind of network-wide phase locking, which is why local-learning alternatives such as predictive coding (PC) keep drawing research interest. Sakana AI researchers propose Augmented Lagrangian Predictive Coding (PC-ALM), a variant of PC that keeps every update layer-local yet recovers backprop-aligned credit signals. The team reports training residual MLPs up to 1,000 layers within roughly 2 percentage points of backpropagation on MNIST.
Is it deployable? Yes, as research code: an MIT-licensed JAX reference implementation runs on CPU and reproduces the paper's width-depth grid. It is a training method, not a model, and has only been tested on small image benchmarks so far.
Why Standard PC Stalls in Deep, Narrow Networks
PC treats every hidden activation as an optimization variable and penalizes the squared mismatch between each layer's activation and the prediction arriving from the layer below. Inference is gradient descent on that energy; learning is a Hebbian-like weight step. The catch is that supervision enters at the output and must diffuse through a chain of local compromises. In deep, narrow networks the credit signal fades long before it reaches the input. Innocenti et al. characterized this PC–BP gap as a function of width and depth, and it is worst when width is smaller than depth.
What PC-ALM Changes
PC-ALM starts from the constrained view of training: minimize the supervised loss subject to the constraint that each layer's activation equals the nonlinear transform of the previous layer's output, expressed as \(hi = \sigma(Wi h_{i-1})\). Where standard PC handles this constraint softly through energy penalties, PC-ALM enforces it with an augmented Lagrangian, introducing per-layer multipliers that carry structured error information through the network. This yields update rules that remain strictly layer-local while producing credit signals closely aligned with those of backpropagation—the key to closing the gap on deep, narrow architectures.
via MarkTechPost