Source-Optimal Training is Transfer-Suboptimal
By: C. Evans Hedges
Potential Business Impact:
Improves computer learning by finding the best way to teach it.
We prove a fundamental misalignment in transfer learning: the source regularization that minimizes source risk almost never coincides with the regularization maximizing transfer benefit. Through sharp phase boundaries for L2-SP ridge regression, we characterize the transfer-optimal source penalty $τ_0^*$ and show it diverges predictably from task-optimal values, requiring stronger regularization in high-SNR regimes and weaker regularization in low-SNR regimes. Additionally, in isotropic settings the decision to transfer is remarkably independent of target sample size and noise, depending only on task alignment and source characteristics. CIFAR-10 and MNIST experiments confirm this counterintuitive pattern persists in non-linear networks.
Similar Papers
Sparse Optimization for Transfer Learning: A L0-Regularized Framework for Multi-Source Domain Adaptation
Machine Learning (Stat)
Makes computer learning faster and more accurate.
Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Machine Learning (Stat)
Teaches computers to learn from new data faster.
Intuitions of Machine Learning Researchers about Transfer Learning for Medical Image Classification
CV and Pattern Recognition
Helps AI learn faster from medical pictures.