1
Introduction
2
Setting
▶
2.1
Feature maps
2.2
Particle measures
2.3
Synthesis, analysis, kernel operator and regularized ridgelet transform
2.4
Risks, Gibbs reference measure and free energy
2.5
Finite width and finite sample
2.6
Samples, empirical features and the risk with labels
2.7
Rademacher complexity
2.8
Synthesis of a signed measure
2.9
Dynamics: log-Sobolev inequality, proximal Gibbs measure, mean-field Langevin flow
2.10
The finite particle system: Gibbs measure and free energy
2.11
Convergence rates: the weighted Tikhonov problem
3
Basic lemmas of the setting
▶
3.1
Second moments on the domain
3.2
Gibbs variational principle
3.3
Synthesis operator, adjoint, kernel and regularized ridgelet transform
3.4
Characterization of the minimizer of the free energy
3.5
Existence of the minimizer
4
Structure of the minimizers: the null-space effect
▶
4.1
Gibbs minimizer of the free energy
4.2
Stationary points of the gradient flow with weight decay
4.3
Without amplitude regularization the null-space component survives
5
Sample-size threshold: from the empirical to the population Gibbs minimizer
▶
5.1
Empirical Gibbs structure, moment bounds and tails
5.2
Uniform deviation over the class of bounded first amplitude moment
5.3
Complexity of the class of tanh ridge functions
5.4
The main estimate, the sample-size threshold and the convergence of the learned quantities
5.5
Improved rate via linearization
6
Regularization limits and the geometry of the limit
▶
6.1
Reference measure and representability
6.2
Energy comparison
6.3
Convergence to the canonical ridgelet transform
6.4
Order of the limits
6.5
The zero-temperature limit: total-variation regularization
7
Convergence rates for the regularization limit
▶
7.1
Rewriting as a weighted Tikhonov problem
7.2
The rate theorem and its corollaries
7.3
Spectral powers of \(T_0\) and the source condition of general order
7.4
Joint schedule of the sample size and the regularization
8
Dynamics: mean-field Langevin, log-Sobolev inequalities
▶
8.1
Entropy sandwich
8.2
Log-Sobolev inequality of the proximal Gibbs measures
8.3
Exponential convergence of the mean-field Langevin dynamics
8.4
Static chaos of the finite-particle Gibbs measure
8.5
The third stage of the order of the limits: \(t\to \infty \) at fixed \(M\) first
9
Explicit geometry of the limit
10
Heavy-tailed hidden laws and Sobolev-type representability
11
Material for lean-operator-ridgelet
12
Material for FoML
13
Material for Mathlib
Dependency graph
Shallow Learning Tends to Ridgelet Transform
Sho Sonoda
1
Introduction
2
Setting
2.1
Feature maps
2.2
Particle measures
2.3
Synthesis, analysis, kernel operator and regularized ridgelet transform
2.4
Risks, Gibbs reference measure and free energy
2.5
Finite width and finite sample
2.6
Samples, empirical features and the risk with labels
2.7
Rademacher complexity
2.8
Synthesis of a signed measure
2.9
Dynamics: log-Sobolev inequality, proximal Gibbs measure, mean-field Langevin flow
2.10
The finite particle system: Gibbs measure and free energy
2.11
Convergence rates: the weighted Tikhonov problem
3
Basic lemmas of the setting
3.1
Second moments on the domain
3.2
Gibbs variational principle
3.3
Synthesis operator, adjoint, kernel and regularized ridgelet transform
3.4
Characterization of the minimizer of the free energy
3.5
Existence of the minimizer
4
Structure of the minimizers: the null-space effect
4.1
Gibbs minimizer of the free energy
4.2
Stationary points of the gradient flow with weight decay
4.3
Without amplitude regularization the null-space component survives
5
Sample-size threshold: from the empirical to the population Gibbs minimizer
5.1
Empirical Gibbs structure, moment bounds and tails
5.2
Uniform deviation over the class of bounded first amplitude moment
5.3
Complexity of the class of tanh ridge functions
5.4
The main estimate, the sample-size threshold and the convergence of the learned quantities
5.5
Improved rate via linearization
6
Regularization limits and the geometry of the limit
6.1
Reference measure and representability
6.2
Energy comparison
6.3
Convergence to the canonical ridgelet transform
6.4
Order of the limits
6.5
The zero-temperature limit: total-variation regularization
7
Convergence rates for the regularization limit
7.1
Rewriting as a weighted Tikhonov problem
7.2
The rate theorem and its corollaries
7.3
Spectral powers of \(T_0\) and the source condition of general order
7.4
Joint schedule of the sample size and the regularization
8
Dynamics: mean-field Langevin, log-Sobolev inequalities
8.1
Entropy sandwich
8.2
Log-Sobolev inequality of the proximal Gibbs measures
8.3
Exponential convergence of the mean-field Langevin dynamics
8.4
Static chaos of the finite-particle Gibbs measure
8.5
The third stage of the order of the limits: \(t\to \infty \) at fixed \(M\) first
9
Explicit geometry of the limit
10
Heavy-tailed hidden laws and Sobolev-type representability
11
Material for lean-operator-ridgelet
12
Material for FoML
13
Material for Mathlib