Let \({\mathcal X}\) and \(Z\) be measurable spaces. A feature map is a jointly measurable function \(\varphi \colon Z\times {\mathcal X}\to {\mathbb R}\), \((z,x)\mapsto \varphi _z(x)\), with \(|\varphi _z(x)|\le 1\) for all \(z\in Z\) and \(x\in {\mathcal X}\). Assumption (A3) of the paper, \(\varphi _{(w,b)}(x)=\tanh (w^\top x-b)\) on \(Z={\mathbb R}^m\times {\mathbb R}\), is the instance ‘tanhFeature‘.

Let \(Z\) be a metric space. The feature map \(\varphi \) is Lipschitz in the hidden parameter if there is \(L\ge 0\) such that \(|\varphi _z(x)-\varphi _{z'}(x)|\le L\, d(z,z')\) for all \(x\in {\mathcal X}\) and \(z,z'\in Z\). Under (A1) and (A3) this holds with \(L=\sqrt{R_X^2+1}\).

For a feature map \(\varphi \) on \(Z\times {\mathcal X}\) and a subset \(A\subseteq {\mathcal X}\), the restriction of \(\varphi \) to \(A\) is the feature map \(Z\times A\to {\mathbb R}\), \((z,x)\mapsto \varphi _z(x)\); it is used with \(A=\{ |x|\le R_X\} \), which is assumption (A1).

For a feature map \(\varphi \) on \(Z\times {\mathcal X}\) and a measurable map \(e\colon Z'\to Z\), the reparametrized feature map on \(Z'\times {\mathcal X}\) is \((z',x)\mapsto \varphi _{e(z')}(x)\). It is used to identify the hidden-parameter space \({\mathbb R}^m\times {\mathbb R}\) of (A3) with the Euclidean space \({\mathbb R}^{m+1}\).

(A3) Let \(m\ge 1\), \({\mathcal X}={\mathbb R}^m\) and \(Z={\mathbb R}^m\times {\mathbb R}\). The hidden parameter is \(z=(w,b)\) and \(\varphi _z(x):=\tanh (w^\top x-b)\). Then \(|\varphi _z|\le 1\) and \((z,x)\mapsto \varphi _z(x)\) is continuous, so \(\varphi \) is a feature map in the sense of Definition 1.

(A3) with the Euclidean hidden-parameter space: the tanh feature \(\varphi _{(w,b)}(x)=\tanh (w^\top x-b)\) of Definition 5 with \(z=(w,b)\) in \({\mathbb R}^{m+1}={\mathbb R}^m\times {\mathbb R}\) carrying the Euclidean norm \(|(w,b)|=\sqrt{|w|^2+b^2}\) (the reparametrization, Definition 4, along the identity \({\mathbb R}^{m+1}\to {\mathbb R}^m\times {\mathbb R}\)).

For \(\rho \in {\mathcal P}(\Theta )\), \(\Theta ={\mathbb R}\times Z\), with \(\int |a|\, \mathrm d\rho {\lt}\infty \) the realization is \(F_\rho (x):=\int _\Theta a\, \varphi _z(x)\, \rho (\, \mathrm da\, \mathrm dz)\), so that \(|F_\rho (x)|\le \int |a|\, \mathrm d\rho \).

The hidden marginal of \(\rho \in {\mathcal P}(\Theta )\) is \(\nu _\rho (\, \mathrm dz):=\int _{{\mathbb R}}\rho (\, \mathrm da\, \mathrm dz)\), the image of \(\rho \) under \((a,z)\mapsto z\).

The conditional mean amplitude of \(\rho \in {\mathcal P}(\Theta )\) is \(m_\rho (z):={\mathbb E}_\rho [a\mid z]\), defined \(\nu _\rho \)-a.e. through the disintegration \(\rho (\, \mathrm da\, \mathrm dz)=\nu _\rho (\, \mathrm dz)\, \kappa (z,\, \mathrm da)\) as \(m_\rho (z)=\int _{{\mathbb R}} a\, \kappa (z,\, \mathrm da)\).

The coefficient measure of \(\rho \in {\mathcal P}(\Theta )\) with \(\int |a|\, \mathrm d\rho {\lt}\infty \) is the signed measure \(\Pi \rho (\, \mathrm dz):=\int _{{\mathbb R}} a\, \rho (\, \mathrm da\, \mathrm dz)\) on \(Z\), i.e. \(\Pi \rho (A)=\int _{{\mathbb R}\times A}a\, \, \mathrm d\rho \); by disintegration \(\Pi \rho =m_\rho \, \nu _\rho \).

For particles \(\theta =(\theta _\ell )_{\ell =1}^M\in \Theta ^M\) the empirical particle measure is \(\rho _\theta :=\frac1M\sum _{\ell =1}^M\delta _{\theta _\ell } \in {\mathcal P}(\Theta )\).

For \(r\in L^2(P_X)\) the analysis map is \((S^*r)(z):=\langle \varphi _z,r\rangle _{L^2(P_X)}=\int _{{\mathcal X}}\varphi _z(x)r(x)\, P_X(\, \mathrm dx)\), a function of \(z\in Z\) that does not depend on the base measure \(\nu \). By Lemma ?? it is the adjoint of \(S_\nu \) for every \(\nu \).

For a base measure \(\nu \in {\mathcal P}(Z)\) and \(u\colon Z\to {\mathbb R}\) the synthesis of \(u\) is the function \((S_\nu u)(x):=\int _Z u(z)\varphi _z(x)\, \nu (\, \mathrm dz)\) on \({\mathcal X}\).

For \(\nu \in {\mathcal P}(Z)\) the synthesis operator is \(S_\nu \colon L^2(\nu )\to L^2(P_X)\), \((S_\nu u)(x):=\int _Z u(z)\varphi _z(x)\, \nu (\, \mathrm dz)\). It is a well-defined bounded linear operator with \(\| S_\nu \| \le 1\), since \(|(S_\nu u)(x)|\le \int |u|\, \mathrm d\nu \le \| u\| _{L^2(\nu )}\) pointwise.

The adjoint \(S_\nu ^*\colon L^2(P_X)\to L^2(\nu )\) of the synthesis operator. By Lemma ??, \((S_\nu ^*r)(z)=\langle \varphi _z,r\rangle _{L^2(P_X)}=(S^*r)(z)\) \(\nu \)-a.e.

The kernel operator on \(L^2(P_X)\) is \(K_\nu :=S_\nu S_\nu ^*\); its integral kernel is \(k_\nu (x,x')=\int \varphi _z(x)\varphi _z(x')\, \nu (\, \mathrm dz)\).

For \(\lambda {\gt}0\) the resolvent \((K_\nu +\lambda )^{-1}\) is the inverse of the bounded operator \(K_\nu +\lambda \) on \(L^2(P_X)\); by Lemma ?? it exists and \(\| (K_\nu +\lambda )^{-1}\| \le 1/\lambda \). (In Lean it is defined as the inverse when \(K_\nu +\lambda \) is invertible and as \(0\) otherwise.)

For \(\lambda {\gt}0\), \(\nu \in {\mathcal P}(Z)\) and \(f\in L^2(P_X)\) the regularized ridgelet transform is \(R_{\lambda ,\nu }f:=S^*(K_\nu +\lambda )^{-1}f\), i.e. \((R_{\lambda ,\nu }f)(z)=\langle \varphi _z,(K_\nu +\lambda )^{-1}f\rangle _{L^2(P_X)}\), a bounded continuous function of \(z\) that lies in \(L^2(\nu )\).

For \(f\in L^2(P_X)\) the population risk of \(\rho \in {\mathcal P}(\Theta )\) is \(L(\rho ):=\tfrac 12\| F_\rho -f\| _{L^2(P_X)}^2=\tfrac 12\int _{{\mathcal X}}(F_\rho (x)-f(x))^2\, P_X(\, \mathrm dx)\), with residual \(r_\rho :=F_\rho -f\).

For a sample \((x_i,y_i)_{i=1}^N\) the empirical risk of \(\rho \in {\mathcal P}(\Theta )\) is \(L_N(\rho ):=\frac1{2N}\sum _{i=1}^N\bigl(F_\rho (x_i)-y_i\bigr)^2\).

Let \(\lambda ,\beta {\gt}0\) and \(\nu _0\in {\mathcal P}(Z)\) (in the paper \(\nu _0(\, \mathrm dz)=Z_0^{-1}e^{-V(z)/\beta }\, \mathrm dz\)). With \(U(\theta )=\frac\lambda 2a^2+V(z)\) the Gibbs reference measure is \(\mu _U(\, \mathrm d\theta ):=Z_U^{-1}e^{-U(\theta )/\beta }\, \mathrm d\theta ={\mathcal N}(0,\beta /\lambda )(\, \mathrm da)\otimes \nu _0(\, \mathrm dz)\).

The domain of the free energy is \({\mathcal D}:=\{ \rho \in {\mathcal P}(\Theta ):\operatorname {KL}(\rho \| \mu _U){\lt}\infty \} \). Elements of \({\mathcal D}\) are not required to satisfy \(\int V\, \mathrm d\rho {\lt}\infty \).

The free energy on \({\mathcal D}\) is \({\mathcal F}(\rho ):=L(\rho )+\beta \, \operatorname {KL}(\rho \| \mu _U)\) (the paper’s definition contains the additional constant \(-\beta \log Z_U\), which is dropped here). By Lemma 183, whenever \(\int U\, \mathrm d\rho {\lt}\infty \) one has \({\mathcal F}(\rho )=L(\rho )+\int U\, \mathrm d\rho +\beta \operatorname {Ent}(\rho )-\beta \log Z_U\).

The empirical free energy \({\mathcal F}_N\) is obtained from \({\mathcal F}\) by replacing \(L\) with \(L_N\): \({\mathcal F}_N(\rho ):=L_N(\rho )+\beta \, \operatorname {KL}(\rho \| \mu _U)\).

For a reference measure \(\mu \) on \(\Theta \), a potential \(W\colon \Theta \to {\mathbb R}\) and \(\beta {\gt}0\) with \(Z_W:=\int e^{-W/\beta }\, \mathrm d\mu {\lt}\infty \), the Gibbs measure is \(\hat\mu _W(\, \mathrm d\theta ):=Z_W^{-1}e^{-W(\theta )/\beta }\, \mu (\, \mathrm d\theta )\). (The paper takes \(\mu \) to be Lebesgue measure on \(\Theta ={\mathbb R}\times Z\).)

For measurable \(s\colon Z\to {\mathbb R}\), \(\lambda ,\beta {\gt}0\), the conditional amplitude kernel is \(\kappa _s(z,\, \mathrm da):={\mathcal N}\bigl(-s(z)/\lambda ,\beta /\lambda \bigr)(\, \mathrm da)\).

For particles \(\theta =(\theta _\ell )_{\ell =1}^M\), \(\theta _\ell =(a_\ell ,z_\ell )\), the finite-width network (mean-field scaling) is \(F_\theta (x):=\frac1M\sum _{\ell =1}^Ma_\ell \varphi _{z_\ell }(x)\).

For a sample \((x_i,y_i)_{i=1}^N\), \(\lambda \ge 0\) and a potential \(V\colon Z\to {\mathbb R}\), the finite-width objective is \(J_N(\theta ):=L_N(\theta )+\frac1M\sum _{\ell =1}^MU(\theta _\ell )\), where \(L_N(\theta )=\frac1{2N}\sum _{i=1}^N(F_\theta (x_i)-y_i)^2\) and \(U(a,z)=\frac\lambda 2a^2+V(z)\).

The empirical inner product on \({\mathbb R}^N\) is \(\langle u,v\rangle _N:=\frac1N\sum _{i=1}^Nu_iv_i\), with norm \(\| u\| _N:=\langle u,u\rangle _N^{1/2}\).

For \(r\in {\mathbb R}^N\) the empirical analysis map is \((S_N^*r)(z):=\frac1N\sum _{i=1}^N\varphi _z(x_i)r_i=\langle \varphi _z,r\rangle _N\) (independent of \(\nu \)).

For a finite measure \(\nu \) on \(Z\) (in the paper \(\nu \in {\mathcal P}(Z)\)) the empirical kernel operator \(K_{N,\nu }\) on \(({\mathbb R}^N,\langle \cdot ,\cdot \rangle _N)\) is \((K_{N,\nu }h)_i:=\int _Z\varphi _z(x_i)\langle \varphi _z,h\rangle _N\, \nu (\, \mathrm dz)\); for \(\nu _M=\frac1M\sum _\ell \delta _{z_\ell }\), \((K_{N,\nu _M}h)_i=\frac1M\sum _\ell \varphi _{z_\ell }(x_i)\langle \varphi _{z_\ell },h\rangle _N\).

For fixed hidden parameters \(z_1,\dots ,z_M\in Z\) and the sample \((x_i)_{i=1}^N\), the fixed-feature synthesis operator is \(S_N\colon ({\mathbb R}^M,\langle \cdot ,\cdot \rangle _M)\to ({\mathbb R}^N,\langle \cdot ,\cdot \rangle _N)\), \((S_N\gamma )_i:=\frac1M\sum _{\ell =1}^M\gamma _\ell \varphi _{z_\ell }(x_i)\), with adjoint \((S_N^*r)_\ell =\frac1N\sum _i\varphi _{z_\ell }(x_i)r_i=(S_N^*r)(z_\ell )\).

For particles \(\theta \in \Theta ^M\) and the sample \((x_i,y_i)_{i=1}^N\) the residual is \(r_i:=F_\theta (x_i)-y_i\), \(r=(r_i)_i\in {\mathbb R}^N\).

For \(\lambda {\gt}0\), \(K_{N,\nu _M}+\lambda \colon {\mathbb R}^N\to {\mathbb R}^N\) is a linear bijection; \((K_{N,\nu _M}+\lambda )^{-1}\) denotes its inverse.

The vector field of the gradient flow \(\dot\theta =-M\nabla J_N(\theta )\) is, with \(r=(F_\theta (x_i)-y_i)_i\), \(\theta \mapsto \bigl(-[(S_N^*r)(z_\ell )+\lambda a_\ell ],\, -[a_\ell \nabla _z(S_N^*r)(z_\ell )+\nabla V(z_\ell )]\bigr)_{\ell =1}^M\).

A curve \(\theta \colon [0,\infty )\to \Theta ^M\) is a solution of the gradient flow ??, \(\dot\theta _t=-M\nabla J_N(\theta _t)\), if it is differentiable on \([0,\infty )\) with \(\dot a_\ell =-[(S_N^*r_t)(z_\ell )+\lambda a_\ell ]\) and \(\dot z_\ell =-[a_\ell \nabla _z(S_N^*r_t)(z_\ell )+\nabla V(z_\ell )]\).

For \(\rho \in {\mathcal P}(\Theta )\) with \(\int |a|\, \mathrm d\rho {\lt}\infty \) (in particular for \(\rho \in {\mathcal D}\)), the realization \(F_\rho \) is a bounded measurable function, hence an element of \(L^2(P_X)\).

For \(f\in L^2(P_X)\) and \(\rho \) with \(\int |a|\, \mathrm d\rho {\lt}\infty \) the residual is \(r_\rho :=F_\rho -f\in L^2(P_X)\).

\({\mathcal{H}}_M:=({\mathbb R}^M,\langle \cdot ,\cdot \rangle _M)\) with \(\langle \gamma ,\gamma '\rangle _M:=\frac1M\sum _{\ell =1}^M\gamma _\ell \gamma '_\ell \).

The adjoint of \(S_N\) is \(S_N^*\colon ({\mathbb R}^N,\langle \cdot ,\cdot \rangle _N)\to ({\mathbb R}^M,\langle \cdot ,\cdot \rangle _M)\), \((S_N^*r)_\ell =\frac1N\sum _i\varphi _{z_\ell }(x_i)r_i=(S_N^*r)(z_\ell )\).

\(S_N\) has finite rank, so \(S_N^\dagger \) is the Moore–Penrose inverse and \(S_N^\dagger y=(S_N^*S_N)^\dagger S_N^*y\in (\ker S_N)^\perp \); in Lean \(S_N^\dagger y\) is the unique \(u\in (\ker S_N)^\perp \) with \(S_N^*S_Nu=S_N^*y\).

The fixed-feature gradient flow with weight decay \(\lambda \ge 0\) and no noise is \(\dot\gamma _t=-S_N^*(S_N\gamma _t-y)-\lambda \gamma _t\), \(\gamma _{t=0}=\gamma _0\) ??.

For \(\lambda {\gt}0\), \(\gamma _\lambda :=(B+\lambda )^{-1}S_N^*y\), where \(B=S_N^*S_N\).

For a sample \(x_1,\dots ,x_N\in {\mathcal X}\) the empirical feature is the feature map \((z,i)\mapsto \varphi _z(x_i)\) on the index space \(\{ 1,\dots ,N\} \), so that \(\Phi _z=(\varphi _z(x_i))_{i=1}^N\in {\mathbb R}^N\).

The uniform probability measure on \(\{ 1,\dots ,N\} \); the empirical inner product is \(\langle u,v\rangle _N=\frac1N\sum _{i=1}^Nu_iv_i\) and \(\| u\| _N^2=\langle u,u\rangle _N\).

For the law \(Q\) of \((X,Y)\) on \({\mathcal X}\times {\mathbb R}\) the risk with labels is \(\tilde L(\rho ):=\tfrac 12\, {\mathbb E}\bigl(F_\rho (X)-Y\bigr)^2\).

For a sample \(\omega =(x_i,y_i)_{i=1}^N\in ({\mathcal X}\times {\mathbb R})^N\) the empirical risk is \(L_N(\rho )=\frac1{2N}\sum _i(F_\rho (x_i)-y_i)^2\), a random functional of the sample.

\({\mathcal F}_N(\rho ):=L_N(\rho )+\beta \, \operatorname {KL}(\rho \| \mu _U)\) (the constant \(-\beta \log Z_U\) of the paper is dropped).

The sample \((x_i,y_i)_{i=1}^N\) consists of i.i.d. copies of \((X,Y)\sim Q\); its law is the product measure \(Q^{\otimes N}\) on \(({\mathcal X}\times {\mathbb R})^N\).

\(f:{\mathcal X}\to {\mathbb R}\) is a regression function of \(Q\), \(f(x)={\mathbb E}[Y\mid X=x]\), if \(f\) is measurable and \({\mathbb E}\bigl[(Y-f(X))\, g(X)\bigr]=0\) for every bounded measurable \(g\) (the tower property).

The identification of \({\mathbb R}^N\) with \(L^2(\text{uniform})\): a vector \(u\in {\mathbb R}^N\) is the function \(i\mapsto u_i\).

For \(\rho \in {\mathcal P}(\Theta )\) and the sample \((x_i,y_i)_{i=1}^N\) the empirical residual is \(r_{N,\rho }:=F_\rho |_x-y=(F_\rho (x_i)-y_i)_i\in {\mathbb R}^N\).

\(\tilde{{\mathcal F}}(\rho ):=\tilde L(\rho )+\beta \, \operatorname {KL}(\rho \| \mu _U)\) (the paper’s constant \(-\beta \log Z_U\) is dropped); \(\tilde{{\mathcal F}}={\mathcal F}+\mathrm{const}\) has the same minimizer \(\rho ^*\) as \({\mathcal F}\).

For a class \(\mathcal G=\{ g_i:i\in \iota \} \) of functions on \(\Xi \) and points \(\xi _1,\dots ,\xi _N\in \Xi \), with i.i.d. Rademacher signs \(\sigma _k\in \{ \pm 1\} \), the empirical Rademacher complexity is \({\widehat{\mathfrak R}}_N(\mathcal G):={\mathbb E}_\sigma \sup _{i}\bigl|\tfrac 1N\sum _{k=1}^N\sigma _kg_i(\xi _k)\bigr|\) (absolute convention, FoML’s ‘empiricalRademacherComplexity‘); the one-sided version \({\mathbb E}_\sigma \sup _i\frac1N\sum _k\sigma _kg_i(\xi _k)\) of the paper is ‘empiricalRademacherOneSided‘. For \(\mathcal G=-\mathcal G\) the two agree.

The one-sided empirical Rademacher complexity \({\mathbb E}_\sigma \sup _i\frac1N\sum _{k=1}^N\sigma _kg_i(\xi _k)\) (without absolute value), FoML’s ‘empiricalRademacherComplexity_without_abs‘.

The Rademacher complexity is the expectation of the empirical one over \(N\) i.i.d. points, \({\mathfrak R}_N(\mathcal G):={\mathbb E}\, {\widehat{\mathfrak R}}_N(\mathcal G)\) (FoML’s ‘rademacherComplexity‘).

The feature class of a feature map \(\varphi \) is \(\Phi :=\{ \varphi _z:z\in Z\} \), a class of functions on \({\mathcal X}\) indexed by \(Z\).

For points \(x_1,\dots ,x_N\in {\mathcal X}\), \({\widehat{\mathfrak R}}_N(\Phi ):={\mathbb E}_\sigma \sup _{z\in Z}\bigl|\tfrac 1N\sum _{k=1}^N\sigma _k\varphi _z(x_k)\bigr|\). For \(\Phi =-\Phi \) (as for \(\tanh \)) this is the one-sided complexity of the paper.

For the law \(Q\) of \((X,Y)\), \({\mathfrak R}_N(\Phi ):={\mathbb E}\, {\widehat{\mathfrak R}}_N(\Phi )\), the expectation over the inputs \(x_1,\dots ,x_N\) of an i.i.d. sample of size \(N\) from \(Q\).

The product class of the feature map is \(\Psi :=\{ \varphi _z\varphi _{z'}:z,z'\in Z\} \), indexed by \(Z\times Z\); its members are bounded by one.

\({\widehat{\mathfrak R}}_N(\Psi ):={\mathbb E}_\sigma \sup _{z,z'\in Z} \bigl|\tfrac 1N\sum _{k=1}^N\sigma _k\varphi _z(x_k)\varphi _{z'}(x_k)\bigr|\), and \({\mathfrak R}_N(\Psi ):={\mathbb E}\, {\widehat{\mathfrak R}}_N(\Psi )\) over the inputs of an i.i.d. sample from \(Q\).

\({\mathfrak R}_N(\Psi ):={\mathbb E}\, {\widehat{\mathfrak R}}_N(\Psi )\), the expectation over the inputs \(x_1,\dots ,x_N\) of an i.i.d. sample of size \(N\) from \(Q\).

For \(B\ge 0\), \({\mathcal P}_B:=\{ \rho \in {\mathcal P}(\Theta ):\int |a|\, \rho (\, \mathrm da\, \mathrm dz)\le B\} \) (the integrability of \(a\) is part of the definition).

For a sample \(\omega =(x_i,y_i)_{i=1}^N\), \(G_+(\omega ):=\sup _{\rho \in {\mathcal P}_B}\bigl(\tilde L(\rho )-L_N(\rho )\bigr)\).

\(G_-(\omega ):=\sup _{\rho \in {\mathcal P}_B}\bigl(L_N(\rho )-\tilde L(\rho )\bigr)\).

\(G(\omega ):=\sup _{\rho \in {\mathcal P}_B}|L_N(\rho )-\tilde L(\rho )| =\max (G_+,G_-)\).

For \(e\colon {\mathcal X}\times {\mathbb R}\to {\mathbb R}\) (in the paper \(e(x,y)=y-f(x)\)), \(G_N(\omega ):=\sup _{z\in Z}\bigl|\tfrac 1N\sum _{i=1}^N\varphi _z(x_i)e(x_i,y_i) -{\mathbb E}[\varphi _z(X)e(X,Y)]\bigr|\).

\(H_N(\omega ):=\sup _{z,z'\in Z}\bigl|\tfrac 1N\sum _{i=1}^N \varphi _z(x_i)\varphi _{z'}(x_i)-{\mathbb E}[\varphi _z(X)\varphi _{z'}(X)]\bigr|\).

For a countable \(Z_0\subseteq Z\), \({\mathcal P}_B^0:=\bigl\{ \tfrac 1M\sum _{j=1}^M\delta _{(\epsilon _jq,z_j)}:M\ge 1, q\in {\mathbb Q},\ \epsilon _j\in \{ \pm 1\} ,\ z_j\in Z_0\bigr\} \cap {\mathcal P}_B\), a countable subclass of \({\mathcal P}_B\).

For finite measures \(\nu _1,\nu _2\) and functions \(m_1,m_2\), with \(\mu :=\nu _1+\nu _2\) and \(w_i:=\, \mathrm d\nu _i/\, \mathrm d\mu \), the mixed density is \(\psi :=w_1m_1-w_2m_2\), so that \(m_1\nu _1-m_2\nu _2=\psi \, \mu \).

\(B:=\frac{{Y_{\max }}}\lambda +\sqrt{\frac{2\beta }{\pi \lambda }}\), the bound on \(\int |a|\, \mathrm d\rho \) for both minimizers (Lemma ??(c)).

\(\Delta _N(\delta ):=4B(B+{Y_{\max }})\, {\mathfrak R}_N(\Phi ) +(B+{Y_{\max }})^2\sqrt{\frac{\log (2/\delta )}{2N}}\).

\(D(\delta ):=4C_{\mathrm P}B(B+{Y_{\max }})\sqrt{m+1} +(B+{Y_{\max }})^2\sqrt{\tfrac 12\log (2/\delta )}\).

\(N_0({\varepsilon },\delta ):=\bigl\lceil (D(\delta )/(\beta {\varepsilon }))^2\bigr\rceil \).

\(\Delta _N':=2(B+{Y_{\max }})\, {\mathfrak R}_N(\Phi ) +(B+{Y_{\max }})\sqrt{\frac{2\log (4/\delta )}{N}}\).

\(E_\delta :=\{ \omega :|y_i|\le {Y_{\max }}\ \forall i,\ G(\omega )\le \tfrac 12\Delta _N(\delta )\} \), the event of Lemma 302 (with the full-measure condition on the labels).

The setting of Section ??: \(Q\in {\mathcal P}({\mathcal X}\times {\mathbb R})\) with \(|Y|\le {Y_{\max }}\) a.s., \(f={\mathbb E}[Y\mid X]\) a regression function with \(|f|\le {Y_{\max }}\), \(\lambda ,\beta {\gt}0\), and \(\rho ^*\in {\mathcal D}\) a minimizer of \({\mathcal F}\) on \({\mathcal D}\) (with target \(f\in L^2(P_X)\)).

For every sample \(\omega =(x_i,y_i)_{i=1}^N\), \(\rho _N^*(\omega )\in {\mathcal D}\) is a minimizer of \({\mathcal F}_N\) on \({\mathcal D}\) (Lemma ?? for \(L_N\)).

\(e(x,y):=y-F_{\rho ^*}(x)\) (the paper’s \(e\) has the opposite sign; only \(|e|\) enters).

\(E'_\delta :=E_{\delta /2}\cap \{ G_N\le \Delta _N'\} \), where \(G_N\) is the deviation of Lemma 305 for \(e(x,y)=y-F_{\rho ^*}(x)\) and \(\Delta '_N\) is 86.

For a sample \(\omega =(x_i,y_i)_i\) and \(\rho \in {\mathcal P}(\Theta )\), \(g_{N,\rho }:=S_N^*(F_\rho |_x-y)\), i.e. \(g_{N,\rho }(z)=\frac1N\sum _i\varphi _z(x_i)(F_\rho (x_i)-y_i)\); for \(\rho =\rho _N^*\) this is \(g_N\) of Lemma ??.

For a sample \(\omega =(x_i,y_i)_i\), \(\nu \in {\mathcal P}(Z)\) and \(\lambda {\gt}0\), \(R^N_{\lambda ,\nu }y:=S_N^*(K_{N,\nu }+\lambda )^{-1}y\), the regularized ridgelet transform of the empirical feature.

\(D'(\delta ):=(B+{Y_{\max }})\bigl[2C_{\mathrm P}\sqrt{m+1} +\sqrt{2\log (4/\delta )}\bigr]\), so that \(\Delta '_N\le D'(\delta )/\sqrt N\).

\(e(x,y):=F_{\rho ^*}(x)-y\), a fixed (sample-independent) function on \({\mathcal X}\times {\mathbb R}\) with \(\sup |e|\le E_*:=M_*+{Y_{\max }}\).

\(h:=F_{\rho _N^*}-F_{\rho ^*}\), with \(\| h\| _N:=\| h|_x\| _N\) and \(\| h\| :=\| h\| _{L^2(P_X)}\).

\(\ell _F(x,y):=\tfrac 12(F(x)-y)^2\), so that \(\tilde L(\rho )=P\ell _{F_\rho }\) and \(L_N(\rho )=P_N\ell _{F_\rho }\).

For \(g\colon {\mathcal X}\times {\mathbb R}\to {\mathbb R}\), \(P_Ng:=\frac1N\sum _{i=1}^Ng(x_i,y_i)\) and \(Pg:={\mathbb E}g(X,Y)\).

\((P_N-P)g:=P_Ng-Pg\).

\({\mathcal{K}}_N:=\operatorname {KL}(\rho ^*\| \rho _N^*)+\operatorname {KL}(\rho _N^*\| \rho ^*)\) (the Jeffreys divergence).

\(D_\varsigma :=|\Pi \varsigma |(Z)\), the total variation of the signed measure \(\Pi \varsigma =\Pi \rho _N^*-\Pi \rho ^*\) on \(Z\).

\(m^*:=-g^*/\lambda \) with \(g^*=S^*r^*\), the everywhere-defined continuous version of \(m_{\rho ^*}\) (Theorem ??(b)).

\(m_N:=-g_N/\lambda \) with \(g_N=S_N^*r_N^*\), the everywhere-defined version of \(m_{\rho _N^*}\) (Lemma ??(a)).

With \(\mu :=\nu _N^*+\nu ^*\), \(w_N:=\, \mathrm d\nu _N^*/\, \mathrm d\mu \), \(w^*:=\, \mathrm d\nu ^*/\, \mathrm d\mu \), the density of \(\Pi \varsigma =m_N\nu _N^*-m^*\nu ^*\) with respect to \(\mu \) is \(\psi :=w_Nm_N-w^*m^*\), so that \(D_\varsigma =\int |\psi |\, \mathrm d\mu \).

\(\eta _N':=2E_*{\mathfrak R}_N(\Phi )+E_*\sqrt{2\log (4/\delta )/N}\), the bound of Lemma 305 with \(\delta _1=\delta /2\); under Lemma 314, \(\eta '_N\le E_*D'(\delta )/\sqrt N\) with \(D'(\delta )=2C_{\mathrm P}\sqrt{m+1}+\sqrt{2\log (4/\delta )}\).

\(\eta _N'':=2{\mathfrak R}_N(\Psi )+\sqrt{2\log (4/\delta )/N}\), the bound of Lemma 303 with \(\delta _2=\delta /2\); under Lemma 315, \(\eta ''_N\le D''(\delta )/\sqrt N\) with \(D''(\delta )=2C'_{\mathrm P}\sqrt{m+1}+\sqrt{2\log (4/\delta )}\).

\(c_1:=\max \{ 1,1/\lambda ,M_*/\sqrt\beta \} \) with \(M_*={Y_{\max }}/\lambda \).

For a bound \(\sup _z|m^*(z)|\le M_1\) on the population coefficient, \(c_1(M_1):=\max \{ 1,1/\lambda ,M_1/\sqrt\beta \} \); the constant \(c_1\) of Theorem  efthm:m3-double-prime is \(c_1(M_*)\) with \(M_*={Y_{\max }}/\lambda \).

\(\Omega _\delta :=\{ G_N\le \eta _N'\} \cap \{ H_N\le \eta _N''\} \); \({\mathbb P}(\Omega _\delta )\ge 1-\delta \) by Lemmas 305 and 303.

For \(A{\gt}0\) and \(c\in \mathbb N\), \(\mathrm{HC}(m,A,c)\) is the statement: for every \(N\), every \(x_1,\dots ,x_N\in {\mathbb R}^m\) and every \(0{\lt}{\varepsilon }\le 1/2\), \(\mathcal N({\varepsilon },\Phi ,L_2(P_N))\le (A/{\varepsilon })^{c(m+1)}\). Haussler’s bound gives \(\mathrm{HC}(m,A,2)\) with an absolute \(A\).

For a finite signed Borel measure \(\gamma \) on \(Z\) the synthesis is \((S\gamma )(x):=\int _Z\varphi _z(x)\, \gamma (\, \mathrm dz)\); if \(\gamma =h\nu \) with \(h\in L^2(\nu )\) then \(S\gamma =S_\nu h\), and \(F_\rho =S\, \Pi \rho \) when \(\int |a|\, \, \mathrm d\rho {\lt}\infty \).

For a finite signed measure \(\gamma \) on \(Z\), \(L(\gamma ):=\tfrac 12\| S\gamma -f\| ^2_{L^2(P_X)}\).

For \(\rho \in {\mathcal D}\) let \(r_\rho =F_\rho -f\), \(s_\rho :=S^*r_\rho \) and \(W_\rho (\theta ):=a\, s_\rho (z)+U(\theta )\). The proximal Gibbs measure is \(\hat\mu _\rho (\, \mathrm d\theta ):=Z_\rho ^{-1}e^{-W_\rho (\theta )/\beta }\, \mathrm d\theta =Z_\rho ^{-1}e^{-a s_\rho (z)/\beta }\mu _U(\, \mathrm d\theta )\), where \(Z_\rho =\int e^{-W_\rho /\beta }\, \mathrm d\theta {\lt}\infty \) since \(|s_\rho |\le c_\rho \).

A probability measure \(\mu \) on \({\mathbb R}^d\) satisfies \(\mathrm{LSI}(\alpha )\) with \(\alpha {\gt}0\) if for all \(g\in C_c^\infty ({\mathbb R}^d)\)

(Here the test functions are \(C^1_c\); by standard approximation the two conventions agree.)

\(\rho \) has a smooth density with respect to \(\mu \) if \(\rho =p\, \mu \) for a positive \(C^1\) function \(p\) with \(\nabla \log p\in L^2(\rho )\).

For \(\rho \ll \mu \) with a smooth density \(p=\frac{\, \mathrm d\rho }{\, \mathrm d\mu }\) the relative Fisher information is \(I(\rho |\mu ):=\int \bigl|\nabla \log \tfrac {\, \mathrm d\rho }{\, \mathrm d\mu }\bigr|^2\, \mathrm d\rho =\int |\nabla \log p|^2\, \mathrm d\rho \).

A probability measure \(\mu \) satisfies the KL form of \(\mathrm{LSI}(\alpha )\) if for every probability measure \(\rho \ll \mu \) with a smooth density and \(\operatorname {KL}(\rho \| \mu ){\lt}\infty \),

This is (L1) of the paper, obtained from \(\mathrm{LSI}(\alpha )\) by substituting \(g=\sqrt{\, \mathrm d\rho /\, \mathrm d\mu }\).

For \(s\colon Z\to {\mathbb R}\) and \(\lambda {\gt}0\) the shear is \(T_s(a,z):=(a+s(z)/\lambda ,\ z)\), a bijection of \(\Theta \) with inverse \(T_s^{-1}(a',z)=(a'-s(z)/\lambda ,\ z)\).

The parameter space \(\Theta ={\mathbb R}\times Z\) is given the Euclidean structure \(|\theta |^2=a^2+|z|^2\); a measure on \(\Theta \) is regarded as a measure on this Euclidean space (formally, its image under the identity map to ‘WithLp 2 (ℝ × Z)‘).

For measures \(\rho ,\mu \) on \(\Theta \), \(I(\rho |\mu )\) is the relative Fisher information with respect to the Euclidean gradient \(\nabla _\theta =(\partial _a,\nabla _z)\).

A probability measure \(\mu \) on \(\Theta ={\mathbb R}\times Z\) satisfies \(\mathrm{LSI}(\alpha )\) if it does so as a measure on the Euclidean space \(\Theta \).

A probability measure \(\mu \) on \(\Theta \) satisfies the KL form of \(\mathrm{LSI}(\alpha )\) if \(\operatorname {KL}(\rho \| \mu )\le \frac1{2\alpha }I(\rho |\mu )\) for every probability measure \(\rho \) on \(\Theta \) with a smooth density with respect to \(\mu \) and \(\operatorname {KL}(\rho \| \mu ){\lt}\infty \).

(A8), Gaussian confinement. \(V(z)=\frac{\lambda _z}2|z|^2\) with \(\lambda _z{\gt}0\), so that \(\nu _0(\, \mathrm dz)=Z_0^{-1}e^{-V(z)/\beta }\, \mathrm dz ={\mathcal N}(0,(\beta /\lambda _z)I)(\, \mathrm dz)\).

For \(c\ge 0\) let \(\alpha (c):=\dfrac {\min \{ \lambda ,\ \lambda _z e^{-c^2/(2\lambda \beta )}\} } {\beta \, (1+\sqrt{R_X^2+1}\, c/\lambda )^2}\), the right-hand side of the LSI constant of \(\hat\mu _\rho \) in Lemma ??(iv) with \(c=c_\rho \); it is nonincreasing in \(c\).

For \(E\in {\mathbb R}\) let \(c_E:=\sqrt{2(E+\beta \log Z_U)}\) (here \(c_E=\sqrt{2E}\), the constant of the free energy being dropped) and

The feature map is smooth with bounded derivatives if \(z\mapsto \varphi _z(x)\) is \(C^\infty \) for every \(x\) and \(\sup _{x,z}\| \nabla _z^n\varphi _z(x)\| {\lt}\infty \) for every \(n\ge 1\). Under (A1) and (A3) this holds with \(\sup _{x,z}\| \nabla _z\varphi _z(x)\| \le \sqrt{R_X^2+1}\) and \(\sup _{x,z}\| \nabla _z^2\varphi _z(x)\| \le c_2(R_X^2+1)\).

(Hypothesis W as the definition of a solution in law.) A curve \((\rho _t)_{t\ge 0}\) of probability measures on \(\Theta \) is an MFLD flow if

• \(\rho _t\in {\mathcal D}\) for all \(t\ge 0\), \(t\mapsto \rho _t\) is weakly continuous, \(\sup _{t\le T}\int |\theta |^2\, \mathrm d\rho _t{\lt}\infty \) for every \(T\), and for a.e. \(t\ge 0\) the measure \(\rho _t\) has a smooth density with respect to \(\hat\mu _{\rho _t}\) with \(\nabla \log \frac{\, \mathrm d\rho _t}{\, \mathrm d\hat\mu _{\rho _t}}\in L^2(\rho _t)\);

• \(u\mapsto I(\rho _u|\hat\mu _{\rho _u})\) is locally integrable on \([0,\infty )\) and for all \(0\le s\le t\)

  \[ {\mathcal F}(\rho _t)={\mathcal F}(\rho _s)-\beta ^2\int _s^tI(\rho _u|\hat\mu _{\rho _u})\, \mathrm du . \]

(W2) is the integrated form of \(\frac{\, \mathrm d}{\, \mathrm dt}{\mathcal F}(\rho _t)=-\beta ^2I(\rho _t|\hat\mu _{\rho _t})\) for a locally absolutely continuous \(t\mapsto {\mathcal F}(\rho _t)\).

For \(\theta =(\theta _\ell )_{\ell =1}^M\in \Theta ^M\) the particle risk is \(L(\rho _\theta )=\tfrac 12\| F_{\rho _\theta }-f\| _{L^2(P_X)}^2\), where \(\rho _\theta =\frac1M\sum _\ell \delta _{\theta _\ell }\) and \(F_{\rho _\theta }(x)=\frac1M\sum _\ell a_\ell \varphi _{z_\ell }(x)\).

The \(M\)-particle Gibbs measure is \(\pi _M(\, \mathrm d\theta ):=Z_M^{-1}e^{-ML(\rho _\theta )/\beta }\, \mu _U^{\otimes M}(\, \mathrm d\theta )\), \(Z_M:=\int e^{-ML(\rho _\theta )/\beta }\, \mathrm d\mu _U^{\otimes M}\). Since \(\mu _U^{\otimes M}\propto e^{-\sum _\ell U(\theta _\ell )/\beta }\, \mathrm d\theta \), this is \(\pi _M\propto e^{-MJ(\theta )/\beta }\, \mathrm d\theta \) with \(J(\theta )=L(\rho _\theta )+\frac1M\sum _\ell U(\theta _\ell )\), the invariant measure of the \(M\)-particle noisy gradient descent.

For \(\mu \in {\mathcal P}(\Theta ^M)\) the \(M\)-particle free energy is \({\mathcal F}^M(\mu ):=M\int _{\Theta ^M}L(\rho _\theta )\, \mu (\, \mathrm d\theta ) +\beta \, \operatorname {KL}(\mu \| \mu _U^{\otimes M})\) (the paper’s \({\mathcal F}^M\) up to the dropped constant \(-\beta \log Z_{\pi _M}\)).

For \(\rho \in {\mathcal P}(\Theta )\), \(\rho ^{\otimes M}\in {\mathcal P}(\Theta ^M)\) is the law of \(M\) i.i.d. particles of law \(\rho \); the comparison measure of static chaos is \(\rho ^{*\otimes M}\) for the minimizer \(\rho ^*\) of \({\mathcal F}\).

(E) A family \((\mu _t)_{t\ge 0}\subset {\mathcal P}(\Theta ^M)\) is ergodic to \(\pi \in {\mathcal P}(\Theta ^M)\) if \(\| \mu _t-\pi \| _{{\mathrm{TV}}}\to 0\) as \(t\to \infty \) and \(\sup _{t\ge 0}\int \frac1M\sum _\ell a_\ell ^2\, \mu _t(\, \mathrm d\theta ){\lt}\infty \). For the laws of the \(M\)-particle noisy gradient descent and \(\pi =\pi _M\) the first property is the ergodic theorem for non-degenerate Langevin diffusions with an invariant probability measure and the second is the moment bound of the particle system.

For a sample \(\omega =(x_i,y_i)_{i=1}^N\) the empirical \(M\)-particle Gibbs measure is \(\pi _M(\, \mathrm d\theta )\propto e^{-ML_N(\rho _\theta )/\beta }\mu _U^{\otimes M} (\, \mathrm d\theta )\), the invariant measure of the \(M\)-particle empirical noisy gradient descent (\(J=J_N\)); it is the particle Gibbs measure of the empirical feature for the uniform measure on \(\{ 1,\dots ,N\} \) and the target \(y\) (Lemma 271).

\(C_g({Y_{\max }},\lambda ,\beta ):=\frac1{2\beta }\Bigl(\frac{{Y_{\max }}^2}{\lambda ^2} +\frac\beta \lambda \Bigr)+\log 2+4\exp \Bigl(2C\Bigl(\frac{{Y_{\max }}}\lambda +\sqrt{\frac{2\beta }{\pi \lambda }}\Bigr)\Bigr)\exp \Bigl(\frac\beta {2\lambda } \bigl(2C+\frac{{Y_{\max }}}\beta \bigr)^2\Bigr)\), a bound on the constant \(C^N_g\) of Theorem 550 that depends on the sample only through \({Y_{\max }}\) (\(A_N\le {Y_{\max }}^2/\lambda ^2+\beta /\lambda \), and ?? with \(\| f\| \to {Y_{\max }}\)).

On \(L^2(\nu _0)\) let \(T_0:=S_{\nu _0}^*S_{\nu _0}\). It is positive semidefinite with \(\langle T_0u,u\rangle =\| S_{\nu _0}u\| ^2\) and \(\| T_0\| \le 1\).

For \(\lambda {\gt}0\) and \(f\in L^2(P_X)\), \(u_\lambda :=(T_0+\lambda )^{-1}S_{\nu _0}^*f=S_{\nu _0}^*(K_{\nu _0}+\lambda )^{-1}f=R_{\lambda ,\nu _0}f\); the two expressions agree by the push-through identity (Lemma ??(4)).

(SC\(_a\)) Source condition of order \(a{\gt}0\). There is \(g_0\in L^2(\nu _0)\) with \(u^\dagger =T_0^ag_0\). Since \(\operatorname {ran}T_0^a\subset \overline{\operatorname {ran}T_0}=(\ker S_{\nu _0})^\perp \), such a \(u^\dagger \) automatically satisfies the minimum-norm condition, and (SC\(_a\)) contains (R) (\(f=S_{\nu _0}u^\dagger \)). For \(a=1/2\), \(\operatorname {ran}T_0^{1/2}=\operatorname {ran}S_{\nu _0}^*\), so (SC\(_{1/2}\)) is equivalent to \(u^\dagger =S^*g\) with \(g\in L^2(P_X)\), i.e. to \(f=K_{\nu _0}g\). (In Lean the condition is formalized for \(a\in \{ 1/2,1\} \) with a bound \(G\) on the norm of the source element: \(f=K_{\nu _0}g\), \(\| g\| \le G\), resp. \(f=S_{\nu _0}T_0g_0\), \(\| g_0\| \le G\); the spectral power \(T_0^a\) of an operator on a real Hilbert space is not available in Mathlib.)

(S\(_\infty \)) Uniform sup-norm bound. For the family of \((\lambda ,\beta )\) under consideration (for instance \(\lambda \le \lambda _0\), \(\kappa \le \kappa _0\)), \(\sup _{(\lambda ,\beta )}\sup _{z\in Z}|m^*_{\lambda ,\beta }(z)|\le B_\infty {\lt}\infty \), where \(m^*_{\lambda ,\beta }\) is the conditional mean amplitude of the minimizer of \({\mathcal F}_{\lambda ,\beta }\).

\(v:=w\, m^*\in L^2(\nu _0)\), where \(w=\, \mathrm d\nu ^*/\, \mathrm d\nu _0\) and \(m^*=m_{\rho ^*}\); \(v\) is bounded, and \(\Pi \rho ^*=m^*\nu ^*=v\, \nu _0\).

\(\Xi :=\| (w-1)m^*\| _{L^2(\nu _0)}\), where \(w=\, \mathrm d\nu ^*/\, \mathrm d\nu _0\) and \(m^*=m_{\rho ^*}\); \(\Xi \) is the whole contribution of the learned base measure to the rate.

\(\lambda _N:=N^{-1/p}\) for \(p{\gt}0\); \(p=4(a+2)\) in Theorem ??, \(p=2(a+3)\) in Corollary ??(iii).

\(C_{\mathrm{stat}}(m,\delta ):=\sqrt{2\bigl[4C_{\mathrm P}\sqrt{m+1} +\sqrt{\tfrac 12\log \tfrac 4\delta }\bigr]}+2C_{\mathrm P}\sqrt{m+1} +\sqrt{2\log \tfrac 4\delta }\).

\(C_1:=\frac12\bigl(B_\infty ^2e^{\kappa _0B_\infty ^2/2}+\| u^\dagger \| ^2\bigr) e^{\kappa _0\| u^\dagger \| ^2/4}\| u^\dagger \| \).

\(C_1':=\frac12\bigl(B_\infty ^2e^{\kappa _0B_\infty ^2/2} +\| u^\dagger \| ^2\bigr)\).

With \(\zeta _{0,0}:=\frac{\kappa _0}2\| u^\dagger \| ^2\), \(C_2:=\| u^\dagger \| ^3e^{\zeta _{0,0}/2}\max \Bigl\{ \frac{e^{\zeta _{0,0}}}8 \exp \Bigl(\frac{e^{\zeta _{0,0}}\| u^\dagger \| ^2}{8\beta _0}\Bigr),\ \frac12\Bigr\} \).

(R) \(f\in \operatorname {ran}S_{\nu _0}\), that is, there is \(u_0\in L^2(\nu _0)\) with \(f=S_{\nu _0}u_0=\int _Zu_0(z)\varphi _z\, \nu _0(\, \mathrm dz)\) in \(L^2(P_X)\). Such a \(u_0\) is called a representer of \(f\) (with respect to \(\nu _0\)).

Under (R), the canonical (minimum-norm) ridgelet transform of \(f\) with respect to \(\nu _0\) is \(u_0^\dagger :=S_{\nu _0}^\dagger f:=P_{(\ker S_{\nu _0})^\perp }u_0\), the orthogonal projection onto \((\ker S_{\nu _0})^\perp \) of any representer \(u_0\) of \(f\) (Lemma 382: it does not depend on \(u_0\)). Even if \(\operatorname {ran}S_{\nu _0}\) is not closed, \(S_{\nu _0}^\dagger f\) is defined in this sense as long as \(f\in \operatorname {ran}S_{\nu _0}\). (In Lean, \(u_0^\dagger :=0\) when \(f\notin \operatorname {ran}S_{\nu _0}\).)

For \(\nu \in {\mathcal P}(Z)\), \(u\colon Z\to {\mathbb R}\) measurable and \(\sigma ^2:=\beta /\lambda \) put \(\rho _{\nu ,u}:=\nu (\, \mathrm dz)\otimes {\mathcal N}(u(z),\sigma ^2)(\, \mathrm da)\), a probability measure on \(\Theta \) with hidden marginal \(\nu \) and conditional mean amplitude \(u\). The competitor of the paper is \(\tilde\rho _u:=\rho _{\nu _0,u}\) for \(u\in L^2(\nu _0)\).

For \(\rho \) with \(\int |a|\, \mathrm d\rho {\lt}\infty \), \(\lambda {\gt}0\) and a finite measure \(\nu \) on \(Z\), the bounded measurable function \(-s_\rho /\lambda =-\lambda ^{-1}S^*r_\rho \) is an element of \(L^2(\nu )\); for \(\nu =\nu ^*\) it is \(m_{\rho ^*}\) (Theorem 228).

\(\tilde Z:=\int _Z\exp \bigl(\tfrac \kappa 2m^{*}(z)^2\bigr)\nu _0(\, \mathrm dz) =\int _Z\exp \bigl(\tfrac {s^*(z)^2}{2\lambda \beta }\bigr)\nu _0(\, \mathrm dz)=Z_*/Z_0\) in the notation of Theorem 226.

For \(\rho \) with \(\int |a|\, \mathrm d\rho {\lt}\infty \) and \(\lambda {\gt}0\), \(m_\rho :=-s_\rho /\lambda =-\lambda ^{-1}S^*r_\rho \colon Z\to {\mathbb R}\), a bounded measurable function; for the minimizer \(\rho ^*\) it is the conditional mean amplitude (Theorem 228).

\(w:=\frac{\, \mathrm d\nu ^*}{\, \mathrm d\nu _0}=\frac{Z_0}{Z_*}\exp \bigl(\frac\kappa 2m^{*2}\bigr) =\tilde Z^{-1}\exp \bigl(\frac\kappa 2m^{*2}\bigr)\) with \(\kappa =\lambda /\beta \) (Theorem 401).

Assume (A1), (A3), (A5), \(f\in L^2(P_X)\), (R) and the convention ?? (\(\nu _0\) fixed). Let \((\lambda _n,\beta _n)_{n\ge 1}\) satisfy \(\lambda _n\to 0\) and \(\kappa _n:=\lambda _n/\beta _n\to 0\), and let \(\rho _n^*\) be the minimizer of \({\mathcal F}\) for \((\lambda _n,\beta _n)\); write \(\nu _n^*:=\nu _{\rho _n^*}\), \(m_n^*:=m_{\rho _n^*}\), \(\gamma _n:=\Pi \rho _n^*=m_n^*\nu _n^*\) and \(\gamma _\infty :=u_0^\dagger \nu _0\).

\(r_N:=\frac1{2\lambda }\Bigl[\sqrt{2D(\delta /2)/\sqrt N} +D'(\delta )/\sqrt N\Bigr]+\frac{{Y_{\max }}}\lambda \sqrt{\frac{D(\delta /2)}{2\beta \sqrt N}}\), the bound of Corollary 340 with \(\Delta _N\le D(\delta /2)/\sqrt N\) and \(\Delta _N'\le D'(\delta )/\sqrt N\) (Corollary 341); \(r_N\to 0\) as \(N\to \infty \).

\({\mathcal P}_2:=\bigl\{ \rho \in {\mathcal P}(\Theta ):\int a^2\, \mathrm d\rho {\lt}\infty \bigr\} \).

For \(\gamma \ne 0\) with \(c:=\| \gamma \| _{\mathcal M}\) and Jordan decomposition \(\gamma =\gamma ^+-\gamma ^-\), the lift \(\bar\rho :=c^{-1}\bigl(\gamma ^+(\, \mathrm dz)\otimes \delta _{c}(\, \mathrm da)+ \gamma ^-(\, \mathrm dz)\otimes \delta _{-c}(\, \mathrm da)\bigr)\) is the measure \(\frac{|\gamma |(\, \mathrm dz)}{c}\otimes \delta _{c\vartheta (z)}(\, \mathrm da)\) of 159, with \(\vartheta =\, \mathrm d\gamma /\, \mathrm d|\gamma |\in \{ \pm 1\} \).

For \(\gamma \ne 0\), \(\bar\rho (\, \mathrm da\, \mathrm dz):=\frac{|\gamma |(\, \mathrm dz)}{c}\otimes \delta _{c\, \vartheta (z)}(\, \mathrm da)\) with \(c=\| \gamma \| _{\mathcal M}\) and \(\vartheta =\, \mathrm d\gamma /\, \mathrm d|\gamma |\in \{ \pm 1\} \); for \(\gamma =0\), \(\bar\rho :=\delta _0\otimes \nu \) with an arbitrary \(\nu \in {\mathcal P}(Z)\).

The functional of the \(\beta =0\) problem ?? is \({\mathcal{E}}_0(\rho ):=L(\rho )+\frac\lambda 2\int a^2\, \mathrm d\rho \) on \({\mathcal P}_2\).

The total-variation regularized functional on \({\mathcal M}(Z)\) is \({\mathcal{E}}_{TV}(\gamma ):=L(\gamma )+\frac\lambda 2\| \gamma \| ^2_{\mathcal M}\).

For \(a\ge 0\), \(T_0^a:=\sum _k\mu _k^a\langle e_k,\cdot \rangle e_k\) is the spectral power of \(T_0\) (Definition 789 with respect to the eigenbasis of Lemma 163); it does not depend on the choice of the eigenbasis (Lemma 794).

(SC\(_a\)) Source condition of order \(a{\gt}0\). There is \(g_0\in L^2(\nu _0)\) with \(u^\dagger =T_0^ag_0\), where \(T_0^a\) is the spectral power of Definition 164. Since \(\operatorname {ran}T_0^a\subset (\ker S_{\nu _0})^\perp \), such a \(u^\dagger \) automatically satisfies the minimum-norm condition, and (SC\(_a\)) contains (R) (\(f=S_{\nu _0}u^\dagger \)). (In Lean, with a bound \(G\) on the source norm: \(f=S_{\nu _0}T_0^ag_0\) with \(\| g_0\| \le G\).)