For \(m=1\) the feature \(\varphi _{(w,b)}(x)=\tanh (w^\top x-b)\) of Definition 5 is \(\tanh (w_0x_0-b)\) under \({\mathbb R}^1\cong {\mathbb R}\).

For \(s\ne 0\), \(2s\coth s=2|s|+\frac{4|s|}{e^{2|s|}-1}\) ??.

For \(\alpha {\gt}0\) and \(L\ge 1\), \(\int _{1/L\le |w|\le L}|w|^{\alpha -1}\, \mathrm dw=2M_L\) (Lemma 649).

With \(M_L:=\int _{1/L}^Lw^{\alpha -1}\, \mathrm dw=\frac{L^\alpha -L^{-\alpha }}\alpha \), \(Z_L=\int _{B_L}|w|^{\alpha -1}\, \mathrm dw\, \mathrm db=2L\cdot 2M_L=4LM_L =\frac{4L(L^\alpha -L^{-\alpha })}\alpha \) ??.

For \(\alpha {\gt}0\) and \(L{\gt}1\), \(\nu _0^{(L)}\) is a probability measure: \(\int _{B_L}|w|^{\alpha -1}\, \mathrm dw\, \mathrm db=Z_L\) (Lemma 649).

For \(\alpha {\gt}0\) and \(L{\gt}1\), \(\nu _0^{(L)}\bigl(\{ |b|\le |w|\} \bigr)=\frac\alpha {1+\alpha }\cdot \frac{1-L^{-2-2\alpha }}{1-L^{-2\alpha }}\) ??; it tends to \(c_\alpha \) as \(L\to \infty \).

For \(u_1,u_2,b\in {\mathbb R}\) and \(s:=u_1-u_2\), \(1-\tanh (u_1-b)\tanh (u_2-b)=\frac{\cosh s}{\cosh (b-u_1)\cosh (b-u_2)}\) ??.

For \(u_1,u_2\in {\mathbb R}\) with \(s:=u_1-u_2\ne 0\), \(\int _{\mathbb R}\frac{\, \mathrm db}{\cosh (b-u_1)\cosh (b-u_2)}=\frac{2s}{\sinh s}\) ??.

For \(u_1,u_2\in {\mathbb R}\) and \(s:=u_1-u_2\), \(1-\tanh (u_1-b)\tanh (u_2-b)=\frac{\cosh s}{\cosh (b-u_1)\cosh (b-u_2)}\), \(\int _{\mathbb R}\frac{\, \mathrm db}{\cosh (b-u_1)\cosh (b-u_2)}=\frac{2s}{\sinh s}\), hence \(\int _{\mathbb R}\bigl[1-\tanh (u_1-b)\tanh (u_2-b)\bigr]\, \mathrm db=2s\coth s =2|s|+\frac{4|s|}{e^{2|s|}-1}\) (\(s\ne 0\); \(=2\) for \(s=0\)).

For \(\varpi :=U[-\ell _\alpha ,\ell _\alpha ]\) and \(x,x'\in [-1,1]\), \(\int _{-\ell _\alpha }^{\ell _\alpha }\operatorname {sgn}(x-t)\operatorname {sgn}(x'-t)\, \frac{\, \mathrm dt}{2\ell _\alpha }=1-\frac{|x-x'|}{\ell _\alpha }=K^{(\infty )}(x,x')\) ??.

For \(|x|,|x'|\le 1\), \(|w|\le L\) and \(g_w(b):=\tanh (wx-b)\tanh (wx'-b)\), \(E_L(w;x,x'):=\int _{|b|{\gt}L}\bigl[1-g_w(b)\bigr]\, \mathrm db\in \bigl[0,\ 4e^{-2(L-|w|)}\bigr]\) ??.

For \(S\ge 0\), \(J_\alpha (S)\ge 0\).

For \(0{\lt}\alpha \) and \(S\ge 0\), \(J_\alpha (S)\le S^\alpha /\alpha \) (from \(\frac{2s}{e^{2s}-1}\le 1\)).

For \(0{\lt}\alpha \) and \(S\ge 0\), \(J_\alpha (S)\le D_\alpha =\int _0^\infty \frac{2s^\alpha }{e^{2s}-1}\, \mathrm ds{\lt}\infty \) ??.

For \(S\ge 0\), \(I_\alpha (S)\ge 0\) (the integrand \((s\coth s-1)s^{\alpha -1}\) is nonnegative).

For \(0{\lt}\alpha \) and \(S\ge 0\), \(I_\alpha (S)\le \frac{S^{2+\alpha }}{3(2+\alpha )}\) (from the Langevin bound \(s\coth s-1\le s^2/3\)).

For \(0{\lt}\alpha \) and \(S\ge 0\), \(I_\alpha (S)=\frac{S^{1+\alpha }}{1+\alpha }-\frac{S^\alpha }\alpha +J_\alpha (S)\) ??: integrate \((s\coth s-1)s^{\alpha -1}=s^\alpha -s^{\alpha -1} +\frac{2s^\alpha }{e^{2s}-1}\), each term being integrable at \(0\).

For \(L\ge 0\) and \(g_w(b)=\tanh (wx-b)\tanh (wx'-b)\), \(\int _{-L}^Lg_w(b)\, \mathrm db=2L-\int _{\mathbb R}(1-g_w)\, \mathrm db+E_L(w;x,x')=2L-2wd\coth (wd)+E_L(w;x,x')\) with \(d=x-x'\) (??, ??).

\(Z_LK^{(L)}(x,x')=\int _{B_L}\tanh (wx-b)\tanh (wx'-b)\, |w|^{\alpha -1}\, \mathrm dw\, \mathrm db\) (Definition 610).

For \(0{\lt}\alpha \), \(L{\gt}1\) and \(x,x'\in [-1,1]\), \(d=x-x'\): \(Z_LK^{(L)}(x,x')=2L\cdot 2M_L-4\Psi _L(d)+{\mathcal{E}}_L(x,x')\) (Fubini over \(B_L\), the inner integral of Lemma 663, and the evenness of \(wd\coth (wd)\) in \(w\)).

For \(0{\lt}\alpha \), \(L{\gt}1\), \(x,x'\in [-1,1]\) and \(d=x-x'\), \(K^{(L)}(x,x')=1-\frac{\Psi _L(d)}{LM_L}+\frac{{\mathcal{E}}_L(x,x')}{4LM_L}\) ??.

For \(0{\lt}\alpha {\lt}1\), \(L\ge 2\) and \(x,x'\in [-1,1]\), \(0\le {\mathcal{E}}_L(x,x')\le 2^{3-\alpha }L^{\alpha -1}+8e^{-L}M_L\) ??: by ??, \({\mathcal{E}}_L\le 8\int _{1/L}^Le^{-2(L-w)}w^{\alpha -1}\, \mathrm dw\); on \([1/L,L/2]\), \(e^{-2(L-w)}\le e^{-L}\), and on \([L/2,L]\), \(w^{\alpha -1}\le (L/2)^{\alpha -1}\) and \(\int _{L/2}^Le^{-2(L-w)}\, \mathrm dw\le \frac12\).

\(\Psi _L(0)=M_L\) ??.

For \(0{\lt}\alpha \), \(L\ge 1\) and \(d\ne 0\), \(\Psi _L(d)=\int _{1/L}^Lwd\coth (wd)\, w^{\alpha -1}\, \mathrm dw =M_L+|d|^{-\alpha }\bigl[I_\alpha (L|d|)-I_\alpha (|d|/L)\bigr]\) ?? (substitute \(s=w|d|\)).

For \(0{\lt}\alpha {\lt}1\), \(L{\gt}1\), \(x,x'\in [-1,1]\) with \(d=x-x'\ne 0\), \(K^{(L)}(x,x')-K^{(\infty )}(x,x')=T_1-T_2-T_3+T_4+T_5\) ??: insert ?? and ?? into ?? and use \(LM_L=L^{1+\alpha }(1-L^{-2\alpha })/\alpha \).

For \(0{\lt}\alpha \), \(L{\gt}1\) and \(x\in [-1,1]\), \(K^{(L)}(x,x)=1-\frac1L+T_5\) (Theorem 676; \(\Psi _L(0)=M_L\)).

For \(0{\lt}\alpha \) and \(L{\gt}1\), \(0\le T_3\le \frac1{L(1-L^{-2\alpha })}\) (from \(J_\alpha (S)\le S^\alpha /\alpha \)).

For \(0{\lt}\alpha \) and \(L{\gt}1\), \(T_3\le \frac{\alpha D_\alpha |d|^{-\alpha }}{L^{1+\alpha }(1-L^{-2\alpha })}\) (from \(J_\alpha \le D_\alpha \)).

For \(0{\lt}\alpha \), \(L{\gt}1\) and \(|d|\le 2\), \(0\le T_4\le \frac{4\alpha L^{-3-2\alpha }}{3(2+\alpha )(1-L^{-2\alpha })}\) (from \(I_\alpha (S)\le \frac{S^{2+\alpha }}{3(2+\alpha )}\)); this is sharper than the bound \(\frac{4\alpha L^{-2-2\alpha }}{3(2+\alpha )(1-L^{-2\alpha })}\) stated in Theorem 676.

For \(0{\lt}\alpha {\lt}1\), \(L\ge 2\) and \(x,x'\in [-1,1]\), \(0\le T_5\le \frac{2^{1-\alpha }\alpha L^{-2}}{1-L^{-2\alpha }}+\frac{2e^{-L}}L\) (from ??).

Let \(K^{(\infty )}(x,x'):=1-c_\alpha |x-x'|\). For \(0{\lt}\alpha {\lt}1\), \(L\ge 2\) and \(x,x'\in [-1,1]\), \(|K^{(L)}(x,x')-K^{(\infty )}(x,x')|\le C_\alpha L^{-\min (1,2\alpha )}\) with \(C_\alpha =\frac1{1-2^{-2\alpha }}\bigl(2+\frac{2\alpha }{1+\alpha } +\frac{4\alpha }{3(2+\alpha )}+2^{1-\alpha }\alpha \bigr)+1\) ??: the sum of the bounds on \(T_1,\dots ,T_5\) with \(1-L^{-2\alpha }\ge 1-2^{-2\alpha }\), \(|d|\le 2\) and \(2e^{-L}\le 1\); on the diagonal \(K^{(L)}(x,x)-1=-\frac1L+T_5\).

For \(0{\lt}\alpha {\lt}1\) and \(L\ge 2\), \(\sup _{x,x'\in [-1,1]}|K^{(L)}(x,x')-K^{(\infty )}(x,x')|\le C_\alpha L^{-\min (1,2\alpha )}\) ??.

For \(\varkappa \ne 0\) and \(x\in [-1,1]\), \(T[\sin \varkappa \, \cdot ](x)=-\frac2{\varkappa ^2}\sin \varkappa x +\frac{2\cos \varkappa }\varkappa x\) (proof of Theorem ??).

For \(\varkappa \ne 0\) and \(x\in [-1,1]\), \(T[\cos \varkappa \, \cdot ](x)=-\frac2{\varkappa ^2}\cos \varkappa x+\frac{2\sin \varkappa }\varkappa +\frac{2\cos \varkappa }{\varkappa ^2}\) (proof of Theorem ??).

(Theorem ??(iii), odd case.) If \(\varkappa {\gt}0\) and \(\cos \varkappa =0\), i.e. \(\varkappa \in {\mathcal{K}}^o=\{ (j+\frac12)\pi \} \), then \(K^{(\infty )}[\sin \varkappa \, \cdot ](x)=\frac{c_\alpha }{\varkappa ^2}\sin \varkappa x\) for \(x\in [-1,1]\).

(Theorem ??(iii), even case.) If \(\varkappa {\gt}0\) and \(\varkappa \sin \varkappa =\alpha \cos \varkappa \), i.e. \(\varkappa \tan \varkappa =\alpha \), then \(K^{(\infty )}[\cos \varkappa \, \cdot ](x)=\frac{c_\alpha }{\varkappa ^2}\cos \varkappa x\) for \(x\in [-1,1]\).

For \(\alpha {\gt}0\) and \(j\ge 0\) the equation \(\varkappa \tan \varkappa =\alpha \) has exactly one root \(\varkappa ^e_j\) in \((j\pi ,j\pi +\frac\pi 2)\): \(\varkappa \tan \varkappa \) is continuous and strictly increasing from \(0\) to \(+\infty \) there (proof of Theorem ??(iii)).

\(\varkappa ^e_j\in (j\pi ,j\pi +\frac\alpha {j\pi })\) for \(j\ge 1\) (Theorem ??(iii)).

\((\varkappa ^e_0)^2\le \alpha \) (Theorem ??(iii)).

(Theorem ??(iv), ??.) \(\frac{k\pi }2\le \varkappa _{(k)}\le \frac{(k+1)\pi }2\) for all \(k\ge 0\), and \(\varkappa _{(k)}\) is strictly increasing (interlacing \(\varkappa ^e_0{\lt}\frac\pi 2{\lt}\varkappa ^e_1{\lt}\frac{3\pi }2{\lt}\cdots \)).

For \(k\ge 1\), \(\frac{4c_\alpha }{\pi ^2(k+1)^2}\le \mu _k(K^{(\infty )})\le \frac{4c_\alpha }{\pi ^2k^2}\), \(c_\alpha =\frac\alpha {1+\alpha }\) ??, and \(\frac1{1+\alpha }\le \mu _0{\lt}1\). In particular \(\mu _k\asymp k^{-2}\), and the exponent does not depend on \(\alpha \).

(Theorem ??(i), injectivity.) If \(g\in C[-1,1]\) and \(K^{(\infty )}g=0\) on \([-1,1]\), then \(g=0\): \(K^{(\infty )}g=\langle g,1\rangle \mathbf1 -\frac{c_\alpha }2Tg\) with \((Tg)''=2g\), so differentiating twice gives \(-c_\alpha g=0\).

(Theorem ??(ii).) If \(\mu {\gt}0\) and \(g\in C[-1,1]\) satisfy \(K^{(\infty )}g=\mu g\) on \([-1,1]\), then \(g\) is twice differentiable on \((-1,1)\) and \(\mu \, g''=-c_\alpha \, g\) there ??. (The paper states \(g\in C^\infty [-1,1]\); the Lean statement records \(C^2\) on the open interval, which is what the exhaustion uses.)

Let \(\varkappa {\gt}0\), \(g\in C[-1,1]\) with \(g'\in C[-1,1]\), \(g''=-\varkappa ^2g\) on \((-1,1)\) and \(g'(\pm 1)=\mp \frac\alpha 2(g(1)+g(-1))\). Then \(g=A\sin \varkappa x+B\cos \varkappa x\) with \(A\cos \varkappa =0\) and \(B(\varkappa \sin \varkappa -\alpha \cos \varkappa )=0\) (proof of Theorem ??(iii)).

(Theorem ??(iii), exhaustion.) If \(\mu {\gt}0\) and \(g\in C[-1,1]\), \(g\not\equiv 0\), satisfy \(K^{(\infty )}g=\mu g\) on \([-1,1]\), then \(\mu =c_\alpha /\varkappa ^2\) for some \(\varkappa {\gt}0\) and either \(\cos \varkappa =0\) (\(\varkappa \in {\mathcal{K}}^o\)) and \(g=A\sin \varkappa x\), or \(\varkappa \tan \varkappa =\alpha \) (\(\varkappa \in {\mathcal{K}}^e\)) and \(g=B\cos \varkappa x\), with \(A,B\ne 0\). Hence the eigenvalues and eigenfunctions are exhausted by Theorem ??(iii) and all eigenvalues are simple.

(Theorem ??(i), positivity.) For \(g\in L^1(P_X)\), \(\iint K^{(\infty )}(x,x')g(x)g(x')\, P_X(\, \mathrm dx)P_X(\, \mathrm dx') =\int \Bigl(\int \operatorname {sgn}(x-t)g(x)\, P_X(\, \mathrm dx)\Bigr)^2\varpi (\, \mathrm dt)\ge 0\) (Remark 655 and Fubini).

(Theorem ??(v).) \(\sum _k\mu _k=\operatorname {Tr}K^{(\infty )}=1\): the odd part is \(\sum _j\frac1{(j+1/2)^2\pi ^2}=\frac12\), the even part \(\sum _j(\varkappa ^e_j)^{-2}=\frac1\alpha +\frac12\) by the Hadamard factorization of \(\varkappa \sin \varkappa -\alpha \cos \varkappa \).

For bounded operators \(A,B\) on a real Hilbert space and all \(k\ge 0\), \(|\mu _k(A)-\mu _k(B)|\le \| A-B\| \) for the min–max values: the min–max principle and \(\langle Ag,g\rangle \le \langle Bg,g\rangle +\| A-B\| \) (proof of Theorem ??(a)).

The min–max values of \(K^{(\infty )}\) on \(L^2(P_X)\) are \(\mu _k(K^{(\infty )})=\frac{c_\alpha }{\varkappa _{(k)}^2}\), the eigenvalues of Theorem ?? in decreasing order (min–max principle for compact self-adjoint operators).

(Theorem ??(a).) If \(\| K^{(L)}-K^{(\infty )}\| \le {\varepsilon }_L\) on \(L^2(P_X)\), then \(|\mu _k(K^{(L)})-\mu _k(K^{(\infty )})|\le {\varepsilon }_L\) for all \(k\ge 0\).

(Theorem ??(b).) Let \(\mu _k(K^{(L)})\) satisfy \(|\mu _k(K^{(L)})-\mu _k(K^{(\infty )})|\le {\varepsilon }_L\) for all \(k\) (part (a)). Then for \(1\le k\le k_{\max }(L)=\lfloor \sqrt{2c_\alpha /(\pi ^2{\varepsilon }_L)}\rfloor -1\), equivalently \({\varepsilon }_L\le \frac{2c_\alpha }{\pi ^2(k+1)^2}\), \(\frac{2c_\alpha }{\pi ^2(k+1)^2}\le \mu _k(K^{(L)})\le \frac{6c_\alpha }{\pi ^2k^2}\) ??.

For \(u\in L^1(\varpi )\) and \(x\in [-\ell _\alpha ,\ell _\alpha ]\), \((S_\varpi u)(x)=\int \operatorname {sgn}(x-t)u(t)\, \varpi (\, \mathrm dt) =\frac1{2\ell _\alpha }\Bigl[\int _{-\ell _\alpha }^xu-\int _x^{\ell _\alpha }u\Bigr]\) (the integrand is \(u\) for \(t{\lt}x\) and \(-u\) for \(t{\gt}x\)).

For \(r\in L^1(P_X)\) and \(t\in [-1,1]\), \((S^*r)(t)=\int \operatorname {sgn}(x-t)r(x)\, P_X(\, \mathrm dx) =\frac12\Bigl[\int _t^1r-\int _{-1}^tr\Bigr]\); for \(t{\lt}-1\) it is \(\frac12\int _{-1}^1r\) and for \(t{\gt}1\) it is \(-\frac12\int _{-1}^1r\).

For \(u\in L^2(\varpi )\), \(h:=S_\varpi u\) is in \(H^1(-1,1)\) with weak derivative \(h'=u/\ell _\alpha \) on \((-1,1)\): \(h(x)-h(-1)=\frac1{\ell _\alpha }\int _{-1}^xu\) for \(x\in [-1,1]\) (Theorem ??(ii), \(\operatorname {ran}S_\varpi \subseteq H^1\)).

For \(f\in H^1(-1,1)\) with \(f'=g\) and the representer \(u_\varpi ^\dagger \) of ??, \((S_\varpi u_\varpi ^\dagger )(x)=f(x)\) for every \(x\in [-1,1]\): with \(a=(1+\alpha )c_0\), \(S_\varpi u_\varpi ^\dagger (x)=\frac1{2\ell _\alpha }\bigl[2a(\ell _\alpha -1) +\ell _\alpha (2f(x)-f(1)-f(-1))\bigr]=f(x)\) (Theorem ??(ii), \(H^1\subseteq \operatorname {ran}S_\varpi \)).

\(K^{(\infty )}=S_\varpi S_\varpi ^*\): for \(r\in L^2(P_X)\) and \(P_X\)-a.e. \(x\), \((S_\varpi S_\varpi ^*r)(x)=\int _{-1}^1K^{(\infty )}(x,x')r(x')\, P_X(\, \mathrm dx')\) with \(K^{(\infty )}(x,x')=1-c_\alpha |x-x'|\) (Remark 655 and the integral representation of \(S_\varpi S_\varpi ^*\)).

\(\operatorname {ran}S_\varpi =H^1(-1,1)\) as classes in \(L^2(P_X)\): \(F\in L^2(P_X)\) is of the form \(S_\varpi u\) with \(u\in L^2(\varpi )\) if and only if \(F\) has a representative \(f\in H^1(-1,1)\). (\(S_\varpi u\) is the primitive of \(u/\ell _\alpha \); conversely \(f=S_\varpi u_\varpi ^\dagger \) with the representer ??.)

\(\ker S_\varpi =\{ u\in L^2(\varpi ):u=0\text{ a.e.\ on }(-1,1), \int _{-\ell _\alpha }^{-1}u=\int _1^{\ell _\alpha }u\} \). (\(S_\varpi u=0\) if and only if the \(H^1\) function \(h=S_\varpi u\) vanishes on \([-1,1]\), i.e. \(h'=u/\ell _\alpha =0\) a.e. on \((-1,1)\) and \(h(1)+h(-1)=\frac1{\ell _\alpha }[\int _{-\ell _\alpha }^{-1}u-\int _1^{\ell _\alpha }u]=0\); the first condition uses the Lebesgue differentiation theorem.)

For \(f\in H^1(-1,1)\) the minimum-norm representer is \(S_\varpi ^\dagger f=u_\varpi ^\dagger \) ??: \(u_\varpi ^\dagger \) is a representer of \(f\) and is orthogonal to \(\ker S_\varpi \) (by (v)), hence is the unique representer in \((\ker S_\varpi )^\perp \) (Lemma 384).

For \(f\in H^1(-1,1)\) with \(c_0=\frac{f(1)+f(-1)}2\), \(\| S_\varpi ^\dagger f\| ^2_{L^2(\varpi )}=\| f\| ^2_{{\mathcal{H}}_\infty } =\frac{1+\alpha }\alpha \| f'\| ^2_{L^2(P_X)}+(1+\alpha )c_0^2\) ??. (\(\| u_\varpi ^\dagger \| ^2=\frac1{2\ell _\alpha }\bigl[\ell _\alpha ^2\int _{-1}^1f'{}^2\, \mathrm dx +\frac{2\ell _\alpha ^2c_0^2}{\ell _\alpha -1}\bigr]\) with \(\ell _\alpha =\frac{1+\alpha }\alpha \), \(\frac{\ell _\alpha }{\ell _\alpha -1}=1+\alpha \).)

\(\| f\| ^2_{{\mathcal{H}}_\infty }\le \frac{1+\alpha }\alpha \| f'\| ^2_{L^2(P_X)} +(1+\alpha )\| f\| _\infty ^2\), since \(|c_0|\le \| f\| _\infty \).

\(\| f\| ^2_{L^2(P_X)}\le 4\| f'\| ^2_{L^2(\, \mathrm dx)}+2c_0^2\), since \(|f(x)-c_0|\le \frac12\bigl(|f(x)-f(1)|+|f(x)-f(-1)|\bigr)\le \frac12\int _{-1}^1|f'| \le \frac1{\sqrt2}\| f'\| _{L^2(\, \mathrm dx)}\).

If \(f=K^{(\infty )}g_0\) with \(g_0\in L^2(P_X)\), then \(f\in H^2(-1,1)\) with \(f''=-c_\alpha g_0\), i.e. \(g_0=(K^{(\infty )})^{-1}f=-\frac{1+\alpha }\alpha f''\), and \(f'(1)=-\frac\alpha 2(f(1)+f(-1))\), \(f'(-1)=+\frac\alpha 2(f(1)+f(-1))\) ??. (\(f=S_\varpi v\) with \(v=S_\varpi ^*g_0\), so \(f'=v/\ell _\alpha \) by Theorem ??(ii) and \(v'=-g_0\); the boundary values come from \(v=\pm \frac12\int g_0\) outside \([-1,1]\).)

If \(f\in H^2(-1,1)\) with \(f'(1)=-\frac\alpha 2(f(1)+f(-1))\) and \(f'(-1)=+\frac\alpha 2(f(1)+f(-1))\), then \(f=K^{(\infty )}g_0\) for \(g_0:=-\frac{1+\alpha }\alpha f''\in L^2(P_X)\) ??. (\(S_\varpi ^*g_0=\ell _\alpha f'-\frac{\ell _\alpha }2(f'(1)+f'(-1))=\ell _\alpha f'\) on \((-1,1)\) and \(=\pm \frac{\ell _\alpha }2(f'(-1)-f'(1))=\pm (1+\alpha )c_0\) outside, i.e. \(S_\varpi ^*g_0=u_\varpi ^\dagger \), and \(S_\varpi u_\varpi ^\dagger =f\) by Theorem ??(ii).)

Let \((\varphi ,\nu )\) and \((\varphi ',\nu ')\) be two feature/hidden-law pairs on the same \(L^2(P_X)\) with kernels \(k_\nu \), \(k_{\nu '}\). If \(|k_\nu (x,x')-k_{\nu '}(x,x')|\le {\varepsilon }\) for \(P_X\)-a.e. \(x\) and \(P_X\)-a.e. \(x'\), then \(\| K_\nu -K_{\nu '}\| _{L^2(P_X)\to L^2(P_X)}\le {\varepsilon }\) (the proof of Lemma 754 only uses the bound almost everywhere).

For \(0{\lt}\alpha {\lt}1\) and \(L\ge 2\), \({\varepsilon }_L:=\| K^{(L)}-K^{(\infty )}\| _{L^2(P_X)\to L^2(P_X)} \le \sup _{x,x'\in [-1,1]}|K^{(L)}(x,x')-K^{(\infty )}(x,x')|\le C_\alpha L^{-\min (1,2\alpha )}\) ??, where \(K^{(L)}=S_{\nu _0^{(L)}}S^*_{\nu _0^{(L)}}\) is the kernel operator of the tanh feature and \(K^{(\infty )}=S_\varpi S_\varpi ^*\) that of the sign network (Remark 655). (Theorem 676, Remark 655 and Lemma 710, since \(P_X\)-a.e. \(x\) lies in \([-1,1]\).)

For \(0{\lt}\alpha {\lt}1\), \(\| K^{(L)}-K^{(\infty )}\| _{L^2(P_X)\to L^2(P_X)}\to 0\) as \(L\to \infty \) (\({\varepsilon }_L\le C_\alpha L^{-\min (1,2\alpha )}\to 0\), Theorem 711).

For \(0{\lt}\alpha {\lt}1\), \(\lambda {\gt}0\) and \(f\in L^2(P_X)\), \(Q^{(L)}_\lambda (f)\to Q^{(\infty )}_\lambda (f)\) as \(L\to \infty \), where \(Q^{(L)}_\lambda (f)=\langle f,(K^{(L)}+\lambda )^{-1}f\rangle \) and \(Q^{(\infty )}_\lambda (f)=\langle f,(K^{(\infty )}+\lambda )^{-1}f\rangle \) (Theorem 740 with \({\varepsilon }_L\to 0\), Theorem 712).

If \(f\in H^1(-1,1)\), then \(Q^{(\infty )}_\lambda (f)\to \| f\| ^2_{{\mathcal{H}}_\infty }\) as \(\lambda \downarrow 0\) (Lemma 749 and Theorem ??(ii),(iii): \(f\in \operatorname {ran}S_\varpi \) and \(\| S_\varpi ^\dagger f\| ^2=\| f\| ^2_{{\mathcal{H}}_\infty }\)).

If \(f\notin H^1(-1,1)\), then \(Q^{(\infty )}_\lambda (f)\to +\infty \) as \(\lambda \downarrow 0\) (Lemma 751 and Theorem ??(ii)).

For \(f\in H^1(-1,1)\) and \(\lambda {\gt}0\), \(Q^{(\infty )}_\lambda (f)\le \| S_\varpi ^\dagger f\| ^2_{L^2(\varpi )}=\| f\| ^2_{{\mathcal{H}}_\infty }\) (Lemma 745 and Theorem 705).

For \(0{\lt}\alpha {\lt}1\) and \(f\in L^2(P_X)\),

in the sense that \(\lambda \mapsto \lim _{L\to \infty }Q^{(L)}_\lambda (f)=Q^{(\infty )}_\lambda (f)\) (nonincreasing in \(\lambda \)) is bounded on \((0,\infty )\) if and only if \(f\) has an \(H^1(-1,1)\) representative. This is the precise meaning of “in the limit \(L\to \infty \), (R) is equivalent to \(f\in H^1\)”. (Theorems 713, 714, 715.)

For \(f\in H^1(-1,1)\), \(\lim _{\lambda \downarrow 0}\lim _{L\to \infty }Q^{(L)}_\lambda (f)=\| f\| ^2_{{\mathcal{H}}_\infty }\); for \(f\notin H^1(-1,1)\) the iterated limit is \(+\infty \).

Let \(0{\lt}\alpha {\lt}1\), \(\nu _0=\nu _0^{(L)}\) (\(L\ge 2\)), \((\lambda ,\beta )\) fixed and \(f\in H^1(-1,1)\), and let \(\rho ^*_L\) be the corresponding minimizer of the free energy. Then

??. (Corollary 760 with \(\| K^{(L)}-K^{(\infty )}\| \le {\varepsilon }_L\to 0\), Theorem 712, and Lemma 716. In Lean the reference law is \(\nu _0^{(L)}\) itself, so no smoothing \(\tilde\nu _0^{(L)}\) and no correction \(1/L\) of \({\varepsilon }_L\) is needed.)

Under the hypotheses of Corollary 720, with \(\nu ^*_L\) the hidden marginal of \(\rho ^*_L\) and \(\kappa =\lambda /\beta \), \(\limsup _L\operatorname {KL}(\nu ^*_L\| \nu _0^{(L)})\le \frac\kappa 2\| f\| ^2_{{\mathcal{H}}_\infty }\); the right-hand side does not depend on \(\lambda ,\beta \) (Corollary 761).

Under the hypotheses of Corollary 720, with \(m^*_L\) the conditional mean amplitude of \(\rho ^*_L\), \(\limsup _L\| m^*_L\| ^2_{L^2(\nu ^*_L)}\le \| f\| ^2_{{\mathcal{H}}_\infty }\): the bounds of Theorem 397 hold asymptotically with \(f\in H^1\) in place of (R) and \(\| f\| _{{\mathcal{H}}_\infty }\) in place of \(\| u_0^\dagger \| \) (Corollary 762).

Let \(K\ge 0\) be a bounded positive operator on a Hilbert space and \(x\in \overline{\operatorname {ran}K}\). Then \(\lambda (K+\lambda )^{-1}x\to 0\) as \(\lambda \downarrow 0\). (For \({\varepsilon }{\gt}0\) write \(x=(x-Kw)+Kw\) with \(\| x-Kw\| {\lt}{\varepsilon }\); then \(\| \lambda (K+\lambda )^{-1}(x-Kw)\| \le {\varepsilon }\) and \(\| \lambda (K+\lambda )^{-1}Kw\| \le \lambda \| w\| \), Lemma 197.)

\(K^{(\infty )}\) is injective on \(L^2(P_X)\): if \(K^{(\infty )}r=0\) then \(\| S_\varpi ^*r\| ^2=\langle K^{(\infty )}r,r\rangle =0\), so \(v:=S^*r\) vanishes \(\varpi \)-a.e.; \(v\) is continuous on \([-1,1]\) with \(v'=-r\) (Lemma 698), so \(v=0\) on \([-1,1]\), \(\int _{-1}^xr=0\) for all \(x\), and \(r=0\) a.e. by Lebesgue differentiation.

\(\overline{\operatorname {ran}K^{(\infty )}}=(\ker K^{(\infty )})^\perp =L^2(P_X)\), since \(K^{(\infty )}\) is self-adjoint and injective (Theorem 724).

If \(f=K^{(\infty )}g\) with \(g\in L^2(P_X)\), then \(R^{(\infty )}_\lambda f\to u^\dagger _\infty =S^*g\) uniformly on \(Z\) as \(\lambda \downarrow 0\). (\(K^{(\infty )}\) is self-adjoint and injective, so \(\overline{\operatorname {ran}K^{(\infty )}}=L^2(P_X)\) and \((K^{(\infty )}+\lambda )^{-1}f=g-\lambda (K^{(\infty )}+\lambda )^{-1}g\to g\) in \(L^2\) by Lemma 723; the bound \(\| S^*r\| _\infty \le \| r\| \) of Lemma ??(2) gives uniform convergence.)

If \(g=-\frac{1+\alpha }\alpha f''\) \(P_X\)-a.e. (i.e. \(f=K^{(\infty )}g\), ??), then \(u^\dagger _\infty (w,b)=-\frac{1+\alpha }{2\alpha }\int _{-1}^1\tanh (wx-b)f''(x)\, \, \mathrm dx\) ??: the limiting canonical ridgelet transform is the ridgelet transform, with the activation itself as filter, of the target sharpened by \(-\Delta \).

If \(f\in H^2(-1,1)\) satisfies the boundary conditions ?? and \(g:=-\frac{1+\alpha }\alpha f''\), then \(f=K^{(\infty )}g\), \(R^{(\infty )}_\lambda f\to u^\dagger _\infty \) uniformly on \(Z\) as \(\lambda \downarrow 0\), and \(u^\dagger _\infty (w,b)=-\frac{1+\alpha }{2\alpha }\int _{-1}^1\tanh (wx-b)f''(x)\, \mathrm dx\) (Theorems 709, 726, 727).

If \(f=K^{(\infty )}g\) with \(g\in L^2(P_X)\), then for \(L\ge 2\), \(\| S_{\nu _0^{(L)}}u^\dagger _\infty -f\| _{L^2(P_X)}\le {\varepsilon }_L\| g\| \) \(\le C_\alpha L^{-\min (1,2\alpha )}\| g\| \) ??, since \(S_{\nu _0^{(L)}}S^*g=K^{(L)}g\) and \(\| K^{(L)}g-K^{(\infty )}g\| \le {\varepsilon }_L\| g\| \) (Theorem 711).

For \(g\in L^2(P_X)\) and every \(L\), \(\| S^*g\| ^2_{L^2(\nu _0^{(L)})}=\langle K^{(L)}g,g\rangle \) ??.

If \(f=K^{(\infty )}g\), then \(\| u^\dagger _\infty \| ^2_{L^2(\nu _0^{(L)})}=\langle K^{(L)}g,g\rangle \to \langle f,g\rangle \) as \(L\to \infty \) ?? (\(|\langle (K^{(L)}-K^{(\infty )})g,g\rangle |\le {\varepsilon }_L\| g\| ^2\)).

If \(f=K^{(\infty )}g\) with \(f\in H^2(-1,1)\) satisfying ?? and \(g=-\frac{1+\alpha }\alpha f''\), then \(\langle f,g\rangle =\langle K^{(\infty )}g,g\rangle =\| S_\varpi ^*g\| ^2 =\| S_\varpi ^\dagger f\| ^2=\| f\| ^2_{{\mathcal{H}}_\infty }\) (Theorem 705; \(S_\varpi ^*g\) is the minimum-norm representer of \(f=S_\varpi S_\varpi ^*g\)).

For \(r\in L^1(P_X)\) and \(t\in {\mathbb R}\), \(\lim _{w\to +\infty }(S^*r)(w,wt)=\lim _{w\to +\infty }\int \tanh (w(x-t))r(x)\, P_X(\, \mathrm dx) =\int \operatorname {sgn}(x-t)r(x)\, P_X(\, \mathrm dx)=(S_\varpi ^*r)(t)\), and \(\lim _{w\to -\infty }(S^*r)(w,wt)=-(S_\varpi ^*r)(t)\) (dominated convergence of \(\tanh (w(x-t))\to \operatorname {sgn}(w)\operatorname {sgn}(x-t)\) for \(x\ne t\), with the dominating function \(|r|\)).

For \(f=K^{(\infty )}g\) and every \(t\in {\mathbb R}\), \(\lim _{w\to \pm \infty }u^\dagger _\infty (w,wt)=\pm (S_\varpi ^*g)(t)\); that is, \(u^\dagger _\infty (w,wt)\to \operatorname {sgn}(w)\, u^\dagger _\varpi (t)\), where \(S_\varpi ^*g=u^\dagger _\varpi \) is the representer ?? of the sign network (extended by the same formula to \(|t|{\gt}\ell _\alpha \); the explicit values are Theorem 735). (Lemma 733.)

For \(f\in H^2(-1,1)\) with the boundary conditions ?? and \(g=-\frac{1+\alpha }\alpha f''\): \((S_\varpi ^*g)(t)=\frac{1+\alpha }\alpha f'(t)\) for \(|t|{\lt}1\), and \((S_\varpi ^*g)(t)=\mp (1+\alpha )\frac{f(1)+f(-1)}2\) for \(t\gtrless \pm 1\); hence \(\lim _{w\to \pm \infty }u^\dagger _\infty (w,wt)=\pm \frac{1+\alpha }\alpha f'(t)\) for \(|t|{\lt}1\) and \(\mp \operatorname {sgn}(t)(1+\alpha )\frac{f(1)+f(-1)}2\) for \(|t|{\gt}1\). (\(\int _{-1}^1\operatorname {sgn}(x-t)f''\, \mathrm dx=f'(1)+f'(-1)-2f'(t)=-2f'(t)\) for \(|t|{\lt}1\), using \(f'(1)+f'(-1)=0\), and \(\int \operatorname {sgn}(x-t)f''=-\operatorname {sgn}(t)(f'(1)-f'(-1)) =\operatorname {sgn}(t)\alpha (f(1)+f(-1))\) for \(|t|{\gt}1\).)

For \(f\in H^2(-1,1)\) with ?? and \(g=-\frac{1+\alpha }\alpha f''\): for \(|t|{\lt}1\), \(\lim _{w\to \pm \infty }u^\dagger _\infty (w,wt)=\pm \frac{1+\alpha }\alpha f'(t)\), and for \(|t|{\gt}1\), \(\lim _{w\to +\infty }u^\dagger _\infty (w,wt)=-\operatorname {sgn}(t)(1+\alpha )\frac{f(1)+f(-1)}2\) (Theorems 734, 735).

Let \(A_1,A_2,R_1,R_2\) be bounded operators on a Hilbert space and \(\lambda \in {\mathbb R}\) with \(R_1(A_1+\lambda )=\mathrm{id}\) and \((A_2+\lambda )R_2=\mathrm{id}\). Then \(R_1-R_2=R_1(A_2-A_1)R_2\).

Under the hypotheses of Lemma 737 with \(\lambda {\gt}0\) and \(\| R_1\| ,\| R_2\| \le 1/\lambda \), \(\| R_1-R_2\| \le \| R_1\| \, \| A_1-A_2\| \, \| R_2\| \le \lambda ^{-2}\| A_1-A_2\| \).

Under the hypotheses of Lemma 738, for every \(f\), \(|\langle f,R_1f\rangle -\langle f,R_2f\rangle |\le \lambda ^{-2}\| A_1-A_2\| \, \| f\| ^2\).

Let \(K_\nu =S_\nu S_\nu ^*\) and \(K_{\nu '}=S_{\nu '}S_{\nu '}^*\) be the kernel operators of two (feature map, hidden law) pairs on the same \(L^2(P_X)\), \(\lambda {\gt}0\) and \(f\in L^2(P_X)\). Then \(|Q^\nu _\lambda (f)-Q^{\nu '}_\lambda (f)|\le \lambda ^{-2}\| K_\nu -K_{\nu '}\| \, \| f\| ^2\). In the setting of Theorem ??, \(K_\nu =K^{(L)}\), \(K_{\nu '}=K^{(\infty )}\) and \(\| K^{(L)}-K^{(\infty )}\| \le {\varepsilon }_L\), this is \(|Q^{(L)}_\lambda (f)-Q^{(\infty )}_\lambda (f)|\le \lambda ^{-2}{\varepsilon }_L\| f\| ^2\). (Resolvent identity \((A+\lambda )^{-1}-(B+\lambda )^{-1}=(A+\lambda )^{-1}(B-A)(B+\lambda )^{-1}\) and \(\| (K+\lambda )^{-1}\| \le \lambda ^{-1}\).)

For \(\lambda {\gt}0\), \(f\in L^2(P_X)\) and two hidden laws \(\nu ,\nu '\in {\mathcal P}(Z)\), \(R_{\lambda ,\nu }f=S^*(K_\nu +\lambda )^{-1}f\) and \(R_{\lambda ,\nu '}f=S^*(K_{\nu '}+\lambda )^{-1}f\) are bounded functions on \(Z\) with \(\sup _{z\in Z}|R_{\lambda ,\nu }f(z)-R_{\lambda ,\nu '}f(z)|\le \lambda ^{-2}\| K_\nu -K_{\nu '}\| \, \| f\| \). With \(\nu =\nu _0^{(L)}\), \(\nu '\) the limit law and \(\| K^{(L)}-K^{(\infty )}\| \le {\varepsilon }_L\) this is ??. (\(\| S^*r\| _\infty \le \| r\| _{L^2(P_X)}\), Lemma ??(2), and Theorem 740.)

If \(\| K_{ u_L}-K_{ u'}\| o0\) along a filter in \(L\), then for every \(\lambda {\gt}0\) and \(f\in L^2(P_X)\), \(Q^{ u_L}_\lambda (f) o Q^{ u'}_\lambda (f)\) (Theorem  efthm:order-of-limits-L-a).

For \(\lambda {\gt}0\) and \(f\in L^2(P_X)\), with \(g=(K_\nu +\lambda )^{-1}f\) and \(u_\lambda =S_\nu ^*g=R_{\lambda ,\nu }f\), \(Q^\nu _\lambda (f)=\langle K_\nu g,g\rangle +\lambda \| g\| ^2=\| u_\lambda \| ^2_{L^2(\nu )} +\lambda \| g\| ^2_{L^2(P_X)}\ge 0\). (\(f=(K_\nu +\lambda )g\) and \(\langle K_\nu g,g\rangle =\| S_\nu ^*g\| ^2\).)

For \(\lambda {\gt}0\), \(f\in L^2(P_X)\) and \(u_\lambda :=S_\nu ^*(K_\nu +\lambda )^{-1}f =R_{\lambda ,\nu }f\), one has \(S_\nu u_\lambda -f=-\lambda (K_\nu +\lambda )^{-1}f\) and \(\frac12\| S_\nu u_\lambda -f\| ^2+\frac\lambda 2\| u_\lambda \| ^2 =\frac\lambda 2\langle f,(K_\nu +\lambda )^{-1}(\lambda +K_\nu )(K_\nu +\lambda )^{-1}f\rangle =\frac\lambda 2Q^\nu _\lambda (f)\); by Lemma ??(4) this is the minimum of the Tikhonov functional \(u\mapsto \frac12\| S_\nu u-f\| ^2+\frac\lambda 2\| u\| ^2\) over \(L^2(\nu )\).

For \(\lambda {\gt}0\) and every \(u\in L^2(\nu )\) with \(S_\nu u=f\), \(Q^\nu _\lambda (f)\le \| u\| ^2_{L^2(\nu )}\); under (R), \(Q^\nu _\lambda (f)\le \| S_\nu ^\dagger f\| ^2\). (Take \(u\) in the Tikhonov minimum, Proposition 744: \(\frac\lambda 2Q^\nu _\lambda (f)\le \frac12\| S_\nu u-f\| ^2+\frac\lambda 2\| u\| ^2 =\frac\lambda 2\| u\| ^2\).)

For \(0{\lt}\lambda \le \lambda '\), \(Q^\nu _{\lambda '}(f)\le Q^\nu _\lambda (f)\): the map \(\lambda \mapsto Q^\nu _\lambda (f)\) is nonincreasing. (Without spectral theory: \(Q^\nu _\lambda (f)=\min _u[\lambda ^{-1}\| S_\nu u-f\| ^2+\| u\| ^2]\) by Proposition 744, and the functional is pointwise nonincreasing in \(\lambda \).)

For \(0{\lt}\lambda '\le \lambda \), \(\| u_{\lambda '}-u_\lambda \| ^2_{L^2(\nu )}\le Q^\nu _{\lambda '}(f)-Q^\nu _\lambda (f)\). (The identity \(J_\lambda (u)=J_\lambda (u_\lambda )+\frac12\| S_\nu (u-u_\lambda )\| ^2 +\frac\lambda 2\| u-u_\lambda \| ^2\) of Lemma ??(4) with \(u=u_{\lambda '}\), \(J_\lambda (u_\lambda )=\frac\lambda 2Q^\nu _\lambda (f)\) and \(J_\lambda (u_{\lambda '})\le \frac\lambda {\lambda '}J_{\lambda '}(u_{\lambda '}) =\frac\lambda 2Q^\nu _{\lambda '}(f)\).)

Assume (R). Then \(u_\lambda =R_{\lambda ,\nu }f\to S_\nu ^\dagger f\) in \(L^2(\nu )\) as \(\lambda \downarrow 0\). (By Lemma 453, \(u_\lambda -u^\dagger =-\lambda (T_0+\lambda )^{-1}u^\dagger \); \(u^\dagger \in (\ker S_\nu )^\perp =\overline{\operatorname {ran}T_0}\), so for \({\varepsilon }{\gt}0\) write \(u^\dagger =(u^\dagger -T_0w)+T_0w\) with \(\| u^\dagger -T_0w\| {\lt}{\varepsilon }\); then \(\| \lambda (T_0+\lambda )^{-1}(u^\dagger -T_0w)\| \le {\varepsilon }\) and \(\| \lambda (T_0+\lambda )^{-1}T_0w\| \le \lambda \| w\| \).)

Assume (R). Then \(Q^\nu _\lambda (f)\uparrow \| S_\nu ^\dagger f\| ^2_{L^2(\nu )}\) as \(\lambda \downarrow 0\): \(\lambda \mapsto Q^\nu _\lambda (f)\) is nonincreasing, bounded by \(\| S_\nu ^\dagger f\| ^2\), and converges to it. (\(Q^\nu _\lambda (f)=\langle f,g_\lambda \rangle =\langle S_\nu u^\dagger ,g_\lambda \rangle =\langle u^\dagger ,u_\lambda \rangle \) and \(u_\lambda \to u^\dagger \), Lemma 748.)

If \(\sup _{\lambda {\gt}0}Q^\nu _\lambda (f){\lt}\infty \), then \(f\in \operatorname {ran}S_\nu \). (Let \(\ell :=\sup _{\lambda {\gt}0}Q^\nu _\lambda (f)=\lim _{\lambda \downarrow 0}Q^\nu _\lambda (f)\). For \(0{\lt}\lambda '\le \lambda \le \lambda _0\), Lemma 747 gives \(\| u_{\lambda '}-u_\lambda \| ^2\le Q_{\lambda '}-Q_\lambda \le \ell -Q_{\lambda _0}\), so \(u_{1/n}\) is Cauchy in \(L^2(\nu )\) with limit \(u\); and \(\| S_\nu u_\lambda -f\| ^2\le \lambda Q_\lambda (f) \le \lambda \sup Q\to 0\) gives \(S_\nu u=f\). No weak compactness is needed.)

If \(f\notin \operatorname {ran}S_\nu \), then \(Q^\nu _\lambda (f)\uparrow +\infty \) as \(\lambda \downarrow 0\): for every \(M\) there is \(\lambda _0{\gt}0\) with \(Q^\nu _\lambda (f)\ge M\) for all \(0{\lt}\lambda \le \lambda _0\). Together with Lemma 749, \(\lim _{\lambda \downarrow 0}Q^\nu _\lambda (f){\lt}\infty \) if and only if (R) holds for \(\nu \). (Otherwise \(Q^\nu _\lambda (f)\) is bounded on \((0,\infty )\) by monotonicity, and Lemma 750 gives \(f\in \operatorname {ran}S_\nu \).)

For every \(\nu \in {\mathcal P}(Z)\) and \(f\in L^2(P_X)\), \(\lambda \mapsto Q^\nu _\lambda (f)\) is nonincreasing on \((0,\infty )\), and as \(\lambda \downarrow 0\), \(Q^\nu _\lambda (f)\to \| S_\nu ^\dagger f\| ^2_{L^2(\nu )}\) if \(f\in \operatorname {ran}S_\nu \), while \(Q^\nu _\lambda (f)\to +\infty \) otherwise. For the limit kernel \(K^{(\infty )}\) of Theorem ??, \(\| S_\nu ^\dagger f\| ^2=\| f\| ^2_{{\mathcal{H}}_\infty }\) and \(\operatorname {ran}S_\nu =H^1(-1,1)\) (Theorem ??).

For \(g\in L^2(P_X)\) and \(P_X\)-a.e. \(x\), \((K_\nu g)(x)=\int _{{\mathcal X}}k_\nu (x,x')g(x')\, P_X(\, \mathrm dx')\) with \(k_\nu (x,x')=\int _Z\varphi _z(x)\varphi _z(x')\, \nu (\, \mathrm dz)\), \(|k_\nu |\le 1\). (Fubini in \((K_\nu g)(x)=\int _Z\varphi _z(x)\int _{{\mathcal X}}\varphi _z(x')g(x')\, P_X(\, \mathrm dx')\, \nu (\, \mathrm dz)\), the integrand being dominated by \(|g(x')|\).)

Let \((\varphi ,\nu )\) and \((\varphi ',\nu ')\) be two feature/hidden-law pairs on the same \(L^2(P_X)\) with kernels \(k_\nu (x,x')=\int \varphi _z(x)\varphi _z(x')\, \nu (\, \mathrm dz)\) and \(k_{\nu '}\). If \(\sup _{x,x'}|k_\nu (x,x')-k_{\nu '}(x,x')|\le {\varepsilon }\), then \(\| K_\nu -K_{\nu '}\| _{L^2(P_X)\to L^2(P_X)}\le {\varepsilon }\). (By Lemma 753, \(|(K_\nu g-K_{\nu '}g)(x)|\le {\varepsilon }\| g\| _{L^1(P_X)}\le {\varepsilon }\| g\| _{L^2(P_X)}\) for a.e. \(x\), and \(P_X\) is a probability measure.) In the setting of Theorem 676, \(\| K^{(L)}-K^{(\infty )}\| \le \sup _{x,x'\in [-1,1]}|K^{(L)}(x,x')-K^{(\infty )}(x,x')| \le {\varepsilon }_L\).

Let \(u\in L^2(\nu _0)\) be arbitrary and \(\tilde\rho _u:=\rho _{\nu _0,u}=\nu _0\otimes {\mathcal N}(u,\beta /\lambda )\) the competitor of ??. Then \(\tilde\rho _u\in {\mathcal D}\), \(F_{\tilde\rho _u}=S_{\nu _0}u\), \(L(\tilde\rho _u)=\frac12\| S_{\nu _0}u-f\| ^2\), \(\operatorname {KL}(\tilde\rho _u\| \mu _U)=\frac\lambda {2\beta }\| u\| ^2_{L^2(\nu _0)}\) and \({\mathcal F}(\tilde\rho _u)=\frac12\| S_{\nu _0}u-f\| ^2+\frac\lambda 2\| u\| ^2_{L^2(\nu _0)}\) (Lemma ?? with \(\nu =\nu _0\)).

Assume (A1), (A3), (A4), (A5) and \(f\in L^2(P_X)\). Let \(\rho ^*\) be the minimizer of \({\mathcal F}\) (Lemma ??). For every \(u\in L^2(\nu _0)\),

(In the proof of Theorem 397, take the competitor \(\tilde\rho _u=\nu _0\otimes {\mathcal N}(u,\beta /\lambda )\) for an arbitrary \(u\): \(L(\tilde\rho _u)=\frac12\| S_{\nu _0}u-f\| ^2\) and \(\operatorname {KL}(\tilde\rho _u\| \mu _U)=\frac\lambda {2\beta }\| u\| ^2\), Lemma 755; minimality \({\mathcal F}(\rho ^*)\le {\mathcal F}(\tilde\rho _u)\).)

Under the hypotheses of Proposition 756, the infimum of the right-hand side of ?? over \(u\in L^2(\nu _0)\) is attained at \(u=R_{\lambda ,\nu _0}f\) with value \(\frac\lambda 2Q^{\nu _0}_\lambda (f)\) (Proposition 744), and therefore \(L(\rho ^*)+\beta \, \operatorname {KL}(\rho ^*\| \mu _U)\le \frac\lambda 2Q^{\nu _0}_\lambda (f)\).

Under (R), \(Q^{\nu _0}_\lambda (f)\le \| u_0^\dagger \| ^2_{L^2(\nu _0)}\) (Lemma 745), and Theorem 397(a) is recovered: \(L(\rho ^*)+\beta \, \operatorname {KL}(\rho ^*\| \mu _U)\le \frac\lambda 2Q^{\nu _0}_\lambda (f) \le \frac\lambda 2\| u_0^\dagger \| ^2_{L^2(\nu _0)}\).

Under the hypotheses of Proposition 756, with \(\nu ^*\) the hidden marginal of \(\rho ^*\), \(m^*\) its conditional mean amplitude and \(\kappa =\lambda /\beta \),

and \(\| m^*\| ^2_{L^2(\nu ^*)}+\frac{2\beta }\lambda \operatorname {KL}(\nu ^*\| \nu _0)+\frac2\lambda L(\rho ^*) \le Q^{\nu _0}_\lambda (f)\). (The chain rule \(\operatorname {KL}(\rho ^*\| \mu _U)=\operatorname {KL}(\nu ^*\| \nu _0) +\frac\kappa 2\| m^*\| ^2_{L^2(\nu ^*)}\) of Theorem 396, whose proof uses only the Gaussian conditional structure of Theorem 225 and not (R), substituted into Proposition 757.)

Let \((\nu _L)_L\) be hidden laws on \(Z\) whose kernel operators converge in operator norm, \(\| K_{\nu _L}-K_\infty \| \to 0\), to the kernel operator \(K_\infty =K_{\varphi ',\nu '}\) of another feature/hidden-law pair on the same \(L^2(P_X)\); let \((\lambda ,\beta )\) be fixed, \(f\in L^2(P_X)\), and \(\rho ^*_L\) the minimizer of the free energy with reference hidden law \(\nu _L\). Then

In the setting of Corollary 720, \(\nu _L=\tilde\nu _0^{(L)}\), \(K_\infty =K^{(\infty )}\) and \(Q^{(\infty )}_\lambda (f)\le \| f\| ^2_{{\mathcal{H}}_\infty }\) for \(f\in H^1(-1,1)\). (Proposition 757 for each \(L\) and Theorem 742.)

Under the hypotheses of Corollary 760, with \(\nu ^*_L\) the hidden marginal of \(\rho ^*_L\) and \(\kappa =\lambda /\beta \), \(\limsup _{L}\operatorname {KL}(\nu ^*_L\| \nu _L)\le \frac\kappa 2Q^{(\infty )}_\lambda (f)\) (Proposition 759).

Under the hypotheses of Corollary 760, with \(m^*_L\) the conditional mean amplitude of \(\rho ^*_L\), \(\limsup _{L}\| m^*_L\| ^2_{L^2(\nu ^*_L)}\le Q^{(\infty )}_\lambda (f)\) (Proposition 759); the right-hand sides of the three bounds do not depend on \(L\).

Assume (A1), (A3), (A4), (A5) and \(f\in L^2(P_X)\) (no (R)). With \(q:=Z_*/Z_0=\int e^{\kappa m^{*2}/2}\, \mathrm d\nu _0\),

??. (Theorem 226, the logarithm of the density integrated against \(\nu ^*\), and \(e^{\kappa m^{*2}/2}\ge 1\).)

Without (R), \(1\le q\le \exp \bigl(\frac\kappa 2\| m^*\| ^2_{L^2(\nu ^*)}\bigr) \le \exp \bigl(\frac\kappa 2Q^{\nu _0}_\lambda (f)\bigr)\): \(\log q=\frac\kappa 2\| m^*\| ^2_{L^2(\nu ^*)} -\operatorname {KL}(\nu ^*\| \nu _0)\le \frac\kappa 2\| m^*\| ^2_{L^2(\nu ^*)}\) (Lemma 763) and \(\| m^*\| ^2_{L^2(\nu ^*)}\le Q^{\nu _0}_\lambda (f)\) (Proposition 759).

Without (R), \(\| \nu ^*-\nu _0\| _{\mathcal M}=\| w-1\| _{L^1(\nu _0)} \le \sqrt{2\operatorname {KL}(\nu ^*\| \nu _0)}\) (Pinsker’s inequality, Lemma 850, in the convention \(\| \mu \| _{\mathcal M}=|\mu |(Z)=2\| \mu \| _{\mathrm{TV}}\)).