For \(\| x\| =\| x'\| =1\) and \(s^2=\sigma _w^2+\sigma _b^2{\gt}0\), the pair \(U=(w\cdot x-b)/s\), \(U'=(w\cdot x'-b)/s\) is, under \(\nu _0={\mathcal N}(0,\sigma _w^2I_2)\otimes {\mathcal N}(0,\sigma _b^2)\), a standard Gaussian pair with correlation \(\rho =(\sigma _w^2\, x\cdot x'+\sigma _b^2)/s^2\). (Both are centered Gaussian vectors with unit variances and covariance \(\rho \); compare the characteristic functions.)

For \(s^2=\sigma _w^2+\sigma _b^2{\gt}0\) and \(\theta ,\theta '\in S^1\), \(k_{\nu _0}(x(\theta ),x(\theta '))=\sum _{k\ge 0}a_k(s)^2\rho ^k=\kappa _s(\rho )\) with \(\rho =(\sigma _w^2\, x(\theta )\cdot x(\theta ')+\sigma _b^2)/s^2\): the pre-activations \(U=(w\cdot x(\theta )-b)/s\), \(U'=(w\cdot x(\theta ')-b)/s\) form a standard Gaussian pair with correlation \(\rho \) (Lemma 570) and \({\mathbb E}[\phi (U)\phi (U')]=\sum _ka_k(\phi )^2\rho ^k\) for \(\phi =\sigma (s\cdot )\in L^2(\gamma )\) (Lemma 919).

For \(f\in C^1({\mathbb R})\) with \(f\) and \(f'\) bounded and every \(n\ge 0\), \(\int f\, He_{n+1}\, \, \mathrm d\gamma =\int f'\, He_n\, \, \mathrm d\gamma \): since \(He_{n+1}=XHe_n-He_n'\) and \((He_n\gamma )'=(He_n'-XHe_n)\gamma \), this is the integration by parts \(\int f\, (XHe_n-He_n')\gamma =\int f'He_n\gamma \) (the boundary terms vanish by the Gaussian decay).

For \(b\ge 0\) and \(v=(1+2b)^{-1}\), \(\int He_n(y)e^{-by^2}\, \gamma (\, \mathrm dy)=\sqrt v\, (1-v)^{n/2}He_n(0)\): the density \(\gamma (y)e^{-by^2}\) is \(\sqrt v\) times the density of \({\mathcal N}(0,v)\), and \({\mathbb E}[He_n(\sqrt vG)]=(1-v)^{n/2}He_n(0)\) by the Gaussian smoothing identity.

\(\frac1{2\pi }\int _0^{2\pi }\cos ^n\phi \cos (\ell \phi )\, \mathrm d\phi =2^{-n}\binom n{(n-|\ell |)/2}\) for \(|\ell |\le n\), \(n\equiv \ell \pmod2\), and \(0\) otherwise (binomial expansion of \(\cos ^n\phi =2^{-n}(e^{i\phi }+e^{-i\phi })^n\) and orthogonality).

For \(s^2=\sigma _w^2+\sigma _b^2{\gt}0\), \(\mu _\ell =\sum _{k\ge 0}a_k(s)^2c_{k,\ell }\) with \(c_{k,\ell }:=\int \rho (\phi )^k\cos (\ell \phi )\, \tau (\, \mathrm d\phi )\), \(\rho (\phi )=(\sigma _w^2\cos \phi +\sigma _b^2)/s^2\) (dominated convergence in \(\mu _\ell =\int \kappa (\phi )\cos (\ell \phi )\, \tau (\, \mathrm d\phi )\) with \(\kappa (\phi )=\sum _ka_k(s)^2\rho (\phi )^k\), the terms being bounded by the summable \(a_k(s)^2\)).

If \(f\in L^2(\gamma )\) is not \(\gamma \)-a.e. equal to a polynomial, then \(a_k(f)\ne 0\) for infinitely many \(k\): a finite Hermite expansion is a polynomial, by the completeness of the Hermite polynomials.

For \(r\in L^2(P_X)\), \(P_X\)-a.e. in \(x\), \((K_\nu r)(x)=\int _{{\mathcal X}}k_\nu (x,x')r(x')\, P_X(\, \mathrm dx')\) with \(k_\nu (x,x')=\int \varphi _z(x)\varphi _z(x')\, \nu (\, \mathrm dz)\) (Fubini in Lemma 194; the integrand is bounded by \(|r(x')|\)).

The hidden law \(\nu _0={\mathcal N}(0,\sigma _w^2I_2)\otimes {\mathcal N}(0,\sigma _b^2)\) is invariant under the rotations \((w,b)\mapsto (Rw,b)\) of the weight, \(R\in SO(2)\).

\(k_{\nu _0}(x(\theta ),x(\theta '))=\kappa (\theta -\theta ')\) with the zonal function \(\kappa \) of Definition 563: rotating the weight \(w\) by the angle \(\theta '\) leaves \(\nu _0\) invariant and moves \(x(\theta ),x(\theta ')\) to \(x(\theta -\theta '),x(0)\).

\(K_{\nu _0}\) is the convolution operator \(g\mapsto \int \kappa (\theta -\theta ')g(\theta ')\, \tau (\, \mathrm d\theta ')\) on \(L^2(\tau )\), so \(\cos (\ell \theta )\) and \(\sin (\ell \theta )\) are eigenfunctions with eigenvalue \(\mu _\ell =\frac1{2\pi }\int _0^{2\pi }\tilde\kappa (\cos \phi )\cos (\ell \phi )\, \mathrm d\phi \), the \(\ell \)th Fourier (cosine, by evenness) coefficient of \(\kappa \): \(K_{\nu _0}=\sum _\ell \mu _\ell P_\ell \).

The real trigonometric system \((e_\ell )_{\ell \in \mathbb Z}\) of Definition 565 is orthonormal in \(L^2(\tau )\).

Let \((e_i)_{i\in I}\) be an orthonormal basis of \(L^2(P_X)\) with \(K_\nu e_i=\mu _ie_i\) (so \(\mu _i=\| S_\nu ^*e_i\| ^2\in [0,1]\)). Then \(f\in \operatorname {ran}S_\nu \iff \sum _i\mu _i^{-1}\langle f,e_i\rangle ^2{\lt}\infty \) and \(\langle f,e_i\rangle =0\) whenever \(\mu _i=0\). In that case \(u_0^\dagger =S_\nu ^\dagger f=\sum _i\mu _i^{-1}\langle f,e_i\rangle S_\nu ^*e_i\), the series converging in \(L^2(\nu )\) with mutually orthogonal terms, and \(\| u_0^\dagger \| ^2_{L^2(\nu )}=\sum _i\mu _i^{-1}\langle f,e_i\rangle ^2\). (Singular system: \(v_i:=\mu _i^{-1/2}S_\nu ^*e_i\) is orthonormal with \(S_\nu v_i=\mu _i^{1/2}e_i\); “\(\Rightarrow \)” is Bessel’s inequality for \(\mu _i^{-1/2}\langle S_\nu u,e_i\rangle =\langle u,v_i\rangle \), “\(\Leftarrow \)” is the termwise synthesis of \(u=\sum _i\mu _i^{-1/2}\langle f,e_i\rangle v_i\), which lies in \(\overline{\operatorname {ran}S_\nu ^*}=(\ker S_\nu )^\perp \).)

\(0\le \mu _\ell \le 1\) for every \(\ell \): \(\mu _\ell =\| S_{\nu _0}^*e_\ell \| ^2\) and \(\| S_{\nu _0}^*\| \le 1\).

Assume \(\mu _\ell {\gt}0\) for all \(\ell \) (by (ii) this holds when \(\sigma _b{\gt}0\) and \(\sigma \) is not a polynomial). Then \(f\in \operatorname {ran}S_{\nu _0}\iff \sum _{\ell \ge 0}\mu _\ell ^{-1}\| P_\ell f\| ^2_{L^2(\tau )}{\lt}\infty \), where \(\| P_\ell f\| ^2=\langle f,e_\ell \rangle ^2+\langle f,e_{-\ell }\rangle ^2\) in the real trigonometric basis \((e_\ell )_{\ell \in \mathbb Z}\) of Definition 565, i.e. \(\sum _{\ell \in \mathbb Z}\mu _\ell ^{-1}\langle f,e_\ell \rangle ^2{\lt}\infty \). (Lemma 584 for the eigenbasis of Theorem 580.)

Under (R), \(u_0^\dagger =S_{\nu _0}^\dagger f=\sum _{\ell \in \mathbb Z}\mu _\ell ^{-1} \langle f,e_\ell \rangle S^*e_\ell =\sum _{\ell \ge 0}\mu _\ell ^{-1}S^*P_\ell f\), with \((S^*e_\ell )(w,b)=\frac1{2\pi }\int _0^{2\pi }\sigma (w_1\cos \theta +w_2\sin \theta -b) e_\ell (\theta )\, \mathrm d\theta \), the series converging in \(L^2(\nu _0)\) with mutually orthogonal terms, and \(\| u_0^\dagger \| ^2_{L^2(\nu _0)} =\sum _{\ell \in \mathbb Z}\mu _\ell ^{-1}\langle f,e_\ell \rangle ^2 =\sum _{\ell \ge 0}\mu _\ell ^{-1}\| P_\ell f\| ^2\).

Under (R), Theorem ?? (\(\lambda _n\to 0\), \(\kappa _n\to 0\)) gives \(m_n^*\to u_0^\dagger \) in \(L^2(\nu _0)\) and \(\Pi \rho _n^*\to u_0^\dagger \nu _0\) in total variation: the learned conditional mean amplitude converges to the explicit weighted ridgelet transform \(u_0^\dagger =\sum _\ell \mu _\ell ^{-1}S^*P_\ell f\) of Theorem 587, the \(\tau \)-weighted ridgelet transform \(S^*\) with the activation as filter applied to the sharpened target \(K_{\nu _0}^{-1}f=\sum _\ell \mu _\ell ^{-1}P_\ell f\).

For all \(x,x'\in {\mathbb R}^m\), \(k_{\nu _0}(x,x')=\int \varphi _z(x)\varphi _z(x')\nu _0(\, \mathrm dz) =\sum _{k\ge 0}a_k(s_x)a_k(s_{x'})\rho _{xx'}^{\, k}\), the series converging absolutely; on the circle \(s_x\equiv s\) and \(\kappa (\phi )=\tilde\kappa (\cos \phi ) =\kappa _s\bigl((\sigma _w^2\cos \phi +\sigma _b^2)/s^2\bigr)\). (In Lean: the case \(\| x\| =\| x'\| =1\).)

\(\mu _\ell =\sum _{n\ge \ell ,\ n\equiv \ell \ (2)}c_n2^{-n}\binom n{(n-\ell )/2}\) with \(c_n:=\sum _{k\ge n}a_k(s)^2\binom kn(\sigma _w^2/s^2)^n(\sigma _b^2/s^2)^{k-n}\): expand \(\tilde\kappa (t)=\sum _nc_nt^n\) (nonnegative coefficients, Tonelli) and insert \(\frac1{2\pi }\int _0^{2\pi }\cos ^n\phi \cos (\ell \phi )\, \mathrm d\phi =2^{-n}\binom n{(n-\ell )/2}\).

If \(\sigma _b{\gt}0\) and \(\sigma \) is not (a.e. equal to) a polynomial, then \(\mu _\ell {\gt}0\) for all \(\ell \), \(K_{\nu _0}\) is injective and \(\overline{\operatorname {ran}S_{\nu _0}}=L^2(\tau )\). (A non-polynomial \(\sigma \) has \(a_k(s)\ne 0\) for infinitely many \(k\), so every \(c_n{\gt}0\) and \(\mu _\ell \ge c_\ell 2^{-\ell }{\gt}0\).)

Assume \(\mu _\ell {\gt}0\) for all \(\ell \). For \(a\in \{ 1/2,1\} \), \(u_0^\dagger \in \operatorname {ran}\bigl((S_{\nu _0}^*S_{\nu _0})^a\bigr)\iff \sum _{\ell \in \mathbb Z}\mu _\ell ^{-2a-1}\langle f,e_\ell \rangle ^2{\lt}\infty \) (\(=\sum _{\ell \ge 0}\mu _\ell ^{-2a-1}\| P_\ell f\| ^2\)), the source condition (SC\(_a\)) of Definition 139 for some source norm \(G\). (In the singular system of Lemma 584: \(a=1/2\) is \(f\in \operatorname {ran}K_{\nu _0}\), \(a=1\) is \(f\in K_{\nu _0}(\operatorname {ran}S_{\nu _0})\), both read off the coordinates \(\langle e_\ell ,K_{\nu _0}g\rangle =\mu _\ell \langle e_\ell ,g\rangle \).)

Assume \(\mu _\ell {\gt}0\) for all \(\ell \). For every \(a{\gt}0\), \(u_0^\dagger \in \operatorname {ran}\bigl((S_{\nu _0}^*S_{\nu _0})^a\bigr)\iff \sum _{\ell \in \mathbb Z}\mu _\ell ^{-2a-1}\langle f,e_\ell \rangle ^2{\lt}\infty \), where \(T_0^a\) is the spectral power of Definition 164. (In the singular system, \(T_0v_\ell =\mu _\ell v_\ell \) and \((T_0)^a\) maps \(v_\ell \mapsto \mu _\ell ^av_\ell \) on \((\ker S_{\nu _0})^\perp \) and vanishes on \(\ker S_{\nu _0}\).)

If \(\sigma _b=0\) and \(\sigma \) is odd, then \(\mu _\ell =0\) for even \(\ell \), and \(\operatorname {ran}S_{\nu _0}\) consists of odd functions. (\(\kappa (\phi +\pi )=-\kappa (\phi )\), so \(\mu _\ell =\int \kappa (\phi +\pi )\cos (\ell (\phi +\pi ))\, \tau (\, \mathrm d\phi )=-\mu _\ell \) for even \(\ell \).)

For \(\sigma =\operatorname {erf}\): \(a_{2j}(s)=0\) and \(a_{2j+1}(s)=\frac2{\sqrt\pi }\frac{(-1)^j(2j)!}{j!\sqrt{(2j+1)!}} \frac{s^{2j+1}}{(1+2s^2)^{j+1/2}}\) (\(j\ge 0\)); in particular \(a_1(s)=\frac2{\sqrt\pi }\frac s{\sqrt{1+2s^2}}\). (Gaussian integration by parts, \(\sqrt{k!}\, a_k(s)=s^k{\mathbb E}[\sigma ^{(k)}(sG)]\), and the generating function of the physicists’ Hermite polynomials.)

For \(\sigma =\operatorname {erf}\), \(\kappa _s(\rho )=\frac2\pi \arcsin \frac{2s^2\rho }{1+2s^2}\), hence on the circle \(\kappa (\phi )=\tilde\kappa (\cos \phi )=\frac2\pi \arcsin \frac{2(\sigma _w^2\cos \phi +\sigma _b^2)}{1+2s^2}\), \(s^2=\sigma _w^2+\sigma _b^2\) (the arcsine kernel of Williams 1998; the series of Lemma 595 is the Taylor series of the arcsine).

For \(\sigma =\operatorname {erf}\) and \(\sigma _b{\gt}0\) there are \(\eta {\gt}0\) and \(C{\lt}\infty \) with \(\mu _\ell \le Ce^{-\eta |\ell |}\): \(\phi \mapsto \tilde\kappa (\cos \phi )\) is \(2\pi \)-periodic and holomorphic on a strip \(|\Im \phi |{\lt}\eta \), since \(|2(\sigma _w^2t+\sigma _b^2)/(1+2s^2)|{\lt}1\) on \([-1,1]\), and its Fourier coefficients decay geometrically.

For \(\sigma =\operatorname {erf}\), \(\sigma _b{\gt}0\), (R) implies \(\sum _\ell e^{\eta |\ell |}\| P_\ell f\| ^2{\lt}\infty \) for some \(\eta {\gt}0\), so \(f\) is real analytic in \(\theta \) (holomorphic extension to a strip \(|\Im \theta |{\lt}\eta /2\)): \(\| P_\ell f\| ^2\le \| u_0^\dagger \| ^2\mu _\ell \le C\| u_0^\dagger \| ^2e^{-\eta |\ell |}\).

For \(\sigma =\tanh \) and \(\sigma _b{\gt}0\), \(\mu _\ell =O(|\ell |^{-p})\) for every \(p{\gt}0\): \(\sum _kk^{2p}a_k(s)^2=\| N^p\tanh (s\cdot )\| ^2_{L^2(\gamma )}{\lt}\infty \) for the Ornstein–Uhlenbeck number operator \(N\), so \(\phi \mapsto \tilde\kappa (\cos \phi )\) is \(C^\infty \) and its Fourier coefficients decay faster than any power.

For \(\sigma =\tanh \), \(\sigma _b{\gt}0\), (R) implies \(\sum _\ell |\ell |^{p}\| P_\ell f\| ^2{\lt}\infty \) for every \(p\), so \(\| P_\ell f\| =O(|\ell |^{-p})\) for every \(p\) and \(f\in C^\infty (S^1)\).