This post summarizes reproducing kernel Hilbert spaces and mainly follows the lecture slides by D. Sejdinovic, A. Gretton.

We will mainly be concerned about the field $\mathbb{R}$.

Elementary Spaces

The following graph from the slides illustrates the reproducing kernel Hilbert spaces (RHKS) pretty well.

We summarize some important spaces in the following table.

For the following content, we will restrict ourselves to the norm vector space, in which some nice properties can be established.

Linear Operators

A linear operator is a function $A:\mathcal{F}\rightarrow\mathcal{G}$ such that $$A(a_1 f_1+a_2f_2)=a_1 A(f_1)+a_2A(f_2),\qquad\forall\ a_1,a_2\in\mathbb{R},f_1,f_2\in\mathcal{F},$$ where $\mathcal{F}$ and $\mathcal{G}$ are two vector spaces over $\mathbb{R}$. When $\mathcal{G}=\mathbb{R}$, the operators are called functionals.

If $\mathcal{F}$ and $\mathcal{G}$ are two normed vector spaces over $\mathbb{R}$, then the operator norm of a linear operator $A$ is defined as $$||A||=\sup\limits_{f\in\mathcal{F},||f||_{\mathcal{F}}\leq 1}||Af||_{\mathcal{G}}.$$ In this case, a bounded linear operator is one with finite operator norm. bounded operator $\neq$ bounded function.

Continuity

If $\mathcal{F}$ and $\mathcal{G}$ are two normed vector spaces over $\mathbb{R}$, then $A:\mathcal{F}\rightarrow\mathcal{G}$ is continuous at $f_0\in\mathcal{F}$ if $\forall\ \epsilon>0$, $\exists\ \delta>0$ such that $||f-f_0||_{\mathcal{F}}<\delta\Rightarrow||Af-Af_0||_{\mathcal{G}}<\epsilon$. If $A$ is continous at any $f\in\mathcal{F}$, then $A$ is continuous on $\mathcal{F}$.

Let $(\mathcal{F},||\cdot||_{\mathcal{F}})$ and $(\mathcal{G},||\cdot||_{\mathcal{G}})$ be normed linear spaces. If $A$ is a linear operator, then the following three conditions are equivalent:

1. $A$ is a bounded operator.
2. $A$ is continuous on $\mathcal{F}$.
3. $A$ is continuous at one point of $\mathcal{F}$.

Duality

If $\mathcal{F}$ is a normed space, then the space $\mathcal{F}^{\ast}$ of continuous linear functionals $A:\mathcal{F} \rightarrow \mathbb{R}$ is called the topological dual space of $\mathcal{F}$.

(Riesz representation) In a Hilbert space $\mathcal{F}$, for every continous linear functional $A \in \mathcal{F}^{\ast}$, there exists a unique $g \in \mathcal{F}$, such that $Af=\langle f,g\rangle.$

Riesz representation gives an isomorphism $g\mapsto\langle \cdot, g\rangle_{\mathcal{F}}$ between $\mathcal{F}$ and $\mathcal{F}^{\ast}$ which preserves the inner product in the two spaces. This implies that dual space of a Hilbert space is also a Hilbert space.

Isometric isomorphism

Two Hilbert spaces $\mathcal{H}_1$ and $\mathcal{H}_2$ is isometrically isomorphic if there exists a linear bijective map $U:\mathcal{H}_1\rightarrow\mathcal{H}_2$ that preserves the inner product, i.e., $\langle h_1,h_2\rangle_{\mathcal{H}_1}=\langle Uh_1,Uh_2\rangle_{\mathcal{H}_1}$.

(Isometric isomorphism between Hilbert spaces and sequence spaces) Every Hilbert space has an orthonormal basis and isometrically isomorphic to $l^2(A)$ for some set $A$.

When the Hilbert space is separable, then $A$ can be taken as $\mathbb{N}$.

RKHS

Evaluation functional $\delta_x:f\mapsto f(x)$ can be discontinuous in a Hilbert space $\mathcal{H}$ of functions $\mathcal{X}\rightarrow\mathbb{R}$.

A Hilbert space $\mathcal{H}$ of functions $\mathcal{X}\rightarrow\mathbb{R}$ is said to be a reproducing kernel Hilbert space (RKHS) if $\delta_x$ is an continuous functional for all $x\in\mathcal{X}$.

Note that we have restrict the objects in the space to be functionals $\mathcal{X}\rightarrow\mathbb{R}$.

Norm convergence in a RKHS implies pointwise convergence. If $\lim_{n\rightarrow\infty}||f_n-f||_{\mathcal{H}}=0$, then $\lim_{n\rightarrow\infty}f_n(x)=f(x)$, for all $x\in\mathcal{X}$.

Reproducing kernel

Inner product between features

Positive definite function