The Invariant Subspace Problem

Few questions in functional analysis have attracted sustained attention across as many decades as this one. It sits at the confluence of operator theory, spectral theory, and complex analysis, and every partial result has opened new territory rather than narrowing the problem to a routine case.

The problem is rated medium importance on the Open Problem Garden. It is old enough to have accumulated a rich history of partial results, yet still open in the Hilbert space setting after more than seventy years.

Trivial Observations and Why They Run Out #

Two subspaces are always invariant: ${0}$ and $\mathcal{H}$ itself. These are the trivial invariant subspaces; the problem asks whether anything else must exist.

On finite-dimensional spaces the answer is immediate: every operator on $\mathbb{C}^n$ has an eigenvector (by the fundamental theorem of algebra applied to the characteristic polynomial), and the span of any eigenvector is a one-dimensional invariant subspace. This argument fails completely in infinite dimensions, where the spectrum can be continuous and eigenvectors need not exist.

On non-separable Hilbert spaces the problem is also trivial but for a different reason: for any non-zero vector $x \in \mathcal{H}$, the closed linear span $\overline{\operatorname{span}{T^n x : n \geq 0}}$ is a closed invariant subspace, and if $\mathcal{H}$ is non-separable it cannot equal all of $\mathcal{H}$. So the problem is genuinely about separable spaces.

Landscape of Known Results #

Positive Results: Classes with Invariant Subspaces #

The compact case was already known to von Neumann in the 1930s for Hilbert spaces, but was never published; Aronszajn and Smith gave the first published proof, extended to Banach spaces. The key idea is that a compact operator can be approximated by finite-rank operators, each of which has invariant subspaces, and a limiting argument produces an invariant subspace for the compact operator.

Lomonosov’s proof is strikingly short, less than a page, and uses the Schauder fixed-point theorem in an unexpected way. It subsumes both the compact case (an operator commutes with itself) and the polynomially compact case (an operator commutes with $p(T)$, which is compact if $p(T)$ is). For several years it seemed that Lomonosov’s theorem might resolve the problem entirely, until Hadwin, Nordgren, Radjavi, and Rosenthal (1980) exhibited an operator that does not commute with any non-zero compact operator yet still has invariant subspaces.

An operator $T$ is subnormal if it is the restriction of a normal operator on a larger Hilbert space. Normal operators are handled by the spectral theorem, which produces a rich lattice of invariant subspaces; subnormal operators inherit invariant subspaces by restriction. Brown’s proof uses techniques from rational approximation theory (the solution of the Halmos problem on subnormal operators).

Beyond these landmark theorems, invariant subspaces are also known for: hyponormal operators with some additional conditions, operators whose spectrum has interior points, operators satisfying growth conditions on the resolvent, and polynomially bounded operators with spectrum containing the unit circle under further constraints (Liu, 2017; Réjasse, 2023).

Beurling’s Theorem: A Complete Classification #

Beurling’s theorem is a landmark because it gives not merely existence but a full classification of all invariant subspaces for a single operator. The shift on $H^2$ is in many senses the canonical operator for the Hilbert space invariant subspace problem: finding a counterexample to the full problem is equivalent to finding an operator with no invariant subspaces, and the shift shows how rich such structure can be even for a single operator.

Negative Results: Counterexamples on Banach Spaces #

Enflo’s counterexample was the first, constructed in 1975 though not published until 1987 due to its length and complexity. Read’s construction (1984) arrived independently and somewhat earlier in print; a further, more explicit example by Read (1985) lives on the classical space $\ell^1$. These results make clear that the answer to the invariant subspace problem is negative for general Banach spaces. The Hilbert space case remains the central open question precisely because no counterexample on any reflexive Banach space, much less a Hilbert space, has been found.

The Hilbert–Banach Gap #

The separation between Hilbert space and general Banach space behaviour is a recurring theme. Several features of Hilbert spaces that Banach spaces lack suggest why counterexamples might not exist in the Hilbert setting:

• The inner product gives every operator an adjoint $T^*$, and the lattice of invariant subspaces of $T$ and of $T^*$ are related by orthogonal complementation.
• The spectral theorem for normal operators provides a complete invariant subspace theory for that class, anchoring intuition.
• Reflexivity and the existence of unconditional bases in specific Hilbert spaces constrain operator behaviour more than in $\ell^1$.

None of these features has yet been converted into a proof for the general case.

Recent Proof Attempts #

The problem has attracted renewed attention in recent years.

In May 2023, Per Enflo, the same mathematician who produced the first Banach space counterexample, posted a preprint to arXiv (2305.15442) claiming a positive resolution for all separable Hilbert spaces. The original preprint was 13 pages; a substantially expanded version (52 KB) appeared in April 2024. Enflo himself has been cautious about the result, noting that expert review is ongoing. As of this writing the preprint has not received a definitive verdict from the community.

In July 2023 an independent preprint by Neville (arXiv:2307.08176) also claimed a positive solution for separable Hilbert spaces.

In September 2024 a peer-reviewed article in Axioms by Khalil, Yousef, Alshanti, and Abu Hammad announced a proof, but basic errors were identified shortly after publication (Ghatasheh, arXiv:2411.19409, November 2024).

The problem therefore remains officially open. The cluster of recent attempts reflects both its difficulty and its continued centrality in functional analysis.

Research Directions #

1. Cyclic Vectors and the Spectral Radius Formula #

A vector $x \in \mathcal{H}$ is cyclic for $T$ if $\mathcal{H} = \overline{\operatorname{span}{T^n x : n \geq 0}}$. An operator with a non-trivial invariant subspace cannot have every non-zero vector be cyclic. The contrapositive is: if every non-zero vector is cyclic, then $T$ is a counterexample.

Read’s Banach-space constructions proceed by building hypercyclic operators whose orbits are dense. On Hilbert spaces, Hilbert space geometry severely constrains the density of orbits. Making this constraint quantitative, via growth estimates on $|T^n x|$ or on the resolvent $|(T-\lambda)^{-1}|$, might close the gap between known positive results and the general case.

2. Dual Algebra Techniques #

A powerful modern approach studies the dual algebra $\mathcal{A} _T$, the weak-$*$ closure of the polynomials in $T$ as a subalgebra of $\mathcal{B}(\mathcal{H})$. If $\mathcal{A} _T = \mathcal{B}(\mathcal{H})$ (the operator is reflexive in this sense), one can sometimes extract invariant subspaces from the structure of the algebra. Results along these lines have been obtained for $C _{00}$ contractions (Bercovici, Foiaş, Pearcy) and for polynomially bounded operators under spectral conditions (Liu, 2017). The key open question is whether every Hilbert space contraction is reflexive in this sense, or whether the dual algebra approach can be made to work for all contractions via Sz.-Nagy–Foiaş theory.

3. Contractions and the Sz.-Nagy–Foiaş Calculus #

Every contraction ($|T| \leq 1$) on a Hilbert space admits a minimal unitary dilation (Sz.-Nagy’s dilation theorem), and Foiaş developed a functional calculus for contractions based on $H^\infty(\mathbb{D})$. The rich structure of this calculus has produced invariant subspace theorems for $C_{11}$ contractions and for contractions whose spectrum is rich enough. The question is whether the calculus can be pushed to all contractions; the general invariant subspace problem for contractions is equivalent to the full problem (by rescaling), so this is not a simplification but a different vantage point that has been productive.

4. Almost Invariant Half-Spaces #

A weaker notion, studied by Androulakis, Popov, Tcaciuc, and Troitsky, asks for almost invariant half-spaces: closed subspaces $\mathcal{M}$ of infinite dimension and infinite codimension such that $T\mathcal{M} \subseteq \mathcal{M} + \mathcal{F}$ for some finite-dimensional subspace $\mathcal{F}$. These exist for every operator on any infinite-dimensional Banach space. Whether every operator on a Hilbert space has a genuinely invariant (not just almost invariant) infinite-dimensional subspace of infinite codimension remains open and is a concrete intermediate target.

5. Hyperinvariant Subspaces #

A subspace is hyperinvariant for $T$ if it is invariant under every operator that commutes with $T$. Every hyperinvariant subspace is invariant, so existence of a hyperinvariant subspace implies a positive answer to the invariant subspace problem. Lomonosov’s 1973 theorem gives hyperinvariant subspaces when $T$ commutes with a compact operator. The hyperinvariant subspace problem, does every operator on a Hilbert space (other than scalar multiples of the identity) have a hyperinvariant subspace?, is also open and may be harder than the invariant subspace problem itself.

References #

1. Aronszajn, N. & Smith, K. T. (1954). Invariant subspaces of completely continuous operators. Annals of Mathematics, 60(2), 345–350.
2. Beurling, A. (1949). On two problems concerning linear transformations in Hilbert space. Acta Mathematica, 81, 239–255.
3. Brown, S. (1987). Hyponormal operators with thick spectra have invariant subspaces. Annals of Mathematics, 125(1), 93–103.
4. Enflo, P. H. (1987). On the invariant subspace problem for Banach spaces. Acta Mathematica, 158, 213–313.
5. Enflo, P. H. (2023). On the invariant subspace problem in Hilbert spaces. arXiv:2305.15442.
6. Lomonosov, V. I. (1973). Invariant subspaces of operators commuting with compact operators. Functional Analysis and Its Applications, 7(3), 213–214.
7. Read, C. J. (1984). A solution to the invariant subspace problem. Bulletin of the London Mathematical Society, 16(4), 337–401.
8. Read, C. J. (1985). A solution to the invariant subspace problem on the space $\ell^1$. Bulletin of the London Mathematical Society, 17(4), 305–317.
9. Radjavi, H. & Rosenthal, P. (2003). Invariant Subspaces (2nd ed.). Dover.
10. Bercovici, H., Foiaş, C., & Pearcy, C. (1985). Dual Algebras with Applications to Invariant Subspaces and Dilation Theory. AMS.