Hostname: page-component-745bb68f8f-grxwn Total loading time: 0 Render date: 2025-02-06T17:45:55.339Z Has data issue: false hasContentIssue false

Crofton formulas in pseudo-Riemannian space forms

Published online by Cambridge University Press:  28 October 2022

Andreas Bernig
Affiliation:
Institut für Mathematik, Goethe-Universität Frankfurt, Robert-Mayer-Str. 10, 60629 Frankfurt, Germany bernig@math.uni-frankfurt.de
Dmitry Faifman
Affiliation:
School of Mathematical Sciences, Tel Aviv University, Tel Aviv 6997801, Israel faifmand@tauex.tau.ac.il
Gil Solanes
Affiliation:
Departament de Matemàtiques, Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain solanes@mat.uab.cat Centre de Recerca Matemàtica, Campus de Bellaterra, 08193 Bellaterra, Spain
Rights & Permissions [Opens in a new window]

Abstract

Crofton formulas on simply connected Riemannian space forms allow the volumes, or more generally the Lipschitz–Killing curvature integrals of a submanifold with corners, to be computed by integrating the Euler characteristic of its intersection with all geodesic submanifolds. We develop a framework of Crofton formulas with distributions replacing measures, which has in its core Alesker's Radon transform on valuations. We then apply this framework, and our recent Hadwiger-type classification, to compute explicit Crofton formulas for all isometry-invariant valuations on all pseudospheres, pseudo-Euclidean and pseudohyperbolic spaces. We find that, in essence, a single measure which depends analytically on the metric, gives rise to all those Crofton formulas through its distributional boundary values at parts of the boundary corresponding to the different indefinite signatures. In particular, the Crofton formulas we obtain are formally independent of signature.

Type
Research Article
Copyright
© 2022 The Author(s). The publishing rights in this article are licensed to Foundation Compositio Mathematica under an exclusive licence

1. Introduction

1.1 Crofton formulas

The classical Crofton formula computes the length of a rectifiable curve $\gamma$ in $\mathbb {R}^{2}$ as

(1)\begin{equation} \mathrm{Length}(\gamma)=\frac{\pi}{2} \int_{\overline{\operatorname{Gr}}_1(\mathbb{R}^{2})} \#(\gamma \cap \bar{L})\, d\bar{L}, \end{equation}

where $\overline {\operatorname {Gr}}_1(\mathbb {R}^{2})$ is the space of lines in $\mathbb {R}^{2}$ with a rigid motion invariant measure (which is normalized in a suitable way).

A higher-dimensional version states that for $M \subset \mathbb {R}^{n}$ a compact submanifold with boundary, we have

\[ \mu_k(M)=c_{n,k} \int_{\overline{\operatorname{Gr}}_{n-k}(\mathbb{R}^{n})} \chi(M \cap \bar{E})\,d\bar{E}, \]

where $\overline {\operatorname {Gr}}_{n-k}(\mathbb {R}^{n})$ is the Grassmann manifold of affine $(n-k)$-planes equipped with a rigid motion invariant measure, $\chi$ is the Euler characteristic, and $\mu _k(M)$ is the $k$th intrinsic volume of $M$, which can be defined via Weyl's tube formula [Reference WeylWey39]. The same formula also holds with the submanifold $M$ replaced by a compact convex body $K$, in which case the $k$th intrinsic volume can be defined via Steiner's tube formula [Reference SteinerSte82]. We refer to [Reference Klain and RotaKR97, Reference SchneiderSch14] for more information about intrinsic volumes of convex bodies.

More generally, we can take an arbitrary translation-invariant measure $m$ on $\overline {\operatorname {Gr}_{n-k}}(\mathbb {R}^{n})$ and consider the integral

\[ \mu(K):=\int_{\overline{\operatorname{Gr}_{n-k}}(\mathbb{R}^{n})} \chi(K \cap \bar{E}) \,dm(\bar{E}). \]

By the additivity of the Euler characteristic, we have

\[ \mu(K \cup L)+\mu(K \cap L)=\mu(K)+\mu(L) \]

whenever $K,L, K \cup L$ are compact convex bodies, hence $\mu$ is a valuation. Clearly $\mu$ belongs to the space $\operatorname {Val}$ of translation-invariant valuations which are continuous with respect to the Hausdorff metric. In addition, $\mu$ is $k$-homogeneous and even, that is, invariant under $-\operatorname {Id}$. We thus get a map

\[ \operatorname{Cr}:\mathcal{M}(\overline{\operatorname{Gr}_{n-k}}(\mathbb{R}^{n}))^{tr} \to \operatorname{Val}_k^{+}, \]

where $\mathcal {M}^{tr}$ denotes the space of translation-invariant measures.

Alesker [Reference AleskerAle01] has shown that the image of this map is dense with respect to the natural Banach space topology on $\operatorname {Val}_k^{+}$. Therefore, Crofton formulas are a central tool in the study of valuations and in integral geometry.

When restricted to smooth measures and valuations (see § 2 for the notion of smoothness of valuations), the map $\operatorname {Cr}$ is, in fact, a surjection

\[ \operatorname{Cr}:\mathcal{M}^{\infty}(\overline{\operatorname{Gr}_{n-k}}(\mathbb{R}^{n}))^{tr} \twoheadrightarrow \operatorname{Val}_k^{\infty,+}, \]

the kernel of which coincides with the kernel of the cosine transform [Reference Alesker and BernsteinAB04].

Among the many applications in integral geometry of such formulas, we mention the construction of a basis of the space of unitarily invariant valuations on $\mathbb {C}^{n}$ by Alesker [Reference AleskerAle03], the interpretation of Alesker's product of smooth and even valuations in terms of Crofton measures [Reference BernigBer07], the Holmes–Thompson intrinsic volumes of projective metrics [Reference Álvarez Paiva and FernandesÁPF07]. Applications outside integral geometry include isoperimetric inequalities in Riemannian geometry [Reference CrokeCro84], symplectic geometry [Reference OhOh90], systolic geometry [Reference TreibergsTre85], algebraic geometry [Reference Akhiezer and KazarnovskiiAK18], and more. Crofton formulas are employed also outside of pure mathematics, in domains such as microscopy and stereology, see [Reference Kiderlen and JensenKJ17].

Crofton formulas do not only exist on flat spaces, but also on manifolds. In this case, we need a family of sufficiently nice subsets, endowed with a measure. Then the Crofton integral is given by the integral of the Euler characteristic of the intersection with respect to the measure. Under certain conditions which are given in [Reference FuFu16], it yields a smooth valuation on the manifold in the sense of Alesker [Reference AleskerAle06b].

On spheres and hyperbolic spaces, a natural class of subsets are the totally geodesic submanifolds of a fixed dimension, endowed with the invariant measure. On the two-dimensional unit sphere, we have a formula similar to (1), with affine lines replaced by equators. This formula is the main ingredient in the proof of the Fáry–Milnor theorem that the total curvature of a knot in $\mathbb {R}^{3}$ is bigger than $4\pi$ if the knot is non-trivial.

In higher dimensions, the formula becomes slightly more complicated. On the $n$-dimensional unit sphere, we have

(2)\begin{equation} \int_{\operatorname{Geod}_{n-k}(S^{n})} \chi(M \cap E)\,dE=\sum_j \frac{1}{\pi \omega_{k+2j-1}} \binom{-{k}/{2}}{j} \mu_{k+2j}(M). \end{equation}

Here $\operatorname {Geod}_{n-k}(S^{n})$ denotes the totally geodesic submanifolds of dimension $(n-k)$, $\mu _j(M)$ is the $j$th intrinsic volume of $M$ (which can be defined as the restriction of the $j$th intrinsic volume on $\mathbb {R}^{n+1}$ under the isometric embedding $S^{n} \hookrightarrow \mathbb {R}^{n+1}$), and $\omega _n$ denotes the volume of the $n$-dimensional unit ball. A similar formula holds on hyperbolic space. See [Reference Fu and WannererFW19, Reference FuFu14] for more on the integral geometry of real space forms.

Moving on to Lorentzian signature, few results are available. The main challenge to overcome is the non-compactness of the isotropy group, which in general renders the Crofton integral divergent. Some special Crofton-type formulas in Lorentzian spaces of constant curvature, applicable under certain rather restrictive geometric conditions, appeared in [Reference BirmanBir84, Reference Langevin, Chaves and BianconiLCB03, Reference Solanes and TeufelST05, Reference Ye, Ma and WangYMW16].

1.2 Results

We are going to prove Crofton formulas on flat spaces, spheres, and hyperbolic spaces of arbitrary signatures. Let us recall the definition of these manifolds, referring to [Reference O'NeillO'Ne83, Reference WolfWol61] for more information.

Definition 1.1

  1. (i) The pseudo-Euclidean space of signature $(p,q)$ is $\mathbb {R}^{p,q}=\mathbb {R}^{p+q}$ with the quadratic form $Q=\sum _{i=1}^{p} \,dx_i^{2}-\sum _{i=p+1}^{p+q} \,dx_i^{2}$.

  2. (ii) The pseudosphere of signature $(p,q)$ and radius $r>0$ is

    \[ S_r^{p,q}=\{v \in \mathbb{R}^{p+1,q}:Q(v)=r^{2}\}, \]
    equipped with the induced pseudo-Riemannian metric. Its sectional curvature equals $\sigma ={1}/{r^{2}}$. The pseudosphere $S_1^{n,1}\subset \mathbb {R}^{n+1,1}$ is called the de Sitter space.
  3. (iii) The pseudohyperbolic space of signature $(p,q)$ and radius $r>0$ is

    \[ H_r^{p,q}=\{v \in \mathbb{R}^{p,q+1}:Q(v)=-r^{2}\}. \]
    Its sectional curvature equals $\sigma =-({1}/{r^{2}})$. The pseudohyperbolic space $H_1^{n,1}$ is called the anti-de Sitter space.

We colloquially call these spaces generalized pseudospheres. The isometry groups of generalized pseudospheres are given by

\begin{align*} \operatorname{Isom}(\mathbb{R}^{p,q}) & \cong \overline{\operatorname{O}}(p,q)=\operatorname{O}(p,q) \ltimes \mathbb{R}^{p,q},\\ \operatorname{Isom}(S_r^{p,q}) & \cong \operatorname{O}(p+1,q),\\ \operatorname{Isom}(H_r^{p,q}) & \cong \operatorname{O}(p,q+1). \end{align*}

In each case the action is transitive, and the isotropy group is conjugate to $\operatorname {O}(p,q)$. These spaces are isotropic in the sense that the isotropy group acts transitively on the level sets of the metric in the tangent bundle.

Definition 1.2 A complete connected pseudo-Riemannian manifold of constant sectional curvature is called a space form.

Up to taking connected components and universal coverings, any space form is a generalized pseudosphere (cf. [Reference O'NeillO'Ne83, Chapter 8, Corollary 24]).

On a generalized pseudosphere $M$, we formulate Crofton formulas using the space $\operatorname {Geod}_{n-k}(M)$ of totally geodesic subspaces. However, there is no isometry-invariant Radon measure on this space. Therefore, we use an isometry-invariant generalized measure (also called distribution). This causes some technical problems, as the function that we want to integrate is not smooth. Nevertheless, in many cases the integral can still be evaluated. The result is not a valuation anymore, but a generalized valuation in the sense of Alesker [Reference AleskerAle07]. The Crofton map is then a map

\begin{align*} \operatorname{Cr}:\mathcal{M}^{-\infty}(\operatorname{Geod}_{n-k}(M)) \to \mathcal{V}^{-\infty}(M). \end{align*}

In the Riemannian case, any isometry invariant valuation admits an invariant Crofton measure. The corresponding statement in other signatures is also true, but much harder to prove. The second named author proved in [Reference FaifmanFai17] the statement first for certain signatures by an explicit, but difficult, computation and then used the behavior of Crofton formulas under restrictions and projections to handle the general case.

Furthermore, with the exception of Riemannian and Lorentzian signatures, the space of isometry-invariant generalized measures is of greater dimension than the space of isometry-invariant valuations. Thus, we are forced to choose a distribution, and must take care to avoid the kernel of the Crofton map.

Using results by Muro [Reference MuroMur99] on analytic families of homogeneous generalized functions on the space of symmetric matrices, one can construct such an invariant generalized measure on $\operatorname {Geod}_{n-k}(M)$. We construct a particular generalized measure $m_k$ with the distinguishing property that it behaves well under restrictions of Crofton measures (see § 3.3), and is independent (in a precise sense) of signature and dimension.

There is some freedom in the normalization of a Crofton measure. We choose the normalization in such a way that the first coefficient in the Crofton formulas will always be $1$.

Our Crofton formula will evaluate

\[ \operatorname{Cr}_k^{M}:=\operatorname{Cr}_M(\widehat{m}_k), \]

where $\sigma \neq 0$ is the curvature of $M$ and $\widehat {m}_k:=\pi \omega _{k-1}\sqrt {\sigma ^{-1}}^{k} m_k$. The flat case $\sigma =0$ appears through a careful limiting procedure.

The right-hand side of the Crofton formula is expressed in terms of the recently introduced intrinsic volumes on pseudo-Riemannian manifolds [Reference Bernig, Faifman and SolanesBFS22], which are complex-valued generalized valuations on $M$. They satisfy a Hadwiger-type classification [Reference Bernig, Faifman and SolanesBFS21], which allows us to use the template method to compute the coefficients in this formula. However, the resulting computations lead to distributional integrals on the space of symmetric matrices, that can be evaluated directly essentially only for the Lorentzian signature. To conclude the general case, we use techniques of meromorphic continuation and distributional boundary values of analytic functions.

Due to the functorial properties of the constructed Crofton distribution mirroring those of the intrinsic volumes, namely their adherence to the Weyl principle, the resulting Crofton formulas are signature independent. Remarkably, they are also holomorphic, i.e. they involve the intrinsic volumes only and not their complex conjugates.

Maintheorem Let $M$ be a generalized pseudosphere of sectional curvature $\sigma$. Then

\[ \operatorname{Cr}_k^{M}= \sum_{j \geq 0} \frac{\omega_{k-1}}{\omega_{k+2j-1}}{-{k}/{2} \choose j}\sigma^{j} \mu_{k+2j}. \]

The Crofton formulas should be understood formally, namely as the correspondence between distributions on the Grassmannian and the intrinsic volumes through an abstractly defined Crofton map. However, they can also be interpreted as explicit Crofton-type formulas applicable to sufficiently nice subsets of the generalized pseudospheres.

By a strictly convex subset of a non-flat generalized pseudosphere $M^{m}$ we mean its intersection with a strictly convex cone in $\mathbb {R}^{m+1}$, with $M\subset \mathbb {R}^{m+1}$ embedded as in Definition 1.1. For the Riemannian round sphere and hyperbolic space, this coincides with the standard definition of strict convexity.

Corollary 1.3 Let $A \subset M$ be a smooth and strictly convex domain in $M$. Then the generalized measure $\widehat {m}_k$ can be applied to the function $E \mapsto \chi (A \cap E)$, $E \in \operatorname {Geod}_{n-k}(M)$, and

\[ \int_{\operatorname{Geod}_{n-k}(M)} \chi(A \cap E) \,d\widehat{m}_k(E)=\sum_{j \geq 0} \frac{\omega_{k-1}} {\omega_{k+2j-1}} {-{k}/{2} \choose j}\sigma^{j} \mu_{k+2j}(A). \]

Note that the spherical Crofton formula (2) is a special case of our theorem. We also note that we prove a slightly more general statement in Corollary 5.14 using the notion of LC-regularity from [Reference Bernig, Faifman and SolanesBFS22].

1.3 Plan of the paper

After covering the preliminaries, we turn in § 3 to study general Crofton formulas with a distributional Crofton measure, utilizing the Alesker–Radon transform on valuations. In particular, we study under which conditions such formulas can be applied directly to a given subset.

In § 4 we consider LC-regular domains and hypersurfaces of space forms, and deduce that they would be in a good position for the evaluation of intrinsic volumes through Crofton integrals, once the corresponding distributions are constructed. The latter construction is carried out in § 5. Moreover, these distributions are embedded in a meromorphic family of measures on a complex domain as a distributional boundary value, and some delicate, though central to our analysis, convergence questions are investigated and settled. Finally, in § 6, the Hadwiger-type description of intrinsic volumes combined with the template method are applied to yield the explicit Crofton formulas in all cases.

2. Preliminaries

2.1 Notation

By

\[ \omega_n=\frac{\pi^{{n}/{2}}}{\Gamma({n}/{2}+1)} \]

we denote the volume of the $n$-dimensional unit ball. The space of smooth complex valued $k$-forms on a manifold is denoted by $\Omega ^{k}(M)$. The space of smooth complex valued measures on $M$ is $\mathcal {M}^{\infty }(M)$. The space of generalized measures, also called distributions, is denoted by

\[ \mathcal{M}^{-\infty}(M):=(C^{\infty}_c(M))^{*}, \]

where here and in the following the subscript $c$ denotes compactly supported objects. Similarly, for $m=\dim M$, we denote the space of $k$-dimensional currents on $M$ by

\[ \Omega_{-\infty}^{m-k}(M):=(\Omega^{k}_c(M))^{*}. \]

The elements of this space can also be thought of as generalized $(m-k)$-forms.

For an oriented $k$-dimensional submanifold $X \subset M$, we let $[\kern-1pt[ X]\kern-1pt]$ be the $k$-current which is integration over $X$.

By $\mathbb {P}_M:=\mathbb {P}_+(T^{*}M)$ we denote the cosphere bundle of $M$, which consists of all pairs $(p,[\xi ])$, $p \in M$, $\xi \in T_p^{*}M \setminus 0$, where $[\xi ]=[\xi ']$ if there is some $\lambda >0$ with $\xi =\lambda \xi '$. When no confusion can arise, we use the same notation for subsets of $\mathbb {P}_M$ and their lifts to $T^{*}M$. The natural involution on $\mathbb {P}_M$ is the fiberwise antipodal map $s(p,[\xi ]):=(p,[-\xi ])$.

The wave front set of a generalized form $\omega \in \Omega _{-\infty }(M)$ is a closed subset of $\mathbb {P}_M$, denoted by $\operatorname {WF}(\omega )$, and we refer the reader to [Reference HörmanderHör03] or [Reference DuistermaatDui11] for details.

For a generalized pseudo-sphere $M$, we denote by $\operatorname {Geod}_k(M)$ the space of totally geodesic $k$-dimensional submanifolds of $M$.

2.2 Smooth valuations

Let $M$ be a smooth manifold of dimension $m$, which we assume oriented for simplicity.

Let $\mathcal {P}(M)$ be the set of compact differentiable polyhedra on $M$. To $A\in \mathcal {P}(M)$ we associate two subsets of $\mathbb {P}_M$. The conormal cycle, denoted $\operatorname {nc}(A)$, is the union of all conormal cones to $A$. It is an oriented closed Lipschitz submanifold of dimension $(m-1)$, and naturally stratified by locally closed smooth submanifolds corresponding to the strata of $A$.

The conormal bundle, denoted $N^{*}A$, is the union of the conormal bundles to all smooth strata of $A$. It holds that $\operatorname {nc}(A) \subset N^{*}A$. By definition, two stratified spaces intersect transversally if all pairs of smooth strata are transversal.

A smooth valuation is a functional $\mu :\mathcal {P}(M)\to \mathbb {R}$ of the form

\[ \mu(A)=\int_A \phi+\int_{\operatorname{nc}(A)} \omega, \quad \phi \in \Omega^{m}(M), \omega \in \Omega^{m-1}(\mathbb{P}_M). \]

We write $\mu =[[\phi,\omega ]]$ in this case.

The space of smooth valuations is denoted by $\mathcal {V}^{\infty }(M)$. It admits a natural filtration

\[ \mathcal{V}^{\infty}(M)=\mathcal{W}_0^{\infty}(M) \supset \mathcal{W}_1^{\infty}(M) \supset \cdots \supset \mathcal{W}_m^{\infty}(M) =\mathcal{M}^{\infty}(M). \]

It is compatible with the Alesker product of valuations.

2.3 Generalized valuations

The space of generalized valuations is

\[ \mathcal{V}^{-\infty}(M):=(\mathcal{V}^{\infty}_c(M))^{*}. \]

By Alesker–Poincaré duality we have a natural embedding $\mathcal {V}^{\infty }(M) \hookrightarrow \mathcal {V}^{-\infty }(M)$.

There is a natural filtration

\[ \mathcal{V}^{-\infty}(M)=\mathcal{W}_0^{-\infty}(M) \supset \mathcal{W}_1^{-\infty}(M) \supset \cdots \supset \mathcal{W}_m^{-\infty}(M) =\mathcal{M}^{-\infty}(M). \]

In particular, we may consider a generalized measure as a generalized valuation.

A compact differentiable polyhedron $A$ defines a generalized valuation $\chi _A$ by

\[ \langle \chi_A,\mu\rangle=\mu(A), \quad \mu \in \mathcal{V}_c^{\infty}(M). \]

A generalized valuation $\psi$ can be represented by two generalized forms $\zeta \in C^{-\infty }(M)$, $\tau \in \Omega _{-\infty }^{m}(\mathbb {P}_M)$ such that

\[ \langle \psi,[[\phi,\omega]]\rangle=\langle \zeta,\phi\rangle+\langle \tau,\omega\rangle. \]

We refer to $\zeta$ and $\tau$ as the defining currents. For instance, the defining currents of $\chi _A$, $A \in \mathcal {P}(M)$, are $\zeta =\mathbb {1}_A, \tau =[\kern-1pt[ \operatorname {nc}(A) ]\kern-1pt]$. The wave front set of $\psi$ is defined as the pair $\Lambda \subset \mathbb {P}_M$, $\Gamma \subset \mathbb {P}_{\mathbb {P}_M}$ of the wave front sets of $\zeta$ and $\tau$. The space of all generalized valuations with wave front sets contained in $\Lambda,\Gamma$ is denoted by $\mathcal {V}^{-\infty }_{\Lambda,\Gamma }(M)$.

Consider $\psi \in \mathcal {V}^{-\infty }(M)$. We say that $A \in \mathcal {P}(M)$ is WF-transversal to $\psi$, denoted $A \pitchfork \psi$, if the conditions of [Reference Alesker and BernigAB12, Theorem 8.3] hold for

\begin{align*} (\Lambda_1, \Gamma_1) & :=\operatorname{WF}(\chi_A),\\ (\Lambda_2,\Gamma_2) & :=\operatorname{WF}(\psi). \end{align*}

These conditions imply that Alesker's product of smooth valuations can be extended to a jointly sequentially continuous product

\[ \mathcal{V}_{\Lambda_1,\Gamma_1}^{-\infty}(M) \times \mathcal{V}_{\Lambda_2,\Gamma_2}^{-\infty}(M) \to \mathcal{V}^{-\infty}(M), \]

and in particular the pairing $\psi (A):=\langle \psi, \chi _A\rangle =\int _M \psi \cdot \chi _A$ is well-defined.

Let us write a sufficient set of conditions in a particular case.

Proposition 2.1 Assume $\operatorname {WF}(\psi ) \subset (N^{*}D, N^{*}L)$ for some submanifolds with corners $D\subset M$, $L\subset \mathbb {P}_M$, and take $A \in \mathcal {P}(M)$. Assume further:

  1. (i) $A\pitchfork D$;

  2. (ii) $\operatorname {nc}(A)\pitchfork \pi ^{-1}D$, where $\pi :\mathbb {P}_M\to M$ is the natural projection;

  3. (iii) $\pi ^{-1}A\pitchfork L$;

  4. (iv) $\operatorname {nc}(A) \pitchfork s(L)$, where $s:\mathbb {P}_M\to \mathbb {P}_M$ is the antipodal map.

Then $A\pitchfork \psi$.

Proof. These conditions imply the conditions in [Reference Alesker and BernigAB12, Theorem 8.3].

2.4 Intrinsic volumes on pseudo-Riemannian manifolds

In [Reference Bernig, Faifman and SolanesBFS22] we constructed a sequence of complex-valued generalized valuations $\mu _0^{M},\ldots,\mu _m^{M}$ naturally associated to a pseudo-Riemannian manifold $M$ of dimension $m$ and extending the Lipschitz–Killing curvatures therein. They are invariant under isometries and called the intrinsic volumes of $M$. The intrinsic volume $\mu _0$ equals the Euler characteristic, whereas $\mu _m$ is the volume measure of $M$, multiplied by $\mathbf {i}^{q}$ where $q$ is the negative index of the signature. For other values of $k$, $\mu _k$ is typically neither real nor purely imaginary.

The wave front set of $\mu _k$ is contained in $(\emptyset,N^{*}(\operatorname {LC}^{*}_M))$, where $\operatorname {LC}^{*}_M \subset \mathbb {P}_M$ is the dual light cone of the metric, i.e. the set of all pairs $(p,[\xi ]) \in \mathbb {P}_M$ such that $g|_p(\xi,\xi )=0$. Here we use the metric to identify $TM$ and $T^{*}M$.

A subset $A\in \mathcal {P}(M)$ is LC-transversal if $\operatorname {nc}(A)\pitchfork \operatorname {LC}^{*}_M$.

Lemma 2.2 Assume $D=\emptyset$, $L=\operatorname {LC}^{*}_M \subset \mathbb {P}_M$, and $A \in \mathcal {P}(M)$. Then the conditions of Proposition 2.1 are equivalent to the LC-transversality of $A$. In particular, the intrinsic volume $\mu _k$ may be evaluated at LC-transversal $A$.

Proof. The first two conditions are empty. The third condition is satisfied for arbitrary $A$, because the tangent space to $\pi ^{-1}A$ contains all vertical directions, whereas the tangent space of $\operatorname {LC}^{*}_M$ contains all horizontal directions. The fourth condition is precisely LC-transversality.

We also need the notion of LC-regularity, which was introduced in [Reference Bernig, Faifman and SolanesBFS22].

Definition 2.3 Let $X$ be a smooth manifold equipped with a smooth field $g$ of quadratic forms over $TX$. We say that $(X,g)$ is LC-regular if $0$ is a regular value of $g \in C^{\infty }(TX \setminus \underline {0})$.

It was shown in [Reference Bernig, Faifman and SolanesBFS22, Proposition 4.9] that the extrinsic notion of LC-transversality and the intrinsic notion of LC-regularity coincide: a submanifold of a pseudo-Riemannian manifold, equipped with the field of quadratic forms induced from the metric, is LC-regular if and only if it is LC-transversal.

The most important property of the intrinsic volumes is that they satisfy a Weyl principle: for any isometric immersion $M \looparrowright \widetilde M$ of pseudo-Riemannian manifolds we have

\[ \mu_k^{\widetilde M}|_M=\mu_k^{M}, \]

in particular, the restriction on the left-hand side is well-defined. Conversely, we have shown in [Reference Bernig, Faifman and SolanesBFS21] that any family of valuations associated to pseudo-Riemannian manifolds that satisfies the Weyl principle must be a linear combination of intrinsic volumes.

3. Distributional Crofton formulas

Let $M^{m}$ be a manifold. A Crofton formula for a smooth valuation $\phi \in \mathcal {V}^{\infty }(M)$ has the form $\phi (A)=\int _{S}\chi (X(s)\cap A)\,d\mu (s)$, where $S$ is a smooth manifold parametrizing a smooth family of submanifolds of $M$, and $\mu$ a smooth measure on $S$. Similarly, a distributional Crofton formula has $\phi \in \mathcal {V}^{-\infty }(M)$, and $\mu$ is a distribution.

In this section we study some general properties of such formulas, when $M\subset V\setminus \{0\}$ is a submanifold without boundary in a $d$-dimensional linear space $V$, and $S=\operatorname {Gr}_{d-k}(V)$, $k < d$, $X(s)=s \cap M$, $s \in S$.

We utilize the Radon transform on valuations, introduced in [Reference AleskerAle10]. Loosely speaking, the Crofton map is but the Radon transform of a measure with respect to the Euler characteristic. However, there are technical difficulties in applying this formalism directly to distributions, and a large part of this section is concerned with resolving those difficulties. The main results to this end are Propositions 3.7 and 3.12. In the last part, we describe the Crofton wave front of sufficiently nice sets in Proposition 3.17, which controls the applicability of an explicit Crofton integral to the given set.

3.1 The general setting

For a submanifold with corners $X\subset M$, define $Z_X\subset X\times \operatorname {Gr}_{d-k}(V)$ by $Z_X=\{(x,E): x\in X\cap E\}$. Then $Z_X$ is a manifold with corners, more precisely it is the total space of the fiber bundle over $X$ with fiber $\operatorname {Gr}_{d-k-1}(V/\mathbb {R} x)$ at $x\in X$. Write

for the natural projections.

Denote by $W_X\subset \operatorname {Gr}_{d-k}(V)$ the set of subspaces intersecting $X$ transversally in $V$.

We need a simple fact from linear algebra, which we state in a rather general form that is useful for us in several places.

Lemma 3.1 Let $V$ be a vector space, $L_0 \in \operatorname {Gr}_l(V)$, $E_0 \in \operatorname {Gr}_k(V)$ with $L_0 \subset E_0$. Denote by $i:L_0\hookrightarrow E_0$ the inclusion, and $\pi : V/L_0\to V/E_0$ the projection.

  1. (i) Let $E(t)\in \operatorname {Gr}_k(V)$ be a smooth path with $E(0)=E_0$ and $A:L_0\to V/L_0$ a linear map. Then there is a smooth path $L(t) \in \operatorname {Gr}_l(E(t))$ with $L(0)=L_0$ and $L'(0)=A$ if and only if the following diagram commutes.

  2. (ii) Let $L(t) \in \operatorname {Gr}_l(V)$ be a smooth path with $L(0)=L_0$. Let $B:E_0\to V/E_0$ be a linear map. Then there is a smooth path $E(t) \in \operatorname {Gr}_k(V)$ with $L(t) \subset E(t)$, $E(0)=E_0$ and $E'(0)=B$ if and only if the following diagram commutes.

Remark 3.2 The ‘only if’ statement obviously remains true if instead of $L(t)\subset E(t)$, we have $\measuredangle (L(t), E(t))=o(t)$ with respect to any Euclidean structure.

Proof. Consider the partial flag manifold $Z=\{ L\subset E\}\subset \operatorname {Gr}_l(V)\times \operatorname {Gr}_k(V)$. The group $\operatorname {GL}(V)$ acts transitively on $Z$, and any smooth path $F(t)=(L(t) \subset E(t))\in Z$ can be lifted to a smooth curve $g(t)\in \operatorname {GL}(V)$ with $g(0)=\operatorname {Id}$ and $F(t)=g(t)F(0)$. Thus, $E'(0):E_0\to V/E_0$ and $L'(0):L_0\to V/L_0$ are both projections of $g'(0):V\to V$, and the diagram commutes.

In the other direction, write $\pi _W:V\to V/W$ for the natural projection. It follows by the above that the set of velocity vectors $L'(0)$ for all curves $L(t)\subset E(t)$ is the affine space $\{\pi _{L_0}\circ T|_{L_0}\in \mathrm {Hom}(L_0, V/L_0): T\in \mathfrak {gl}(V), \pi _{E_0}\circ T|_{E_0}=E'(0)\}$, which is of dimension ${\binom {k}{2}-\binom {l}{2}-\binom { k-l}{2}=l(k-l)}$. This is also the dimension of the affine space of all $A$ such that the diagram commutes, which finishes the proof of the first part. The second part follows from the first one by taking orthogonal complements.

We need the following technical statement appearing in [Reference AleskerAle10, Proposition 5.1.3].

Lemma 3.3 The natural projection $\pi :N^{*}Z_M\setminus 0\to T^{*}M\setminus 0$ is a submersion.

Proof. Let $(p_t,\xi _t)$ be a smooth path in $T^{*}M\setminus 0$. We lift it to a smooth path $(p_t,E_t,\xi _t,\eta _t)\in T^{*}(M\times \operatorname {Gr}_{d-k}(V))$ such that $p_t \in E_t$, and $(\xi _t,\eta _t)\in N^{*}_{p_t,E_t}Z_M$. Now for $v\in T_pM$, $B \in T_E \operatorname {Gr}_{d-k}(V)= \mathrm {Hom}(E,V/E)$, we have by Lemma 3.1 (applied with $l=1, L_0=\mathbb {R} p$) that $(v,B)\in T_{p,E}Z_M$ if and only if $v+E=B(p)$.

Hence,

\begin{align*} N^{*}_{p,E} Z_M=\{(\xi,\eta) \in T_p^{*}M \times T_E^{*}\operatorname{Gr}_{d-k}(V): \langle \xi,v\rangle+\langle \eta,B\rangle=0\text{ whenever } v+E=B(p)\}. \end{align*}

Fix a Euclidean structure on $V$, inducing Euclidean structures on the spaces $\mathrm {Hom}(E_t, V/E_t)$. Let us choose some $E_t$ such that $p_t\in E_t$, and $T_{p_t}M\cap E_t\subset \operatorname {Ker}(\xi _t)$, which evidently can be done. Consider the linear subspace

\[ W_t=\{ B \in T_{E_t}\operatorname{Gr}_{d-k}(V): B(p_t)\in (T_{p_t}M+E_t)/E_t\}, \]

and recall the natural isomorphism $q_t:(T_{p_t}M+E_t)/E_t \xrightarrow {\sim } T_{p_t}M/(T_{p_t}M\cap E_t)$. We now may define $\eta _t\in W_t^{*}$ by $\langle \eta _t,B\rangle =-\langle \xi _t,q_t(B(p_t))\rangle$ for each $B \in W_t$, as $T_{p_t}M\cap E_t \subset \ker \xi _t$. Extend $\eta _t$ by zero to $W_t^{\perp }$. It follows that $(\xi _t,\eta _t)\in N^{*}_{p_t,E_t}Z_M$, completing the proof.

It follows by [Reference AleskerAle10, Corollary 4.1.7] that the Radon transforms with respect to the Euler characteristic, $\mathcal {R}_M=(\tau _M)_*\pi _M^{*}:\mathcal {V}_c^{-\infty }(M)\to \mathcal {V}^{-\infty }(\operatorname {Gr}_{d-k}(V))$ and $\mathcal {R}_M^{T}=(\pi _M)_*\tau _M^{*}:\mathcal {V}^{\infty }(\operatorname {Gr}_{d-k}(V))\to \mathcal {V}^{\infty }(M)$, are well-defined and continuous.

Definition 3.4 For any $\phi \in \mathcal {V}_c^{-\infty }(M)$, let $\widehat \phi \in C^{-\infty }(\operatorname {Gr}_{d-k}(V))$ be the defining current of $\mathcal {R}_M\phi$ (on the base manifold). Equivalently, using [Reference AleskerAle07, Proposition 7.3.6] we have

\[ \widehat{\phi}=[\mathcal{R}_M\phi]\in\mathcal{W}_0^{-\infty}(\operatorname{Gr}_{d-k}(V))/\mathcal{W}_1^{-\infty}(\operatorname{Gr}_{d-k}(V))=C^{-\infty}(\operatorname{Gr}_{d-k}(V)). \]

Remark 3.5 It is false, in general, that $\widehat \phi$ is a smooth function when $\phi$ is a smooth valuation, see Remark 3.10.

Definition 3.6 The Crofton map $\operatorname {Cr}_M: \mathcal {M}^{\infty }(\operatorname {Gr}_{d-k}(V))\to \mathcal {W}_k^{\infty }(M)$ is the restriction of $\mathcal {R}_M^{T}$ to $\mathcal {M}^{\infty }(\operatorname {Gr}_{d-k}(V))$. More explicitly,

\[ \operatorname{Cr}_M(\mu)(X)=\int_{\operatorname{Gr}_{d-k}(V)}\widehat{\chi}_X(E)\,d\mu(E), \quad X \in \mathcal{P}(M). \]

We show in Proposition 3.12 that $\widehat {\chi }_X(E)=\chi (X \cap E)$.

3.2 Distributional Crofton measures

To allow distributional Crofton measures, it seems essential to require that all intersections $E\cap M$ are transversal, for $E\in \operatorname {Gr}_{d-k}(V)$. This is easily seen to be equivalent, for any $k>0$, to having $\mathbb {R} x\oplus T_xM=V$ for all $x\in M$. In particular, $\dim V=\dim M+1=m+1$. We deduce that $M$ is a hypersurface that is locally diffeomorphic to an open subset of $\mathbb {P}_+(V)$ through the radial projection. In other words, $M$ is locally a strictly star-shaped hypersurface around the origin.

Proposition 3.7

  1. (i) For all $0\leq k\leq m$ and $\psi \in \mathcal {V}^{\infty }_c(M)$, it holds that $E \mapsto \psi (E\cap M)$ is a smooth function on $\operatorname {Gr}_{m+1-k}(V)$.

  2. (ii) The image in $C^{-\infty }(\operatorname {Gr}_{m+1-k}(V))$ of this function equals $\widehat {\psi }$.

Proof. (i) Let us first show $E\mapsto \psi (E\cap M)$ is smooth. By choosing an open cover of $M$ by star-shaped charts, and using the partition of unity property of smooth valuations [Reference AleskerAle07], we may assume $M$ projects diffeomorphically to an open subset of $\mathbb {P}_+(V)$, which we henceforth identify with $M$.

By Boman's theorem [Reference BomanBom67], it suffices to prove that $\psi (E_t \cap M)$ is a smooth function of $t\in (-\epsilon,\epsilon )$ for all smooth curves $E_\bullet :(-\epsilon,\epsilon )\to \operatorname {Gr}_{m+1-k}(V)$. It suffices, in fact, to show smoothness in some open interval around $0$ for any such given curve.

Let us lift $E_t$ to a smooth curve $g_t\in \mathrm {GL}(V)$ with $g_0=\operatorname {Id}$ and $E_t=g_tE_0$. Then $\psi (E_t\cap M)= g_t^{*}\psi (E_0 \cap M)$ for sufficiently small $t$ such that $g_t(\operatorname {Supp}(\psi ))\subset M$, establishing the first part.

(ii) Let us check $\psi (\bullet \cap M)=\widehat {\psi }$ in $C^{-\infty }(\operatorname {Gr}_{m+1-k}(V))$. Take $\mu \in \mathcal {M}^{\infty }(\operatorname {Gr}_{m+1-k}(V))$, and write

\[ \mu=\int_{\operatorname{Gr}_{m+1-k}(V)}\delta_E\,d\mu(E)= \int_{\operatorname{Gr}_{m+1-k}(V)}\chi_{\{E\}}\,d\mu(E)\in\mathcal{V}^{\infty}(\operatorname{Gr}_{m+1-k}(V)). \]

Claim We claim that $\tau _M^{-1}E\pitchfork \pi _M^{*}\psi$.

To see this, write $Z=Z_M$ and identify $W:=Z\times _M\mathbb {P}_M$ with its image in $\mathbb {P}_{Z}$ under $d\pi _M^{*}$. Explicitly, $W|_{(x,E)}=\mathbb {P}_+(\mathrm {Ker}(d_{(x,E)}\pi _M)^{\perp })$, so $W$ is the union of the conormal bundles to all fibers of $\pi _M$. It follows from [Reference AleskerAle10, Proposition 3.3.3] that $\operatorname {WF}(\pi _M^{*}\psi )\subset (\emptyset, N^{*}W)$. By Proposition 2.1, it suffices to check that two intersections in $\mathbb {P}_{Z}$ are transversal: $\pi ^{-1}(\tau _M^{-1}E)\pitchfork W$ and ${N^{*}(\tau _M^{-1}E)\pitchfork W}$.

Denote $z=(x,E)$, let $(z,\zeta )$ be an intersection point. The first intersection is easy to analyze: $T_{z,\zeta }\pi ^{-1}(\tau _M^{-1}E)$ contains all vertical directions of $\mathbb {P}_{Z}$, whereas $T_{z,\zeta }W$ contains all horizontal directions.

To analyze the second intersection, we lift all manifolds from $\mathbb {P}_Z$ to $T^{*}Z$, and retain all notation for the corresponding objects. As in the previous case, the image of $T_{z,\zeta }W$ under the natural projection $\pi :T_{z,\zeta } T^{*}Z\to T_zZ$ is all of $T_zZ$, and so it suffices to show $T_{z,\zeta }(T_z^{*}Z)\subset T_{z,\zeta }W+ T_{z,\zeta }N^{*}(\tau _M^{-1}E)$.

As $N_z^{*}(\pi _M^{-1}x) \subset W$, it suffices to show that

\[ T_{z,\zeta}(T_z^{*}Z)\subset T_{z,\zeta}N_z^{*}(\pi_M^{-1}x)+ T_{z,\zeta}N_z^{*}(\tau_M^{-1}E), \]

which is the same as

\[ T_z^{*}Z\subset N_z^{*}(\pi_M^{-1}x)+ N_z^{*}(\tau_M^{-1}E)=N_z^{*}(T_z\pi_M^{-1}x\cap T_z\tau_M^{-1}E). \]

The proof of the claim is completed by noting that the intersection $T_z\pi _M^{-1}x\cap T_z\tau _M^{-1}E$ is trivial.

Consider the set

\[ X:=\left\{(E,[\xi]): E \in \operatorname{Gr}_{m+1-k}(V), [\xi] \in \operatorname{WF}(\chi_{\tau_M^{-1}E})\right\} \subset \operatorname{Gr}_{m+1-k}(V) \times \mathbb{P}_{\mathbb{P}_Z}. \]

We claim that it is compact. If $X \subset \bigcup _{i \in I} U_i$ is an open cover, then for each $E \in \operatorname {Gr}_{m+1-k}(V)$ we find a finite subcover $X_E \subset \bigcup _{i \in I_E} U_i$ of the compact set

\[ X_E:=X\cap (\{E\} \times \mathbb{P}_{\mathbb{P}_Z})=\{E\} \times \operatorname{WF}(\chi_{\tau_M^{-1}E}). \]

The map $g \mapsto X_{gE}$ is $\operatorname {GL}(V)$-equivariant, hence there exists some open neighborhood $V_E \subset \operatorname {Gr}_{m+1-k}(V)$ of $E$ such that $X_{E'} \subset \bigcup _{i \in I_E} U_i$ for all $E' \subset V_E$. Now $\operatorname {Gr}_{m+1-k}(V)$ is compact, hence finitely many $V_{E_j}$ cover $\operatorname {Gr}_{m+1-k}(V)$. Then $X \subset \bigcup _j \bigcup _{i \in E_j} U_i$ is a finite subcover, proving the claim. The image of $X$ in $\mathbb {P}_{\mathbb {P}_Z}$ is then a compact set disjoint from $\operatorname {WF}(\pi _M^{*}\psi )$.

Thus, we can find a closed cone $\Gamma \subset T^{*}\mathbb {P}_{Z}\setminus \underline 0$ such that for all $E\in \operatorname {Gr}_{m+1-k}(V)$, $\chi _{\tau _M^{-1}E}\in \mathcal {V}^{-\infty }_{(\emptyset, \Gamma )}(Z)$, and $\pi _M^{*}\psi$ acts as a sequentially continuous functional on the latter space. Thus, we can write

\begin{align*} \langle \widehat{\psi},\mu\rangle&= \langle \pi_M^{*}\psi, \tau_M^{*}\int_{\operatorname{Gr}_{m+1-k}(V)} \chi_{\{E\}}\,d\mu(E)\rangle\\ & = \int_{\operatorname{Gr}_{m+1-k}(V)} \langle \pi_M^{*}\psi,\chi_{\tau_M^{-1}(E)}\rangle \,d\mu(E). \end{align*}

It remains to check that

(3)\begin{equation} \langle \pi_M^{*}\psi,\chi_{\tau_M^{-1}(E)}\rangle= \psi(E\cap M). \end{equation}

For a compact submanifold with boundary $A \subset M$ that is transversal to $E\cap M$, we have by [Reference Alesker and BernigAB12, Theorem 5],

\[ \langle \pi_M^{*}\chi_A,\chi_{\tau_M^{-1}(E)}\rangle=\chi(\pi_M^{-1}A\cap \tau_M^{-1}(E))=\chi(A\cap E)=\chi_A(E\cap M). \]

It follows by linearity that any smooth valuation of the form $\psi =\int _{\mathcal {A}} \chi _A\,d\nu (A)$, where $\mathcal {A}$ is a family of submanifolds $A$ as previously and $\nu$ a smooth measure, satisfies (3). This family $\mathbf {Cr}_E$ of valuations spans a dense subset in $\mathcal {V}^{\infty }(M)$. Indeed, we may approximate $\chi _A$ in $\mathcal {V}^{-\infty }(M)$ by a sequence in $\mathbf {Cr}_E$ for any $A$ transversal to $E\cap M$. Were $\mathbf {Cr}_E$ not dense, by Alesker–Poincaré duality one could find a non-zero smooth valuation $\phi$ annihilating $\mathbf {Cr}_E$ and, thus, also vanishing on all submanifolds with boundary that are transversal to $E\cap M$. By the genericity of transversality and continuity, $\phi$ would vanish on all submanifolds with boundary. However, this is impossible by [Reference Bernig and BröckerBB07]. It follows that equality in (3) holds for all $\psi$.

Corollary 3.8 The map $\operatorname {Gr}_{m+1-k}(V)\to \mathcal {V}^{-\infty }(M)$, $E\mapsto \chi _{E\cap M}$ is smooth, and for $\mu \in \mathcal {M}^{\infty }(\operatorname {Gr}_{m+1-k}(V))$ it holds that $\operatorname {Cr}(\mu )=\int _{\operatorname {Gr}_{m+1-k}(V)}\chi _{E\cap M}\,d\mu (E)$.

Proposition 3.9 The map $\operatorname {Cr}:\mathcal {M}^{\infty }(\operatorname {Gr}_{m+1-k}(V))\to \mathcal {W}_k^{\infty }(M)$ extends to a continuous map $\operatorname {Cr}: \mathcal {M}^{-\infty }(\operatorname {Gr}_{m+1-k}(V))\to \mathcal {W}_k^{-\infty }(M)$, by setting, for all $\psi \in \mathcal {V}^{\infty }_c(M)$,

\[ \left\langle \operatorname{Cr}(\mu), \psi\right\rangle := \int_{\operatorname{Gr}_{m+1-k}(V)} \psi(E\cap M)\,d\mu(E). \]

Proof. The right-hand side is well-defined for a generalized measure $\mu$, because the function $E \mapsto \psi (E \cap M)$ is smooth by Proposition 3.7. Take $\mu \in \mathcal {M}^{\infty }(\operatorname {Gr}_{m+1-k}(V))$, $\psi \in \mathcal {V}_c^{\infty }(M)$. To verify this new definition extends the smooth one, we ought to check that

\[ \langle (\pi_M)_*\tau_M^{*}\mu, \psi\rangle = \int_{\operatorname{Gr}_{m+1-k}(V)} \psi(E\cap M)\,d\mu(E), \]

which is the content of Corollary 3.8. Continuity is equally evident.

Remark 3.10 It is tempting to define $\operatorname {Cr}(\mu )$ as a Radon transform, $\operatorname {Cr}(\mu )=\mathcal {R}_M^{T}\mu$, as defined in [Reference AleskerAle10]. Unfortunately the conditions of [Reference AleskerAle10, Corollary 4.1.7], which guarantee that the transform is well-defined on generalized valuations, do not hold for general $k$, as can be seen by a simple dimension count.

3.3 Functorial properties of Crofton measures

The following is a partial summary of the results of [Reference FaifmanFai17, Appendix B] (adapted from the affine to the linear Grassmannian), whereto we refer the reader for further details.

Let $j: U^{r}\hookrightarrow V^{d}$ be an inclusion of a linear subspace. There is then a well-defined operation of restriction

\[ j^{*}:\mathcal{M}^{\infty}(\operatorname{Gr}_k(V)) \to \mathcal{M}^{\infty}(\operatorname{Gr}_{k-(d-r)}(U)), \]

which is the push-forward under the (almost everywhere defined) map ${J_U: E\mapsto j^{-1}(E)=E\cap U}$.

Let $S_U\subset \operatorname {Gr}_{k}(V)$ be the collection of subspaces intersecting $U$ non-generically, and fix a closed cone $\Gamma \subset T^{*}\operatorname {Gr}_k(V)\setminus 0$ such that $\Gamma \cap N^{*}S_U=\emptyset$. Given $k\geq d-r$, let $\mathcal {M}^{-\infty }_\Gamma (\operatorname {Gr}_k(V))$ denote the set of generalized measures (distributions) $\mu$ whose wave front sets lie in $\Gamma$, equipped with the Hörmander topology.

The map $j^{*}$ extends as a sequentially continuous map

\[ j^{*}:\mathcal{M}_\Gamma^{-\infty}(\operatorname{Gr}_k(V)) \to \mathcal{M}^{-\infty}(\operatorname{Gr}_{k-(d-r)}(U)). \]

Similarly, if $\pi :V\to W$ is a quotient map, there is a natural push-forward operation

\[ \pi_*: \mathcal{M}^{\infty}(\operatorname{Gr}_k(V))\to \mathcal{M}^{\infty}(\operatorname{Gr}_{k}(W)), \]

which is the push-forward under the (almost everywhere defined) map $\Pi _W: E\mapsto \pi (E)$. It extends to distributions whose wave front sets are disjoint from the conormal cycle of the collection of subspaces intersecting $\operatorname {Ker} \pi$ non-generically.

The following proposition captures the intuitively obvious fact that the pullback of distributions/valuations under embeddings commutes with the Crofton map. We prove a weak version which suffices for our purposes.

Recall that $M$ is a locally star-shaped hypersurface around the origin.

Proposition 3.11 Take a submanifold $M^{r}\subset V^{d}$, a subspace $j: U\hookrightarrow V$ such that $Z:=M\cap j(U)$ is a submanifold, and a distribution $\mu \in \mathcal {M}_\Gamma ^{-\infty }(\operatorname {Gr}_{d-k}(V))$. Assume $\operatorname {Cr}_M(\mu )$ is transversal to $Z$ in the sense of [Reference AleskerAle10, Definition 3.5.2]. Then $\operatorname {Cr}_M(\mu )|_Z=\operatorname {Cr}_Z(j^{*}\mu )$.

Proof. Choose an approximate identity $\rho _i\in \mathcal {M}^{\infty }(\textrm {GL}(V))$ as $i\to \infty$, and set $\mu _i=\mu \ast \rho _{i}\in \mathcal {M}^{\infty }(\operatorname {Gr}_{d-k}(V))$. For all $A\in \mathcal {P}(Z)$ we have

\[ \operatorname{Cr}_M(\mu_i)(A)= \int_{\operatorname{Gr}_{d-k}(V)} \chi(A \cap E)\,d\mu_i(E)=\int_{\operatorname{Gr}_{r-k}(U)} \chi(A \cap E)d((J_U)_*\mu_i)(E), \]

and, therefore, $\operatorname {Cr}_M(\mu _i)|_Z=\operatorname {Cr}_Z(j^{*}\mu _i)$. The restriction of valuations to a submanifold is continuous in the Hörmander topology on the space of valuations with wave front set contained in $\textrm {WF}(\operatorname {Cr}_M(\mu ))$, see [Reference AleskerAle10, Claim 3.5.4]. Thus, the left-hand side weakly converges to $\operatorname {Cr}_M(\mu )|_Z$. The right-hand side weakly converges to $\operatorname {Cr}_Z(\mu )$.

3.4 Applying generalized Crofton formulas to subsets

Let $M^{m}\subset V=\mathbb {R}^{m+1}$ be a strictly star-shaped hypersurface around the origin. Given $A\in \mathcal {P}(M)$ and a Crofton distribution $\mu \in \mathcal {M}^{-\infty }(\operatorname {Gr}_{m+1-k}(V))$, we would like to evaluate $\operatorname {Cr}(\mu )$ on $A$ using an explicit Crofton integral, whenever $A\pitchfork \operatorname {Cr}(\mu )$.

The following proposition provides some a priori regularity for $\widehat \chi _A$.

Proposition 3.12 For $A\in \mathcal {P}(M)$, it holds that $\chi (A\cap \bullet )\in L^{1}(\operatorname {Gr}_{m+1-k}(V))$, is finite and locally constant on $W_A:=\{E:E\pitchfork A\}$. Furthermore, $\widehat {\chi }_A=\chi (A \cap \bullet )$.

Proof. Let us first check that $\chi (A \cap \bullet )\in L^{1} ( \operatorname {Gr}_{m+1-k}(V))$. Fix a Euclidean structure on $V$ and identify $M$ with the unit sphere. By [Reference Bernig, Fu and SolanesBFS14, Lemma A.2], for a fixed $E_0\in \operatorname {Gr}_{m+1-k}(V)$ we have $[g \mapsto \chi (A\cap gE_0)] \in L^{1}(\operatorname {SO}(V))$. Let $dg, dE$ be the Haar measures on $\operatorname {SO}(V)$ and $\operatorname {Gr}_{m+1-k}(V)$, respectively, and $p:\operatorname {SO}(V)\to \operatorname {Gr}_{m+1-k}(V)$ given by $g\mapsto gE_0$. Then $p_*(\chi (A\cap gE_0)dg)=\chi (A\cap E)\,dE$, and so $\chi (A\cap E)$ is integrable. It is evidently finite and locally constant on $W_A$

It remains to check that $\widehat {\chi }_A=\chi (A\cap \bullet )$. Take an approximate identity ${ \rho }_j\in \mathcal {M}^{\infty }(\operatorname {SO}(V))$, which for convenience we assume invariant under inversion.

Consider the convolution $\phi _j:=\chi _A\ast { \rho _j}\in \mathcal {V}^{-\infty }(M)$. As $\operatorname {SO}(V)$ is transitive on $M$ and $\mathbb {P}_M$, it follows that the defining currents of $\phi _j$ are smooth, and therefore $\phi _j\in \mathcal {V}^{\infty }(M)$.

By [Reference Bernig, Fu and SolanesBFS14, Theorem A.1], $\tilde \phi _j:=\int _{\operatorname {SO}(V)}\chi (gA\cap \bullet )\,d{\rho }_j(g)$ is a well-defined smooth valuation. Let us show that $\tilde \phi _j=\phi _j$. Take $\psi \in \mathcal {V}_c^{\infty }(M)$ and compute

\[ \langle\tilde\phi_j, \psi\rangle= \int_{\operatorname{SO}(V)}\psi(gA\cap M)\,d{ \rho}_j(g) =\int_{\operatorname{SO}(V)}\psi(gA)\,d{ \rho}_j(g) \]

by [Reference Bernig, Fu and SolanesBFS14], whereas

\[ \langle \phi_j, \psi\rangle=\langle \chi_A, \psi\ast { \rho}_j\rangle =(\psi\ast\nu_j)(A)=\int_{\operatorname{SO}(V)}\psi(gA)\,d{ \rho}_j(g). \]

Equality now follows by Alesker–Poincaré duality. We thus have the following equalities of functions on $\operatorname {Gr}_{m+1-k}(V)$:

\[ \phi_j( { \bullet} \cap M)=\tilde\phi_j({ \bullet}\cap M)=\int_{\operatorname{SO}(V)}\chi(gA\cap { \bullet})\,d{ \rho}_j(g)=\chi(A\cap \bullet)\ast { \rho}_j, \]

where the right-hand side is the convolution of $\chi (A\cap \bullet )\in L^{1}(\operatorname {Gr}_{m+1-k}(V))$ with ${ \rho }_j$. It follows that $\phi _j(E\cap M)\to \chi (A\cap E)$ in $L^{1}(\operatorname {Gr}_{m+1-k}(V))$.

Fix $\mu \in \mathcal {M}^{\infty }(\operatorname {Gr}_{m+1-k}(V))$. By Proposition 3.7 and $\operatorname {GL}(V)$-equivariance,

\begin{align*} \langle \widehat{\chi}_A,\mu \rangle &= \lim_{j\to\infty} \langle \widehat{\chi}_A \ast { \rho}_j, \mu\rangle = \lim_{j\to\infty} \langle \widehat {\chi_A\ast { \rho}_j}, \mu\rangle\\ &=\lim_{j\to\infty} \int_{\operatorname{Gr}_{m+1-k}(V)} \phi_j(E\cap M) \,d\mu(E) =\int_{\operatorname{Gr}_{m+1-k}(V)} \chi(A\cap E) \,d\mu(E), \end{align*}

and so $\widehat {\chi }_A=\chi (A\cap \bullet )$.

Definition 3.13 The $k$-Crofton wave front of $A\in \mathcal {P}(M)$ is $\operatorname {Cr}\operatorname {WF}^{k}(A):=\operatorname {WF}(\widehat \chi _A)\subset T^{*}\operatorname {Gr}_{m+1-k}(V)$.

As $\widehat {\chi }_A$ is real-valued, $\operatorname {Cr}\operatorname {WF}^{k}(A)$ must be symmetric under the fiberwise antipodal map.

Proposition 3.14 Assume $\operatorname {Cr}\operatorname {WF}^{k}(A)\cap \operatorname {WF}(\mu )=\emptyset$. Then

(4)\begin{equation} \operatorname{Cr}(\mu)(A)=\int_{\operatorname{Gr}_{m+1-k}(V)}\chi(A\cap E)\,d\mu(E). \end{equation}

Proof. We identify $M$ with $\mathbb {P}_+(V)$. Then $\mathcal {V}^{-\infty }(M)\to C^{-\infty }(\operatorname {Gr}_{m+1-k}(V))$, $\phi \mapsto \widehat \phi$ is $\operatorname {GL}(V)$-equivariant. Consider the sequence of smooth valuations $\psi _j$ given by $\psi _j=\int _{\operatorname {GL}(V)}g^{*}\chi _A\cdot d \rho _j(g)$, where $\rho _j$ is a compactly supported approximate identity on $\operatorname {GL}(V)$. Clearly $\psi _j\to \chi _A$ in the Hörmander topology of $\mathcal {V}^{-\infty }_{\operatorname {WF}(\chi _A)}(M)$. By $\operatorname {GL}(V)$-equivariance we have that

\begin{align*} \widehat{\psi}_j= \int_{\operatorname{GL}(V)}g^{*}\widehat{\chi}_A \cdot d \rho_j(g) \to\widehat\chi_A \end{align*}

in $C^{-\infty }_{\operatorname {WF}(\widehat \chi _A)}(\operatorname {Gr}_{m+1-k}(V))$.

It holds by Propositions 3.9 and 3.7 that

\[ \langle \operatorname{Cr}(\mu), \psi_j\rangle = \int_{\operatorname{Gr}_{m+1-k}(V)} \psi_j(E\cap M)\,d\mu(E)=\langle \mu,\widehat{\psi}_j\rangle. \]

As $j \to \infty$, the left-hand side converges to $\langle \operatorname {Cr}(\mu ),\chi _A\rangle =\operatorname {Cr}(\mu )(A)$, as $A \pitchfork \operatorname {Cr}(\mu )$. The right-hand side converges to $\langle \mu,\widehat {\chi }_A\rangle$ (because $\operatorname {WF}(\mu ) \cap \operatorname {WF}(\widehat {\chi }_A)=\emptyset$), which is the same as $\int _{\operatorname {Gr}_{m+1-k}(V)}\chi (A\cap E)\,d\mu (E)$ by Proposition 3.12.

Determining $\operatorname {Cr}\operatorname {WF}^{k}(A)$ precisely appears to be difficult in general. Let us focus on a subset $A\in \mathcal {P}(M)$ which is either a compact domain with smooth boundary, or a compact hypersurface without boundary.

For the following, we write $H=H(A)$ for $\partial A$ if $A$ is of full dimension, and for $A$ when it is a hypersurface. Write $\widehat E:=E\cap M$, and note that $E$ intersects $H$ transversally in $V$ if and only if $\widehat E$ intersects $H$ transversally in $M$. We use the notation

\begin{align*} \widetilde B_H & :=\{(x,E)\in Z_H: T_x\widehat E\subset T_xH\}, \\ B_H & : =\tau_H(\widetilde B_H)\subset\operatorname{Gr}_{m+1-k}(V). \end{align*}

It is not hard to see that $\widetilde B_H$ is an embedded submanifold of $Z_H$ of dimension

(5)\begin{align} \dim \widetilde B_H&= \dim H+(m-k)(\dim H-(m-k))\nonumber\\ &=k(m+1-k)-1=\dim \operatorname{Gr}_{m+1-k}(V)-1. \end{align}

If $(x,E)\in \widetilde B_H$, we say that $x\in H$ is a tangent point for $E$. Observe also that $W_A= B_H^{c}$.

Write $\tilde \tau _H$ for the restriction $\tau _H|_{\widetilde B_H}:\widetilde B_H\to \operatorname {Gr}_{m+1-k}(V)$. We sometimes write $B^{m+1-k}_H$, etc., to specify the dimension.

Definition 3.15 We say that $E\in \operatorname {Gr}_{m+1-k}(V)$ is a regular tangent to $A$ if $\tilde \tau _H$ is immersive on $\tilde \tau _H^{-1}(E)$.

Note that if $E\notin B_H$, then it is automatically regular.

For a subset $A\subset V$ we denote by $\mathbb {P}(A)$ its image in the projective space $\mathbb {P}(V)$. The regularity of the tangent is equivalent to the non-vanishing of the Gauss curvature of the corresponding section, as follows.

Lemma 3.16 Fix $(p,E)\in \widetilde B_H$. Choose any line $N\subset T_pM\setminus T_pH$, and set $F=E\oplus N$. Then $\tilde \tau _H:\widetilde B_H\to \operatorname {Gr}_{m+1-k}(V)$ is an immersion at $(p,E)$ if and only if $\mathbb {P}(H\cap F)\subset \mathbb {P}( F)$ has non-degenerate second fundamental form at $p$.

In particular, all tangents to $A$ are regular if and only if $\mathbb {P}(H)\subset \mathbb {P}(V)$ is a strictly convex hypersurface.

Proof. Let us sketch the argument, see [Reference TeufelTeu90, Lemma 1(ii)] for details. Clearly $d\tilde \tau _H$ is injective on the subspace of directions where $p$ moves transversally to $E$. Namely, fixing any subspace $\bar {E}\subset T_pH$ such that $E\oplus \bar {E}=T_pH\oplus \mathbb {R} p$, $d\tilde \tau _H$ is injective on $\{ (v,A)\in T_pH\times T_E\operatorname {Gr}_{m+1-k}(V): v\in \bar {E}\}\cap T\widetilde B_H$. That injectivity is retained as the remaining directions are added, corresponds to the non-degeneracy of the Gauss map of the section $H\cap F$.

We now describe the Crofton wave front near regular tangents. For an immersed manifold $i:X\looparrowright Y$ and $y\in i(X)$, we denote

\[ N_y^{*}i(X)=\bigcup_{x\in i^{-1}y}(d_xi(T_xX))^{\perp}\subset T_y^{*}Y,\quad N^{*}i(X)=\bigcup_{y\in i(X)}N^{*}_yi(X). \]

Proposition 3.17 Assume $E_0\in B^{m+1-k}_H$ is a regular tangent. Then $\operatorname {Cr}\operatorname {WF}^{k}_{E_0}(A)\subset N^{*}_{E_0}B_H$.

That $\operatorname {Cr}\operatorname {WF}^{k}_{E_0}(A)$ is contained in the sum of the conormal spaces of the embedded parts of $B_H$ follows from the fact that $\widehat \chi _A$ is locally constant on the complement of $B_H$. However, to show that it is actually contained in the union of those conormal spaces, in the following proof we need a more precise description of $\widehat \chi _A$.

Proof. In the following, by a ball (centered at a point) we mean a compact contractible neighborhood (of the point) with smooth boundary. As $\widetilde B_H$ is a submanifold of $Z_H$, by assumption $B_H\subset \operatorname {Gr}_{m+1-k}(V)$ is an immersed submanifold in a neighborhood around $E_0$, which is a hypersurface by (5).

The preimage $\tilde \tau _H^{-1}(E_0)$ must be finite, or otherwise we could find a sequence of distinct points $(q_j, E_0)\in \widetilde B_H$, which then has a limit point $(q_0, E_0)$, and $\tilde \tau _H$ would fail to be injective in a neighborhood of $(q_0, E_0)$, contradicting the assumed immersivity of $\tilde \tau _H$ there. We use the notation $\tilde \tau _H^{-1}(E_0)=\{(q_j, E_0), 1\leq j\leq N\}$.

We can now find a ball $W \subset \operatorname {Gr}_{m+1-k}(V)$ centered at $E_0$, such that $B_H\cap W$ is the finite union of embedded hypersurfaces $F_j$, each diffeomorphic to a Euclidean ball, with $E_0\in F_j$ and $\partial F_j\subset \partial W$ for all $j$. Note that we have no control on how these hypersurfaces intersect each other. Denote by $C_j^{\pm }$ the connected components of $W\setminus F_j$. The indices are matched by requiring that a neighborhood of $(q_j, E_0)\in \widetilde B_H$ is mapped to $F_j$ by $\tilde \tau _H$.

Fix small balls $K_j\subset M$ around $q_j$ such that

\[ \tilde\tau_H:\pi_H^{-1}(K_j)\cap \tilde\tau_H^{-1}(W) \to F_j \]

is an embedding, $\partial K_j\pitchfork H$ and $\partial K_j\pitchfork \widehat E_0$ in $M$. As $Z_0:=\{q_j:1\leq j\leq N\}$ is the subset of all points in $H$ where $\widehat E_0$ fails to intersect $H$ transversally, it holds that $\widehat E_0\pitchfork (H\setminus Z_0)$, and so $\widehat E_0\cap (H\setminus Z_0)$ is a locally closed submanifold in $H$. We assume the $K_j$ small enough so that they are pairwise disjoint, and in particular $\partial K_j\cap Z_0=\emptyset$. We may, moreover, assume that $H\cap K_j$ is diffeomorphic to a Euclidean ball. By the transversality theorem, we may perturb $K_j$ if necessary to have $(\widehat E_0\cap H) \pitchfork (\partial K_j \cap H)$ in $H$.

Denote by $\tfrac {1}{2} K_j$ a smaller ball centered at $q_j$. Taking $W$ sufficiently small, we may assume that

(6)\begin{equation} \widehat E \pitchfork \biggl(H\setminus \bigcup_j \frac 12 K_j\biggr),\quad\forall E\in W. \end{equation}

This follows by the stability of transversal intersections, because $\widehat E$ is a smooth perturbation of $\widehat E_0$, which intersects $H$ transversally in an open neighborhood of $H\setminus \bigcup _j \textrm {int}(\frac 12 K_j)$. Similarly, we have

(7)\begin{align} \widehat E \pitchfork \partial K_j\textrm{ in } M,\quad \forall E\in W, 1\leq j\leq N \end{align}

and

(8)\begin{equation} (\widehat E\cap H) \pitchfork (\partial K_j\cap H) \textrm{ in } H,\quad \forall E\in W, 1\leq j\leq N. \end{equation}

For $\epsilon \in \{\pm \}^{N}$, denote $C_\epsilon =\cap _{j=1}^{N} C_j^{\epsilon _j}$. Recall that $\widehat {\chi }_A$ is locally constant on $W_A=B_H^{c}$, and so is constant on any connected component of a non-empty set $C_\epsilon$. Let us show there are integers $e_j=e_j(E_0)$ such that for any $\epsilon,\epsilon ' \in \{\pm \}^{N}$ and any $E\in C_{\epsilon }, E'\in C_{\epsilon '}$ one has

(9)\begin{equation} \widehat{\chi}_A(E')-\widehat\chi_A(E)=\sum_{j:\epsilon_j<\epsilon_j'}e_j-\sum_{j:\epsilon_j'<\epsilon_j}e_j. \end{equation}

For $E\in W$, denote $\Sigma _j(E):=\widehat E\cap \partial K_j\cap H$. As it is the transversal intersection of $\widehat E\cap H$ and $\partial K_j \cap H$ in $H$, it is a closed manifold of dimension $(m-k-2)$, and $\chi (\Sigma _j(E))$ is independent of $E\in W$.

Let us distinguish the two cases under consideration. Assume first $A=H$ is a hyper- surface. As $K_i \cap K_j = \emptyset$, we have

\[ \mathbb{1}_M=\sum_{j=1}^{N} (\mathbb{1}_{K_j}-\mathbb{1}_{\partial K_j})+\mathbb{1}_{\overline {K^{c}}}, \]

with $K:=\bigcup _{j=1}^{N} K_j$. Hence, for $E\in W\setminus B_H$ we have

\[ \chi(E\cap A) =\sum_{j=1}^{N} \chi(E\cap K_j\cap A) -\sum_{j=1}^{N}\chi(\Sigma_j(E))+\chi(E\cap \overline{K^{c}} \cap A). \]

The last summand is constant on $W$ by properties (6) and (8). Consequently, for $E\in C_\epsilon$, $E'\in C_{\epsilon '}$ we have

\begin{align*} \widehat\chi_A(E')-\widehat{\chi}_A(E)=\sum_{j=1}^{N} \big(\chi(E'\cap A\cap K_j)- \chi(E\cap A\cap K_j)\big). \end{align*}

The function $\chi (\bullet \cap A\cap K_j)$ is locally constant on $W\setminus F_j$, and it remains to define

(10)\begin{equation} e_j:=\chi(\bullet\cap A\cap K_j)|_{C_j^{+}}- \chi(\bullet\cap A\cap K_j)|_{C_j^{-}}. \end{equation}

The case of full-dimensional $A$ is only slightly more involved. If $(m-k)$ is odd, we have $\widehat {\chi _A}=\tfrac 12\widehat {\chi _{\partial A}}$, reducing to the previous case. Thus, assume $(m-k)$ is even.

Write, as before, whenever $E\in W\setminus B_H$,

\[ \chi(E\cap A)=\sum_{j=1}^{N} \chi(E\cap K_j\cap A) -\sum_{j=1}^{N}\chi(E\cap \partial K_j\cap A)+\chi(E\cap \overline{K^{c}}\cap A). \]

Note that for $E\in W\setminus B_H$, all intersections are manifolds with corners. We have

\[ \chi(E\cap \partial K_j\cap A)=\tfrac 12\chi(E\cap \partial K_j\cap \partial A)=\tfrac 12\chi(\Sigma_j(E)) \]

thus it is constant in $W$.

Set

\[ S_j:=\widehat E\cap \partial K_j, \quad S:=\widehat E\cap \partial A. \]

Here $S_j$ is a transversal intersection in $M$ for all $E\in W$ and, hence, a smooth hyper- surface, whereas $S$ is given by a transversal intersection in $M$ and, hence, smooth for $E\in W\setminus B_H$. Moreover, $S$ is a smooth hypersurface outside of $\tfrac 12 K_j$ for all $E\in W$.

We claim that the intersection $S_j \cap S=\Sigma _j(E)$ is transversal in $\widehat E$ for all $E\in W$. If the intersection is not transversal at $x$, then $T_x( \widehat E\cap \partial K_j)=T_x( \widehat E\cap \partial A)$. However, by assumption $\widehat E \cap \partial A$ and $\partial K_j \cap \partial A$ intersect transversally in $\partial A$, in particular

\[ T_x(\widehat E \cap \partial A)+T_x(\partial K_j \cap \partial A)=T_x\partial A. \]

In conjunction with the previous equality, we get $T_x\partial A \subset T_x\partial K_j$, which is false.

Let $X, Y\subset P$ be smooth domains in a manifold $P$, and assume $\partial X\pitchfork \partial Y$ and $X$ is compact. Let $Z\subset X\cap Y$ be the closure of a connected component of $X\cap Y$. Then $\chi (Z)$ is constant as $X, Y$ are perturbed while maintaining transversality.

Taking $P=\widehat E$, $X=E\cap A$ with $\partial X=S$ (which is a manifold for $E \in W \setminus B_H$), and $Y=E \cap K_j$ with $\partial Y=S_j$ we obtain that $\chi (E\cap A\cap K_j)$ is locally constant in $W\setminus B_H$. Taking $X=E\cap A$ with $\partial X=S$ (which is a manifold outside $\bigcup _j \tfrac 12 K_j$ for all $E \in W$), $Y=E\cap \overline {K^{c}}$ with $\partial Y=\bigcup _j S_j$, it follows that $\chi (E\cap A\cap \overline {K^{c}})$ is constant in $W$. Thus, we may define $e_j$ as in the previous case by (10).

It follows from (9) that for $E\in W$, $\widehat {\chi }_A(E)$ is a linear combination of the indicator functions of the connected components of the complements of the hypersurfaces $F_j$ in $W$. Therefore, $\operatorname {WF}_{E_0}(\widehat \chi _A)\subset \bigcup _j N^{*}F_j$, concluding the proof.

4. The Crofton wave front of LC-regular hypersurfaces

Let $(W,Q)$ be a vector space equipped with a quadratic form. We denote by $\Lambda _{k}^{\nu }(W) \subset \operatorname {Gr}_{k}(W)$ the collection of subspaces $E\subset W$ where $Q|_E$ has nullity $\nu$. We need to describe those sets in several cases.

Proposition 4.1 Assume $(V,Q)$ has dimension $d$.

  1. (i) If $Q$ is non-degenerate, then $\Lambda _k^{\nu }(V)\subset \operatorname {Gr}_k(V)$ is a submanifold of dimension

    (11)\begin{equation} \dim \Lambda_{k}^{\nu}(V)=k(d-k)-\binom{\nu+1}{2}. \end{equation}
    Writing $E_0:=E \cap E^{Q}$, we have
    (12)\begin{equation} T_E \Lambda_{k}^{\nu}(V)=\{A \in \operatorname{Hom}(E,V/E): Q(Au,u)=0, \forall u \in E_0\}. \end{equation}
  2. (ii) If $Q$ has nullity $1$ and $E \in \Lambda _k^{\nu }(V)$ is such that $\operatorname {Ker} Q \cap E=\{0\}$, then $\Lambda _k^{\nu }(V)$ is a manifold near $E$ whose dimension is given by (11).

Proof. (i) See [Reference Bernig and FaifmanBF17, Proposition 4.2].

(ii) Write $L_0:=\operatorname {Ker}(Q)$. Consider $W:=V \oplus \mathbb {R}$, and extend $Q$ as a non-degenerate quadratic form $\widetilde Q$ on $W$. Let us verify that the submanifolds $\Lambda _k^{\nu }(W)$ and $\operatorname {Gr}_k (V)$ intersect transversally in $\operatorname {Gr}_k(W)$ at $E$.

As $L_0 \not \subset E$, also $E^{\widetilde Q} \not \subset L_0^{\widetilde Q}=V$. Thus, we can find a line $L \subset E^{\widetilde Q} \setminus V$. Now any linear map $A:E \to W/E$ decomposes as a sum $A=A_1+A_2$ with $A_1 \in \operatorname {Hom}(E,V/E)=T_E \operatorname {Gr}_k(V)$, $A_2 \in \operatorname {Hom}(E,(E+L)/E) \subset \operatorname {Hom}(E,W/E)=T_E \operatorname {Gr}_k(W)$. As $\widetilde Q(A_2u,v)=0$ for all $u,v \in E_0$, we have $A_2\in T_E \Lambda ^{\nu }_k(W)$ by (12).

This proves the claim. As $\Lambda _k^{\nu } (V)=\Lambda _k^{\nu }(W) \cap \operatorname {Gr}_k (V)$, it is a manifold near $E$. The formula for the dimension then follows from the previous case.

Corollary 4.2 Let $B$ be a smooth manifold, and $W$ a real vector bundle of rank $d$ over $B$. Let $Q\in \Gamma (B, \operatorname {Sym}^{2}(W^{*}))$ be a smooth field of quadratic forms, of nullity at most $1$ for all $x$. Let $\operatorname {Gr}_k(W)$ be the corresponding bundle of $k$-subspaces over $B$, and consider $\Lambda _k^{\nu } (W)=\{(x,E)\in \operatorname {Gr}_k(W): E\in \Lambda ^{\nu }_k(W_x,Q_x)\}$. If $(p,E)\in \Lambda _k^{\nu } (W)$ and $\operatorname {Ker}(Q_ x)\cap E=\{0\}$, then $\Lambda _k^{\nu } (W)$ is a manifold near $(p,E)$, of dimension

\[ \dim \Lambda_k^{\nu} (W)=\dim B+k(d-k)-{\nu+1 \choose 2}. \]

Proof. Using a local trivialization, this reduces to Proposition 4.1.

Lemma 4.3 Let $W$ be a $d$-dimensional vector space equipped with a quadratic form $Q$ of nullity $1$ with kernel $L_0$. Assume $L_0\subset E_0\in \Lambda _k^{\nu }(W)$, and define the set $C \subset T_{E_0} \operatorname {Gr}_k(W)$ of all velocity vectors $E'(0)$ of smooth curves $E(t) \in \Lambda _k^{\nu }(W)$ with $E(0)=E_0$. Then $C$ is a cone over a closed manifold, and has dimension at most $k(d-k)-{\nu +1\choose 2}$.

Proof. If $Q$ is either positive or negative semi-definite, then $\Lambda _k^{\nu }(W)$ is empty if $\nu \geq 2$, whereas $\Lambda _k^{1}(W)=\{E\in \operatorname {Gr}_k(W):L_0\subset E\}$ is a manifold of dimension $(k-1)(d-k)$, whence the statement is trivial. We henceforth assume that is not the case, that is $Q$ has both positive and negative directions.

Fix $W_0\subset W$ such that $W=L_0\oplus W_0$. Denote

\[ \Lambda_k^{\nu}(W, W_0)=\{E\in \Lambda_k^{\nu}(W): \operatorname{Ker}(Q|_E)\subset W_0\}. \]

Consider the map $I:\operatorname {Gr}_k(W)\setminus \operatorname {Gr}_k(W_0)\to \operatorname {Gr}_{k-1}(W_0)$ given by $I(E)=E\cap W_0$.

We claim that the restriction

\[ \tilde I:=I:\Lambda_k^{\nu}(W)\setminus\Lambda_k^{\nu}(W, W_0)\to \Lambda_{k-1}^{\nu-1}(W_0) \]

is well-defined. That is, the nullity of $I(E)$ is $(\nu -1)$ when $E\notin \Lambda _k^{\nu }(W, W_0)$. Indeed, for such $E$ of nullity $\nu$, the nullity of $I(E)$ is clearly at least $(\nu -1)$. As $\operatorname {Ker}(Q|_E) \not \subset W_0$, one can find $w_E\in \operatorname {Ker}(Q|_E)\setminus W_0$ so that $E=I(E)\oplus \operatorname {Span}(w_E)$. If $I(E)$ contains a $\nu$-dimensional subspace $U$ that is $Q$-orthogonal to $I(E)$, then $U$ is also $Q$-orthogonal to $E$ as $Q(w_E, I(E))=0$. Hence, $U \oplus \operatorname {Span}(w_E) \subset \operatorname {Ker}(Q|_E)$, and consequently the nullity of $E$ is at least $(\nu +1)$, a contradiction.

We will describe the fiber $\tilde I^{-1}(F)$ of $F\in \Lambda _{k-1}^{\nu -1}(W_0)$. Denote

\[ \Lambda_k^{\nu}(W,L_0)=\{E\in \Lambda_k^{\nu}(W): L_0\subset E\}, \]

and $\pi _0:W \twoheadrightarrow W_0$ is the projection along $L_0$. Clearly $E\in \tilde I^{-1}(F)\cap \Lambda ^{\nu }_k(W, L_0)$ if and only if $E=L_0 \oplus F$.

As $Q|_{W_0}$ is non-degenerate, $F_K:=F^{Q}\cap F\subset W_0$ is the kernel of $Q$ on $F^{Q}\cap W_0$. Thus, the quotient space

\[ V_F:=F^{Q}\cap W_0/F_K \]

inherits a non-degenerate quadratic form, also denoted $Q$. Let $\pi _K:F^{Q}\cap W_0 \twoheadrightarrow V_F$ be the projection, and observe that $Q(\pi _Kx)=Q(x)$, so that

\[ \pi_K(\Lambda^{1}_1(F^{Q}\cap W_0)\setminus \mathbb{P}(F_K))=\Lambda^{1}_1(V_F). \]

Denote by $\tilde \pi _0:L_0\oplus V_F \twoheadrightarrow V_F$ the projection to the second summand, and observe that $L_0\oplus V_F$ is naturally equipped with a quadratic form $Q$ with nullity $1$ and kernel $L_0$.

The manifold $\Lambda ^{1}_1(V_F)$ embeds naturally into $\Lambda ^{1}_1(L_0\oplus V_F)$ as the image of all lines of the form $0\oplus L$, $L\in \Lambda ^{1}_1(V_F)$. Put

\[ U_0(F)=\Lambda^{1}_1(L_0\oplus V_F)\setminus\Lambda^{1}_1(V_F), \]

which is a neighborhood of $L_0$.

Define a smooth map

\[ \Phi_F:\mathbb{P}(L_0\oplus V_F)\setminus \mathbb{P}(V_F) \to\operatorname{Gr}_k(W) \]

as follows. For $N \in \mathbb {P}(L_0\oplus V_F)\setminus \mathbb {P}(V_F)$, choose any $0 \neq w \in N \subset L_0 \oplus V_F$. Let $\tilde w \in L_0 \oplus (F^{Q} \cap W_0)$ be a lift of $w$, and set $\tilde N:=\operatorname {Span}(\tilde w)$, $\Phi _F(N):=\tilde N+F$. If $\tilde w'$ is another lift, then $\tilde w'-\tilde w \in F_K \subset F$ and, hence, $\Phi _F(N)$ is well-defined. In particular, $\Phi _F(L_0)=L_0+F$.

Claim We claim that $\Phi _F(U_0(F))=\tilde I ^{-1}(F)$, and the restriction $\Phi _F:U_0(F)\to \tilde I^{-1}(F)$ is bijective.

Proof. For the first statement, we consider two cases. If $N=L_0$, then $L_0\oplus F\in \Lambda ^{\nu }_k(W)$, and $I(L_0\oplus F)=F$. If $N\neq L_0, N\notin \mathbb {P}(V_F)$ and $E=\widetilde N+F$, then one easily verifies that $Q|_N=0$ implies $Q|_{\widetilde N}=0$ and, consequently, $\operatorname {Ker}(Q|_E)=\operatorname {Ker}(Q|_F)\oplus \widetilde N$, so that again $E\in \Lambda ^{\nu }_k(W)$, and clearly $I(E)=F$.

For injectivity, first note that $\Phi _F(N)=L_0 \oplus F \iff N=L_0$. All other points $N\in U_0(F)$ lie inside a unique projective line $\mathbb {P}(L_0\oplus L)$ with $L\in \Lambda ^{1}_1(V_F)$, and $N\neq L, L_0$. Put $E=\Phi _F(N)\in \tilde I^{-1}(F)\setminus \Lambda ^{\nu }_k(W, L_0)$.

Note that if $\tilde L, \tilde L'\in \Lambda _1^{1}(F^{Q}\cap W_0)$ are two lines such that $\pi _0(E)=\tilde L+F=\tilde L'+F$, the projections $\pi _K\tilde L,\pi _K\tilde L'\in \Lambda _1^{1}(V_F)$ must coincide: choosing $\tilde v'\in \tilde L'$ we can find $\tilde v \in \tilde L$ such that $\tilde v'=\tilde v+f$ for some $f\in F$. However, $\tilde v,\tilde v'\in F^{Q}$, and so $\tilde v-\tilde v'\in F^{Q}\cap F=F_K$.

As $\pi _0(E)=\pi _0(\widetilde N)+F$, it follows that $L_E:= \pi _K\pi _0(\widetilde N)\in \Lambda ^{1}_1(V_F)$ is uniquely defined by $E$, and it holds that $N\in \mathbb {P}(L_0\oplus L_E)$. It remains to observe that $\Phi _F$ is obviously injective when restricted to $\mathbb {P}(L_0\oplus L_E)$.

We verify that $\Phi _F$ is onto in the non-trivial case $E\in \tilde I^{-1}(F)\setminus \Lambda _1^{1}(W, L_0)$. Choose $w_E\in \operatorname {Ker}(Q|_E)\setminus W_0$, and decompose $w_E=w_1+w_0$, $w_1 \in W_0, w_0 \in L_0$. In particular, $Q(w_1, w_1)=Q(w_E, w_E)-2Q(w_E, w_0)+Q(w_0, w_0)=0$. As $L_0\not \subset E$, we have $w_1 \neq 0$ and $w_1 = w_E-w_0 \in E^{Q}\subset F^{Q}$. As $w_0=w_E- w_1\notin E$ whereas $F=E\cap W_0\subset E$, it follows that $w_1\in \pi _0(E)\setminus F$ and so we can write

\[ \pi_0(E)=F+\tilde L, \quad \tilde L=\operatorname{Span}(w_1)\in\Lambda_1^{1}(F^{Q}\cap W_0). \]

Put $L=\pi _K\tilde L\in \Lambda ^{1}_1(V_F)$. Now take $N\in \mathbb {P}(L_0\oplus L)$ to be the projection of the line $\operatorname {Span}(w_E)\in \mathbb {P}(L_0\oplus \tilde L\oplus F_K)$. Note also that $\mathbb {P}(L_0\oplus L)\subset \Lambda ^{1}_1(L_0\oplus V_F)$. Clearly, $\Phi _F(N)=E$, concluding the proof of the claim.

Denote by $\Psi _F:\tilde I^{-1}(F)\to U_0(F)$ the inverse map of $\Phi _F$. We may fix an auxiliary Euclidean structure on $W$, and choose $w_E\in \operatorname {Ker}(Q|_E)$ as the unit vector forming the least angle with $L$. Now the map $E\mapsto \operatorname {Ker}(Q|_E)$, $\Lambda ^{\nu }_k(W)\to \operatorname {Gr}_\nu (W)$ is smooth, in the sense that it restricts to a smooth map on every smooth curve in $\Lambda ^{\nu }_k(W)\subset \operatorname {Gr}_\nu (W)$. This is because $\operatorname {Ker}(Q|_E)$ is the eigenspace of $0$, which has fixed dimension along the smooth curve. It follows that $E_t\in \Lambda ^{\nu }_k(W)$ is a smooth curve in $\operatorname {Gr}_k(W)$ through $F\oplus L_0$, if and only if $\Psi _F(E_t)\in \Lambda ^{1}_1(L_0\oplus V_F)$ is a smooth curve in $\mathbb {P}(L_0\oplus V_F)$ through $L_0$.

Define the cone $C_0\subset T_{L_0}\mathbb {P}(L_0\oplus V_F)$ of tangent vectors to all curves $L_t$ through $L_0$ belonging to $\Lambda _1^{1}(L_0\oplus V_F)$, as well as the cone $C_F\subset T_{L_0 \oplus F}\operatorname {Gr}_k(W)$ that consists of all tangent vectors to smooth curves through $L_0 \oplus F$ inside $\Lambda ^{\nu }_k(W)$.

We conclude that the differential $D_{L_0}\Phi _F:T_{L_0}\mathbb {P}(L_0 \oplus V_F)\to T_{L_0 \oplus F}\operatorname {Gr}_k(W)$ restricts to a bijective map $A_F:C_0\to C_F$.

Considering all subspaces $F\in \Lambda _{k-1}^{\nu -1}(W_0)$ simultaneously, we have the following fibration.

The image of the section $F\mapsto L_0 \oplus F$ coincides with $\Lambda _k^{\nu }(W,L_0)$.

Therefore, the cone $C\subset T_{E_0}\operatorname {Gr}_k(W)$ has a linear factor that can be identified with $T_{E_0\cap W_0}\Lambda _{k-1}^{\nu -1}(W_0)$. Putting $F=E_0\cap W_0$, the cone $C/T_{F}\Lambda _{k-1}^{\nu -1}(W_0)$ is then identified with $C_F$.

The dimension of $C_0$ can be readily computed. It can be identified with the abstract cone with base $\Lambda _{1, +}^{1}(V_F)$, the manifold of oriented null lines in $V_F$. As the form $Q$ on $V_F$ is non-degenerate and indefinite, we have $\dim \Lambda _{1,+}^{1}(V_F)=\dim V_0-2=d-k-(\nu -1)-2$. Hence,

\[ \dim C_0=\dim \Lambda_{1,+}^{1}(V_F)+1=d-k-\nu. \]

We have $\dim \Lambda _{k-1}^{\nu -1}(W_0)=(k-1)(d-k)-\binom {\nu }{2}$ by Proposition 4.1, and so

\[ \dim C=\dim C_F+\dim\Lambda_{k-1}^{\nu-1}(W_0)= \dim C_0+ \dim\Lambda_{k-1}^{\nu-1}(W_0)=k(d-k)-\binom{\nu+1}{2}. \]

We now turn to LC-regular submanifolds. First, we need a simple fact on LC-regular metrics.

Lemma 4.4 Let $(M,g)$ be LC-regular, and assume $g$ is degenerate on $T_pM$. Let $v_1(x),\ldots,v_m(x)$ be any local frame near $p$, with Gram matrix $A(x)=(g(v_i, v_j))_{i,j=1}^{m}\in \operatorname {Sym}_m(\mathbb {R})$. Then the condition $d_p(\det A)\neq 0$ is independent of the choice of the frame $(v_j)$. Moreover, if the nullity of $g_p$ is $\nu =1$, then $d_p(\det A)\neq 0$, and the degenerate subset of the metric near $p$ is a smooth hypersurface.

Proof. Let $\tilde v_1(x),\ldots,\tilde v_m(x)$ be a different local frame with corresponding Gram matrix $\widetilde A(x)$. Then the change of basis matrix $U(x)\in \operatorname {GL}(m)$ satisfies $\widetilde A(x)=U(x)^{T}A(x) U(x)$. By assumption, $\det A(p)=\det \widetilde A(p)=0$. Thus, $d_p(\det \widetilde A)=\det U(p)^{2}d_p(\det A)$, which implies the first statement.

For the second statement, choose coordinates $x_1,\ldots,x_m$ on $M$ near $p$, and take $v_j={\partial }/{\partial x_j}$. We may assume that $\operatorname {Ker}(g_p)=\operatorname {Ker} A(p)=\operatorname {Span}(v_m)$, and by assumption $A(p)$ has non-degenerate principal $(m-1)$-minor. By LC-regularity, we can choose a curve $p(t)\in M$ with $p(0)=p$ and a smooth vector field $v(t)$ along it with $v(0)=v_m$ such that $({d}/{dt})|_{t=0}g(v(t),v(t))=({d}/{dt})|_{t=0}\langle A(p(t))e_m, e_m\rangle \neq 0$. It follows that

\[ \frac{d}{dt}\biggr|_{t=0}\det A(p(t))=\frac{d}{dt}\biggr|_{t=0}\langle A(p(t))e_{m},e_{m}\rangle\cdot \det (A(p))_{i,j=1}^{m-1}\neq 0. \]

As the degenerate subset of $g$ near $p$ is $\{x:\det A(x)=0\}$, the last assertion follows.

The following is the main result of the section. We use the notation and terminology of §§ 3.1 and 3.4.

Proposition 4.5 Let $(V,Q)$ be a pseudo-Euclidean vector space of dimension $(n+1)$, and $M=Q^{-1}(r)$ with $r \in \{\pm 1\}$ a pseudo-Riemannian space form. Let $H \subset M$ be an LC-regular hypersurface, and $E \in \operatorname {Gr}_{n-k+1}(V)$. Assume $E \in B_H$ is a regular tangent to $H$ of nullity $\nu$. Then each embedded part of $B_H$ through $E$ intersects $\Lambda ^{\nu }_{n-k+1}(V)$ transversally at $E$.

Proof. Denote the signature of $M$ by $(p_M,q_M)$. Write $g=Q|_H$, $\widehat E=E\cap M$. As $H$ is a hypersurface, the nullity of $g_x$ is at most one for all $x\in H$. Define $\widetilde \Lambda ^{\nu }_{n+1-k}:=\tau _M^{-1}\Lambda ^{\nu }_{n+1-k}(V)$, which is a submanifold of $Z_M$ as $\tau _M:Z_M\to \operatorname {Gr}_{n+1-k}(V)$ is a submersion. The relevant maps are given by the following diagram.

As $B_H$ is a hypersurface, one should show for every embedded part $F$ of $B_H$ through $E$ that $T_E\Lambda ^{\nu }_{n+1-k}(V) \not \subset T_E F$. Assuming the contrary, there is $p\in H\cap E$ such that $T_E \Lambda ^{\nu }_{n+1-k}(V) \subset d\tau _H(T_{p,E}\widetilde B_H)$. Observe that $\operatorname {Ker}(Q|_E)\subset T_p\widehat E$.

Before proceeding with the more complicated general case, we consider the case ${\nu =1}$. As $\dim \Lambda ^{1}_{n+1-k}(V) =\dim B_H=k(n+1-k)-1$ by Proposition 4.1, we have $T_E \Lambda ^{1}_{n+1-k}(V) =d\tau _H(T_{p,E}\widetilde B_H)$. Let $v_0 \in E \cap T_pH$ be in the kernel of ${g|_{T_p\widehat E}}$. For any smooth curve $v_t\in TH$ through $v_0$, we may find a smooth curve $E_t\subset B_H$ through $E$ such that $v_t \in E_t$. Then $A:=({d}/{dt})|_{t=0} E_t\in \operatorname {Hom}(E, V/E)$ satisfies $Q(Av_0, v_0)=0$, and by (12) we have $A\in T_E \Lambda ^{1}_{n+1-k}(V)$.

It follows that $({d}/{dt})|_{t=0} g(v_t)=2Q(v_0, Av_0)=0$, contradicting the LC-regularity of $H$.

Let us now consider the general case. Fix an auxiliary Riemannian metric $h$ on $M$, and let $\rho _H$ be the least distance projection to $H$, defined and smooth in a neighborhood of $p$.

By our assumption $\tilde \tau _H$ is an immersion at $(p,E)$. Using Proposition 4.1 we see that

(13)\begin{equation} \dim (d\tilde\tau_H)^{-1}T_{E}\Lambda^{\nu}_{n+1-k}(V)=\dim \Lambda^{\nu}_{n+1-k}(V)=k(n-k+1)-{\nu+1\choose 2}. \end{equation}

Claim 1 Let $X,Y$ be manifolds, let $F:X \to Y$ be a submersion at $p\in Z$. Let $Z \subset X$ and $W \subset Y$ be embedded submanifolds, and $p \in Z$. Denote $f:=F|_Z$. Then

\[ (d_pf)^{-1} T_{f(p)}W=T_pZ \cap T_p(F^{-1}W). \]

As $F^{-1}W$ is a submanifold near $p$ and $T_p(F^{-1}W)=(d_pF)^{-1}T_{F(p)}W$, the statement is clear. Applying the claim to $X=Z_M$, $Y=\operatorname {Gr}_{n+1-k}(V)$, $Z=\tilde B_H$, $W=\Lambda _{n+1-k}^{\nu }(V)$ and $F=\tau _M$ yields

\[ (d_{(p,E)}\tilde \tau_H)^{-1} T_E \Lambda_{n+1-k}^{\nu}(V)=T_{(p,E)}\tilde B_H \cap T_{(p,E)} \widetilde\Lambda_{n+1-k}^{\nu}(V). \]

Case 1: $\operatorname {Ker}(g_p)\cap E=\{0\}$. Define

\[ \widetilde B^{\nu}_H:=\tilde \tau_H^{-1}\Lambda^{\nu}_{n-k+1}(V)=\{(q,F)\in \widetilde B_H:T_q\widehat F\in\Lambda^{\nu}_{n-k}(T_qH)\}. \]

By Corollary 4.2 we see that $\widetilde B^{\nu }_H$ is a smooth manifold near $(p,E)$, of dimension

(14)\begin{align} \dim T_{p,E}\widetilde B_H^{\nu}&= \dim H + (n-k)(n-1-(n-k))-{\nu+1\choose 2}\nonumber\\ &=k(n-k+1)-1 -{\nu+1\choose 2}. \end{align}

Claim 2 We claim that $(d_{p,E}\tilde \tau _H)^{-1}T_{E}\Lambda ^{\nu }_{n+1-k}(V)\subset T_{p,E}\widetilde B_H^{\nu }$.

We postpone the proof. Combined with (13) and (14) we get a contradiction.

Case 2: $T_pH$ has nullity one, and $\operatorname {Ker}(g_p)\subset E$. In this case, let $S\subset H$ be the degenerate subset of $g$. It follows from Lemma 4.4 that $S$ is a smooth hypersurface. Define $\widetilde B^{\nu }_H(S)=\{(q,F)\in \widetilde B_H: q\in S,F\in \Lambda ^{\nu }_{n-k+1}(V)\}$. It is a fiber bundle over $S$ with fiber $\Lambda ^{\nu }_{n-k}(\mathbb {R}^{p_M-1,q_M-1,1})$.

Claim 3 For any $(w,\xi )\in d_{p,E}\tilde \tau _H^{-1}T_E\Lambda ^{\nu }_{n-k+1}(V)$ with $w\in T_pS$ there is a curve $(q(t), F(t))\in \widetilde B^{\nu }_H(S)$ with $(q'(0), F'(0))=(w, \xi )$.

Again we postpone the proof of the claim. The set of all vectors $(q'(0), F'(0))$ as in the claim defines, by Lemma 4.3, a cone in $T_{p,E}\widetilde B_H$ of dimension

\begin{align*} N & =\dim S+ (n-k)((n-2)-(n-k-1))-{\nu +1\choose 2} \\ & = (n-k)(k-1)-{\nu+1\choose 2}+n-2<\dim d_{p,E}\tilde\tau_H^{-1}T_E\Lambda^{\nu}_{n-k+1}(V). \end{align*}

It follows by the claim that we can find a curve $(p(t), E(t))\in \widetilde B_H$ through $(p,E)$ with $E'(0)\in T_{E}\Lambda _{n-k+1}^{\nu }(V)$ and $p'(0)\notin T_pS$.

Let $v_{n-1}\in T_pH$ span $\operatorname {Ker}(g_p)$, and recall that $v_{n-1}\in T_p\widehat E$, in particular, $v_{n-1}\in \operatorname {Ker}(Q|_{E})$. Choosing any smooth vector field $v_{n-1}(t)\in T_{p(t)}\widehat E(t)$, we find $({d}/{dt})|_{t=0}Q(v_{n-1}(t))=2Q(v_{n-1}'(0), v_{n-1}(0))=0$.

Choose a frame $(v_j(t))_{j=1}^{n-1}$ for $H$ along $p(t)$ with $v_{n-1}(0)=v_{n-1}$, such that $T_{p(t)}\widehat E(t)=\operatorname {Span}(v_k(t),\ldots,v_{n-1}(t))$. Then

\[ \frac{d}{dt}\biggr|_{t=0}\det (Q(v_i(t),v_j(t)))_{i,j=1}^{n-1}=\det (Q(v_i(t),v_j(t)))_{i,j=1}^{n-2} \frac{d}{dt}\biggr|_{t=0}Q(v_{n-1}(t))=0, \]

which means that $d_p(\det g)(p'(0))=0$ by Lemma 4.4. As $\operatorname {Ker} d_p(\det g)=T_pS$ and $p'(0) \notin T_pS$, we get a contradiction. This completes the proof of the proposition, modulo the two claims we now proceed to prove.

Proof of Claim 2 Consider a curve $(p(t), E(t))\in \widetilde \Lambda ^{\nu }_{n-k+1}$ with $(p'(0), E'(0))\in T_{p, E}\widetilde B_H$. We ought to find a curve $(q(t), F(t))\in \widetilde B^{\nu }_H$ through $(p,E)$ with $(q'(0), F'(0))=(p'(0), E'(0))$.

Set $q(t)=\rho _H(p(t))$, evidently $q'(0)=p'(0)$. Fix a subspace $W_0\subset T_pH$ which is non-degenerate and contains $T_p \widehat E$. If $T_pH$ is non-degenerate, we can just take $W_0=T_pH$. Otherwise, $\dim \ker g_p=1$ and we may take any hyperplane $W_0 \subset T_pH$ which contains $T_p\widehat E$ and satisfies $W_0 \cap \ker g_p=\{0\}$. Now fix any linear map $A:W_0\to V/W_0$ which makes the following diagram commutative.

In addition, we use Lemma 3.1 to find smooth paths $W(t) \supset E(t)$, $\widetilde W(t)\subset T_{q(t)}H\oplus \mathbb {R} q(t)$ with $W(0)=\widetilde W(0)=W_0$ and $W'(0)=\widetilde W'(0)=A$. For small $t$, $W(t), \widetilde W(t)$ are non-degenerate of fixed signature $(\alpha, \beta )$.

Consider the manifold $Z=\{(x, W)\in M\times \operatorname {Gr}_{\alpha +\beta }(V), x\in W, \mathrm {sign}(Q|_W)=(\alpha, \beta )\}$. Clearly $Z$ is a homogeneous space for $\operatorname {O}(V, Q)$, with the equivariant projection $\pi _Z:\operatorname {O}(V,Q)\to Z$ normalized by $\pi _Z(\operatorname {Id})=(p,W_0)$. We can fix a smooth section $X_Z:Z\to \operatorname {O}(V,Q)$ near $(p, W_0)$ with $X_Z(p, W_0)=\operatorname {Id}$ such that $\pi _Z\circ X_Z=\operatorname {Id}$. Now define the smooth path $R_t\in \operatorname {O}(V,Q)$ by

\[ R_t=X_Z(q(t),\widetilde W(t))\circ X_{ Z}(p(t), W(t))^{-1}. \]

Then $R_tp(t)=q(t)$, and $({d}/{dt})|_{t=0}R_t=0$ because $({d}/{dt})|_{t=0}W(t)=({d}/{dt})|_{t=0}\widetilde W(t)$.

Setting $F(t)=R_tE(t)$, we have $(q'(0), F'(0))=(p'(0), E'(0))$, and $(q(t), F(t)) \in \widetilde B^{\nu }_H$. This proves the claim.

Proof of Claim 3 Consider a curve $(p(t), E(t))\in \widetilde \Lambda ^{\nu }_{n-k+1}(V)$ through $(p, E)$, with $p'(0)=w\in T_pS$ and $(p'(0), E'(0))=(w,\xi )\in T_{p,E}\widetilde B_H$. Let $\rho _S:M\to S$ be the least distance projection with respect to $h$, well-defined and smooth in some neighborhood of $p$. Set $q(t)=\rho _S( p(t))$, clearly $q'(0)=p'(0)$. Denote $L_0=\operatorname {Ker}(g_p)\subset T_pH\cap E$, and extend to a smooth path of lines $L_t\subset E(t)\cap E(t)^{Q}\in \Lambda _\nu ^{\nu }(T_{p(t)}M)$. Consider the manifold of pairs

\[ Z=\{(x, L): x\in M,L\in\Lambda^{1}_1(T_xM)\}. \]

Clearly $Z$ is a homogeneous space for $\operatorname {O}(V, Q)$, with the equivariant projection $\pi _Z:\operatorname {O}(V,Q)\to Z$ normalized by $\pi _Z(\operatorname {Id})=(p,L_0)$. We can fix a smooth section $X_Z:Z\to \operatorname {O}(V,Q)$ near $(p, L_0)$ with $X_Z(p, L_0)=\operatorname {Id}$ such that $\pi _Z\circ X_Z=\operatorname {Id}$. Now define the smooth path $R_t\in \operatorname {O}(V,Q)$ by

\[ R_t=X_Z(q(t),\operatorname{Ker}(g_{q(t)}))\circ X_Z(p(t),L_t)^{-1}. \]

Then $R_tp(t)=q(t)$, and $({d}/{dt})|_{t=0}R_t=0$, provided that $({d}/{dt})|_{t=0}L_t=({d}/{dt})|_{t=0}\operatorname {Ker}(g_{q(t)})$.

Let us verify that $L_t$ can be chosen in this fashion. In the following, we fix some Riemannian metric on various manifolds, and write $|x-y|_X$ for the corresponding distance between $x,y\in X$. We also write, for two subspaces $E, F\subset V$, $\measuredangle (E,F)$ for the angle between them with respect to some Euclidean metric. This should not create ambiguity, as we are concerned only with rough small-scale asymptotics.

As $(p'(0), E'(0))\in T_{p,E}\widetilde B_H$, we may find a curve $(\tilde p(t), \widetilde E(t))\in \widetilde B_H$ through $(p,E)$ with $(p'(0), E'(0))= (\tilde p'(0), \widetilde E'(0))$. Define $\widetilde H(t):=T_{\tilde p(t)}H\oplus \mathbb {R} \tilde p(t)$. It follows that ${\measuredangle (E(t),\widetilde H(t))=O(t^{2})}$, and by Lemma 3.1 we have the following commutative diagram.

Taking the dual diagram and identifying $V=V^{*}$ using $Q$, we obtain

(15)

where $f_0= (\widetilde H^{Q})'(0)$. As $\tilde p'(0)=p'(0)=q'(0)$, it is clear that

\[ f_0=\frac{d}{dt}\biggr|_{t=0}(T_{\tilde p(t)}H\oplus \mathbb{R} \tilde p(t))^{Q}= \frac{d}{dt}\biggr|_{t=0}(T_{q(t)}H\oplus \mathbb{R} q(t))^{Q}= \frac{d}{dt}\biggr|_{t=0}\operatorname{Ker} g_{q(t)}. \]

By Lemma 3.1, we can find $L_t\subset E(t)^{Q}$ with $({d}/{dt})|_{t=0}L_t=f_0$. Note that $L_t$ is not, in general, a null line of $Q$. We now proceed to modify the definition of $L_t$ to force it to be a null line.

Observe that if $q\in S$, $\widetilde E\in \operatorname {Gr}_{n+1-k}(V)$ and $\operatorname {Ker}(g_{q})\subset \widetilde E$, then $\widetilde E^{Q}\subset T_qH\oplus \mathbb {R} q$. We have

\begin{align*} |\tilde p(t)-q(t)|_M & =O(t^{2}), \\ |T_{\tilde p(t)}H-T_{q(t)}H|_{\operatorname{Gr}_{n-1}(V)} & =O(t^{2}),\\ |L_t-\operatorname{Ker}(g_{q(t)})|_{\mathbb{P}(V)} & =O(t^{2}). \end{align*}

It follows that $\measuredangle (E(t)^{Q}, \widetilde H(t))=O(t^{2})$, and so we may apply Lemma 3.1 to obtain the commutative square

and by duality also

(16)

We use the notation $K(t)=E(t)\cap E(t)^{Q}$, $K_0=K(0)$. Observe there is a natural inclusion $\alpha _K: V/K_0\hookrightarrow V/E\oplus V/E^{Q}$. It follows from Lemma 3.1 applied to the inclusions $K(t)\subset E(t)$, $K(t)\subset E(t)^{Q}$ that $\alpha _K\circ K'(0):K_0\to V/E\oplus V/E^{Q}$ coincides with $E'(0)\oplus (E^{Q})'(0):K_0 \to V/E \oplus V/E^{Q}$.

Combining diagrams (15) and (16) then yields the commutative diagram

and so also

By Lemma 3.1, we may redefine $L_t$ such that $({d}/{dt})|_{t=0}L_t=f_0$ and $L_t \subset K(t)=E(t)\cap E(t)^{Q}$. In particular, $L_t$ is a null line of $Q$.

Setting $F(t):=R_tE(t)$ we have $q(t) \in F(t)$, $T_{q(t)}\widehat F(t)\subset T_{q(t)}H$ because $E(t)\subset L_t^{Q}$, and $F'(0)=E'(0)$ because $({d}/{dt})|_{t=0}R_t=0$. This proves the claim.

Remark 4.6 It is easy to see that the conclusion of the proposition with $k=n-1$ is equivalent to the LC-regularity of $H$.

Corollary 4.7 Let $V,M, H, E$ be as in Proposition 4.5 with $H$ compact without boundary, and $A\subset M$ is either $H$ itself or a domain with $\partial A=H$. Then $\operatorname {Cr}\operatorname {WF}_{E}(A)\cap N_E^{*}\Lambda ^{\nu }_{n-k+1}(V)=\emptyset$.

Proof. This follows from Propositions 4.5 and 3.17.

5. Construction of an invariant measure on the Grassmannian

For $X \in \operatorname {Sym}_r(\mathbb {R})$ and $\lambda \in \mathbb {C}$ we set, as in [Reference MuroMur99],

\[ |\det X|_p^{\lambda}:=\begin{cases} |\det X|^{\lambda} & \text{if } \mathrm{sign}(X)=(p,r-p)\\ 0 & \text{otherwise.} \end{cases} \]

It is well-known, essentially due to Cayley and Gärding [Reference GårdingGår48], that $|\det X|_q^{\lambda }$ extends as a meromorphic in $\lambda$ family of generalized functions, which are, in fact, tempered distributions.

We use the set

\[ U_{\mathbb{C}}:= \bigl\{\mathrm{Re}\,\zeta>\tfrac 12 \bigr\} \cup \{\mathrm{Im}\,\zeta>0\} \subset \mathbb{C}, \]

and write $\sqrt {z}$ for the unique branch of the square root function on $U_\mathbb {C}$ such that $\sqrt z>0$ for $z>\tfrac 12$.

5.1 A holomorphic family of Crofton measures

For the following, let $\operatorname {Sym}_r^{+}(\mathbb {R})\subset \operatorname {Sym}_r(\mathbb {R})$ be the cone of positive-definite matrices, and $\mathfrak h_r=\operatorname {Sym}_r(\mathbb {R})\oplus \mathbf {i}\operatorname {Sym}_r^{+}(\mathbb {R})\subset \operatorname {Sym}_r(\mathbb {C})$ the Siegel upper half space. The following is well-known.

Lemma 5.1 For $Z\in \mathfrak h_r$, $\det Z\neq 0$. In particular, we can define for every $\lambda \in \mathbb {C}$ the holomorphic function $Z \mapsto (\det Z)^{\lambda }$, normalized by $\lim _{\epsilon \to 0^{+}} \det (I_r+\mathbf {i}\epsilon I_r)^{\lambda }=1$. Moreover, all eigenvalues of $Z\in \mathfrak h_r$ lie in the upper half plane of $\mathbb {C}$.

Proof. Write $Z=X+\mathbf {i} Y$, $Y>0$. Let $Q_X(v)=\langle Xv, v\rangle$, $Q_Y=\langle Yv,v\rangle$ be the corresponding quadratic forms. Choose a basis $u_j$ such that the Gram matrix of $Y$ is $I_r$, and of $X$ is diagonal: $D=\mathrm {diag}(d_j)$. As $D+\mathbf {i} I_r=U^{T}ZU$ with $U$ invertible, and $\det (D+\mathbf {i} I_r)=\prod (d_j+\mathbf {i})\neq 0$, it follows that $\det Z\neq 0$. As $\mathfrak h_r$ is simply connected, the second statement follows.

For the last statement, we first note there can be no real eigenvalues. Indeed by the first statement, $\det (X+\mathbf {i} Y-\lambda I_r)=\det ((X-\lambda I_r) + \mathbf {i} Y)\neq 0$ for $\lambda \in \mathbb {R}$. Next we argue as before and select a diagonalizing basis, given by $U\in \mathrm {GL}(r)$. We furthermore may assume that $\det U>0$, by interchanging two basis elements. Choose a smooth path $U_t\in \mathrm {GL}(r)$ with $U_0=\mathrm {Id}$ and $U_1=U$. Then $U_t^{T}ZU_t\in \mathfrak h_r$ is a smooth path. For $t=1$, the endpoint is $D+\mathbf {i} I_r$, which has all eigenvalues in the upper half plane. If $Z$ has eigenvalues in the lower half-plane, then by continuity for some $t$ there will be a real eigenvalue, a contradiction.

Recall for the following that given a non-degenerate quadratic form $Q$ on $V$, a compatible Euclidean form is any positive-definite form $P$ such that $V$ admits a decomposition $V=V_+\oplus V_-$ which is both $P$- and $Q$-orthogonal, and $Q|_{V_\pm }=\pm P|_{V_{\pm }}$.

From here on, let $V=\mathbb {R}^{p} \oplus \mathbb {R}^{q}=\mathbb {R}^{n+1}$ with the standard quadratic form $Q$ of signature $(p,q)$ and the corresponding compatible Euclidean form $P_0$. Define a family of complex-valued quadratic forms $Q_\zeta$ on $V$ with $\zeta \in \mathbb {C}$, by

\[ Q_\zeta:=Q+2\zeta P_0. \]

We then have

\[ Q_\zeta(x,y):= \begin{cases} (2\zeta+1) P_0(x,y) & x,y \in \mathbb{R}^{p},\\ (2\zeta-1) P_0(x,y) & x,y \in \mathbb{R}^{q},\\ 0 & x \in \mathbb{R}^{p}, y \in \mathbb{R}^{q}. \end{cases} \]

Observe that $Q_\zeta$ is real and positive-definite for $\zeta >\tfrac 12$, and $Q_0=Q$. Furthermore, by Lemma 5.1, $\det Q_\zeta \neq 0$ for $\zeta \in U_{\mathbb {C}}$, as either $Q_\zeta$ or $\mathbf {i} Q_\zeta$ lies in $\mathfrak h_{n+1}$. Note that a complex-valued non-degenerate quadratic form $Q$ on a real vector space $E$ defines an element $\operatorname {vol}_Q^{2}\in \operatorname {Dens}_\mathbb {C}(E)^{2}$, and given a branch of square root we also get a complex-valued density $\operatorname {vol}_Q\in \operatorname {Dens}_\mathbb {C}(E)$.

By a ($P$-)frame on an open subset $U\subset \operatorname {Gr}_{n+1-k}(V)$ we understand a smooth section of the Stiefel manifold of $P$-orthonormal $(n+1-k)$-frames in $V$ defined over $U$. For a subspace $E\in \operatorname {Gr}_{n+1-k}(V)$ and $\zeta \in U_{\mathbb {C}}$, choose a frame $u_i(E)$ on $U$ and define $X^{P}_\zeta :U\to \operatorname {Sym}_{n+1-k}(\mathbb {C})$ to be the corresponding Gram matrix of $Q_\zeta$, namely $X^{P}_\zeta (E)=(Q_\zeta (u_i(E), u_j(E)))_{i,j=1}^{n+1-k}$.

Note that if $\widetilde X_{\zeta }$ is the corresponding matrix for a different frame $\widetilde u_i(E)$ on $U$, then

(17)\begin{equation} \widetilde X_\zeta(E)=B(E)^{T} X_\zeta(E) B(E) \end{equation}

for some smooth map $B:U\to \operatorname {O}(n+1-k)$.

Observe that by (17), $\det (X^{P}_\zeta )$ is independent of the choice of $P$-orthonormal bases of $E$. Moreover, either the real or imaginary part of $Q_\zeta |_E$ is positive-definite, and consequently by Lemma 5.1, $\det X^{P}_\zeta (E)\neq 0$.

The function $\det (X^{P}_\zeta )^{\lambda }\in C^{\infty }(\operatorname {Gr}_{n+1-k}(V),\mathbb {C})$ is thus well-defined for all $P$ and $\lambda \in \mathbb {C}$, and analytic in $\zeta \in U_{\mathbb {C}}$, once the normalization $\det (X^{P}_1)^{\lambda }>0$ is fixed, as $U_{\mathbb {C}}$ is simply connected.

Define the smooth measure $\widetilde m^{\zeta, P}_k$ on the Grassmannian $\operatorname {Gr}_{n+1-k}(V)$ by

\[ d\widetilde m^{\zeta, P}_k:=\det(X^{P}_\zeta)^{-({n+1})/{2}}(E)\,d\sigma_P(E), \]

where $d\sigma _P(E)$ is the $\operatorname {O}(P)$-invariant probability measure on the Grassmannian.

Proposition 5.2 The complex-valued smooth measure

\[ m^{\zeta}_k:=(2\zeta+1)^{{p(n+1-k)}/{2}}(2\zeta-1)^{{q(n+1-k)}/{2}}\widetilde m_k^{\zeta,P_0}, \quad \zeta\in U_{\mathbb{C}} \]

depends analytically on $\zeta$ and is normalized, i.e.

\[ \int_{\operatorname{Gr}_{n+1-k}(V)} \,dm^{\zeta}_k=1. \]

Proof. The first statement is clear. For the second, we first see how $\widetilde m^{\zeta, P}_k$ depends on $P$. Let $P_1,P_2$ be two Euclidean structures on $V$. From the natural identification $T_E\operatorname {Gr}_{n+1-k}(V)=E^{*}\otimes V/E$ we obtain that

\[ \operatorname{Dens}(T_E\operatorname{Gr}_{n+1-k}(V))=\operatorname{Dens}^{*}(E)^{n+1}\otimes \operatorname{Dens}(V)^{n+1-k}. \]

Spelling this out gives

\[ \frac{d\sigma_{P_1}(E)}{d\sigma_{P_2}(E)}=\biggl(\frac{\operatorname{vol}_{P_1|_E}}{\operatorname{vol}_{P_2|_E}}\biggr)^{-(n+1)} \biggl(\frac{\operatorname{vol}_{P_1}}{\operatorname{vol}_{P_2}}\biggr)^{n+1-k}, \]

As

\[ \det X_\zeta^{P_i}(E)=\frac{\operatorname{vol}^{2}_{Q_\zeta|_E}}{\operatorname{vol}^{2}_{P_i|_E}}, \]

we find that

\[ \widetilde m^{\zeta, P_1}_k=\biggl(\frac{\operatorname{vol}_{P_1}}{\operatorname{vol}_{P_2}}\biggr)^{n+1-k}\widetilde m^{\zeta, P_2}_k. \]

For $\zeta >\tfrac {1}{2}$, $Q_\zeta$ is a Euclidean structure. Then

\begin{align*} 1 & =\int \widetilde m^{\zeta, Q_\zeta}_k= \biggl(\frac{\operatorname{vol}_{Q_\zeta}}{\operatorname{vol}_{P_0}}\biggr)^{n+1-k} \int \widetilde m^{\zeta, P_0}_k\\ & = \sqrt{2\zeta+1}^{p(n+1-k)} \sqrt{2\zeta-1}^{q(n+1-k)} \int \widetilde m^{\zeta, P_0}_k=\int m^{\zeta}_k. \end{align*}

By uniqueness of the analytic continuation, this formula also holds for general $\zeta \in U_{\mathbb {C}}$.

5.2 Homogeneous distributions on the space of symmetric matrices

Lemma 5.3 The meromorphic family of generalized functions

\[ f_\lambda(X):=\sum_{h=0}^{r} e^{\mathbf{i} \pi h\lambda}|\det X|^{\lambda}_{r-h} \in C^{-\infty}(\operatorname{Sym}_r(\mathbb{R})) \]

is analytic in $\lambda \in \mathbb {C}$ and satisfies

(18)\begin{equation} f_\lambda(-X)=e^{\mathbf{i} \pi r \lambda} \overline{f_\lambda(X)}. \end{equation}

Proof. We recall some results from [Reference MuroMur99]. Consider a linear combination

\[ g_\lambda(X):=\sum_{h=0}^{r} a_h |\det X|^{\lambda}_{r-h} \]

with constant coefficients $a_h \in \mathbb {C}$ and set $\vec a:=(a_0,\ldots,a_r) \in \mathbb {C}^{r+1}$. Then $g_\lambda \in C^{-\infty }(\operatorname {Sym}_r(\mathbb {R}))$ is meromorphic with possible poles in the set $\{-m, -({2m+1})/{2}: m\geq 1\}$.

The order of the pole at $s$ in this set can be obtained as follows. Set $\epsilon =-1$ if $s$ is an even integer and $\epsilon =1$ otherwise. Define inductively linear maps $d^{(m)}=(d^{(m)}_0,\ldots,d^{(m)}_{r+1-m}):\mathbb {C}^{r+1} \to \mathbb {C}^{r+1-m}$ by setting

\begin{align*} d^{(0)}_h(\vec a) & := a_h,\\ d^{(1)}_h(\vec a) & :=a_h+\epsilon a_{h+1} ,\\ d_h^{(2l+1)}(\vec a) & := d^{(2l-1)}_h-d^{(2l-1)}_{h+2}, \quad l=1,2,\ldots,\\ d_h^{(2l)}(\vec a) & := d^{(2l-2)}_h+d^{(2l-2)}_{h+2}, \quad l=1,2,\ldots. \end{align*}

Then $g_\lambda$ has a pole of order $p$ at $s=-({2m+1})/{2}$ if and only if $d^{2p}(\vec a) \neq 0, d^{2p+2}(\vec a)=0$. Similarly, $g_\lambda$ has a pole of order $p$ at $s=-m$ if and only if $d^{2p-1}(\vec a) \neq 0, d^{2p+1}(\vec a)=0$. Here we use the convention that $d^{(m)}=0$ if $m >r+1$ and that a pole of order $0$ is a point of analyticity.

In our situation, the coefficients $a_h=a_h(\lambda )=e^{\mathbf {i} \pi h \lambda }$ depend on $\lambda$ and we cannot apply Muro's result directly. However, writing

\[ f_\lambda(X)=\sum_{h=0}^{r} a_h(\lambda) |\det X|^{\lambda}_{r-h}=\sum_{j=0}^{\infty} \frac{(\lambda-s)^{j}}{j!} \sum_{h=0}^{r} a_h^{(j)}(s) |\det X|^{\lambda}_{r-h}, \]

we see that it is enough to prove that the order of the pole of $\sum _{h=0}^{r} a_h^{(j)}(s) |\det X|^{\lambda }_{r-h}$ at $\lambda =s$ is at most $j$ for all $j$.

By induction, we find that for all $l=0,1,\ldots$

\begin{align*} d^{2l}_h(\vec a(\lambda)) & = e^{\mathbf{i} \pi h \lambda} (1+e^{2\pi \mathbf{i} \lambda})^{l}, \\ d^{2l+1}_h(\vec a(\lambda)) & = e^{\mathbf{i} \pi h \lambda} (1+\epsilon e^{\mathbf{i} \pi \lambda}) (1-e^{2\pi \mathbf{i} \lambda})^{l}, \end{align*}

and, hence,

\begin{align*} d^{2j+2}_h\biggl(\vec a^{(j)}\biggl(-\frac{2m+1}{2}\biggr)\biggr) & = \frac{d^{j}}{d\lambda^{j}}\biggr|_{\lambda=-({2m+1})/{2}} e^{\mathbf{i} \pi h \lambda} (1+e^{2\pi \mathbf{i} \lambda})^{j+1}=0, \\ d^{2j+1}_h\big(\vec a^{(j)}(-m)\big) & = \frac{d^{j}}{d\lambda^{j}}\biggr|_{\lambda=-m} e^{\mathbf{i} \pi h \lambda} (1+\epsilon e^{\mathbf{i} \pi \lambda}) (1-e^{2\pi \mathbf{i} \lambda})^{j}=0, \end{align*}

which finishes the proof.

Proposition 5.4 Let $C' \subset \operatorname {Sym}_r^{+}(\mathbb {R})$ be a closed convex cone. Then

\[ \lim_{Y\to 0, Y \in C'} \det(X+\mathbf{i} Y)^{\lambda}=f_\lambda(X) \]

in the strong topology on tempered distributions on $\operatorname {Sym}_r(\mathbb {R})$.

Proof. For $\mathrm {Re}\,\lambda \geq 0$ the statement is easy, so in the following we assume $\mathrm {Re}\,\lambda <0$.

First we claim that the limit exists. We show that there are constants $\alpha =\alpha (\lambda )\geq 0$ and $b'=b(\lambda,C')$ such that

(19)\begin{equation} |\det(X+\mathbf{i} Y)^{\lambda}| \leq b' \|Y\|^{-\alpha},\quad \forall X\in\operatorname{Sym}_r(\mathbb{R}), \forall Y\in C'\setminus\{0\}. \end{equation}

First take $Y=I_r$. Letting $(\mu _j)_{j=1}^{r}\subset \mathbb {R}$ be the eigenvalues of $X$, we obtain

\begin{align*} |\det(X+\mathbf{i} I_r)^{\lambda}| &= \prod|\mu_j+\mathbf{i} |^{{\mathrm{Re}}\,\lambda} e^{-\mathrm{Im}\,\lambda\cdot\mathrm{Arg}(\mu_j+\mathbf{i})}\\ & \leq e^{\pi r|\mathrm{Im}\,\lambda|}\prod(\mu_j^{2}+1)^{{\mathrm{Re}\,\lambda}/{2}}\\ & \leq e^{\pi r|\mathrm{Im}\,\lambda|}. \end{align*}

Now for general $Y$, we have

\[ |\det (X+\mathbf{i} Y)^{\lambda}|=|\det Y|^{\mathrm{Re}\,\lambda}|\det(\sqrt Y^{-1} X \sqrt Y^{-1} +\mathbf{i} I)^{\lambda}| \leq e^{\pi r {|\mathrm {Im}\lambda|}}|\det Y|^{\mathrm{Re}\,\lambda}, \]

and letting $c:=\sup \{{\|Y\|^{r}}/{|\det Y|}: Y\in C'\}$, we conclude that (19) holds with $b'=c^{ -\mathrm {Re}\,\lambda } e^{\pi r |\mathrm {Im}\lambda |}$, and $\alpha =-r\mathrm {Re}\,\lambda \geq 0$.

It now follows from [Reference VladimirovVla66, § 26.3] that the limit

\begin{align*} \det(X+\mathbf{i} 0)^{\lambda}:=\lim_{Y\to 0, Y \in C'}\det(X+\mathbf{i} Y)^{\lambda}\in \mathcal{S}' \end{align*}

exists in the strong topology on the space of tempered distributions of order $\lceil r|\mathrm {Re}\,\lambda |\rceil +{r+1\choose 2}+3$.

It remains to verify that $\det (X+\mathbf {i} 0)^{\lambda }=f_\lambda (X)$ for $\mathrm {Re}\,\lambda <0$. We use the notation

\[ H_\epsilon=\{X+\mathbf{i}\epsilon I_r: X\in \operatorname{Sym}_r(\mathbb{R})\}\subset \operatorname{Sym}_r(\mathbb{C}). \]

Let $\psi (X)$ be a Schwartz function on $\operatorname {Sym}_r(\mathbb {R})$, which is the Fourier transform of a compactly supported smooth function, in particular it has an analytic extension to $\operatorname {Sym}_r(\mathbb {C})$. Writing $dZ=\Large \wedge _{i=1}^{r} \Large \wedge _{j=i}^{r} \, dz_{ij}$, the integral $\int _{H_\epsilon } \psi (Z)\det (Z)^{\lambda } \, dZ$ is convergent, because $\psi$ is rapidly decaying at infinity and $\det (Z)^{\lambda }$ of polynomial growth. It is clearly analytic in $\lambda \in \mathbb {C}$. Furthermore, its value is independent of $\epsilon$ as the integrand is a closed form, rapidly decaying at infinity. For $\lambda >0$, we have

\[ (\mu_j+\mathbf{i} \epsilon)^{\lambda}=|\mu_j+\mathbf{i} \epsilon|^{\lambda} e^{\mathbf{i} \lambda\mathrm{Arg}(\mu_j+\mathbf{i} \epsilon)}\to |\mu_j|^{\lambda} e^{\mathbf{i}\lambda ({\pi}/{2})(1-\operatorname{sign}(\mu_j))}. \]

Hence,

\[ \det(X+\mathbf{i} \epsilon I_r)^{\lambda}=\prod_{j=1}^{r} (\mu_j+\mathbf{i} \epsilon)^{\lambda} \to \prod |\mu_j|^{\lambda} e^{\mathbf{i}\pi \#\{\mu_j<0\}\lambda}=f_\lambda(X), \]

and so

\[ \int_{H_\epsilon} \psi(Z)\det(Z)^{\lambda} \,dZ\to \int_{\operatorname{Sym}_r(\mathbb{R})} \psi(X) f_\lambda(X)\,dX \]

for $\lambda >0$. By analytic extension we conclude that for all $\lambda \in \mathbb {C}$ and $\epsilon >0$,

\[ \int_{H_\epsilon} \psi(Z)\det(Z)^{\lambda} \,dZ =\int_{\operatorname{Sym}_r(\mathbb{R})} \psi(X) f_\lambda(X)\,dX, \]

that is,

\[ \int_{\operatorname{Sym}_r(\mathbb{R})} \psi(X+\mathbf{i} \epsilon I_r)\det(X+\mathbf{i} \epsilon I_r)^{\lambda} \,dX =\int_{\operatorname{Sym}_r(\mathbb{R} )} \psi(X) f_\lambda(X)\,dX. \]

As $\epsilon \to 0$, we have $\psi (X+\mathbf {i} \epsilon I_r)\to \psi (X)$ in $\mathcal {S}$, whereas $\det (X+\mathbf {i} \epsilon I_r)^{\lambda }\to \det (X+\mathbf {i} 0)^{\lambda }$ in $\mathcal {S}'$. It follows by continuity that

\[ \int_{\operatorname{Sym}_r(\mathbb{R})} \psi(X)\det(X+\mathbf{i} 0)^{\lambda} \,dX = \int_{\operatorname{Sym}_r(\mathbb{R})} \psi(X) f_\lambda(X)\,dX. \]

Finally, noting that the set of Schwartz functions such as $\psi$ is dense, we conclude that $\det (X+\mathbf {i} 0)^{\lambda }=f_\lambda$ for all $\lambda \in \mathbb {C}$, as claimed.

Henceforth, we use $f_\lambda$ and $\det (X+\mathbf {i} 0)^{\lambda }$ interchangeably.

The following statement shows that the convergence along $Y\in \mathbb {R}_+ I_r$ holds in a finer topology, namely the normal Hörmander topology. We refer to [Reference Brouder, Dang and HéleinBDH16] for its definition (where it is called normal topology). The main point for us is that the operation of pull-back of generalized sections is continuous in this topology, provided some condition on wave fronts is satisfied.

We do not know whether an analogue of the following proposition holds for an arbitrary distributional boundary value; the proof below is tailored to our particular case, and in essence leverages strong convergence by induction on dimension.

Proposition 5.5 Denote $N^{*}\Gamma ^{r}=\bigcup _{\nu =0}^{r} N^{*} \Gamma ^{r}_\nu \subset T^{*}\operatorname {Sym}_r(\mathbb {R})$, where $\Gamma ^{r}_\nu$ consists of all matrices of nullity $\nu$. It then holds for all $\lambda \in \mathbb {C}$ that $\det (X+\mathbf {i}\epsilon I_r)^{\lambda }\to f_\lambda (X)$ in $C^{-\infty }_{N^{*}\Gamma ^{r}}(\operatorname {Sym}_r(\mathbb {R}))$ in the normal Hörmander topology.

Proof. First note that $g^{*}f_\lambda =\det (g)^{2\lambda }f_\lambda$ for all $g\in \operatorname {GL}(r)$. Thus, we have the differential equations $(\underline A-2\lambda \operatorname {tr}(A))f_\lambda =0$, where $\underline A$ is the vector field defined by the infinitesimal action of $A\in \mathfrak {gl}(r)$. It follows from [Reference HörmanderHör03, Theorem 8.3.1] that $\operatorname {WF}(f_\lambda )\subset N^{*}\Gamma ^{r}$.

We proceed by induction on $r$, the case $r=1$ being trivial. As $\det (X+\mathbf {i}\epsilon I_r)^{\lambda }\to \det (X+\mathbf {i}0)^{\lambda }$ in the strong topology by Proposition 5.4 and $N^{*}_0\Gamma ^{r}_r=T_0^{*}\operatorname {Sym}_r(\mathbb {R})$, it remains to consider convergence in $\operatorname {Sym}_r(\mathbb {R})\setminus \{0\}$. Consider a matrix $Y\in \operatorname {Sym}_r(\mathbb {R})$ of nullity $\nu < r$. Let $E_0$ be its kernel, and $F_0=E_0^{\perp }$. There is then a unique map $E:U\to \operatorname {Gr}_\nu (\mathbb {R}^{r})$ in a neighborhood $U$ of $Y$ such that $E(Y)=E_0$, and $E(X)$ is an invariant subspace of $X$. Here and in the following, $U$ is assumed sufficiently small for various purposes.

We claim $E=E(X)$ is smooth. Indeed, consider $Z=\{(X,F): X(F)= F\}\subset \operatorname {Sym}_r(\mathbb {R})\times \operatorname {Gr}_{r-\nu }(\mathbb {R}^{r})$. Clearly $Z$ is the graph of a unique function $F=F(X)$ near $(Y, F_0)$. Let us check that $Z$ is a manifold near $(Y, F_0)$. Define $\alpha :U\times \operatorname {Gr}_{r-\nu }(\mathbb {R}^{r})\to \operatorname {Gr}_{r-\nu }(\mathbb {R}^{r})\times \operatorname {Gr}_{r-\nu }(\mathbb {R}^{r})$ by $\alpha (X, F)=(F, X(F))$. Then $Z=\alpha ^{-1}( \Delta )$, where $\Delta$ is the diagonal. Let us verify that $\alpha$ is a submersion at $(Y, F_0)$.

For $M\in \operatorname {Sym}_r(\mathbb {R})$ and $H\in T_{F_0}\operatorname {Gr}_{r-\nu }(\mathbb {R}^{r})=\mathrm {Hom}(F_0,\mathbb {R}^{r}/F_0)$, one computes $d_{Y,F_0}\alpha (M, H)=(H, Y\circ H+M|_{F_0\to \mathbb {R}^{r}/F_0})=(H, M|_{F_0\to \mathbb {R}^{r}/F_0})$, because by construction $Y: \mathbb {R}^{r}/F_0\to \mathbb {R}^{r}/F_0$ is the zero map. Noting that any linear map $F_0\to \mathbb {R}^{r}/F_0$ is induced by a symmetric matrix mapping $M:\mathbb {R}^{r} \to \mathbb {R}^{r}$, it follows that $\alpha$ is submersive and $Z$ is a manifold. Further,

\[ T_{Y,F_0}Z=\{(M,H):M\in\operatorname{Sym}_r(\mathbb{R}), H=M|_{F_0\to \mathbb{R}^{r}/F_0} \}. \]

In particular, if $(0, H)\in T_{Y,F_0}Z$, then we must have $H=0$. It follows that $F(X)$ is smooth in $U$, and therefore so is $E(X)=F(X)^{\perp }$.

Choose arbitrary orthonormal frames $e_i(X)$ for $E(X)$ and $f_i(X)$ for $F(X)=E(X)^{\perp }$ depending smoothly on $X$. Define

\[ A:U\to\operatorname{Sym}_\nu(\mathbb{R}),\quad B:U\to \operatorname{Sym}_{r-\nu}(\mathbb{R}) \]

by

\[ A(X)=(\langle Xe_i(X), e_j(X)\rangle ),\quad B(X)=(\langle X f_i(X), f_j(X)\rangle ). \]

Then $A$ is a submersion in $U$. Indeed one has

\[ d_YA(M)_{i,j}=\langle M e_i(Y), e_j(Y)\rangle+\langle Ye_i(Y), d_Ye_j(M)\rangle +\langle Ye_j(Y), d_Ye_i(M)\rangle, \]

and the last two summands vanish as $e_i(Y), e_j(Y)\in E_0$. It follows that $d_YA:\operatorname {Sym}_r(\mathbb {R})\to \operatorname {Sym}_\nu (\mathbb {R})$ is surjective, and so $A$ is submersive near $Y$.

It holds that

\[ \det(X+\mathbf{i}\epsilon I_r)^{\lambda}=A^{*}\det(X_1+\mathbf{i}\epsilon I_\nu) ^{\lambda} \det(B(X)+\mathbf{i}\epsilon I_{r-\nu})^{\lambda}, \quad X_1 \in \operatorname{Sym}_{\nu}(\mathbb{R}). \]

As $B(X)$ is non-degenerate, the second factor is a smooth function in $(X,\epsilon )\in U\times \mathbb {R}$.

For the first factor, we have by the induction assumption that $\det (X_1+\mathbf {i}\epsilon I_\nu ) ^{\lambda }\to \det (X_1+\mathbf {i}0)^{\lambda }$ in the normal topology on $C^{-\infty }_{N^{*}\Gamma ^{\nu }}(\operatorname {Sym}_\nu (\mathbb {R}))$. It then holds that

\[ \operatorname{WF}(A^{*}\det(X_1+\mathbf{i}0)^{\lambda})\subset A^{*}(N^{*}\Gamma^{\nu}) =N^{*}(A^{-1}\Gamma^{\nu})= N^{*}(\Gamma^{r}\cap U), \]

and by [Reference Brouder, Dang and HéleinBDH16], $A^{*}\det (X_1+\mathbf {i}\epsilon I_\nu )^{\lambda }\to A^{*}\det (X_1+\mathbf {i}0)^{\lambda }$ in the normal Hörmander topology on $C^{-\infty }_{N^{*}\Gamma ^{r}}(\operatorname {Sym}_r(\mathbb {R}))$. We conclude that

\[ \det(X+\mathbf{i}\epsilon I_r)^{\lambda}\to \det(X+\mathbf{i}0)^{\lambda} \]

in the normal Hörmander topology on $C^{-\infty }_{N^{*}\Gamma ^{\nu }}(\operatorname {Sym}_r(\mathbb {R}))$.

Remark 5.6 Using the Hilbert–Schmidt inner product to identify $T_0^{*}\operatorname {Sym}_r(\mathbb {R})=\operatorname {Sym}_r(\mathbb {R})$, the statement of the proposition in fact holds with all conormal cones intersected with $\operatorname {Sym}_r^{+}(\mathbb {R})$, which follows from [Reference HörmanderHör03, Theorem 8.1.6].

5.3 Construction of an $\operatorname {O}(p,q)$-invariant Crofton distribution

In [Reference FaifmanFai17, Proposition 4.9], an $\operatorname {O}(p,q)$-invariant distribution was constructed on $\operatorname {Gr}_{n+1-k}(V)$. To avoid singularities, it made use of several auxiliary Euclidean structures that gave rise to several locally defined distributions that were then patched together. For the present paper, we need an alternative construction making use of a single Euclidean structure. To handle the singularities, we must carefully monitor the wave front set.

Write $P=P_0$ for the Euclidean structure on $V$. Let $dE=d\sigma _{P}$ denote the $\operatorname {O}(P)$-invariant probability measure on $\operatorname {Gr}_{n+1-k}(V)$. Decompose $V=V_P^{+}\oplus V_P^{-}$ such that $Q|_{V_P^{\pm }}=\pm P|_{V_P^{\pm }}$.

We use the notation $\kappa =n+1-k$. An orthonormal basis $u_1,\ldots, u_\kappa$ spanning $E\in \operatorname {Gr}_{\kappa }(V)$ is called adapted if, using the notation $s=\dim E\cap V_P^{+}$, $t=\dim E\cap V_P^{-}$, the vectors $u_{\kappa -s-t+1},\ldots, u_{\kappa -t}$ form a basis of $E\cap V_P^{+}$, whereas $u_{\kappa -t+1},\ldots, u_{\kappa }$ form a basis of $E\cap V_P^{-}$. A frame $u_i(E')$, $i=1,\ldots, \kappa$ given near $E$ and adapted at $E$ is well-adapted in a neighborhood $W$ if, whenever $E'\in W$ is such that $E'\cap E$ is spanned by the subset $(u_{i}(E))_{i\neq j}$ for some $1\leq j\leq \kappa$, then $u_{i}(E')=u_{i}(E)$ for all $i\neq j$.

It is easy to see that a well-adapted frame can always be chosen to extend a given adapted orthonormal basis $u_i(E)$ of $E$ to a small neighborhood: define the frame $u_i(E')$ by orthogonally projecting $u_i(E)$ to $E'$, and then applying the Gram–Schmidt process.

For $X\in \operatorname {Sym}_\kappa (\mathbb {R})$ and $\mu \in \mathbb {R}$, denote

\[ E_\mu(X)=\{v\in\mathbb{R}^{\kappa}: Xv=\mu v\},\quad \mathrm{mult}(\mu, X)=\dim E_\mu(X). \]

Define for $a\geq 0$

\[ B^{a}_\mu=\{X: \mathrm{mult}(\mu, X)=a\}\subset\operatorname{Sym}_\kappa(\mathbb{R}). \]

We make use of the Hilbert–Schmidt Euclidean structure $\langle X,Y\rangle =\operatorname {tr}(XY)$ to identify $T_X\operatorname {Sym}_\kappa (\mathbb {R})=T_X^{*}\operatorname {Sym}_\kappa (\mathbb {R})=\operatorname {Sym}_\kappa (\mathbb {R})$.

Lemma 5.7 The subset $B^{a}_\mu \subset \operatorname {Sym}_\kappa (\mathbb {R})$ is a locally closed submanifold. It holds that $N^{*}_XB_\mu ^{a}\ {=}\ \{\Xi \in \operatorname {Sym}_\kappa (\mathbb {R}): X\Xi \ {=}\ \mu \Xi \}\ {=}\ \{\Xi \in \operatorname {Sym}_\kappa (\mathbb {R}): \Xi X\,{=}\,\mu \Xi \}\,{=}\,\operatorname {Span}\{vv^{T}: v\in E_\mu (X)\}$, and $\mathrm {codim} B_\mu ^{a}=\binom {a+1}{2}$.

Proof. The subset $B^{a}_0$ locally coincides with an orbit of the action of $\operatorname {GL}(\kappa )$ on $\operatorname {Sym}_\kappa (\mathbb {R})$ by $(g, X)\mapsto g^{T}Xg$, and $B^{a}_\mu =\mu I+B^{a}_0$. Now $B_\mu ^{a}$ fibers over $\operatorname {Gr}_a(\mathbb {R}^{\kappa })$ with fiber $\operatorname {Sym}_{\kappa -a}(\mathbb {R})$. Consequently,

\[ \dim B_\mu^{a}=a(\kappa-a)+{\kappa-a+1\choose 2} \]

and one computes that $\mathrm {codim} B_\mu ^{a}=\binom {a+1}{2}$.

Let us describe the set $N^{*}_XB_\mu ^{a}$. As $T_XB_0^{a}=\{A^{T}X+XA: A\in \mathfrak {gl}_\kappa (\mathbb {R})\}$, we have $\Xi \in N^{*}_XB_0^{a}\iff \operatorname {tr}(\Xi A^{T} X+\Xi X A)=0$ for all $A$ or, equivalently, $N^{*}_XB_0^{a}=\{\Xi \in \operatorname {Sym}_\kappa (\mathbb {R}): \Xi X=0\}$. It follows that

\begin{align*} N^{*}_XB_\mu^{a}= N^{*}_{X-\mu I}B_0^{a}=\{\Xi: \Xi X=\mu \Xi\}, \end{align*}

and the second form follows by transposition. Finally, $\Xi =uv^{T}+vu^{T}$ is easily checked to satisfy $\Xi X=\mu \Xi$ when $u,v\in E_\mu (X)$. By a simple dimension count we conclude that $N^{*}_XB_\mu ^{a}=\operatorname {Span}\{uv^{T}+vu^{T}: u,v\in E_\mu (X)\}$, which coincides with $\operatorname {Span}\{vv^{T}: v\in E_\mu (X)\}$ as $uv^{T}+vu^{T}=(u+v)(u+v)^{T}-uu^{T}-vv^{T}$.

Lemma 5.8 For any $Y\in \operatorname {Sym}_\kappa (\mathbb {R})$ with $Y\in B_0^{r}$ and for every $\epsilon >0$, there is a neighborhood $W_Y$ of $Y$ such that for all $X\in W_Y$, if $X\in B^{r'}_0$, then $T_{X}B^{r'}_0$ contains a subspace that is $\epsilon$-close to $T_Y B^{r}_0$.

Proof. Recall that by Lemma 5.7, $T_XB^{r'}_0=\{\Xi \in \operatorname {Sym}_\kappa (\mathbb {R}): X\Xi = 0\}^{\perp }$. The statement now follows from the following general fact.

Claim Let $M_0\in \mathrm {Mat}_{n\times n}(\mathbb {R})$ be a matrix. Then for any $\epsilon >0$, there is a neighborhood $W_\epsilon$ of $M_0$ such that for any $M\in W_\epsilon$, $\operatorname {Ker}(M)^{\perp }$ contains a subspace that is $\epsilon$-close to $\operatorname {Ker}(M_0)^{\perp }$.

Proof. Assume that $\mathrm {rank}M_0=r$, and the first $r$ rows $u_1(M_0)^{T},\ldots, u_r(M_0)^{T}$ are linearly independent. Therefore, $\operatorname {Ker}(M_0)^{\perp }=\operatorname {Span}(u_1(M_0),\ldots, u_r(M_0))$. By choosing $W_\epsilon$ small enough, we may ensure that $\operatorname {Span}(u_1(M),\ldots, u_r(M))$ is $r$-dimensional, and $\epsilon$-close to $\operatorname {Ker}(M_0)^{\perp }$. As $\operatorname {Span}(u_1(M),\ldots, u_r(M))\subset \operatorname {Ker}(M)^{\perp }$, this concludes the proof.

Lemma 5.9 Let $M_\tau$ be a smooth curve in $\operatorname {Sym}_\kappa (\mathbb {R})$ such that the spectrum of $M_\tau$ lies in $[-1,1]$ for all $\tau$. If $\epsilon \in \{-1,1\}$ and $\mathrm {mult}(\epsilon, M_0)=s$, then ${d}/{d\tau }|_0M_\tau \in T_{M_0}B^{s}_\epsilon$.

Proof. Write $\dot M_0={d}/{d\tau }|_0M_\tau$. By Lemma 5.7, we ought to show that for all unit vectors $v\in E_\epsilon (M_0)$, $\langle \dot M_0, vv^{T}\rangle =0$. Now for any such $v$, $\langle M_0 v, v\rangle =\epsilon$. We know by assumption $|\langle M_\tau v, v\rangle |\leq 1$ for all $\tau$, and so

\[ \langle \dot M_0, vv^{T}\rangle=\langle \dot M_0 v, v\rangle =\frac{d}{d\tau}\biggr|_0\langle M_\tau v, v\rangle=0. \]

Proposition 5.10 Let $X_0$ be locally defined by a well-adapted frame at $E\in \operatorname {Gr}_{n-k+1}(V)$. Then there is a neighborhood $W_{E}$ of $E$ where the distribution

\[ \widetilde m^{0}_k(E'):= X_0^{*}f_{-({n+1})/{2}}(E')\cdot dE' \]

is well-defined. Furthermore, the corresponding distributions agree on non-empty intersections $W_{E_1}\cap W_{E_2}$ for all $E_1, E_2$, giving rise to a globally defined distribution $\widetilde m^{0}_k\in \mathcal {M}^{-\infty }(\operatorname {Gr}_{n+1-k}(V))$. Moreover, $\widetilde m^{0}_k$ is $\operatorname {O}(Q)$-invariant.

Fix $E$, and a well-adapted to $E$ frame $u_j(E')$ defined in a neighborhood $W'_E$. Set $Y=X_0(E)$, $s=\dim E\cap V_P^{+}=\mathrm {mult}(1, Y)$, $t=\dim E\cap V_P^{-}=\mathrm {mult}(-1, Y)$, $r=\mathrm {mult}(0, Y)$.

Claim It holds that $\operatorname {Image}(d_E X_0)+T_YB_{0}^{r}=T_Y\operatorname {Sym}_\kappa (\mathbb {R})$.

Proof. Let us prove the claim. We may assume that $Y$ lies in the singular support of $f_{\lambda }$, that is $r\geq 1$. Thus, $Y\in B^{r}_0\cap B^{s}_1\cap B^{t}_{-1}$.

The intersection $B^{s,t}_{1,-1}:=B^{s}_1\cap B^{t}_{-1}$ is transversal. To this end, simply observe that by Lemma 5.7, $N_Y^{*}B_1^{s}\cap N_Y^{*}B_{-1}^{t}=\{\Xi : \Xi Y=\Xi =-\Xi \}=\{0\}$, so $B^{s}_1\pitchfork B^{t}_{-1}$, and $B^{s,t}_{1,-1}$ is a submanifold. It holds that

\[ \dim B^{s,t}_{1,-1}={\kappa+1\choose 2}-{s+1\choose 2}-{t+1\choose 2}. \]

Similarly, the intersection $B^{r}_0\cap B^{s,t}_{1,-1}$ is transversal. Indeed, $N^{*}_YB^{s,t}_{1,-1}=\{ \Xi _1+\Xi _2: \Xi _1Y=\Xi _1, \Xi _2Y=-\Xi _2\}$ and

\[ \mathrm{codim}T_YB^{s,t}_{1,-1}=\mathrm{codim}T_Y B_1^{s}+\mathrm{codim}T_Y B_{-1}^{t}={s+1\choose 2}+{t+1\choose 2}. \]

If $\Xi = \Xi _1+\Xi _2\in N^{*}_YB^{s,t}_{1,-1}\cap N_Y^{*}B_0^{r}$, then $\Xi _1-\Xi _2=\Xi _1Y+ \Xi _2Y=\Xi Y=0$, so that $\Xi _1= \Xi _2$, which can only happen if $\Xi _1=\Xi _2=0$ because $\Xi _1Y=\Xi _1$, $\Xi _2Y=-\Xi _2$, therefore $\Xi =0$. Thus, $B_0^{r} \pitchfork B_{1,-1}^{s,t}$ as claimed.

Set $E_Y:=E_1(Y)\oplus E_{-1}(Y)$, and define

\[ W_Y=\{X\in B^{s,t}_{1,-1}:E_\mu(X)=E_\mu(Y)\text{, } \forall\mu\neq \pm 1\}. \]

As $X\in W_Y$ is uniquely determined by its eigenspace $E_X(1)$, $W_Y$ is evidently a manifold that can be identified with $\operatorname {Gr}_{s}(E_Y)=\operatorname {Gr}_s(\mathbb {R}^{s+t})$, in particular $\dim W_Y=st$. By definition, $W_Y\subset B^{r}_0$.

Now because $-P\leq Q\leq P$, the spectrum of $X_0(E)$ lies in $[-1,1]$. By Lemma 5.9 we have $\mathrm {Image}(d_E X_0)\subset T_YB^{s,t}_{1,-1}$.

For $1\leq j\leq \kappa$, choose a smooth curve $\gamma _j(\tau )$ through $E$ given by

\[ \gamma_j(\tau)=\operatorname{Span}(u_1(E),u_2(E),\ldots,\cos\tau u_j(E)+\sin\tau \xi,\ldots,u_\kappa(E)), \]

where $\xi \in E^{P}$ is arbitrary. Observe that $T_E\operatorname {Gr}_{\kappa }(V)=\operatorname {Span}\{\gamma _j'(0):1\leq j\leq \kappa \}$. As the frame is well-adapted to $E$, $u_i(\gamma _j(\tau ))=u_i(E)$ for $i\neq j$, and so $u_j(\gamma _j(\tau ))=\cos \tau u_j(E)+\sin \tau \xi$. One computes

\[ d_E X_0 (\gamma_j'(0))= \begin{pmatrix} 0 & \cdots & 0 & Q(\xi, u_1) & 0 & \cdots & 0 \\ \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & \cdots & 0 & Q(\xi, u_{j-1}) & 0 & \cdots & 0\\ Q(\xi, u_1) & \cdots & Q(\xi, u_{j-1}) & Q(\xi, 2u_j) & Q(\xi, u_{j+1}) & \cdots & Q(\xi,u_\kappa) \\ 0 & \cdots & 0 & Q(\xi, u_{j+1}) & 0 & \cdots & 0\\ \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & \cdots & 0 & Q(\xi, u_\kappa) & 0 & \cdots & 0 \end{pmatrix}. \]

Note that $E\cap (E^{P})^{Q}=(E\cap V_P^{+}) \oplus (E\cap V_P^{-})$. Hence, $Q(\xi, u_1), Q(\xi,u_2),\ldots, Q(\xi,u_{\kappa -s-t})\in (E^{P})^{*}$ are linearly independent functionals, whereas the bottom right $(s+t)\times (s+t)$ minor of $d_E X_0 (\gamma _j'(0))$ vanishes.

Therefore, for $1\leq j\leq \kappa -s-t$, we may choose $\xi _j\in E^{P}$ such that $Q(\xi _j, u_i)=0$ for $1\leq i\leq j-1$, whereas $Q(\xi _j, u_i)$ is arbitrary for $j\leq i\leq \kappa -s-t$. For $\kappa -s-t+1\leq j\leq \kappa$, we may choose $\xi _j\in E^{P}$ to get arbitrary $\kappa -s-t$ first entries in the $j$th row and column. Thus, the entries of a matrix in $\operatorname {Image}(d_E X_0)$ can be made arbitrary outside of the bottom right $(s+t)\times (s+t)$ minor. Consequently, $\mathrm {codim}(\mathrm {Image}(d_E X_0))= {s+t+1\choose 2}$.

We claim that $\mathrm {Image}(d_E X_0)\cap T_YW_Y=\{0\}$. This is because $T_YW_Y$ consists of all matrices that vanish outside of the bottom right $(s+t)\times (s+t)$-minor $M$, which has zeros in its top left $s\times s$ minor and bottom right $t\times t$ minor.

One easily verifies that ${s+1\choose 2}+{t+1\choose 2}+st={s+t+1\choose 2}$, so that

\[ \dim \mathrm{Image}(d_E X_0)+\dim T_{Y}W_Y\geq\dim T_Y B^{s,t}_{1,-1}. \]

Hence,

\[ \mathrm{Image}(d_E X_0)\oplus T_YW_Y=T_Y B^{s,t}_{1,-1}. \]

As $T_YW_Y\subset T_YB^{r}_0$ and $T_YB^{r}_0 + T_Y B^{s,t}_{1,-1}=T_Y\operatorname {Sym}_\kappa (\mathbb {R})$, we conclude that $\mathrm {Image}(d_E X_0)+T_YB^{r}_0=T_Y\operatorname {Sym}_\kappa (\mathbb {R})$ as claimed.

Fix $\epsilon >0$. For $E'$ in a sufficiently small neighborhood $W_E$ of $E$, $\operatorname {Image} d_{E'} X_0$ must contain a subspace that is $\epsilon$-close to $\mathrm {Image}(d_E X_0)$. We may moreover by Lemma 5.8 assume $W_E$ is such that for all $E'\in W_E$, $Y'= X_0(E')$ has nullity $r'\leq r$, and $T_{Y'}B_0^{r'}$ contains a subspace that is $\epsilon$-close to $T_YB_0^{r}$. Thus, for sufficiently small $\epsilon$ we find a neighborhood $W_E$ such that for all $E'\in W_E$ with $Y'= X_0(E')$ of nullity $r'$,

\[ \operatorname{Image}(d_{E'} X_0)+T_{ X_0(E')}B^{r'}_0=\operatorname{Sym}_\kappa(\mathbb{R}). \]

Define

\[ L_{E',Y'}=\mathrm{Ker}\big(d X_0^{*}:T^{*}_{Y'}\operatorname{Sym}_\kappa(\mathbb{R})\to T_{E'}^{*}\operatorname{Gr}_{\kappa}(V)\big). \]

Thus, $N_Y^{*}B_0^{r'}\cap L_{E',Y'}=\{0\}$ for $E'\in W_E$.

By Proposition 5.5, $\operatorname {WF}_Y(f_\lambda )\subset N_{Y'}^{*}B_0^{r'}$. We conclude that $\operatorname {WF}(f_{\lambda })\cap L_{E',Y'}=\emptyset$.

By [Reference DuistermaatDui11, Proposition 1.3.3] $X_0^{*}$ defines a sequentially continuous linear operator on $C^{-\infty }_{N^{*}\Gamma ^{\kappa }}(\operatorname {Sym}_\kappa (\mathbb {R}))$, where $f_\lambda$ lies. Moreover, $X_0^{*}f_\lambda$ must itself be an analytic family: for a smooth compactly supported test measure $\psi$ on $W_{E}$ we have $\langle X_0^{*}f_\lambda, \psi \rangle =\langle f_\lambda, ( X_0)_*\psi \rangle$, and by [Reference Guillemin and SternbergGS77, Chapter VI, Proposition 3.9] we have $\operatorname {WF}(( X_0)_*\psi )\subset \bigcup _{E'\in W_E} L_{E', X_0(E')}$. Now analyticity of a vector-valued function coincides with weak analyticity in quasi-complete locally convex vector spaces, and $C^{-\infty }_{N^{*}\Gamma ^{\kappa }}(\operatorname {Sym}_\kappa (\mathbb {R}))$ is quasi-complete [Reference Dabrowski and BrouderDB14, Proposition 29]. Furthermore $(X_0)_*\psi$ defines a continuous linear functional on $C^{-\infty }_{N^{*}\Gamma ^{\kappa }}(\operatorname {Sym}_\kappa (\mathbb {R}))$ by [Reference Dabrowski and BrouderDB14, Lemma 3], confirming that the family of generalized functions $X_0^{*}f_\lambda \in C^{-\infty }(W_E)$ is analytic.

Now observe that the continuous functions $X_0^{*}f_{\lambda }$ defined separately for $W_{E_1}$ and $W_{E_2}$ coincide on non-empty intersections $W_{E_1}\cap W_{E_2}$ for $\mathrm {Re}\,\lambda >0$ by (17). If follows by uniqueness of the analytic extension that this holds for all $\lambda \in \mathbb {C}$. In particular, $\widetilde m^{0}_k$ is a globally well-defined distribution.

For invariance, we note that for $g\in \operatorname {O}(Q)$, $g^{*}X_0^{*}f_\lambda =\psi _g(E)^{\lambda } X_0^{*}f_\lambda$ for $\lambda >0$, and consequently by uniqueness of analytic extension for all $\lambda \in \mathbb {C}$. Here $\psi _g(E)=\operatorname {Jac}(g:E\to gE)^{-2}$, which for $g\in \operatorname {O}(Q)$ satisfies $\psi _g(E)={\det X_0(gE)}/{\det X_0(E)}$, see [Reference Bernig and FaifmanBF17, Proposition 4.7] for details. Using the identification

\[ \operatorname{Dens}(T_E\operatorname{Gr}_{n+1-k}(V))=\operatorname{Dens}(E^{*}\otimes V/E)=\operatorname{Dens}^{*}(E)^{\otimes (n+1)}\otimes\operatorname{Dens}(V)^{\otimes n+1-k}, \]

it follows that $X_0^{*}f_\lambda \,dE$ is an $\operatorname {O}(Q)$-invariant distribution on $\operatorname {Gr}_{n+1-k}(V)$ when $\lambda =-({n+1})/{2}$.

Henceforth whenever $X_0$ appears, a local well-adapted frame should be chosen arbitrarily unless an explicit choice is provided.

5.4 Some properties of the invariant distributions

We use the following rescaling of the invariant distribution constructed previously, which brings the total integral to $1$ as we show later.

Definition 5.11 Set

\[ m_k:= e^{({\mathbf{i}\pi}/{2}) (n+1-k)q} \widetilde m_k^{0} \in\mathcal{M}^{-\infty}(\operatorname{Gr}_{n+1-k}(V))^{\operatorname{O}(Q)}. \]

Lemma 5.12 Let $j\colon \mathbb {R}^{p,q}\to \mathbb {R}^{q,p}$ be given by $j(x,y)=(y,x)$ where $x\in \mathbb {R}^{p},y\in \mathbb {R}^{q}$. Let us also denote by $j$ the induced map $\operatorname {Gr}_{p+q-k}(\mathbb {R}^{p,q})\to \operatorname {Gr}_{p+q-k}(\mathbb {R}^{q,p})$. Then

\[ j^{*}m_k=\overline{m_k}. \]

Proof. We have $m_k=\mathbf {i}^{(n+1-k)p}{X_0^{*}f_{-({n+1})/2}}\,d\sigma _{P_0}$ on $\mathbb {R}^{q,p}$. As $X_{ 0} \circ j=- X_{ 0}$ (where $X_0$ is defined using $j$-corresponding frames), (18) implies that $j^{*}X_{ 0}^{*}f_{-({n+1})/2}=\mathbf {i}^{-(n+1-k)(n+1)}\overline { X_{ 0}^{*}f_{-({n+1})/2}}$. It follows that

\begin{align*} j^{*}m_k&=\mathbf{i}^{(n+1-k)p-(n+1-k)(n+1)} \overline{ X_{ 0}^{*}f_{-({n+1})/2}}\,d\sigma_{P_0}\\ &=\mathbf{i}^{-(n+1-k) q}\overline{ X_{ 0}^{*}f_{-({n+1})/2}}\,d\sigma_{P_0}, \end{align*}

which is the conjugate of $m_k$ in $\mathbb {R}^{p,q}$.

Proposition 5.13 Define

\begin{align*} N^{*}\Lambda:=\bigcup_{\nu\geq 1}N^{*}\Lambda^{\nu}_{n+1-k}(V). \end{align*}
  1. (i) The wave front set of $m_k$ is contained in $N^{*}\Lambda$.

  2. (ii) We have $m^{\mathbf {i}\epsilon }_k\to m_k$ in $\mathcal {M}_{N^{*}\Lambda }^{-\infty }(\operatorname {Gr}_{n+1-k}(V))$ as $\epsilon \to 0^{+}$ in the normal Hörmander topology.

Proof. Write $\lambda =-({n+1})/{2}$. For $\zeta \in U_{\mathbb {C}}$ we have

\[ (2\zeta+1)^{-(({n+1-k})/{2})p}(2\zeta-1)^{-(({n+1-k})/{2})q}m^{\zeta}_k(E)= \det (X^{P_0}_\zeta)^{\lambda}\,dE. \]

We compute, using a well-adapted frame,

\[ \det(X^{P_0}_{\mathbf{i}\epsilon}(E))^{\lambda} = \det(X_{0}(E)+2\mathbf{i}\epsilon I_{n+1-k} )^{\lambda}=X_0^{*}\det(X+2\mathbf{i}\epsilon I_{n+1-k})^{\lambda}. \]

By Proposition 5.5, we have $\det (X+2\mathbf {i}\epsilon I_{n+1-k})^{\lambda }\to f_{\lambda }(X)$ in the normal Hörmander topology on $C^{-\infty }_{\Gamma ^{n+1-k}}(\operatorname {Sym}_{n+1-k}(\mathbb {R}))$. By the proof of Proposition 5.10, we may use the continuity of the pull-back $X_0^{*}$ in the normal Hörmander topology [Reference Brouder, Dang and HéleinBDH16]. Noting that $X_0^{-1}(\Gamma ^{n+1-k}\nu )\subset \Lambda _{n+1-k}^{\nu }(V)$ so that $X_0^{*}N^{*}\Gamma ^{n+1-k}\subset N^{*}\Lambda$, we find that

\[ X_0^{*}\det(X+2\mathbf{i}\epsilon I_{n+1-k})^{\lambda} \,dE\to \widetilde m_k^{0} \]

in the normal Hörmander topology as stated.

Corollary 5.14 Let $M \subset V^{n+1}$ be a pseudosphere or a pseudohyperbolic space, and $A\subset M$ either a smooth domain with LC-regular boundary, or a smooth LC-regular hypersurface without boundary. Assume all $Q$-degenerate tangents to $A$ of codimension $k$ are regular. Then

(20)\begin{equation} \operatorname{Cr}(m_k)(A)=\int_{\operatorname{Gr}_{n+1-k}(V)}\chi(A\cap E)\,dm_k(E). \end{equation}

Proof. First note that $\operatorname {Cr}(m_k)\in \mathcal {V}^{-\infty }(M)$ is isometry invariant and by [Reference Bernig, Faifman and SolanesBFS21, Theorem C] is given by a linear combination of the intrinsic volumes. Thus, $A$ is WF-transversal to $\operatorname {Cr}(m_k)$. The assertion now follows from Corollary 4.7, and Propositions 5.13(i) and 3.14.

Corollary 5.15 Let $M^{n}\subset V^{n+1}$ be a pseudosphere or a pseudohyperbolic space, and $A\subset M$ either a smooth domain or a hypersurface without boundary. Denote $H=H(A)$.

  1. (i) Assume that for each $x\in H$, $H$ is either pseudo-Riemannian at $x$ or tangentially regular at $x$. Then (20) holds for $k=1$.

  2. (ii) If $\mathbb {P}(H)\subset \mathbb {P}(V)$ is strictly convex, then (20) holds for all $k$.

Proof. In both cases, it follows from [Reference Bernig, Faifman and SolanesBFS22, Lemma 4.7] that $H$ is LC-regular, and we can apply Corollary 5.14.

Example The complex-valued distribution $m_{n}\in \mathcal {M}^{-\infty }(\mathbb {P}(V))$ is invariant under the group of projective transformations preserving the quadric $[Q]=\{Q=0\}$. Its singular support is $[Q]$, and $\operatorname {WF}(m_{n})\subset N^{*}[Q]$. In particular, $m_{n}(A)$ is well-defined for any domain $A\subset \mathbb {P}(V)$ that is smooth near $[Q]$ and transversal to it. When $Q$ is definite, the quadric $[Q]$ has no real points and $m_{n}$ is the Haar measure on the round projective space.

5.5 The flat case

Next we construct a translation- and $\operatorname {O}(p,q)$-invariant distribution on the affine Grassmannian $\overline{\operatorname{Gr}}_{p+q-k}(\mathbb {R}^{p,q})$.

Proposition 5.16 Let $P$ be a $Q$-compatible Euclidean structure in $W=\mathbb {R}^{p+1,q}=W_+\oplus W_-$. Let $x\in W_+\cap S^{p,q}$, $T=T_xS^{p,q}$, and define

\[ s\colon\overline{\operatorname{Gr}}_{p+q-k}(T) \longrightarrow \operatorname{Gr}_{p+q+1-k}(W) ,\quad s(v+F)=F\oplus \mathbb{R}(x+v), {F\in\operatorname{Gr}_{p+q-k}(T)}, \]

which is a diffeomorphism onto its open image. Given $t>0$, consider the homothety $v\mapsto tv$ on $T$ and the induced map $h_t$ on $\overline{\operatorname{Gr}}_{p+q-k}(T)$.

  1. (i) Let $dE$ be an $\operatorname {O}(P)$-invariant measure on $\operatorname {Gr}_{p+q+1-k}(V)$, thus given by a smooth density. Then

    \[ d\overline F=\frac 1{k!} \frac{d^{k}}{dt^{k}}\biggr|_{t=0}h_{t}^{*} s^{*}\,dE \]
    is an $\overline {\operatorname {O}(P|_T)}$-invariant measure on $\overline{\operatorname{Gr}}_{p+q-k}(T)$.
  2. (ii) Let $X_0\colon \operatorname {Gr}_{p+q+1-k}(W)\to \operatorname {Sym}_{p+q+1-k}(\mathbb {R})$ be as in Proposition 5.10, and let $X_0'$ be the corresponding map on $\operatorname {Gr}_{p+q-k}(T)$. Then

    \[ \frac 1{k!}\frac{d^{k}}{dt^{k}}\biggr|_{t=0} (h_{1/t})_* (s^{-1})_* ( X_0^{*}f_{-({p+q+1})/2}(E)\,dE)= (X_0')^{*} f_{-({p+q+1})/2}(F) \,d\overline F , \]
    and this generalized measure is $\overline {\operatorname {O}(Q|_T)}$-invariant. Here $(s^{-1})_*$ denotes the push-forward by $s^{-1}$ of the restriction to the open set $\mathrm {Image}(s)$.

Proof. (i) For $g \in \operatorname {O}(P|_T)=\operatorname {Stab}_{\operatorname {O}(P)}(x)\subset \operatorname {O}(P)$, because $g\circ s\circ h_t=s\circ h_t\circ g$ and $g^{*} \,dE=dE$, we have $g^{*}h_{t}^{*} s^{*}\,dE=h_t^{*} s^{*}g^{*}\,dE=h_t^{*} s^{*}\,dE$, which yields $\operatorname {O}(P|_T)$-invariance. As for translation invariance, let $\rho _U\colon U\times \mathbb {R}^{k}\to \overline {\operatorname {Gr}}_{p+q-k}(T)$ be a local trivialization of the bundle $\pi \colon \overline {\operatorname {Gr}}_{p+q-k}(T)\to {\operatorname {Gr}_{p+q-k}}(T)$, and put $\eta =\rho _U^{*}s^{*}\,dE$. Then $h_t\circ \rho _U(F,w)=\rho _U(F,tw)$ and, thus,

\[ \rho_U^{*}h_t^{*}s^{*}(dE)_{(F,w)}=t^{k}\eta_{(F,tw)}=t^{k}\eta_{(F,0)} +O(t^{k+1}). \]

As the induced action of a translation of $T$ on $U\times \mathbb {R}^{k}$ has the form $(F,w)\mapsto (F, w+\varphi (F))$, the translation invariance of $\overline {dF}$ follows.

(ii) Let $f_1,\ldots, f_{p+q-k}$ be a $P$-orthonormal basis of $F\in \operatorname {Gr}_{p+q-k}(T)$, and let $w\in T$ be $P$-orthogonal to $F$. A $P$-orthonormal basis of $s(tw+F)$ is $(1+t^{2}P(w))^{-1/2}(x+tw), f_1,\ldots, f_{p+q-k}$. Hence, the Gram matrices $X_0'$ and $X_0$ of $Q$ restricted to $F$ and $s(tw+F)$, respectively, satisfy

\[ \det X_0=(1+t^{2}P(w))^{-1}Q(x+tw)\det X_0'. \]

Therefore, for $\lambda >0$, $F\in \operatorname {Gr}_{p+q-k}(T)$ and $w\in F^{P}\cap T$ we have

\[ f_\lambda(X_0(s(tw+F)))=(1+t^{2}P(w))^{-\lambda}Q({x+tw})^{\lambda} f_\lambda(X_0'(F)) \]

whenever $Q(x+tw)>0$, where $f_\lambda$ is defined by Lemma 5.3 on $\operatorname {Sym}_{p+q+1-k}(\mathbb {R})$ or $\operatorname {Sym}_{p+q-k}(\mathbb {R})$ depending on the argument.

By analytic continuation we get

\[ s^{*} X_0^{*}f_{\lambda}(tw+F)=(1+t^{2}P(w))^{-\lambda}Q({x+tw})^{\lambda} (X_0')^{*}f_\lambda(tw+F) \]

for all $\lambda$. Hence,

\[ \lim_{t\to 0} h_t^{*} s^{*} X_0^{*} f_\lambda= ( X_0')^{*} f_\lambda. \]

In the proof of part (i) we have seen $(h_{1/t})_*(s^{-1})_*\,dE=O(t^{k})$. Hence, by continuity

\begin{align*} &\frac{d^{k}}{dt^{k}}\biggr|_{t=0} (h_{1/t})_* (s^{-1})_* ( X_0^{*}f_{-({p+q+1})/2}(E)\,dE)\\ &\qquad =\lim_{t\to 0}h_t^{*}s^{*} X_0^{*} f_{-({p+q+1})/2}\frac{d^{k}}{dt^{k}} \biggr|_{t=0} (h_{1/t})_* (s^{-1})_* \,dE\\ &\qquad =k!( X_0')^{*} f_{-({p+q+1})/2} \,d\overline F. \end{align*}

Translation invariance is clear. Further, if $g\in \operatorname {O}(Q|_T)\subset \operatorname {O}(Q)$, then $g\circ s\circ h_t=s\circ h_t\circ g$. As $X_0^{*}f_{-({p+q+1})/2}(E)\,dE$ is $\operatorname {O}(Q)$-invariant, this yields $\operatorname {O}(Q|_T)$-invariance.

Translation-invariance and ${\operatorname {O}(P|_T)}$-invariance characterize $d\overline F$ uniquely up to normalization. The normalization can be deduced from Theorem 6.5 in the case $q=0$. As for the translation-invariant and $\operatorname {O}(Q|_T)$-invariant generalized measure obtained on $T\cong \mathbb {R}^{p,q}$, we take the normalization of Definition 5.11 as follows.

Definition 5.17 On $\overline{\operatorname {Gr}}_{p+q-k}(\mathbb {R}^{p,q})$ we fix the following translation-invariant and ${\operatorname {O}(p,q)}$-invariant generalized measure

\[ \check{m}_k:=e^{({\mathbf{i}\pi}/2) (p+q+1-k)q} (X_0')^{*}f_{-({p+q+1})/{2}}(F) \,d\overline F. \]

The Crofton map in the flat case is

\[ \operatorname{Cr}: \mathcal{M}^{-\infty}(\overline{\operatorname{Gr}}_{p+q-k}(V))\to \mathcal{V}^{-\infty}(V), \]

given by $\langle \operatorname {Cr}(\mu ), \psi \rangle = \int _{\overline {\operatorname {Gr}}_{p+q-k}(V)}\psi (\bar {E})\,d\mu (\bar {E})$ for all $\psi \in \mathcal {V}_c^{\infty }(V)$.

The results of the present section and §§ 3 and 4 can be easily adapted to the flat pseudo-Euclidean setting. Let us state explicitly Corollary 5.15 in the flat case.

Corollary 5.18 Let $A\subset \mathbb {R}^{p,q}$ be either a smooth domain or a hypersurface without boundary. Denote by $H=H(A)$ the corresponding closed hypersurface.

  1. (i) Assume that for each $x\in H$, $H$ is either pseudo-Riemannian near $x$ or has non-zero Gauss curvature at $x$. Then

    (21)\begin{equation} \operatorname{Cr}(\check m_k)(A)=\int_{\overline{\operatorname{Gr}}_{p+q-k}(V)}\chi(A\cap E)\,d\check m_k(E). \end{equation}
    holds for $k=1$.
  2. (ii) If $H$ is strictly convex, then (21) holds for all $k$.

6. Crofton formulas for generalized pseudospheres

For the de Sitter space embedded in Lorentz space, one can compute the Crofton formulas through a direct computation of the restriction of the measures to subspaces, combined with the Hadwiger theorem and the template method. However, for general signatures, an explicit computation appears to be hard. Instead, we carry out an analytic extension argument, which recovers the Crofton formulas for all signatures in a unified fashion.

For $\zeta >\tfrac 12$, we denote by $S_\zeta ={Q_\zeta ^{-1}(1)}\subset \mathbb {R}^{p+q}=\mathbb {R}^{n+1}$ the unit sphere in the Euclidean space $(\mathbb {R}^{n+1},Q_\zeta )$. For $\zeta =0$ we have $S^{p-1,q}=Q_0^{-1}(1)$ with the induced pseudo-Riemannian metric $Q_{0}$. We also denote by $S^{n}$ the unit sphere in $\mathbb {R}^{n+1}$ with respect to some fixed Euclidean structure (which is independent of $\zeta$).

In the following we make use of the operation of restriction of Crofton distributions, as described in § 3.1.

Proposition 6.1 For the standard inclusion $e\colon \mathbb {R}^{p,q}\hookrightarrow \mathbb {R}^{p+l,q+r}$, we have $e^{*}m_k=m_k$.

Proof. Note first that the restriction $e^{*}m_k$ is well-defined by [Reference FaifmanFai17, Remark 2.13]. For $\zeta >\tfrac 12$, $Q_\zeta$ is positive-definite, and so $e^{*}m_k^{\zeta }=m_k^{\zeta }$ by the uniqueness of probability measure on the Grassmannian invariant under the positive-definite orthogonal group, as

\[ e^{*}:\mathcal{M}^{\infty}(\operatorname{Gr}_{p+q+l+r-k}(\mathbb{R}^{p+l,q+r}))\to \mathcal{M}^{\infty}(\operatorname{Gr}_{p+q-k}(\mathbb{R}^{p,q})) \]

is essentially the push-forward operation under intersection with $\mathbb {R}^{p,q}$. By analytic extension in $\zeta$, we get $e^{*}m_k^{\mathbf {i} \epsilon } = m_k^{\mathbf {i} \epsilon }$. The statement then follows from Proposition 5.13(ii).

Proposition 6.2 Given $A\in \mathcal {P}( S^{p-1,q})$, let $\overline A\in \mathcal {P}(S^{n})$ be its radial projection. Assume $A$ is either an LC-regular hypersurface, or a smooth domain with LC-regular boundary. Assume further that either all $Q_0$-degenerate tangents of codimension $k$ are regular, or $\chi (A\cap E)$ is constant for almost every plane $E$ of codimension $k$. Then

\[ \lim_{\epsilon\to 0^{+}} \operatorname{Cr}_{S^{n}}(m_k^{\mathbf{i}\epsilon})(\overline A)=\operatorname{Cr}_{S^{p-1,q}}(m_k)(A). \]

Proof. We have by Proposition 3.14

\[ \operatorname{Cr}_{S^{n}}(m_k^{\mathbf{i}\epsilon})(\overline A)=\langle m_k^{\mathbf{i}\epsilon}, \chi(\overline A\cap \bullet)\rangle. \]

By Corollary 5.14, it holds that

\[ \operatorname{Cr}_{S^{p-1,q}}(m_k)(A)=\langle m_k, \chi(A\cap \bullet)\rangle. \]

By Proposition 5.13(ii), $m_k^{\mathbf {i} \epsilon }$ tends to $m_k$ as $\epsilon \to 0^{+}$ in the normal Hörmander topology on $\mathcal {M}^{-\infty }_{N^{*}\Lambda }(\operatorname {Gr}_{n+1-k}(V))$. Combining Corollary 4.7 and Proposition 5.13(i), we see that evaluating at $\chi (\overline A\cap \bullet )=\chi ( A\cap \bullet )$ is continuous in this topology, and the statement follows.

We consider for a moment the case $q=1$. We use two types of templates in the de Sitter sphere $S^{p-1,1}$. The first is the Riemannian $(p-1)$-unit sphere

\begin{align*} R^{p-1,0}=S^{p-1,1}\cap \{x_{p+1}=0\}. \end{align*}

Fix $\theta \in (0,\pi / 4)$. Our second template is

\[ R^{p-1,1}=R^{p-1,1}(\theta)=\{x\in S^{p-1,1}\colon x_{p+1}^{2} \leq \tan^{2}\theta (x_1^{2}+\cdots+x_{p}^{2})\}. \]

The points of $\partial R^{p-1,1}$ lie at (time-like) distance of $\rho =\mathrm {arctanh}(\tan \theta )$ from $R^{p-1,0}$.

For each $\zeta >\tfrac 12$ and $s=0,1$, we denote by $T_\zeta ^{p-1,s}$ the radial projection of $R^{p-1,s}$ on $S_\zeta$. Thus, $T_\zeta ^{p-1,0}$ is a totally geodesic $(p-1)$-sphere in $S_\zeta$, and the points of $\partial T_\zeta ^{p-1,1}$ lie at distance $\varepsilon =\arctan (\sqrt \xi \tan \theta )$ from $T_\zeta ^{p-1,0}$ where $\xi =({2\zeta -1})/({2\zeta +1})$. We then have

\[ \frac{d\rho}{d\theta}=\frac{1+\tan^{2} \theta}{1-\tan^{2} \theta}, \quad \frac{d\epsilon}{d\theta}=\sqrt{\xi} \frac{1+\tan^{2} \theta}{1+\xi\tan^{2} \theta}. \]

We denote by $\mu _k^{\zeta }\in \mathcal {V}^{\infty }(S_\zeta )$ the Riemannian intrinsic volumes in $S_\zeta$, and by $\mu _k\in \mathcal {V}^{-\infty }(S^{p-1,q})\otimes \mathbb {C}$ the (complex-valued) intrinsic volumes on $S^{p-1,1}$. Note that $\mu _k^{\zeta }(T_\zeta ^{p-1,0})=\mu _k(R^{p-1,0})$ for all $\zeta >\tfrac 12$.

Proposition 6.3 For $s=0,1$, the function $\zeta \mapsto \mu _k^{\zeta }(T_\zeta ^{p-1,s})$ extends to a holomorphic function $f_{k,s}(\zeta )$ on $U_\mathbb {C}$ such that $\lim _{\zeta \to 0}f_{k,s}(\zeta )=\mu _k(R^{p-1,s})$.

Proof. For $s=0$, the statement is trivial as $\mu _k^{\zeta }(T_\zeta ^{p-1,0})$ does not depend on $\zeta$. Let us consider $s=1$. The radial projections $\pi _\zeta :S^{n}\to S_\zeta$ and $\pi _{0}:S^{n}\to S^{p-1,1}$ have Jacobians

\begin{align*} \mathrm{Jac}\,\pi_\zeta &= \biggl(\frac{\cos\epsilon}{\cos\theta}\biggr)^{p-1}\frac{d\epsilon}{d\theta} =\xi^{1/2}\biggr(\frac{1+\tan^{2}\theta}{1+\xi\tan^{2}\theta}\biggr) ^{({p-1})/2+1}, \quad \xi=\xi(\zeta)=\frac{2\zeta-1}{2\zeta+1},\\ \mathrm{Jac}\,\pi_{0} &= \biggl(\frac{\cosh\rho}{\cos\theta}\biggr)^{p-1} \frac{d\rho}{d\theta}=\biggl(\frac{1+\tan^{2}\theta}{1-\tan^{2}\theta}\biggr)^{({p-1})/2+1}. \end{align*}

As $\xi (\zeta )=({2\zeta -1})/({2\zeta +1})$ is continuous on $\mathbb {C}\setminus \{-\tfrac 12\}$, it maps $U_\mathbb {C}$ to a simply connected region in $\mathbb {C}\setminus \{0\}$. Moreover, $\xi (\zeta )\in \mathbb {R}$ if and only if $\zeta \in \mathbb {R}$, and $\xi (\zeta )>0$ for $\zeta >\tfrac 12$, so that $\xi (\zeta ),1+\xi (\zeta )\tan ^{2}\theta \neq 0$ for $\zeta \in U_\mathbb {C}$. It follows that the right-hand side of the first equation extends to a holomorphic function in $U_\mathbb {C}$ whose limit as $\zeta \to 0$ equals the right-hand side of the second equation multiplied by $\mathbf {i}$. The statement follows for $k=p$ because $\mu _{p}^{\zeta }=\operatorname {vol}_{p}$, and $\mu _{p}=\mathbf {i} \operatorname {vol}_{p}$ on $S^{p-1,1}$.

Now consider $k=p-1$. As

\[ \mu_{p-1}(R^{p-1,1})=\tfrac 12\operatorname{vol}_{p-1}(\partial R^{p-1,1}),\quad \mu_{p-1}(T_\zeta^{p-1,1})=\tfrac 12\operatorname{vol}_{p-1}(\partial T_\zeta^{p-1,1}),\]

and

\begin{align*} \frac{d}{d\theta}\operatorname{vol} (R^{p-1,1}(\theta))&=\frac{d\rho}{d\theta}\frac{d}{d\rho}\operatorname{vol} (R^{p-1,1}(\theta))=\frac{1+\tan^{2}\theta}{1-\tan^{2}\theta}\operatorname{vol}_{p-1}(\partial R^{p-1,1}),\\ \frac{d}{d\theta}\operatorname{vol} (T_\zeta^{p-1,1}(\theta)) &=\frac{d\epsilon}{d\theta}\frac{d}{d\epsilon}\operatorname{vol} (T_\zeta^{p-1,1}(\theta)) =\sqrt\xi\frac{1+\tan^{2}\theta}{1+\xi\tan^{2}\theta}\operatorname{vol}_{p-1}(\partial T_\zeta^{p-1,1}), \end{align*}

this case follows from the previous one.

For $(k-p-1)$ positive and odd, because $N^{*}R^{p-1,1}$ is contained in the time-like orbit of the cosphere bundle of $S^{p-1,1}$, we have

(22)\begin{align} \mu_k(R^{p-1,1}) & = \sum_\nu \mathbf{i}^{p-1-k-2\nu}c_{p,k,\nu} [[0,\phi_{k+2\nu,\nu}^{-}]](R^{p-1,1}) +\mathbf{i} d_{p,k} \operatorname{vol}(R^{p-1,1}), \end{align}
(23)\begin{align} \mu_k^{\zeta}(T_\zeta^{p-1,1}) & = \sum_\nu c_{p,k,\nu} [[0,\phi_{k+2\nu,\nu}^{\zeta}]] (T_\zeta^{p-1,1}) +d_{p,k} \operatorname{vol}(T_\zeta^{p-1,1}), \end{align}

for certain constants $c_{p,k,\nu },d_{p,k}$, where $\phi _{k,r}^{-}$ is the smooth form given in Lemma 5.1 of [Reference Bernig, Faifman and SolanesBFS22] when $M=S^{p-1,1}$, and $\phi _{k,r}^{\zeta }$ is the form $\phi _{k,r}^{+}$ in the same lemma when $M=S_\zeta$. For $k-p-1\geq 0$ and even, (22) and (23) hold with the volume term removed.

As $S_\zeta$ and $S^{p-1,q}$ have constant curvature 1, we have $\phi _{k,r}^{-}=\phi _{k,0}^{-}$ and $\phi _{k,r}^{\zeta }=\phi _{k,0}^{\zeta }$. By the structure equations (see [Reference Bernig, Faifman and SolanesBFS22, equations (32) and (33)]) we have

\begin{align*} d\phi_{k,0}^{-}&=\theta_0 \wedge (- k\phi_{k-1,0}^{-} -(p-1-k) \phi_{k+1,0}^{-}),\\ d\phi_{k,0}^{\zeta}&=\theta_0 \wedge (k\phi_{k-1,0}^{\zeta} -(p-1-k) \phi_{k+1,0}^{\zeta}), \end{align*}

where $\theta _0$ is the contact 1-form defined by the pseudo-Riemannian metric.

Now take $M=S^{p-1,1}$ and assume $\omega \in \Omega ^{\dim M-1}(\mathbb {P}_M)$, and $d\omega =\theta _0\wedge \omega '$. Let $\nu :\partial R^{p-1,1}(\theta )\to \mathbb {P}_M$ be the outer normal map, and extend it smoothly to $M$. We then have

\begin{align*} \frac{d}{d\theta}[[{ 0,}\omega]](R^{p-1,1}(\theta)) & =\frac{d}{d\theta} \left\langle \omega, [\kern-1pt[ N^{*}R^{p-1,1}(\theta) ]\kern-1pt] \right\rangle\\ & = \frac{d}{d\theta} \left\langle \nu^{*}\omega, [\kern-1pt[ \partial R^{p-1,1}(\theta) ]\kern-1pt] \right\rangle\\ &=\frac{d}{d\theta} \left\langle \nu^{*}\theta_0\wedge \nu^{*}\omega', [\kern-1pt[ R^{p-1,1}(\theta) ]\kern-1pt] \right\rangle\\ & =\biggl\langle \nu^{*}\theta_0\wedge \nu^{*}\omega', \frac{\partial}{\partial \theta}\cdot [\kern-1pt[ \partial R^{p-1,1}(\theta) ]\kern-1pt] \biggr\rangle\\ & = \biggl\langle \nu^{*}\theta_0, \frac{\partial}{\partial \theta} \biggr\rangle \cdot \left\langle \nu^{*}\omega', [\kern-1pt[ \partial R^{p-1,1}(\theta) ]\kern-1pt] \right\rangle\\ & =\frac{d\rho}{d\theta} \cdot [[0,\omega']] (R^{p-1,1}(\theta)). \end{align*}

Hence, $({d}/{d\theta })[[0,\phi ^{-}_{k,0}]] (R^{p-1,1}(\theta ))$ equals

\[ \frac{1+\tan^{2}\theta}{1-\tan^{2}\theta}\Bigl(- k[[0,\phi_{k-1,0}^{-}]] (R^{p-1,1}(\theta)) -(p-1-k) [[0,\phi_{k+1,0}^{-}]] (R^{p-1,1}(\theta))\Bigr), \]

and similarly $({d}/{d\theta })[[0,\phi _{k,0}^{\zeta }]] (T_\zeta ^{p-1,1}(\theta ))$ is

\[ \sqrt\xi \frac{1+\tan^{2}\theta}{1+\xi\tan^{2}\theta} \Bigl(k[[0,\phi_{k-1,0}^{\zeta}]](T_\zeta^{p-1,1}(\theta)) -(p-1-k) [[0,\phi_{k+1,0}^{\zeta}]] (T_\zeta^{p-1,1}(\theta))\Bigr). \]

It follows by induction on $k=p,\ldots,0$ that $[[0,\phi _{k,0}^{\zeta }]](T_\zeta ^{p-1,1}(\theta ))$ is holomorphic in $\zeta \in U_\mathbb {C}$ and

(24)\begin{equation} \lim_{\zeta\to 0} [[0,\phi_{k,0}^{\zeta}]](T_{\zeta}^{p-1,1}(\theta))=\mathbf{i}^{p-1-k} [[0,\phi_{k,0}^{-}]](R^{p-1,1}(\theta)). \end{equation}

By (22) and (23) this completes the proof.

In order to normalize the leading coefficient in the Crofton formulas, we rescale the measures $m_k, \check m_k$ as follows.

Definition 6.4 Let $M\subset \mathbb {R}^{p,q}$ be the pseudosphere of curvature $\sigma >0$, or the pseudohyperbolic space of curvature $\sigma <0$. We define

\[ \operatorname{Cr}_k^{M}={\pi\omega_{k-1}}\sqrt{\sigma^{-1}}^{k}\operatorname{Cr}_M(m_k). \]

In the flat pseudo-Euclidean space $M=\mathbb {R}^{p,q}$ we take

\[ {\operatorname{Cr}_k^{M}={\pi\omega_{k-1}}\operatorname{Cr}_M(\check m_k).} \]

Theorem 6.5 (Crofton formula)

Let $M$ be a pseudosphere, a pseudohyperbolic space, or a pseudo-Euclidean space. Then, independently of the signature of $M$,

(25)\begin{equation} {\operatorname{Cr}_k^{M}= \sum_{j=0}^{\lfloor({n-k})/{2}\rfloor}\frac {\omega_{k-1}}{\omega_{k+2j-1}} {-{k}/{2} \choose j}\sigma^{j} \mu_{k+2j}}, \end{equation}

where $\sigma$ is the sectional curvature of $M$ and $n$ its dimension.

Proof. Take first the pseudosphere $M=S^{p-1,q}$ of curvature $\sigma =1$. We can assume $q>0$ as the formula is known in $S^{n}$ (cf., e.g., [Reference Fu and WannererFW19]). We know that

\[ \operatorname{Cr}_k^{M}=\sum_{j=0}^{\lfloor({n-k})/2\rfloor} (a_{j,p,q}\mu_{k+2j}+b_{j,p,q}\overline{\mu_{k+2j}}) \]

for certain coefficients $a_{j,p,q},b_{j,p,q}\in \mathbb {C}$. Indeed, by [Reference Bernig, Faifman and SolanesBFS21, Theorem C] we may express $\operatorname {Cr}_k^{M}$ as a linear combination of the intrinsic volumes and their complex conjugates. As both $\mu _r$ and $\operatorname {Cr}_r^{M}$ are the restrictions of elements in $\operatorname {Val}_r^{-\infty,+}$ and thus belong to the $(-1)^{r}$-eigenspace of the Euler–Verdier involution, only the displayed terms appear.

Let $e\colon S^{p-1,q}{\hookrightarrow }S^{p-1+l,q}$ and $\tilde e\colon S^{p-1+l,1}{\hookrightarrow }S^{p-1+l,q}$ be standard inclusions. By Proposition 6.1, we have

\begin{align*} \sum_{j=0}^{\lfloor({p-1+q-k})/2\rfloor} a_{j,p,q}\mu_{k+2j}+b_{j,p,q}\overline{\mu_{k+2j}} &= \operatorname{Cr}_k^{S^{p-1,q}}=e^{*}(\operatorname{Cr}_k^{S^{p-1+l,q}})\\ &=\sum_{j=0}^{\lfloor({p-1+q-k})/2\rfloor} a_{j,p+l,q}\mu_{k+2j}+b_{j,p+l,q}\overline{\mu_{k+2j}},\\ \sum_{j=0}^{\lfloor({p+l-k})/2\rfloor} a_{j,p+l,1}\mu_{k+2j}+b_{j,p+l,1}\overline{\mu_{k+2j}}&= \operatorname{Cr}_k^{S^{p-1+l,1}}=\tilde e^{*}(\operatorname{Cr}_k^{S^{p-1+l,q}})\\ &=\sum_{j=0}^{\lfloor({p+l-k})/2\rfloor} a_{j,p+l,q}\mu_{k+2j}+b_{j,p+l,q}\overline{\mu_{k+2j}}. \end{align*}

By the linear independence of $\{\mu _{i}\}_i\bigcup \{\overline \mu _i\}_i$ [Reference Bernig, Faifman and SolanesBFS22, Corollary 7.4], and taking $l\geq q-1$, this yields

\begin{align*} a_{j,p,q}&=a_{j,p+l,q}=a_{j,p+l,1},\\ b_{j,p,q}&=b_{j,p+l,q}=b_{j,p+l,1} \end{align*}

for all $j\leq ({p-1+q-k})/2$.

It suffices then to determine $a_j:=a_{j,p,1},b_j:=b_{j,p,1}$, i.e. to prove the statement in the de Sitter sphere $M=S^{p-1,1}$. To this end, we evaluate both sides on the templates $R^{p-1,s}\subset M$ with $s=0,1$. In order to compute $\operatorname {Cr}_k^{M}(R^{p-1,s})$ we use the spherical Crofton formula:

(26)\begin{equation} \operatorname{Cr}_{S_\zeta}(\pi\omega_{k-1}m_k^{\zeta})= \sum_{j\geq 0}\frac{\omega_{k-1}} {\omega_{k+2j-1}} {-{k}/{2} \choose j}\mu^{\zeta}_{k+2j}=:\sum_{j\geq 0}c_{j}\mu^{\zeta}_{k+2j}, \end{equation}

for $\zeta >\tfrac 12$. Given $p\geq k$ and $s=0,1$, let $S^{p}$ be the unit sphere of an arbitrary Euclidean structure in $\mathbb {R}^{p,1}=\mathbb {R}^{p+1}$ and let $T^{p-1,s}$ be the radial projection on $S^{p}$ of $R^{p-1,s}$. By Definition 6.4, Proposition 6.2, and applying analytic continuation to (26) via Proposition 6.3, we have

\begin{align*} \operatorname{Cr}_k^{M}(R^{p-1,s}) &={\pi \omega_{k-1}}\lim_{\epsilon\to 0^{+}} \operatorname{Cr}_{S^{p}}(m_k^{\mathbf{i}\epsilon})(T^{p-1,s})\\ &= \lim_{\epsilon\to 0^{+}} \sum_{j\geq 0} c_j f_{k+2j,s}(\mathbf{i}\epsilon)\\ &= \sum_{j\geq 0} c_j \mu_{k+2j}(R^{p-1,s}). \end{align*}

Now, for $s=0,1$, taking $p=k+2l-s+1$, we obtain

\begin{align*} \sum_{j\geq 0} c_j \mu_{k+2j}(R^{k+2l-s,s})&=\sum_{j\geq 0} a_j \mu_{k+2j}(R^{k+2l-s,s})+b_j \overline{\mu_{k+2j}(R^{k+2l-s,s})}\\ &=\sum_{j\geq 0} (a_j+(-1)^{s}b_j) \mu_{k+2j}(R^{k+2l-s,s}), \end{align*}

because $\mu _{k+2j}(R^{k+2l-s,s})\in \mathbf {i}^{s}\mathbb {R}$ by (22).

For $l=0$ we have $\mu _{k+2j}(R^{k -s,s})=0$ for all $j \geq 1$ and, hence, $c_0=a_0+(-1)^{s}b_0$ for $s=0,1$ and, thus, $a_0= c_0,b_0=0$. Suppose that $a_j= c_j, b_j=0$ for all $j< j_0$. Taking $l=j_0$ we deduce $c_{j_0}=a_{j_0}+(-1)^{s}b_{j_0}$ for $s=0,1$. By induction, we deduce $a_j= c_j$ and $b_j=0$ for all $j$, which completes the proof for $\sigma =1$.

For $\sigma >0$ the theorem follows by the homogeneity of the $\mu _k$ (cf. [Reference Bernig, Faifman and SolanesBFS22, Proposition 1.2(iii)]).

Let us now turn to $\sigma =-1$, i.e. to $H^{p,q-1}\subset \mathbb {R}^{p,q}$. Note that the anti-isometry $j\colon \mathbb {R}^{q,p}\to \mathbb {R}^{p,q}$ of Lemma 5.12 maps $S^{q-1,p}$ to $H^{p,q-1}$. Therefore, by Lemma 5.12 and the homogeneity of the $\mu _k$,

\begin{align*} \operatorname{Cr}_k^{H^{p,q-1}}(j(A)) &=\pi\omega_{k-1}\mathbf{i}^{k}\int_{\operatorname{Gr}_{n+1-k}} \chi(E\cap j(A)) {dm_k(E)}\\ &=\pi\omega_{k-1}\mathbf{i}^{k}\int_{\operatorname{Gr}_{n+1-k}} \chi(E\cap A) dj^{*}m_k(E)\\ &=\mathbf{i}^{k}\overline{\operatorname{Cr}_k^{S^{{q-1,p} }}(A)}\\ &=\mathbf{i}^{k}\sum_\nu c_{\nu} \overline{\mu_{k+2\nu}(A)}\\ &=\mathbf{i}^{k}\sum_\nu c_{\nu} \mathbf{i}^{-k-2\nu}{\mu_{k+2\nu}(j(A))}. \end{align*}

This proves the statement for $\sigma =-1$. The case $\sigma <0$ follows as before from the homogeneity of the $\mu _k$.

Finally, we consider the case $\sigma =0$. Let us identify $M=\mathbb {R}^{p-1,q}$ with the tangent space $T_x S^{p-1,q}$ at some $x\in S^{p-1,q}$. Let $\Lambda _k^{x}\colon \mathcal {V}^{-\infty }(S^{p-1,q})^{\operatorname {O}(p,q)}\to \operatorname {Val}^{-\infty }(T_x S^{p-1,q})^{O(p-1,q)}$ be given by (cf. [Reference AleskerAle06a, Proposition 3.1.5])

\[ \Lambda_k^{x}(\varphi)=\frac{1}{k!}\frac{d^{k}}{dt^{k}}\biggr|_{t=0} h_t^{*}\phi^{*}\varphi, \]

where $\phi \colon U\subset T_x S^{p-1,q}\to S^{p-1,q}$ is defined on a neighborhood of $x$ by

\[ \phi(w)=Q(x+w)^{-1/2}(x+w) \]

and $h_t(w)=tw$. By Proposition 5.16 we have

\[ \Lambda_k^{x} \operatorname{Cr}_k^{S^{p-1,q}}=\operatorname{Cr}_k^{\mathbb{R}^{p-1,q}}. \]

On the other hand, denoting by $g$ the metric on $S^{p-1,q}$, because $\mu _k\in \mathcal {W}_k^{-\infty }$, behaves naturally with respect to isometries and is $k$-homogeneous, we have

\[ \Lambda_k^{x} \mu_k^{g}= \lim_{t\to 0} t^{-k}(\phi\circ h_t)^{*}\mu_k^{g}=\lim_{t\to 0} \mu_k^{(\phi\circ h_t)^{*}g/t^{2}}. \]

As $(\phi \circ h_t)^{*}g/t^{2}$ converges, $C^{\infty }$-uniformly on compact sets, to the flat metric $g_0$, we conclude by [Reference Bernig, Faifman and SolanesBFS22, Proposition 1.2(ii)] that

\[ \Lambda_k^{x} \mu_k^{g} =\mu_k^{g_0}. \]

Applying $\Lambda _k^{x}$ to both sides of (25) the case $\sigma =0$ follows.

Recall from [Reference Bernig, Faifman and SolanesBFS22] that the intrinsic volumes $\mu _k$ were defined in terms of certain generalized curvature measures $C_{k,p}^{0},C_{k,p}^{1}$. On a manifold of constant curvature $\sigma$, these fulfill $C_{k,p}^{i}=\sigma ^{p} C_{k,0}^{i}$. Using this and [Reference Bernig, Faifman and SolanesBFS22, Eq. (61)], the Crofton formula (25) becomes

\[ \operatorname{Cr}_k^{M} =\mathbf{i}^{q} \sum_j d_{k,j}\sigma^{j} \operatorname{glob}(C_{k+2j,0}^{0}+\mathbf{i} C_{k+2j,0}^{1}), \]

where $\operatorname {glob}:\mathcal {C}^{-\infty }(M)\to \mathcal {V}^{-\infty }(M)$ is the globalization map (cf. [Reference Bernig, Faifman and SolanesBFS22, § 2]). The constants $d_{k,j}$ are independent of the signature and the curvature and can, thus, be deduced from the case of Euclidean spheres. Therefore, by [Reference Fu and WannererFW19, § 3.2] we obtain

(27)\begin{equation} \operatorname{Cr}_k^{M} =\frac{\pi^{k}}{k!\omega_k}\mathbf{i}^{q} \sum_j \biggl(\frac{\sigma}4\biggr)^{j} \operatorname{glob}(C_{k+2j,0}^{0}+\mathbf{i} C_{k+2j,0}^{1}). \end{equation}

Remark 6.6 It is interesting to note that (27) yields

\[ \chi -\frac{\sigma}{2\pi} \operatorname{Cr}_2^{M} = \mathbf{i}^{q} \operatorname{glob}(C_{0,0}^{0} + \mathbf{i} C_{0,0}^{1}), \]

which can be seen as a generalization of the fact that the angular excess of a spherical triangle is proportional to its area.

Acknowledgements

We wish to thank Gautier Berck for several inspiring talks and discussions, and the referee for numerous valuable comments which helped improve the exposition. A.B. was supported by DFG grant BE 2484/5-2. D.F. was supported by ISF Grant 1750/20. G.S. was supported by FEDER/MICINN grant PGC2018-095998-B-I00 and the Serra Húnter Programme.

References

Akhiezer, D. and Kazarnovskii, B., Average number of zeros and mixed symplectic volume of Finsler sets, Geom. Funct. Anal. 28 (2018), 15171547.CrossRefGoogle Scholar
Alesker, S., Description of translation invariant valuations on convex sets with solution of P. McMullen's conjecture, Geom. Funct. Anal. 11 (2001), 244272.CrossRefGoogle Scholar
Alesker, S., Hard Lefschetz theorem for valuations, complex integral geometry, and unitarily invariant valuations, J. Differential Geom. 63 (2003), 6395.CrossRefGoogle Scholar
Alesker, S., Theory of valuations on manifolds. I. Linear spaces, Israel J. Math. 156 (2006), 311339.CrossRefGoogle Scholar
Alesker, S., Theory of valuations on manifolds. II, Adv. Math. 207 (2006), 420454.CrossRefGoogle Scholar
Alesker, S., Theory of valuations on manifolds. IV. New properties of the multiplicative structure, in Geometric aspects of functional analysis, Lecture Notes in Mathematics, vol. 1910 (Springer, Berlin, 2007), 144.CrossRefGoogle Scholar
Alesker, S., Valuations on manifolds and integral geometry, Geom. Funct. Anal. 20 (2010), 10731143.CrossRefGoogle Scholar
Alesker, S. and Bernig, A., The product on smooth and generalized valuations, Amer. J. Math. 134 (2012), 507560.CrossRefGoogle Scholar
Alesker, S. and Bernstein, J., Range characterization of the cosine transform on higher Grassmannians, Adv. Math. 184 (2004), 367379.CrossRefGoogle Scholar
Álvarez Paiva, J. C. and Fernandes, E., Gelfand transforms and Crofton formulas, Selecta Math. (N.S.) 13 (2007), 369390.CrossRefGoogle Scholar
Bernig, A., Valuations with Crofton formula and Finsler geometry, Adv. Math. 210 (2007), 733753.CrossRefGoogle Scholar
Bernig, A. and Bröcker, L., Valuations on manifolds and Rumin cohomology, J. Differ. Geom. 75 (2007), 433457.CrossRefGoogle Scholar
Bernig, A. and Faifman, D., Valuation theory of indefinite orthogonal groups, J. Funct. Anal. 273 (2017), 21672247.CrossRefGoogle Scholar
Bernig, A., Faifman, D. and Solanes, G., Uniqueness of curvature measures in pseudo-Riemannian geometry, J. Geom. Anal. 31 (2021), 1181911848.CrossRefGoogle Scholar
Bernig, A., Faifman, D. and Solanes, G., Curvature measures of pseudo-Riemannian manifolds, J. Reine Angew. Math. 788 (2022), 77127.CrossRefGoogle Scholar
Bernig, A., Fu, J. H. G. and Solanes, G., Integral geometry of complex space forms, Geom. Funct. Anal. 24 (2014), 403492.CrossRefGoogle Scholar
Birman, G. S., Crofton's and Poincaré's formulas in the Lorentzian plane, Geom. Dedicata 15 (1984), 399411.CrossRefGoogle Scholar
Boman, J., Differentiability of a function and of its compositions with functions of one variable, Math. Scand. 20 (1967), 249268.CrossRefGoogle Scholar
Brouder, C., Dang, N. V. and Hélein, F., Continuity of the fundamental operations on distributions having a specified wave front set (with a counterexample by Semyon Alesker), Studia Math. 232 (2016), 201226.Google Scholar
Croke, C. B., A sharp four-dimensional isoperimetric inequality, Comment. Math. Helv. 59 (1984), 187192.CrossRefGoogle Scholar
Dabrowski, Y. and Brouder, C., Functional properties of Hörmander's space of distributions having a specified wavefront set, Comm. Math. Phys. 332 (2014), 13451380.CrossRefGoogle Scholar
Duistermaat, J. J., Fourier integral operators, Modern Birkhäuser Classics (Birkhäuser/Springer, New York, 2011). Reprint of the 1996 edition [MR 1362544], based on the original lecture notes published in 1973 [MR 0451313].CrossRefGoogle Scholar
Faifman, D., Crofton formulas and indefinite signature, Geom. Funct. Anal. 27 (2017), 489540.CrossRefGoogle Scholar
Fu, J. H. G., Algebraic integral geometry, in Integral geometry and valuations, Advanced Courses in Mathematics – CRM Barcelona, eds. E. Gallego and G. Solanes (Springer, Basel, 2014), 47112.Google Scholar
Fu, J. H. G., Intersection theory and the Alesker product, Indiana Univ. Math. J. 65 (2016), 13471371.CrossRefGoogle Scholar
Fu, J. H. G. and Wannerer, T., Riemannian curvature measures, Geom. Funct. Anal. 29 (2019), 343381.CrossRefGoogle Scholar
Gårding, L., Extension of a formula by Cayley to symmetric determinants, Proc. Edinb. Math. Soc. (2) 8 (1948), 7375.CrossRefGoogle Scholar
Guillemin, V. and Sternberg, S., Geometric asymptotics, Mathematical Surveys, vol. 14 (American Mathematical Society, Providence, RI, 1977).CrossRefGoogle Scholar
Hörmander, L., The analysis of linear partial differential operators. I, Classics in Mathematics (Springer, Berlin, 2003). Distribution theory and Fourier analysis, reprint of the second (1990) edition [Springer, Berlin; MR 1065993 (91m:35001a)].CrossRefGoogle Scholar
Kiderlen, M. and Jensen, E. B. V., (eds.), Tensor valuations and their applications in stochastic geometry and imaging, Lecture Notes in Mathematics, vol. 2177 (Springer, 2017).Google Scholar
Klain, D. A. and Rota, G.-C., Introduction to geometric probability, Lezioni Lincee. [Lincei Lectures] (Cambridge University Press, Cambridge, 1997).Google Scholar
Langevin, R., Chaves, R. M. B. and Bianconi, R., Formulas of Cauchy and Crofton in Lorentz-Minkowski and de Sitter spaces. Homage to Luis Santaló. Vol. 1 (Spanish), Math. Notae 41(2001/02) (2003), 99113.Google Scholar
Muro, M., Singular invariant hyperfunctions on the space of real symmetric matrices, Tohoku Math. J. (2) 51 (1999), 329364.CrossRefGoogle Scholar
Oh, Y.-G., Second variation and stabilities of minimal Lagrangian submanifolds in Kähler manifolds, Invent. Math. 101 (1990), 501519.CrossRefGoogle Scholar
O'Neill, B., Semi-Riemannian geometry with applications to relativity, Pure and Applied Mathematics, vol. 103 (Academic Press, New York, 1983).Google Scholar
Schneider, R., Convex bodies: the Brunn-Minkowski theory, expanded edition, Encyclopedia of Mathematics and its Applications, vol. 151 (Cambridge University Press, Cambridge, 2014).Google Scholar
Solanes, G. and Teufel, E., Integral geometry in constant curvature Lorentz spaces, Manuscripta Math. 118 (2005), 411423.CrossRefGoogle Scholar
Steiner, J., Über parallele Flächen, Monatsber. Preuß. Akad. Wiss., 114–118 (1840). Ges. Werke, vol. 2 (Reimer, Berlin, 1882), 171–176.Google Scholar
Teufel, E., Kinematische Berührung im Äquiaffinen, Geom. Dedicata 33 (1990), 317323.CrossRefGoogle Scholar
Treibergs, A., Estimates of volume by the length of shortest closed geodesics on a convex hypersurface, Invent. Math. 80 (1985), 481488.CrossRefGoogle Scholar
Vladimirov, V. S., Methods of the theory of functions of many complex variables (MIT Press, Cambridge, MA, 1966).Google Scholar
Weyl, H., On the volume of tubes, Amer. J. Math. 61 (1939), 461472.CrossRefGoogle Scholar
Wolf, J., Homogeneous manifolds of constant curvature, Comment. Math. Helv. 36 (1961), 112147.CrossRefGoogle Scholar
Ye, N., Ma, X. and Wang, D., The Fenchel-type inequality in the 3-dimensional Lorentz space and a Crofton formula, Ann. Global Anal. Geom. 50 (2016), 249259.CrossRefGoogle Scholar