Quadratic forms of signature (2, 2) and eigenvalue spacings on flat 2-tori

Alex Eskin^{∗}Gregory Margulis^{†}and Shahar Mozes^{‡}

June 28, 2001

1 Introduction

The Oppenheim conjecture proved by Margulis [Mar1] (see also [Mar2]) asserts that for a nondegenerate indefinite irrational quadratic form Q in n ≥ 3 variables the set Q(ℤ^{n}) is dense. In [EMM] (using the lower bounds established in [DM]) a quantitative version of the conjecture was established. Namely:

Let ρ be a continuous positive function on the sphere {v ∈ ℝ^{n} | ∥v∥ = 1}, and let Ω = {v ∈ ℝ^{n} | ∥v∥ < ρ(v/∥v∥)}. We denote by T Ω the dilate of Ω by T. For an indefinite quadratic form Q in n variables, let N_{Q,Ω}(a, b, T) denote the cardinality of the set

{x ∈ ℤ^{n} : x ∈ T Ω and a < Q(x) < b}.

We recall from [EMM] that for any such Q there exists a constant λ_{Q,Ω} such that

Vol({x ∈ ℝ^{n} : x ∈ T Ω and a ≤ Q(x) ≤ b}) ∼ λ_{Q,Ω}(b − a)T^{n−2}.  (1)

Theorem 1.1. ([EMM, Theorem 2.1]) Let Q be an indefinite quadratic form of signature (p, q), with p ≥ 3 and q ≥ 1. Suppose Q is not proportional to a rational form. Then for any interval (a, b), as T → ∞,

N_{Q,Ω}(a, b, T) ∼ λ_{Q,Ω}(b − a)T^{n−2},  (2)

∗Research partially supported by BSF grant 94-00060/1, GIF grant G-454.213.06.9, the Sloan Foundation and the Packard Foundation.

†Research partially supported by NSF grant DMS-9424613

‡Research partially supported by the Israel Science Foundation and by BSF grant 94-00060/1

where n = p + q, and λ_{Q,Ω} is as in (1).

If the signature of Q is (2, 1) or (2, 2) then Theorem 1.1 fails; in fact there are irrational forms for which along a subsequence T_{j}, N_{Q,Ω}(a, b, T_{j}) > T_{j}^{n−2}(log T_{j})^{1−ϵ}. Such forms may be obtained by considering irrational forms which are very well approximated by split rational forms. It should be noted that the asymptotically exact lower bounds established by Dani and Margulis (see [DM]) hold for any irrational indefinite quadratic form in n ≥ 3 variables.

Observe also that whenever a form of signature (2, 2) has a rational isotropic subspace L then L∩T Ω contains on the order of T^{2} integral points x for which Q(x) = 0, hence N_{Q,Ω}(−ϵ, ϵ, T) ≥ cT^{2}, independently of the choice of ϵ. Thus to obtain an asymptotic formula similar to (2) in the signature (2, 2) case, we must exclude the contribution of the rational isotropic subspaces. We remark that an irrational quadratic form may have at most 4 rational isotropic subspaces (see Lemma 10.3).

The space of quadratic forms in 4 variables is a linear space of dimension 10. Fix a norm ∥ · ∥ on this space.

Definition 1.2. (EWAS) A quadratic form Q is called extremely well approximable by split forms (EWAS) if for any N > 0 there exists a split integral form Q′ and 2 ≤ k ∈ ℝ such that

∥Q − \frac{1}{k}Q′∥ ≤ \frac{1}{k^{N}} .

Our main result is:

Theorem 1.3. Suppose Ω is as above. Let Q be an indefinite quadratic form of signature (2, 2) which is not EWAS. Then for any interval (a, b),

Ñ_{Q,Ω}(a, b, T) ∼ λ_{Q,Ω}(b − a)T^{2}  (3)

where the constant λ_{Q,Ω} is as in (1), and Ñ_{Q,Ω} counts the points not contained in isotropic subspaces.

As observed above, lattice points belonging to isotropic rational 2-dimensional subspaces have to be excluded. It turns out that also points belonging to a wider class of subspaces (which we shall call “quasinull”) have to be treated separately. Given the form Q consider the orthogonal group SO(Q) ∈ SL(4, ℝ) of all the orientation preserving linear transformations preserving Q. It acts on the 6-dimensional space ∧^{2}ℝ^{4}. This representation is reducible and ∧^{2}ℝ^{4} decomposes into a direct sum of two irreducible 3-dimensional spaces, ∧^{2}ℝ^{4} = V_{1} ⊕ V_{2} (see Lemma 2.1). We observe that a 2-dimensional subspace L ∈ ℝ^{4} is isotropic if and only if the corresponding 1dimensional subspace ∧^{2}L ∈ ∧^{2}ℝ^{4} lies in one of the subspace V_{i}. Equivalently, if we let {w_{1}, w_{2}} be a basis of L then L is isotropic if and only if ∥π_{1}(w_{1}∧w_{2})∥·∥π_{2}(w_{1}∧w_{2})∥ = 0, where π_{i} : ∧^{2}ℝ^{4} → V_{i}, i = 1, 2 are the projections to V_{i} so that v = π_{1}(v) + π_{2}(v). Given a rational 2-dimensional subspace L let {w_{1}, w_{2}} be an integral basis of L ∩ ℤ^{4}. The subspace will be called μ_{1}-quasinull if

∥π_{1}(w_{1} ∧ w_{2})∥ · ∥π_{2}(w_{1} ∧ w_{2})∥ < μ_{1},

where μ_{1} > 0 is a fixed constant.

Since most results do not depend on the choice of the parameter μ_{1}, we will often use the term quasinull subspace to refer to a μ_{1}-quasinull subspace. We also define the norm of a 2-dimensional rational subspace L to be the norm of w_{1} ∧ w_{2} where {w_{1}, w_{2}} is any integral basis of L ∩ ℤ^{4}.

The following theorem is valid without any diophantine conditions:

Theorem 1.4. Suppose Ω is as above. Let Q be any indefinite quadratic form of signature (2, 2) not proportional to a rational form. Then for any interval (a, b), as T → ∞,

lim sup \frac{1}{T^{2}} N′_{Q,Ω}(a, b, T) ≤ λ_{Q,Ω}(b − a)  (4)

where the constant λ_{Q,Ω} is as in (1), and N′_{Q,Ω} counts the points not contained in quasinull subspaces of norm at most T.

With a diophantine condition we have:

Theorem 1.5. Suppose Q is a quadratic form of signature (2, 2) which is not EWAS. Let X_{Q,Ω}(a, b, T) denote the number of integral points v ∈ T Ω such that a < Q(v) < b and v lies in some non-isotropic quasinull subspace of norm at most T. Then, as T → ∞,

X_{Q,Ω}(a, b, T) = o(T^{2})  (5)

We also recall:

Theorem 1.6 (Dani-Margulis). Suppose Ω is as above. Let Q be any indefinite quadratic form in n ≥ 3 variables, not proportional to a rational form. Then for any interval (a, b), as T → ∞,

lim inf \frac{1}{T^{n−2}} N_{Q,Ω}(a, b, T) ≥ λ_{Q,Ω}(b − a)  (6)

where the constant λ_{Q,Ω} is as in (1).

To deduce Theorem 1.3 from Theorem 1.4, Theorem 1.6 and Theorem 1.5 one can argue as follows: Suppose Q of signature (2, 2) is not EWAS (see Definition 1.2); then by Theorem 1.5, X_{Q,Ω}(a, b, T) is o(T^{2}). Hence, if 0 ∉ (a, b) we get

lim sup_{T →∞} \frac{N_{Q,Ω}(a,b,T)}{T^{2}} = lim sup_{T →∞} \frac{Ñ_{Q,Ω}(a,b,T)}{T^{2}} = lim sup_{T →∞} \frac{N′_{Q,Ω}(a,b,T)}{T^{2}} ≤ λ_{Q,Ω}(b − a)

However, the universal lower bound of Dani-Margulis (Theorem 1.6) implies that for any irrational Q, lim inf_{T →∞} \frac{1}{T^{2}} N_{Q,Ω}(a, b, T) = λ_{Q,Ω}(b − a). Hence for Q not EWAS and 0 ∉ (a, b) we get N_{Q,Ω}(a, b, T) ∼ λ_{Q,Ω}(b−a)T^{2} which is equivalent to Theorem 1.3.

Eigenvalue spacings on flat 2-tori

It has been suggested by Berry and Tabor that the distribution of the local spacings between eigenvalues of the quantization of a completely integrable Hamiltonian is Poisson. For the Hamiltonian which is the geodesic flow on the flat 2-torus, it was noted by P. Sarnak [Sar] that this problem translates to one of the spacing between the values at integers of a binary quadratic form, and is related to the quantitative Oppenheim problem in the signature (2, 2) case. We briefly recall the connection following [Sar].

Let Δ ⊂ ℝ^{2} be a lattice and let M = ℝ^{2}/Δ denote the associated flat torus. The eigenfunctions of the Laplacian on M are of the form f_{v}(·) = e^{2πi⟨v,·⟩}, where v belongs to the dual lattice Δ^{∗}. The corresponding eigenvalues are 4π^{2}∥v∥^{2}, v ∈ Δ^{∗}. These are the values at integral points of the binary quadratic B(m, n) = 4π^{2}∥mv_{1} + nv_{2}∥^{2}, where {v_{1}, v_{2}} is a ℤ-basis for Δ^{∗}. We will identify Δ^{∗} with ℤ^{2} using this basis.

We label the eigenvalues (with multiplicity) by

0 = λ_{0}(M) < λ_{1}(M) ≤ λ_{2}(M) …

It is easy to see that Weyl’s law holds, i.e.

|{j : λ_{j}(M) ≤ T }| ∼ c_{M}T

where c_{M} = (area M)/(4π|). We are interested in the distribution of the local spacings λ_{j}(M) − λ_{k}(M). In particular, for 0 ∉ (a, b), set

R_{M}(a, b, T) = \frac{|\{(j, k) : λ_{j}(M) ≤ T, λ_{k}(M) ≤ T, a ≤ λ_{j}(M) − λ_{k}(M) ≤ b\}|}{T}

The quantity R_{M} is called the pair correlation. The random number (Poisson) model predicts that

lim_{T →∞} R_{M}(a, b, T) = c_{M}^{2}(b − a).  (7)

Note that the differences λ_{j}(M) − λ_{k}(M) are precisely the integral values of the quadratic form Q_{M}(x_{1}, x_{2}, x_{3}, x_{4}) = B(x_{1}, x_{2}) − B(x_{3}, x_{4}).

P. Sarnak showed in [Sar] a that (7) holds on a set of full measure in the space of tori. Some remarkable related results for forms of higher degree and higher dimensional tori were proved in [V1], [V2] and [V3]. These methods, however, cannot be used to construct a specific torus for which (7) holds. A corollary of Theorem 1.3 is the following:

Theorem 1.7. Let M be a 2 dimensional flat torus rescaled so that one of the coefficients in the associated binary quadratic form B is 1. Let A_{1}, A_{2} denote the two other coefficients of B. Suppose that there exists N > 0 such that for all triples of integers (p_{1}, p_{2}, q),

max_{i=1,2} |A_{i} − \frac{p_{i}}{q} | > \frac{1}{q^{N}}

Then, for any interval (a, b) not containing 0, (7) holds, i.e.

lim_{T →∞} R_{M}(a, b, T) = c_{M}^{2}(b − a).

In particular, the set of (A_{1}, A_{2}) ⊂ ℝ^{2} for which (7) does not hold has zero Hausdorff dimension.

Thus, if one of the A_{i} is diophantine (e.g. algebraic), then M has a spectrum whose pair correlation satisfies the Berry-Tabor conjecture.

This establishes the pair correlation for the flat torus or “boxed oscillator” considered numerically by Berry and Tabor. We note that without some diophantine condition, (7) may fail.

Let Ω_{M} ⊂ ℝ^{4} denote the set {x : max(B(x_{1}, x_{2})^{1/2}, B(x_{3}, x_{4})^{1/2}) ≤ 1}. By construction, if (m_{1}, n_{1}, m_{2}, n_{2}) ∈ ℤ^{4} ∩ T Ω, then B(m_{1}, n_{1}) and B(m_{2}, n_{2}) are two eigenvalues size at most T^{2}, and thus Q_{M}(m_{1}, n_{1}, m_{2}, n_{2}) is a difference between two points in the spectrum of size at most T^{2}. An elementary calculation (cf. [Sar]) shows that λ_{Q_{M} ,Ω_{M}} = (area M/4π)^{2} = c_{M}^{2}. Thus, one arrives at the following:

Proposition 1.8. (see [Sar]) The equation (7) holds for M if and only if (3) holds with Q = Q_{M}, Ω = Ω_{M}.

Thus in view of Proposition 1.8, Theorem 1.7 is indeed a corollary of Theorem 1.3.

Remark. The case where M is rectangular is simpler since in that case the quasinull subspaces can be easily described, allowing for an elementary proof of Theorem 1.5. This is written out in §9.

Outline of the proofs

In order to estimate N_{Q,Ω}(a, b, T) we make (as in [EMM]) a transition to considering certain integrals on the space of unimodular lattices in ℝ^{4}, i.e. SL(4, ℝ)/SL(4, ℤ). This transition is based on the transitivity of the action of the orthogonal group SO(Q) on the level sets of the quadratic form Q. We fix a suitably chosen compact set U ⊂ ℝ^{4} -{0} such that for v ∈ U, a ≤ Q(v) ≤ b. Let f̂ denote the function on the space of lattices SL(n, ℝ)/SL(n, ℤ) which associates to each lattice Δ the number of points in Δ ∩ U. Because of the transitivity of the action of SO(Q), any vector v ∈ ℤ^{n} with T/2 ≤ ∥v∥ ≤ T and a ≤ Q(v) ≤ b can be brought into U by an appropriate element g ∈ SO(Q). Note that the number of points in the lattice gℤ^{4} lying in U is exactly equal to the number of points in the lattice ℤ^{4} lying in g^{−1}U. Hence the number of points in ℤ^{4} lying in g^{−1}U is equal to f̂(gℤ^{4}). By varying g ∈ SO(Q) in an appropriate way, the sets g^{−1}U can be made to cover the set R = {v ∈ ℝ^{4} : a ≤ Q(v) ≤ b and T/2 ≤ ∥v∥ ≤ T }. Thus, the number of integer points in R, i.e. N_{Q}(a, b, T) − N_{Q}(a, b, T/2) may be approximated by an integral of the form T^{2} ∫_{H} f̂(gℤ^{4}) dg, where H is a suitably chosen subset of SO(Q). Note that one needs on the order of T^{2} translates of U to cover R, since g is volume preserving and the volume of R is asymptotic to T^{2}.

Observe that for any choice of U, f̂ is unbounded. In [DM] S.G. Dani and G.A. Margulis have used integrals of the form ∫_{H} ϕ(gℤ^{n}) dg where ϕ is a bounded function with ϕ ≤ f̂ to give asymptotically exact lower bounds on the number of lattice points in sets of the form {v ∈ ℝ^{n} : a < Q(v) < b} ∩ T Ω for indefinite irrational quadratic forms, n ≥ 3. The proof of these estimates uses M. Ratner’s measure classification theorem (see [Ratner]) as well as the methods developed by Dani and Margulis for studying unipotent flows via “linearization”, see [DM]. Here, as in [EMM], one needs to have an estimate of the contribution of elements of lattices lying at the “cusps” of SL(4, ℝ)/SL(4, ℤ), i.e., outside of large compact subsets. The contributions of lattices having either a single short vector or three independent short vectors is estimated as in [EMM]. In estimating the contribution of those lattices having a two dimensional sublattice spanned by short vectors one has to exclude the lattice points belonging to quasinull subspaces. One of the main technical tools of the current paper is the estimate provided by Theorem 2.6 for the contribution of these lattices excluding the points coming from quasinull subspaces. The proof of Theorem 2.6 (which is outlined in §5) occupies §5-§8. The contribution of quasinull spaces is estimated in §9 for the case of rectangular tori, and in §10 in the general case.

Acknolegements: We would like to thank P. Sarnak for his encouragement and for many useful conversations.

2 Passage to the space of lattices

Fix a quadratic form Q of signature (2, 2) and discriminant 1. Then there exists an element g_{Q} ∈ SL(4, ℝ) such that for any v ∈ ℝ^{n}, Q(v) = B(g_{Q}v), where B is the “standard” quadratic form B(z_{1}, z_{2}, z_{3}, z_{4}) = z_{1}z_{4} − z_{2}z_{3}. Let Λ denote g_{Q}ℤ^{4} so that Q(ℤ^{4}) = B(Λ).

We identify ℝ^{4} with M_{2}(ℝ) by sending (z_{1}, z_{2}, z_{3}, z_{4}) to (\begin{matrix}z_{1} & z_{2} \\ z_{3} & z_{4} \end{matrix}). Then the

determinant function on M_{2}(ℝ) becomes the quadratic form B of signature (2, 2). This shows that the orthogonal group H = SO(B) is locally isomorphic to SL(2, ℝ)× SL(2, ℝ) with the action of g = (g_{1}, g_{2}) ∈ SL(2, ℝ) × SL(2, ℝ) on v ∈ M_{2}(ℝ) given by v → g_{1}vg_{2}^{−1}. Write H = H_{1} × H_{2} (almost direct product).

Let e_{ij} ∈ M_{2}(ℝ) denote the elementary matrix with 1 in row i column j and 0’s elsewhere. Then e_{11}, e_{12}, e_{21} and e_{22} form a basis for M_{2}(ℝ) ≈ ℝ^{4}. The following lemma is standard:

Lemma 2.1 (Reducibility of the representation of SO(2,2) on ∧^{2}ℝ^{4}).

(a) The space ∧^{2}ℝ^{4} ≈ ∧^{2}M_{2}(ℝ) splits as a direct sum of two invariant subspaces V_{1} ⊕ V_{2} each of dimension 3. A basis for V_{1} is given by f_{1}^{(1)} = e_{11} ∧ e_{12}, f_{2}^{(1)} = e_{21} ∧ e_{22}, f_{3}^{(1)} = e_{11} ∧ e_{22} − e_{12} ∧ e_{21}. A basis for V_{2} is given by f_{1}^{(2)} = e_{11} ∧ e_{21}, f_{2}^{(2)} = e_{12} ∧ e_{22}, f_{3}^{(2)} = e_{11} ∧ e_{22} + e_{12} ∧ e_{21}.

(b) H_{1} fixes each vector in V_{2} and H_{2} fixes each vector in V_{1}. On V_{i}, H_{i} fixes the quadratic form Q(xf_{1}^{(i)} + yf_{2}^{(i)} + zf_{3}^{(i)}) = z^{2} − xy.

(c) Let π_{i} denote the projection ∧^{2}ℝ^{4} → V_{i}. Let L be a two-dimensional subspace of ℝ^{4}, and let v_{1}, v_{2} be a basis for L. Then

Q(π_{1}(v_{1}∧v_{2})) = Q(π_{2}(v_{1}∧v_{2})) = the discriminant of the restriction of B to L .

The restriction of B to L is identically 0 if and only if π_{1}(v_{1} ∧ v_{2}) = 0 or π_{2}(v_{1} ∧ v_{2}) = 0.

Let Λ be a lattice in ℝ^{4}. A subspace L of ℝ^{4} is called Λ-rational if L ∩ Λ is a lattice in L. Let L be a 2-dimensional Λ-rational subspace L of ℝ^{4}, and let v_{1}, v_{2} be an integral basis for L ∩ Λ. Write v^{L} = v_{1} ∧ v_{2}. Note that up to sign, v^{L} depends only on L and Λ and does not depend on the choice of integral basis. In view of Lemma 2.1 (c), the restriction of B to L is identically 0 if and only if ∥π_{1}(v^{L})∥∥π_{2}(v^{L})∥ = 0.

Definition 2.2 (Quasinull subspace). Let Λ be a lattice in ℝ^{4}. Fix μ_{1}, 0 < μ_{1} < 1. A Λ-rational subspace L of ℝ^{4} is called μ_{1}-quasinull with respect to Λ if dim(L) = 2 and

∥π_{1}(v^{L})∥∥π_{2}(v^{L})∥ < μ_{1}  (8)

Instead of the standard quadratic form B and the lattice Λ = g_{Q}ℤ^{4}, we often consider the given form Q(v) = B(g_{Q}v) and the standard lattice ℤ^{4}. We will then say that a ℤ^{4}-rational subspace L is μ_{1}-quasinull if and only if g_{Q}L is μ_{1}-quasinull with respect to Λ. We will also occasionally omit μ_{1}.

Instead of working directly with SO(2, 2) we work with SL(2, ℝ) × SL(2, ℝ),

which is locally isomorphic. Let b_{t} ∈ SL(2, ℝ) denote the matrix (\begin{matrix}e^{t/2} & 0 \\ 0 & e^{−t/2} \end{matrix}), let

a_{t} = (b_{t}, b_{t}) and let K = SO(2) × SO(2) denote the standard maximal compact subgroup of SL(2, ℝ) × SL(2, ℝ). Let dk denote the normalized Haar measure on K.

Let X_{T} (Λ) denote the set of v ∈ Λ which do not belong to any quasinull subspace of Λ of norm at most T. As in [EMM], for a lattice Δ ⊂ ℝ^{4} and a function f ∈ C_{0}^{∞}(ℝ^{4} − {0}), let

f̂(Δ) = ∑_{v∈Δ} f(v).

We will also need the following modification:

f̃(g; Λ) = ∑_{v∈X_{T} (Λ)} f(gv).  (9)

In order to prove Theorem 1.4 we prove the following theorem:

Theorem 2.3. Let Q be any quadratic form of signature (2, 2) and discriminant 1 which is not proportional to a rational form. Let g_{Q} ∈ SL(4, ℝ) be such that for all v ∈ ℝ^{4}, Q(v) = B(g_{Q}v), and let Λ = g_{Q}ℤ^{4}. Then, for any function f ∈ C_{0}^{∞}(ℝ^{4}−{0}), and any continouous function ν on K,

lim sup_{t→∞} ∫_{K} f̃(a_{t}k; Λ)ν(k) dk ≤ ∫_{SL(4.ℝ)/SL(4,ℤ)} f̂(Δ)ν(k) dμ(Δ)  (10)

where μ denotes the normalized Haar measure on the space of lattices SL(4, ℝ)/SL(4, ℤ).

Theorem 1.4 follows from Theorem 2.3 by an argument identical to that [EMM, §3]. Essentially, the proof is based on the identity of the form

∫_{K} f̃(a_{t}k; Λ)ν(k) dk = ∑_{v∈X_{T} (Λ)} ∫_{K} f(a_{t}kv)ν(k) dk,  (11)

obtained by integrating (9). The right hand side is then related to the number of lattice points v ∈ [e^{t}/2, e^{t}]∂Ω with a < Q(v) < b which are not contained in quasinull subspaces of norm at most T (where T = e^{t}).

Let Δ be a lattice in ℝ^{4}. As in [EMM], for any Δ-rational subspace L, we denote by d_{Δ}(L) or simply by d(L) the volume of L/(L∩Δ). Let us note that d(L) is equal to the norm of e_{1} ∧ ⋯ ∧ e_{ℓ} in the exterior power ∧^{ℓ}ℝ^{4} where ℓ = dim L and {e_{1}, ⋯, e_{ℓ}} is a basis over ℤ of L ∩ Δ. If L = {0} we write d(L) = 1.

We recall the following:

Lemma 2.4 ([EMM], Lemma 5.6). For any two Δ-rational subspaces L and M

d(L)d(M) ≥ d(L ∩ M)d(L + M).  (12)

As in [EMM] we use the notation:

α_{i}(Δ) = sup \{\frac{1}{d(L)}| L is a Δ-rational subspace of dimension i \}, 0 ≤ i ≤ 4,
α(Δ) = max_{0≤i≤4} α_{i}(Δ).  (13)

We recall the following theorem:

Theorem 2.5. Suppose i = 1 or 3. Then for any ξ > 0, sup_{t≥0} ∫_{K} α_{i}(a_{t}kΛ)^{2−ξ} dk < ∞. Hence there exists a constant c depending only on ξ and Λ such that for all t > 0 and all δ > 0,

|{k ∈ K : α_{i}(a_{t}kΛ) > 1/δ}| < cδ^{2−ξ}.

Proof. The first assertion is proved in [EMM, § 5] (see the proof of equation (5.75)). The second assertion follows from the first and Chebechev’s inequality.

The analogous assertion for α_{2} is false even when ξ = 1; this accounts for the failure of Theorem 1.1 in the signature (2, 2) case. However, in this situation, we prove a substitute: see Theorem 2.6 below.

Suppose g ∈ SL(4, ℝ), and Λ is as above. Let

α̂_{2}(g; Λ) = sup \{\frac{1}{d(gL)} : dim(L) = 2, L is rational and not μ_{1}-quasinull \} .

Theorem 2.6. There exists a constant c depending only on μ_{1} and Λ such that for all 0 < δ ≪ 1 and all t > 0,

|{k ∈ K : α̂_{2}(a_{t}k; Λ) > 1/δ}| < cδ^{1.05}.  (14)

Hence, for any θ < 1.05,

sup_{t≥0} ∫_{K} α̂_{2}(a_{t}k; Λ)^{θ} < ∞.

In §3 we derive Theorem 2.3 from Theorem 2.5 and Theorem 2.6. After making some preliminary calculations in §4, we will begin the proof of Theorem 2.6 (which is the main technical point of this paper) in §5.

3 Proof of Theorem 2.3

In this section, we deduce Theorem 2.3 from Theorem 2.5, and Theorem 2.6. This argument is very similar to the proof of [EMM, Theorem 3.5] in [EMM, §5].

Lemma 3.1. For every function f ∈ C_{0}^{∞}(ℝ^{4} -{0}) there exists a constant c = c(f) such that for all g ∈ SL(4, ℝ),

f̃(g; Λ) ≤ cα̂(g; Λ)

where we use the notation α̂(g; Λ) = max (1, α_{1}(gΛ), α̂_{2}(g; Λ), α_{3}(gΛ)).

Proof of Lemma 3.1. This is an easy modification of a well known lemma of Schmidt, see [Sch, Lemma 2].

Proof of Theorem 2.3. We may assume that f̃ is non-negative. Let A(r) = {Δ ∈ SL(4, ℝ)/SL(4, ℤ) : α_{1}(Δ) > r}. Choose a continuous nonnegative function g_{r} on SL(4, ℝ)/SL(4, ℤ) such that g_{r}(Δ) = 1 if Δ ∈ A(r + 1), g_{r}(Δ) = 0 if Δ ∉ A(r) and 0 ≤ g_{r}(Δ) ≤ 1 if Δ ∈ A(r) − A(r + 1). Then

\begin{matrix}∫_{K} f̃(a_{t}k; Λ) dk = \\ = ∫_{K} f̃(a_{t}k; Λ)g_{r}(a_{t}kΛ) dk + ∫_{K} f̃(a_{t}k; Λ)(1 − g_{r}(a_{t}kΛ)) dk. \end{matrix}  (15)

But,

f̃(a_{t}k; Λ)g_{r}(a_{t}kΛ) ≤ cα̂(a_{t}k; Λ)g_{r}(a_{t}kΛ) = = cα̂(a_{t}k; Λ)^{−1.04}g_{r}(a_{t}kΛ)α̂(a_{t}k; Λ)^{1.04} ≤ B_{1}r^{−1.04}α̂(a_{t}k; Λ)^{1.04}

(the last inequality is true because α̂(a_{t}k; Λ) ≥ α_{1}(a_{t}kΛ) and g_{r}(a_{t}kΛ) = 0 if α_{1}(a_{t}kΛ) ≤ r). Therefore

∫_{K} f̃(a_{t}k; Λ)g_{r}(a_{t}kΛ) dk ≤ B_{1}r^{−1.04} ∫_{K} α̂(a_{t}k; Λ)^{1.04} dk.  (16)

According to Theorem 2.5 and Theorem 2.6 there exists B such that

∫_{K} α̂(a_{t}k; Λ)^{1.04} dk < B

for any t ≥ 0. Then (16) implies that

∫_{K} f̃(a_{t}k; Λ)g_{r}(a_{t}kΛ) dk ≤ BB_{1}r^{−1.04}.  (17)

Let h(Δ) = f̂(Δ)(1 − g_{r}(Δ)). Then for all t > 0 and all k ∈ K,

f̃(a_{t}k; Λ)(1 − g_{r}(a_{t}kΛ)) ≤ h(a_{t}kΛ).  (18)

Since the function h is continuous and has compact support, [EMM, Theorem 4.5] implies that for every ϵ > 0 and every quadratic form Q not proportional to a rational form, there exists t_{0} > 0 such that for Λ = g_{Q}ℤ^{4} and every t > t_{0},

|∫_{K} h(a_{t}kΛ) dk − ∫_{SL(4.ℝ)/SL(4,ℤ)} h(Δ) dμ(Δ)| < \frac{ϵ}{2}.  (19)

It is easy to see that (15), (17), (18) and (19) imply (10) if r is sufficiently large.

4 The Rectangles

We will eventually establish (14), but first we will need to study carefully the sets {k ∈ K : d(a_{t}kL) < η}, where L is a two-dimensional subspace. This will be done in this section.

Let b_{t} = (\begin{matrix}e^{t/2} & 0 \\ 0 & e^{−t/2} \end{matrix}), and k_{θ} = (\begin{matrix}cos θ & sin θ \\ − sin θ & cos θ \end{matrix}).

Then a_{t} = (b_{t}, b_{t}) and k = (k_{θ}, k_{θ′}). If we use the norm ∥v∥′ = max(∥π_{1}(v)∥, ∥π_{2}(v)∥) for defining α_{2} and d then the set {k ∈ K : d(a_{t}kL) < η} is the direct product of the sets {θ : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < η} and {θ′ : ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < η} As we shall see below, each set {θ : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < η} is either an interval or a union of two intervals; thus our set {k ∈ K : d(a_{t}kL) < η} is a union of up to four rectangles. Note that these rectangles (and all rectangles appearing in this paper) have sides parallel to the coordinate axes.

More generally, we may also consider sets of the form {(θ, θ′) : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < η_{1}, ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < η_{2}}. Let M_{t}(L) = min_{k∈k} ∥a_{t}kv^{L}∥′.

Implied constants. In this section, all implied constants are either absolute, or depend only on μ_{1} and Λ.

Lemma 4.1. There is a constant c ≪ 1 such that the following holds: Suppose L is not μ_{1}-quasinull with respect to Λ, t ≫ 1 and M_{t}(L) ≪ 1. Then, both of the sets {θ : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < c} and {θ′ : ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < c} are not all of SO(2). Also, if ∥v^{L}∥ ≫ 1 then for some j ∈ {1, 2} the set {ϕ : ∥b_{t}k_{ϕ}π_{j}(v^{L})∥ < M_{t}(L)^{−0.1}} is not all of SO(2).

Proof. It is easy to check that for any v ∈ ℝ^{3} and t ≫ 1,

∥b_{t}∥^{−1}∥v∥ ≤ min_{θ} ∥b_{t}k_{θ}v∥ ≤ max_{θ} ∥b_{t}k_{θ}v∥ ≈ ∥b_{t}∥∥v∥  (20)

Write v^{L} = π_{1}(v^{L}) + π_{2}(v^{L}). Without loss of generality we may assume ∥π_{2}(v^{L})∥ ≥ ∥π_{1}(v^{L})∥. Since M_{t}(L) ≪ 1, from (20) we see that ∥π_{2}(v^{L})∥ ≪ ∥b_{t}∥. Hence since L is not μ_{1}-quasinull with respect to Λ, ∥π_{1}(v^{L})∥ ≥ ∥b_{t}∥^{−1}. Then, using (20), max_{θ} ∥b_{t}k_{θ}π_{1}(v^{L})∥ ≈ ∥b_{t}∥∥π_{1}(v^{L})∥ ≥ c as required. Since L is Λ-rational, ∥v^{L}∥ ≥ c_{0} where c_{0} = c_{0}(Λ), hence ∥π_{2}(v^{L})∥ ≥ c_{0}/2. This implies (for any θ < 1),

max_{θ} ∥b_{t}k_{θ}π_{2}(v^{L})∥ ≈ ∥b_{t}∥∥π_{2}(v^{L})∥ ≥ (c_{0}/2)∥b_{t}∥ ≥ (c_{0}/2)∥b_{t}∥^{−θ} ≥ (c_{0}/4)(∥b_{t}∥∥π_{2}(v^{L})∥)^{−θ} ≥ cM_{t}(L)^{−θ}

as required. (here c depends only on μ_{1} and Λ).

For an interval I ⊂ S^{1} and c > 0, let cI denote the interval with the same center as I and length c|I|. Similarly for a rectangle R = I_{1} × I_{2} and c > 0, let cR denote the rectangle cI_{1} × cI_{2}

We approximate the sets {(θ, θ′) : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < η_{1}, ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < η_{2}} by sets R_{t}^{L,±,±}(η_{1}, η_{2}) which are rectangles (rather then unions of rectangles), and whose center does not depend on t. A major difficulty is that the aspect ratio of the rectangles is not bounded (even if η_{1} = η_{2}). However, the modified family has the following properties:

Proposition 4.2. Suppose L is not μ_{1}-quasinull with respect to Λ, t ≫ 1, and M_{t}(L) ≪ 1. There exist rectangles R_{t}^{L,±±}(δ_{1}, δ_{2}) = I_{t}^{π_{1}(L),±}(δ_{1}) × I_{t}^{π_{2}(L),±}(δ_{2}), such that the following properties hold:

(a) “The rectangles R_{t}^{L,±±}(η_{1}, η_{2}) approximate the sets {(θ, θ′) : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < η_{1}, ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < η_{2}}:” i.e. there exists 0 < c < 1 such that if M_{t}(L) ≪ δ_{i} ≪ max_{k} ∥π_{i}(a_{t}kv^{L})∥ then

R_{t}^{L,±±}(cδ_{1}, cδ_{2}) ⊂ {(θ, θ′) : ∥b_{t}k_{θ}π_{1}(v^{L})∥ < δ_{1}, ∥b_{t}k_{θ′}π_{2}(v^{L})∥ < δ_{2}} ⊂ R_{t}^{L,++}(c^{−1}δ_{1}, c^{−1}δ_{2})∪

∪ R_{t}^{L,+−}(c^{−1}δ_{1}, c^{−1}δ_{2}) ∪ R_{t}^{L,−+}(c^{−1}δ_{1}, c^{−1}δ_{2}) ∪ R_{t}^{L,−−}(c^{−1}δ_{1}, c^{−1}δ_{2}) (21)

(b) “The rectangles R_{t}^{L} shrink as t increases;” i.e. there exists an absolute constant C < ∞ such that if max_{k} ∥π_{i}(a_{t}kv^{L})∥ ≫ δ_{i} ≫ M_{t}(L), τ ≫ 0 and δ_{i} ≫ M_{t+τ}(L), then

R_{t+τ}^{L,±±}(δ_{1}, δ_{2}) ⊂ Ce^{−τ/2}R_{t}^{v,±±}(δ_{1}, δ_{2})  (22)

(c) “Measure ratio estimate:” If max_{k} ∥π_{i}(a_{t}kv^{L})∥ ≫ η_{i} ≫ δ_{i} ≫ M_{t}(L), then

\frac{|R_{t}^{L,±±}(δ_{1}, δ_{2})|}{|R_{t}^{L,±±}(η_{1}, η_{2})|} ≪ (\frac{δ_{1}δ_{2}}{η_{1}η_{2}} )^{1/2}  (23)

(d) “The alternative”. Either “one side is determined” i.e. for max_{k} ∥π_{i}(a_{t}kv^{L})∥ ≫ δ_{i} ≫ M_{t}(L),

R_{t}^{L,±±}(δ_{1}, δ_{2}) has at least one side of length ≈ e^{−t}M_{t}(L)^{−1/2}δ_{i}^{1/2}  (24)

or “the minimum does not decrease” i.e. for τ > 0,

M_{t+τ}(L) ≥ M_{t}(L).  (25)

Proposition 4.2 is proved in Appendix A.

Notation: We use the notation R_{t}^{L,±±}(η) to denote R_{t}^{L,±±}(η, η).

5 The scheme of the proof of Theorem 2.6

Let L_{t}(δ) denote the set of non-μ_{1}-quasinull Λ-rational 2-dimensional subspaces L such that for some k ∈ K, d(a_{t}kL) < δ. We have:

{k ∈ K : α̂_{2}(a_{t}k; Λ) > 1/δ} = ⋃_{L∈L_{t}(δ)}{k ∈ K : d(a_{t}kL) < δ} ⊂ ⋃_{ρ∈±^{σ∈±}} ( ⋃_{L∈L_{t}(δ)} R_{t}^{L,σρ}(c^{−1}δ))

(26)

where c is as in Proposition 4.2 (a). Let τ > 0 be a parameter to be chosen later. Let

L_{t,new}(δ) = L_{t}(δ) - L_{t−τ}(δ),  and  L_{t,old}(δ) = L_{t}(δ) ∩ L_{t−τ}(δ)

We will consider old and new subspaces separately. The reason for this will become apparent in §7. The proof of Theorem 2.6 consists of the following steps:

Step 1 (Contribution of old subspaces). There exists a constant τ > 0 (depending only on μ_{1} and Λ) such that for any ω > 1 and any ξ > 0 there exist constants κ = κ(ω, ξ, μ_{1}, Λ) < 1, t_{0} = t_{0}(μ_{1}, Λ), δ_{0} = δ_{0}(ω, ξ, τ, μ_{1}, Λ) > 0, and B = B(ω, ξ, τ, μ_{1}, Λ) > 0 such that for any δ < δ_{0}, any t > t_{0} + τ, and any σ, ρ ∈ {+, −},

| ⋃_{L∈L_{t,old}(δ)} R_{t}^{L,σρ}(ωδ)| ≤ κ | ⋃_{L∈L_{t−τ} (δ)} R_{t−τ}^{L,σρ}(ωδ)| + Bδ^{2−ξ}

Step 2 (Contribution of new subspaces). For any ω > 1 there exist constants t_{0} = t_{0}(μ_{1}, Λ), δ_{0} = δ_{0}(μ_{1}, Λ, ω) > 0, and B_{1} = B_{1}(μ_{1}, Λ, ω) such that for any τ > 0, any δ < δ_{0}, any t > t_{0} + τ, and any σ, ρ ∈ {+, −},

| ⋃_{L∈L_{t,new}(δ)} R_{t}^{L,σρ}(ωδ)| ≤ B_{1}δ^{1.05}.

Proof of Theorem 2.6 assuming Step 1 and Step 2 : To prove (14) it is clearly enough to estimate the measure of the right hand side of (26) with fixed σ and ρ. Henceforth we fix σ and ρ and drop the ±± from the notation.

Choose ξ > 0 such that 1.05 < 2−ξ, and choose τ such that Step 1 holds. Choose ω_{1} = ω_{1}(τ) > 1 such that for any v ∈ ∧^{2}ℝ^{4} and any s, 0 < s < τ, ω_{1}^{−1}∥v∥ ≤ ∥a_{s}v∥ ≤ ω_{1}∥v∥. Choose ω = c^{−1}ω_{1}, where c is as in Proposition 4.2 (a). Suppose δ is sufficiently small and t is sufficiently large so that both Step 1 and Step 2 hold. In the argument below, all implied constants depend on μ_{1}, Λ, τ and ω. Let

h(t) = | ⋃_{L∈L_{t}(δ/ω_{1})} R_{t}^{L}(ωδ)|

h_{old}(t) = | ⋃_{L∈L_{t,old}(δ/ω_{1})} R_{t}^{L}(ωδ)|

and

h_{new}(t) = | ⋃_{L∈L_{t,new}(δ/ω_{1})} R_{t}^{L}(ωδ)|

so that h(t) ≤ h_{old}(t) + h_{new}(t). By Step 2, h_{new}(t) = O(δ^{1.05}). By Step 1,

h_{old}(t + τ) ≤ κh(t) + O(δ^{2−ξ}) ≤ κh_{old}(t) + O(δ^{1.05})  (27)

We may assume that δ is small enough so that h(0) = 0. Hence, since κ < 1, (27) implies that for n ∈ ℕ, h_{old}(nτ) = O(δ^{1.05}). Then, for n ∈ ℕ, h(nτ) = O(δ^{1.05}).

Now let t > 0 be arbitrary. We may write t = nτ + s, where 0 < s < τ. Then, by the definition of ω_{1}, for each L ∈ L_{t}(δ/ω_{1}), L ∈ L_{nτ}(δ), and R_{t}^{L}(c^{−1}δ) ⊂ R_{nτ}^{L}(ωδ). Hence, the measure of the right hand side of (26) is bounded by h(nτ) = O(δ^{1.05}). This proves (14).

6 Proof of Step 1

The proof is based on Proposition 4.2 (b); indeed, if the rectangles R^{L}(ωδ) were disjoint, then Step 1 would follow directly from Proposition 4.2 (b). In general, these rectangles are not disjoint, but the overlap is contained in the region where either α_{1} or α_{3} is large. In fact, we have the following lemma (which follows immediately from Lemma 2.4):

Lemma 6.1. Suppose L and M are two distinct 2-dimensional Δ-rational lattices with d(L) < δ, d(M) < δ. Then α_{1}(Δ) > 1/δ or α_{3}(Δ) > 1/δ. Hence, by Proposition 4.2 (a),

R_{t}^{L}(ωδ)∩R_{t}^{M}(ωδ) ⊂ {k ∈ K : α_{1}(a_{t}kΛ) > c/(ωδ)}∪{k ∈ K : α_{3}(a_{t}kΛ) > c/(ωδ)} (28) where c is as in Proposition 4.2 (a).

Now the measure of the right hand side of (28) can be bounded by using Theorem 2.5. However, to complete the proof, we will need a certain covering lemma (Lemma 6.4 below).

6.1 Two covering lemmas

Definition 6.2. A rectangle R ∈ ℝ^{2} will be called a quasi-square if the ratio of its sides is between 1/2 and 2.

Lemma 6.3 (Covering by quasi-squares). Let {S_{j}} be countable collection of quasi-squares in ℝ^{2}, with sides parallel to the coordinate axes, and let E = ⋃_{j} S_{j}. Then there is a subcollection D of pairwise disjoint quasi-squares such that E ⊂ ⋃_{D} 6S_{j} and so that the total area of the remaining quasi-squares satisfies

| ⋃_{S∉D} S| ≤ |⋃_{i≠j}(6S_{i}) ∩ (6S_{j})|  (29)

Proof. This is well known. The proof consists of ordering the quasi-squares by size, and repeatedly adding to P the largest quasi-square which is disjoint from the union of the quasi-squares already in D. Note that each remaining quasi-square S is contained in (6S) ∩ (6T) where T is a quasi-square already in D.

Lemma 6.4 (Covering by rectangles). Let {R_{j}} be a countable collection of rectangles in ℝ^{2} with sides parallel to the coordinate axes. Suppose

|⋃_{j≠k}(6R_{j}) ∩ (6R_{k})| ≤ ϵ  (30)

Then for ν ≪ 1,

|⋃_{j} νR_{j}| ≤ 42ν |⋃_{j} R_{j}| + 21ϵ  (31)

Proof. Using cuts perpendicular to its longer side, each of the rectangles R_{j} may be divided into a finite set of quasi-squares. Partitioning these into 21 classes P_{j}^{(k)}, k = 0 … 20, so that a quasi-square from class P_{j}^{(k+1)} follows a quasi-square from class P_{j}^{(k)} (mod 21), we obtain R_{j} = ⊔_{k=0}^{20} ⊔_{S∈P_{j}^{(k)}} S. For each 1 ≤ k ≤ 21 let P^{(k)} = ⋃ P_{j}^{(k)}.

If S is a quasi-square [−a, a]×[−b, b], let S^{†} denote the “cross” [−νa, νa]×[−b, b]∪ [−a, a] × [−νb, νb]. Clearly |S^{†}| ≤ 2ν|S|, and

⋃_{j}(νR_{j}) ⊂ ⋃_{j,k} ⋃_{S∈P_{j}^{(k)}} S^{†} = ⋃_{k} ⋃_{S∈P^{(k)}} S^{†}

Hence to estimate |⋃_{j}(νR_{j})|, we need to estimate |⋃_{S∈P^{(k)}} S^{†}| for each k. Note that

if S, T ∈ P^{(k)} and S ≠ T, then (6S) ∩ (6T) ⊂ ⋃_{i≠j}(6R_{i}) ∩ (6R_{j}).

Thus, by Lemma 6.3, we obtain a subcollection D^{(k)} ⊂ P^{(k)} of disjoint quasisquares such that

| ⋃_{S∈P^{(k)} - D^{(k)}} S| ≤ | ⋃_{S≠T^{S,T ∈P^{(k)}}}(6S) ∩ (6T)| ≤ |⋃_{i≠j}(6R_{i}) ∩ (6R_{j})| ≤ ϵ.  (32)

Observe that |⋃_{S∈D^{(k)}} S^{†}| ≤ 2ν |⋃_{S∈D^{(k)}} S|. Thus for each k,

\begin{aligned}| ⋃_{S∈P^{(k)}} S^{†}| & ≤ | ⋃_{S∈D^{(k)}} S^{†}| + | ⋃_{S∈P^{(k)} - D^{(k)}} S^{†}| \\ & ≤ 2ν | ⋃_{S∈D^{(k)}} S| + | ⋃_{S∈P^{(k)} - D^{(k)}} S| \end{aligned}

Using (32) the lemma follows.

6.2 The proof of Step 1

We may assume that ω is sufficiently large and δ sufficiently small so that by Proposition 4.2 (b), for L ∈ L_{t,old}(δ), R_{t}^{L}(ωδ) ⊂ Ce^{−τ}R_{t−τ}^{L}(ωδ). Also by Lemma 6.1 and Theorem 2.5, for any ξ > 0,

| ⋃_{L≠M^{L,M∈L_{t−τ} (δ)}} R_{t−τ}^{L}(ωδ) ∩ R_{t−τ}^{M}(ωδ)| ≤ |{k : α_{1}(a_{t}kΛ) > c/(ωδ)}| +

+ |{k : α_{3}(a_{t}kΛ) > c/(ωδ)}| = O(δ^{2−ξ})

where the implied constant depends only on μ_{1}, Λ, ξ and ω. Choose τ > 0 such that Ce^{−τ/2} < 1/2, and let κ = 2Ce^{−τ/2}. Then, by Lemma 6.4,

| ⋃_{L∈L_{t,old}(δ)} R_{t}^{L}(ωδ)| ≤ 2Ce^{−τ/2} | ⋃_{L∈L_{t,old}(δ)} R_{t−τ}(ωδ)| + O(δ^{2−ξ})

≤ κ | ⋃_{L∈L_{t−τ} (δ)} R_{t−τ}(ωδ)| + O(δ^{2−ξ})

7 The nesting property

In this section we begin the proof of Step 2. In this section (and the rest of the proof of Step 2) τ and t are fixed; we also assume that t is sufficiently large and δ sufficiently small so that Proposition 4.2 holds. In this section all constants are independent of t and δ, but may depend on μ_{1}, Λ, ω, τ and the constants in §4.

Let c < 1 be as in Lemma 4.1. Let N > 1 be a parameter to be chosen later (see Proposition 7.4 below), and let c_{0} = c/N^{3}. If L is not μ_{1}-quasinull, then by Lemma 4.1 at least one of the rectangles R_{1}^{L} = R_{t}^{L}(c_{0}δ^{−0.1}, c_{0}) or R_{2}^{L} = R_{t}^{L}(c_{0}, c_{0}δ^{−0.1}) does not wrap around the torus. Let’s call this rectangle R_{big}^{L}. We will also use the “intermediate” rectangles R_{int}^{L} = R^{L}(δ^{0.1}).

The strategy of the proof of Step 2 is as follows: For every L ∈ L_{t,new}(δ), R^{L}(ωδ) ⊂ R_{int}^{L} ⊂ R_{big}^{L}. We have the following estimates (which follow immediately from Proposition 4.2 (c)):

Lemma 7.1. Suppose L ∈ L_{t}(δ). Then,

|R_{t}^{L}(ωδ)| = O_{N} (δ^{0.9}|R_{int}^{L}|)  (33)

and

|R_{t}^{L}(ωδ)| = O_{N} (δ^{1.05}|R_{big}^{L}|)  (34)

If the rectangles {R_{big}^{L} : L ∈ L_{t}(δ)}, were disjoint, Step 2 would follow immediately from Lemma 7.1. However, in general, these rectangles may intersect. Their main combinatorial property is the following:

Lemma 7.2. Suppose L and M are two distinct elements in L_{t}(δ), and R_{t}^{L}(ωδ) intersects R_{big}^{M}. Then L intersects M non-trivially (i.e. dim(L ∩ M) > 0). More generally, for any N > 1, there exists δ_{0} = δ_{0}(N, μ_{1}, Λ, ω) such that if δ < δ_{0} and R_{t}^{L}(ωδ) intersects NR_{big}^{M}, then L and M intersect non-trivially.

Proof of Lemma 7.2. Pick k ∈ R_{t}^{L}(ωδ) ∩ NR_{big}^{M}. Then d(a_{t}kL) = O_{N}(δ), and d(a_{t}kM) = O_{N}(δ^{−0.1}). In particular, d(a_{t}kL)d(a_{t}kM) = O_{N}(δ^{0.9}) ≪ 1. Hence, by Lemma 2.4 a_{t}kL and a_{t}kM intersect non-trivially. Thus, L and M intersect nontrivially.

Note that Lemma 7.2 can be applied only when a “small” rectangle R_{t}^{L}(ωδ) intersects a “big” rectangle R_{big}^{M}; there is no information gained when two “big” rectangles overlap. The following property is useful for overcoming this difficulty:

Definition 7.3 (Nesting Property). Let R_{j} be a collection of rectangles in ℝ^{2}. This collection has the nesting property if there is a constant N > 1 such that for any i, j, i ≠ j, one of the following holds:

(i) R_{i} and R_{j} are disjoint.

(ii) R_{j} ⊂ NR_{i}.

(iii) R_{i} ⊂ NR_{j}.

For example, collection of squares have the nesting property. However, in general, collections of rectangles do not (unless for example the aspect ratio is bounded).

The main result of this section is the following proposition:

Proposition 7.4 (New subspaces have the nesting property). There is a constant N > 1, independent of t and δ and a decomposition of L_{t,new}(δ) into four disjoint subsets L_{t}^{q}(δ) such that for each q, the collections {R_{int}^{L} : L ∈ L_{t}^{q}(δ)} and {R_{big}^{L} : L ∈ L_{t}^{q}(δ)} both have the nesting property.

Proposition 7.4 is the main advantage of working with L_{t,new}(δ) instead of L_{t}(δ). Before beginning the proof of Proposition 7.4 we state a corollary:

Corollary 7.5. Suppose L, M ∈ L_{t}^{q}(δ) and R_{big}^{L} intersects R_{big}^{M}. Then L intersects M non-trivially.

Proof of Corollary 7.5. Suppose R_{big}^{L} intersects R_{big}^{M}. Using Proposition 7.4 we may assume without loss of generality that R_{big}^{L} ⊂ NR_{big}^{M}, hence R_{t}^{L}(ωδ) ⊂ NR_{big}^{M}. Now the corollary follows from Lemma 7.2.

We now start the proof of Proposition 7.4: We first note the following:

Lemma 7.6. Suppose L ∈ L_{t,new}(δ). Then M_{t}(L) ≈ δ (where M_{t}(L) is defined in §4).

Proof. Since L ∈ L_{t}(δ), M_{t}(L) ≤ δ. Since L ≠ L_{t−τ}(δ), M_{t−τ}(L) ≥ δ; then M_{t}(L) ≥ δ/∥a_{τ}∥.

The advantage of L_{t,new}(δ) compared to L_{t}(δ) is the following:

Lemma 7.7 (Fixed side). Suppose L ∈ L_{t,new}(δ). Then for any η ≫ δ, the rectangle R_{t}^{L}(η) has at least one side of length ≈ e^{−t}δ^{−1/2}η^{1/2}. For any η_{1}, η_{2} ≫ δ, R_{t}^{L}(η_{1}, η_{2}) has at least one side of length either ≈ e^{−t}δ^{−1/2}η_{1}^{1/2} or ≈ e^{−t}δ^{−1/2}η_{2}^{1/2}.

Proof. Since L ∈ L_{t}(δ), M_{t}(L) ≤ δ. Since L ∉ L_{t−τ}(δ), M_{t−τ}(L) > δ. Hence the second alternative of part (d) of Proposition 4.2 cannot hold. Hence the first alternative holds, and thus R_{t−τ}^{L}(η) has a side of length ≈ e^{−t}M_{t−τ}(L)^{−1/2}η^{1/2} ≈ e^{−t}δ^{−1/2}η^{1/2}. Thus the same holds for R_{t}^{L}(η). The proof of the second assertion is identical.

Proof of Proposition 7.4. We only prove the assertion for the “big” rectangles; the proof for the “intermediate” rectangles is identical. Suppose L ∈ L_{t,new}(δ). By Lemma 7.7, there exist constants c_{3}, c_{4} depending only on μ_{1}, Λ, ω and τ such that R_{big}^{L} either has a side of length between c_{3}e^{−t}δ^{−0.55} and c_{4}e^{−t}δ^{−0.55} or has a side of length between c_{3}e^{−t}δ^{−1/2} and c_{4}e^{−t}δ^{−1/2}; this side can be either “vertical” or “horizontal”. Thus there are four possibilities for each L. For 1 ≤ q ≤ 4 let L_{t}^{q}(δ) denote the set of L ∈ L_{t,new}(δ) for which the q’th possibility occurs. Now let N = 2c_{4}/c_{3}; the proposition follows.

We will also use the following version of the nesting property:

Lemma 7.8 (Nesting Lemma). Suppose L_{1}, … L_{m} ∈ L_{t}^{q}(δ). Then one of the following holds:

(a) ⋂_{j=1}^{m} R_{big}^{L_{j}} = ∅.

(b) One can reorder the sets so that for 1 ≤ i < j ≤ m,

R_{big}^{L_{i}} ⊂ NR_{big}^{L_{j}}.  (35)

Proof. In view of Lemma 7.7, one can just order the rectangles R_{big}^{L} by increasing size.

8 Proof of Step 2

In this section all implied constants depend on μ_{1}, Λ, τ, ω and N (and hence ultimately only on μ_{1} and Λ).

8.1 Preliminaries

In view of Corollary 7.5, if some point belongs to many sets of the form R_{big}^{L_{i}} where all L_{i} ∈ L_{t}^{q}(δ), then for i ≠ j, dim(L_{i} ∩ L_{j}) = 1. The following lemma is immediately obvious:

Lemma 8.1. Suppose L_{1}, L_{2}, …, L_{m} are distinct two-dimensional subspaces of ℝ^{4} and for 1 ≤ i < j ≤ m, dim(L_{i} ∩ L_{j}) = 1. Then at least one of the following holds:

(a) dim(L_{1} ∩ L_{2} ∩ ⋯ ∩ L_{m}) = 1.

(b) dim(L_{1} + L_{2} + ⋯ + L_{m}) = 3.

Thus, if many rectangles R^{L_{j}} have a common point, then either all the L_{j} have a common vector, or all the L_{j} lie in a common 3-dimensional subspace, or both. In view of this, we will need some information about the sets F (η) = {k : d(a_{t}kH) < η}, where H is either a vector or a three-dimensional subspace. Thus, we will use the following lemma:

Lemma 8.2. Let H be a Λ-rational subspace of dimension 1 or 3. For η > 0 let F (η) denote the set {k ∈ K : d(a_{t}kH) < η}. Suppose R_{1} and R_{2} are disjoint rectangles such that for i = 1, 2, |R_{i} ∩ F (η)| ≥ 0.9|R_{i}|, and let λ > 1 be such that 3λR_{1} and 3λR_{2} are still disjoint and of diameter at most π/8. Then, for some i ∈ {1, 2},

|R_{i}| ≤ c_{1}λ^{−1}|(λR_{i}) ∩ F (c_{2}η)|

where c_{1} > 0 and c_{2} > 1 depend only on Λ.

Proof. See Appendix B.

8.2 Proof of Step 2

Notational convention: In this subsection, we use the notation α_{13}(Δ) = max(α_{1}(Δ), α_{3}(Δ)). Let L_{t}^{q}(δ) be as in §7. We fix q.

Definition 8.3 (The class Ω_{0}). Let Ω_{0} ⊂ L_{t}^{q}(δ) denote the subspaces L ∈ L_{t}^{q}(δ) such that

R_{t}^{L}(ωδ) ⊂ {k ∈ K : α_{13}(a_{t}kΛ) > b_{0}δ^{−0.55}},  (36)

where b_{0} is a constant to be chosen later, depending only on ω, τ, μ_{1}, Λ and N (and independent of t and δ).

Lemma 8.4 (The contribution of Ω_{0}).

| ⋃_{L∈Ω_{0}} R_{t}^{L}(ωδ)| ≤ |{k ∈ K : α_{13}(a_{t}kΛ) ≥ b_{0}δ^{−0.55}}| = O(δ^{1.05}).

Proof. This follows immediately from (36) and Theorem 2.5.

Lemma 8.5 (Intermediate rectangles are disjoint). Suppose L, M ∈ L_{t}^{q}(δ) - Ω_{0}, and L ≠ M. Then R_{int}^{L} and R_{int}^{M} are disjoint.

Proof. Suppose not. Then by Proposition 7.4, we may assume that R_{int}^{L} ⊂ NR_{int}^{M}. Then R_{t}^{L}(ωδ) ⊂ NR_{int}^{M}. Let k ∈ R_{t}^{L}(ωδ) ⊂ NR_{int}^{M} be arbitrary. Then d_{a_{t}kΛ}(a_{t}kL) = O(δ) and d_{a_{t}kΛ}(a_{t}kM) = O(δ^{0.1}). Hence

d_{a_{t}kΛ}(a_{t}kL) d_{a_{t}kΛ}(a_{t}kM) = O(δ^{1.1}).

Then by Lemma 2.4,

d_{a_{t}kΛ}(a_{t}k(L ∩ M)) d_{a_{t}kΛ}(a_{t}k(L + M)) = O(δ^{1.1}).

Thus, either d_{a_{t}kΛ}(a_{t}k(L ∩ M)) = O(δ^{0.55}) or d_{a_{t}kΛ}(a_{t}k(L + M)) = O(δ^{0.55}). In either case, α_{13}(a_{t}kΛ) ≥ b_{0}δ^{−0.55} as long as b_{0} is sufficiently small. Since k ∈ R_{t}^{L}(ωδ) is arbitrary, this implies that L belongs to Ω_{0} which is a contradiction.

We now choose b_{0} in Definition 8.3 so that Lemma 8.5 holds.

Definition 8.6 (The class Ω_{1}). Let Ω_{1} ⊂ L_{t}^{q}(δ) - Ω_{0} denote the set of subspaces L ∈ L_{t}^{q}(δ) - Ω_{0} such that

|R_{t}^{L}(ωδ) ∩ {k ∈ K : α_{13}(a_{t}kΛ) > b_{1}δ^{−0.55}}| ≥ 0.1|R_{t}^{L}(ωδ)|  (37)

where b_{1} is a constant to be chosen later, depending only on ω, τ, μ_{1}, Λ and N (and independent of t and δ).

Lemma 8.7 (The contribution of Ω_{1}).

| ⋃_{L∈Ω_{1}} R_{t}^{L}(ωδ)| = O(δ^{1.05}).

Proof. In view of Lemma 8.5 and the fact that R_{t}^{L}(ωδ) ⊂ R_{int}^{L}, the rectangles {R_{t}^{L}(ωδ) : L ∈ Ω_{1}} are pairwise disjoint. Now the estimate follows immediately from (37) and Theorem 2.5.

The following lemma is a refinement of Lemma 7.2:

Lemma 8.8. Suppose L, M ∈ L_{t}^{q}(δ) - Ω_{0} - Ω_{1}, L ≠ M and R_{t}^{L}(ωδ) ⊂ NR_{big}^{M}. Then dim(L ∩ M) = 1, and for all k ∈ R_{t}^{L}(ωδ) ∩ {k ∈ K : α_{13}(a_{t}kΛ) ≤ b_{1}δ^{−0.55}},

d_{a_{t}kΛ}(a_{t}k(L ∩ M)) = O(δ^{0.35})  and  d_{a_{t}kΛ}(a_{t}k(L + M)) = O(δ^{0.35}).

Proof. Suppose k ∈ R_{t}^{L}(ωδ). Then d_{a_{t}kΛ}(a_{t}kL) = O(δ) and d_{a_{t}kΛ}(a_{t}kM) = O(δ^{−0.1}). Hence, by Lemma 2.4,

d_{a_{t}kΛ}(a_{t}k(L ∩ M)) d_{a_{t}kΛ}(a_{t}k(L + M)) = O(δ^{0.9}).  (38)

Hence, at least one of the factors in (38) is O(δ^{0.45}) ≪ 1. This implies that a_{t}kL and a_{t}kM intersect non-trivially, hence L and M intersect non-trivially, so dim(L ∩ M) = 1.

Now suppose k ∈ R_{t}^{L}(ωδ) ∩ {k : α_{13}(a_{t}kΛ) ≤ b_{1}δ^{−0.55}}. Then,

d_{a_{t}kΛ}(a_{t}k(L ∩ M)) ≥ α_{3}^{−1}(a_{t}kΛ) ≥ b_{1}^{−1}δ^{0.55}  (39)

and

d_{a_{t}kΛ}(a_{t}k(L + M)) ≥ α_{1}^{−1}(a_{t}kΛ) ≥ b_{1}^{−1}δ^{0.55}.  (40)

Now from (38), (39) and (40) it follows that for k ∈ R_{t}^{L}(ωδ) ∩ {k : α_{13}(a_{t}kΛ) ≤ b_{1}δ^{−0.55}},

d_{a_{t}kΛ}(a_{t}k(L ∩ M)) = O(δ^{0.35})  and  d_{a_{t}kΛ}(a_{t}k(L + M)) = O(δ^{0.35}). (41)

But, since L ∉ Ω_{1}, |R_{t}^{L}(ωδ) ∩ {k : α_{13}(a_{t}kΛ) ≤ b_{1}δ^{−0.55}}| ≥ 0.9|R_{t}^{L}(ωδ)|.

We now choose b_{1} in Definition 8.6 so that Lemma 8.8 holds.

Definition 8.9 (The class Ω_{2}). Let Ω_{2} ⊂ L_{t}^{q}(δ) - Ω_{0} - Ω_{1} denote the set of subspaces L ∈ L_{t}^{q}(δ) - Ω_{0} - Ω_{1} such that

|R_{t}^{L}(ωδ)| ≤ b_{3}δ^{0.45}|R_{int}^{L} ∩ {k ∈ K : α_{13}(a_{t}kΛ) ≥ b_{4}δ^{−0.35}}|

where the constants b_{3} < ∞ and b_{4} > 0 are to be chosen later, depending only on ω, τ, μ_{1}, Λ and N (and independently of t and δ).

Lemma 8.10 (The contribution of Ω_{2}).

| ⋃_{L∈Ω_{2}} R_{t}^{L}(ωδ)| = O(δ^{1.05}).

Proof. This is a formal consequence of Lemma 8.5 and Theorem 2.5.

Lemma 8.11. Suppose L_{1}, L_{2}, L_{3} ∈ L_{t}^{q}(δ) - Ω_{0} - Ω_{1} - Ω_{2}. Then R_{big}^{L_{1}} ∩ R_{big}^{L_{2}} ∩ R_{big}^{L_{3}} = ∅.

Proof. Suppose not. Then by Lemma 7.8, after possibly renumbering the L_{j}, we may assume that for 1 ≤ j < k ≤ 3, R_{big}^{L_{j}} ⊂ NR_{big}^{L_{k}}. Then by the first assertion of Lemma 8.8 (or by Lemma 7.2), for 1 ≤ j < k ≤ 3, dim(L_{j} ∩ L_{k}) = 1. Now by Lemma 8.1, either dim(L_{1} ∩L_{2} ∩L_{3}) = 1 or dim(L_{1} +L_{2} +L_{3}) = 3 (or both). Without loss of generality, we may assume that the former holds; then let H = L_{1} ∩ L_{2} ∩ L_{3}.

Let R_{j} = R_{t}^{L_{j}}(ωδ). Since for j = 1, 2, R_{j} ⊂ NR_{big}^{L_{3}}, by Lemma 8.8 for every k ∈ R_{j} either α_{13}(a_{t}kΛ) > b_{1}δ^{−0.55} or d_{a_{t}kΛ}(a_{t}kH) ≤ b_{5}δ^{0.35} where b_{5} is the implied constant in Lemma 8.8. Since L_{j} ∉ Ω_{1}, the set R_{j} ∩ {k : α_{13}(a_{t}kΛ) ≤ b_{1}δ^{−0.55}} is of area at least 0.9|R_{j}|, and for each of its points k, d_{a_{t}kΛ}(a_{t}kH) ≤ b_{5}δ^{0.35}. Hence |R_{j} ∩ {k : d_{a_{t}kΛ}(a_{t}kH) ≤ b_{5}δ^{0.35}}| ≥ 0.9|R_{j}|.

We now choose λ to be the largest number such that 3λR_{j} ⊂ R_{int}^{L_{j}}. Then by Lemma 8.5, 3λR_{1} and 3λR_{2} are disjoint. In view of the construction of the rectangles R^{L}(η) we have λ^{−1} = O(δ^{0.45}). Observe that the diameters of 3λR_{j} are smaller then π/8. We may now apply Lemma 8.2 with η = b_{5}δ^{0.35} and conclude (as long as b_{3} in Definition 8.9 is sufficiently big and b_{4} in Definition 8.9 is sufficiently small), that for some j ∈ {1, 2}, L_{j} ∈ Ω_{2}. This is a contradiction.

We now choose b_{3}, b_{4} in Definition 8.9 so that Lemma 8.11 holds.

Lemma 8.12 (The contribution of the rest of the subspaces). Let L′ denote L_{t}^{q}(δ) - Ω_{0} - Ω_{1} - Ω_{2}. Then

| ⋃_{L∈L′} R_{t}^{L}(ωδ)| = O(δ^{1.05}).

Proof. In view of Lemma 8.11, the rectangles {R_{big}^{L} : L ∈ L′} cover each point at most twice. Now the estimate follows from Lemma 7.1.

Now Step 2 follows immediately from Lemma 8.4, Lemma 8.7, Lemma 8.10 and Lemma 8.12. This completes the proof of Theorem 2.6.

9 Quasinull subspaces and rectangular tori

For the case of the rectangular torus of sides π and π/β, the eigenvalues are values at integers of the binary quadratic form q_{β}(m, n) = m^{2} + β^{2}n^{2}. Let Q_{β}(x_{1}, x_{2}, x_{3}, x_{4}) = q_{β}(x_{1}, x_{2})−q_{β}(x_{3}, x_{4}). In this case we have 4 isotropic subspaces, i.e. {x_{1} = ±x_{3}, x_{2} = ±x_{4}}.

Let B(·) be the standard form as defined in §2. Then Q_{β}(v) = B(g_{Q_{β}}v), where g_{Q_{β}} is given by the change of variable z_{1} = x_{1} − x_{3}, z_{2} = β(x_{2} − x_{4}), z_{3} = β(x_{2} + x_{4}), z_{4} = x_{1} +x_{3}. Let Λ_{β} = g_{Q_{β}}ℤ^{4}; then Λ_{β} consists of the vectors (w_{1}, βw_{2}, βw_{3}, w_{4}), with w_{i} ∈ ℤ, and also w_{1} ± w_{4} ∈ 2ℤ, w_{2} ± w_{3} ∈ 2ℤ. By construction, Q_{β}(ℤ^{4}) = B(Λ_{β}).

In this section, we give an elementary proof of Theorem 1.5 for the special case of rectangular tori (i.e. Q = Q_{β}). We first make the following:

Definition 9.1 (Exceptional subspaces). Fix a constant 0 < μ < 1/2. A subspace L of ℝ^{4} is called exceptional if dim L = 2 and L spanned by the vectors u_{1} = (A, B, −A, B) and u_{2} = (C, D, C, −D) or by the vectors u_{1} = (A, B, −A, −B) and u_{2} = (C, D, C, D) where A, B, C, D ∈ ℤ − {0} satisfy the condition

|\frac{AD}{BC} − β^{2}| < \frac{μ}{(BC)^{2}}  (42)

Observe that (42) implies that AD/BC is a convergent p_{n}/q_{n} in the continued faction expansion of b^{2}. Note also that n is determined by L and let us denote Φ(L) = n.

Note that Q_{β}(u_{1}) = Q_{β}(u_{2}) = 0, but the restriction of Q_{β} to L is not-zero. In fact,

Q_{β}(λ_{1}u_{1} + λ_{2}u_{2}) = (AD − β^{2}BC)λ_{1}λ_{2}.  (43)

Lemma 9.2 (Quasinull implies exceptional). Let Q_{β},g_{Q_{β}}, Λ_{β} be as in the beginning of this section. There exist constants μ_{1} > 0, c > 1 (depending only on μ in Definition 9.1 and β) such that the following holds: Let L be any Λ_{β}-rational subspace with ∥v^{L}∥ > c which is μ_{1}-quasinull with respect to Λ_{β}. Then the ℤ^{4}-rational subspace g_{Q_{β}}^{−1}L is an exceptional subspace in the sense of Definition 9.1.

Proof. Let v_{1} = (w_{1}, βw_{2}, βw_{3}, w_{4}) and v_{2} = (w′_{1}, βw′_{2}, βw′_{3}, w′_{4}) be a reduced integral basis for L ∩ Λ_{β}. From the definition of Λ_{β}, w_{1}, w_{2}, w_{3}, w_{4} ∈ ℤ and w′_{1}, w′_{2}, w′_{3}, w′_{4} ∈ ℤ. Since the basis is reduced, ∥v^{L}∥ = ∥v_{1} ∧ v_{2}∥ ≥ c_{0}∥v_{1}∥∥v_{2}∥ where c_{0} is an absolute constant. Let μ_{1} = c_{0}μ/4 and let c = 4μ_{1}/β.

Since ∥v^{L}∥ = ∥π_{1}(v^{L})∥ + ∥π_{2}(v^{L})∥ > c, (8) implies that either

∥π_{1}(v^{L})∥ < \frac{2μ_{1}}{∥v^{L}∥} < \frac{β}{2}  (44)

or

∥π_{2}(v^{L})∥ < \frac{2μ_{1}}{∥v^{L}∥} < \frac{β}{2}  (45)

Without loss of generality, we may assume that (45) holds. Using Lemma 2.1, we get

π_{2}(v^{L}) = β(w_{1}w′_{3} − w_{3}w′_{1})e_{11} ∧ e_{21} + β(w_{2}w′_{4} − w_{4}w′_{2})e_{12} ∧ e_{22}+

+ \frac{1}{2}((w_{1}w′_{4} − w_{4}w′_{1}) + β^{2}(w_{2}w′_{3} − w_{3}w′_{2}))(e_{11} ∧ e_{22} + e_{12} ∧ e_{21}) (46)

Considering the coefficient of e_{11} ∧ e_{21} we get β|w_{1}w′_{3} − w_{3}w′_{1}| < β. Hence |w_{1}w′_{3} − w_{3}w′_{1}| < 1. But w_{1}w′_{3} − w_{3}w′_{1} ∈ ℤ, thus w_{1}w′_{3} − w_{3}w′_{1} = 0. Similarly, by considering the coefficient of e_{12} ∧ e_{22} we get w_{2}w′_{4} − w_{4}w′_{2} = 0. Thus, v_{1} and v_{2} may be written as

v_{1} = λ_{1}(y_{1}, 0, βy_{3}, 0) + λ_{2}(0, y_{2}, 0, βy_{4}) and v_{2} = λ′_{1}(y_{1}, 0, βy_{3}, 0) + λ′_{2}(0, y_{2}, 0, βy_{4}),

(47)

where λ_{1}, λ_{2}, λ′_{1}, λ′_{2}, y_{1}, y_{2}, y_{3}, y_{4} ∈ ℤ. Substituting into the coefficient of e_{11} ∧ e_{22} + e_{12} ∧ e_{21} in (46) and using (45) we get

|(λ_{1}λ′_{2} − λ_{2}λ′_{1})(y_{1}y_{4} + β^{2}y_{2}y_{3})| < \frac{4μ_{1}}{∥v^{L}∥}

Since λ_{1}λ′_{2} − λ_{2}λ′_{1} ∈ ℤ and λ_{1}λ′_{2} − λ_{2}λ′_{1} ≠ 0 (otherwise v_{1} and v_{2} would be linearly dependent), |λ_{1}λ′_{2} − λ_{2}λ′_{1}| ≥ 1. Also ∥v^{L}∥ ≥ c_{0}∥v_{1}∥∥v_{2}∥ ≥ c_{0}|y_{2}y_{3}|. Hence

|y_{1}y_{4} + β^{2}y_{2}y_{3}| < \frac{4μ_{1}}{c_{0}|y_{2}y_{3}|} < \frac{μ}{|y_{2}y_{3}|}  (48)

Now the lemma follows from (47), (48) and the definition of g_{Q_{β}}.

Lemma 9.3 (Contribution of exceptional lattices). Suppose β^{2} is diophantine, i.e. there exists N > 0 such that for all relatively prime pairs of integers (p, q), |β^{2} − p/q| > q^{−N}. Then, for any interval (a, b), lim sup_{T →∞} \frac{1}{T^{2}} X_{β}(a, b, T) = 0, where X_{β} counts the points in the exceptional subspaces.

Proof. Let p_{n}/q_{n} denote the continued fraction approximations to β^{2}. Then, since β^{2} is diophantine, ϵ_{n} = |β^{2} − p_{n}/q_{n}| > q_{n}^{−N} for all n.

Suppose L is an exceptional subspace. Without loss of generality, we may assume that L is spanned by the vectors u_{1} = (A, B, −A, B) and u_{2} = (C, D, C, −D) where A, B, C, D ∈ ℤ and because of (42), (AD)/(BC) = p_{n}/q_{n} for n = Φ(L). Hence, there exists ν > 0 such that AD = νp_{n} and BC = νq_{n}.

Let N_{L}(a, b, T) denote the number of non-diagonal solutions in L, i.e the number of vectors v = λ_{1}u_{1} +λ_{2}u_{2} with λ_{1}, λ_{2} ∈ ℤ -{0} such that ∥v∥ < T and a < Q_{β}(v) < b.

Note that u_{1} and u_{2} are orthogonal. Since we are considering non-diagonal solutions both λ_{1} and λ_{2} are non-zero. Then the condition that ∥v∥ < T implies (by Schmidt’s lemma) that N_{L} < T^{2}/(∥u_{1}∥∥u_{2}∥) ≪ T^{2}/(νq_{n}).

By (43), the quadratic form restricted to L is Q(λ_{1}u_{1}+λ_{2}u_{2}) = (AD−β^{2}BC)λ_{1}λ_{2} = νq_{n}ϵ_{n}λ_{1}λ_{2}, where ϵ_{n} = \frac{AD}{BC} − β^{2} satisfies ∥ϵ_{n}∥ > q_{n}^{−N}. Thus, if there exists a nondiagonal solution, (i.e. with |λ_{1}λ_{2}| ≥ 1), we must have νq_{n}ϵ_{n} < ρ where ρ = max(|a|, |b|). Hence we need only consider exceptional subspaces with ν < ρ/(q_{n}ϵ_{n}). The total contribution N_{n} = ∑_{Φ(L)=n} N_{L} is bounded by

N_{n}(a, b, T) = ∑_{Φ(L)=n} N_{L}(a, b, T) ≤ ∑_{ν=1}^{ρ/(q_{n}ϵ_{n})} ∑_{BC=νq_{n}^{AD=νpn}} \frac{T^{2}}{νq_{n}} ≤ ∑_{ν=1}^{ρ/(q_{n}ϵ_{n})} τ(νp_{n})τ(νq_{n})\frac{T^{2}}{νq_{n}}

(49)

where τ is the divisor function. Using the estimate τ(q) < c_{ϵ}q^{ϵ} which holds for any ϵ > 0 as well as the diophantine condition on β by which ∥ϵ_{n}∥ > q_{n}^{−N}, one deduces that

N_{n}(a, b, T) ≤ C_{ϵ} \frac{T^{2}}{q_{n}^{1−(2N+4)ϵ}}  (50)

where C_{ϵ} is independent of T and n. Fix ϵ < 1/(2N +4) so that θ = 1−(2N +4)ϵ > 0. Also note that because of the form of Q_{β} restricted to L, any N_{n} is bounded by a constant R_{n} independent of T. Pick any M > 0. Then

lim sup_{T →∞} \frac{1}{T^{2}} X_{n}(a, b, T) = lim sup_{T →∞} (\frac{1}{T^{2}} ∑_{n=1}^{M−1} N_{n}(a, b, T) + \frac{1}{T^{2}} ∑_{n=M}^{∞} N_{n}(a, b, T)) ≤

≤ lim sup_{T →∞} \frac{1}{T^{2}} ∑_{n=1}^{M−1} R_{n} + C_{ϵ} ∑_{n=M}^{∞} q_{n}^{−θ} ≤ C_{ϵ} ∑_{n=M}^{∞} q_{n}^{−θ}

Letting M → ∞ we obtain the desired estimate, since the sum ∑_{n=1}^{∞}(1/q_{n}^{θ}) converges by the properties of continued fractions.

Now Theorem 1.5 in the special case Q = Q_{β}, follows from Lemma 9.2 and Lemma 9.3. This completes the proof of Theorem 1.3 in the special case of rectangular tori.

10 The contribution of quasinull subspaces

10.1 Structural preliminaries

Lemma 10.1. Given three transversal 2-dimensional subspaces V_{1}, V_{2} and V_{3} of ℝ^{4}, there is a unique (up to proportionality) quadratic form Q such that the restriction of Q to each V_{i} is zero. If the subspaces are defined over ℚ then Q is rational and split.

Proof. We may choose a basis {e_{1}, e_{2}, e_{3}, e_{4}} for ℝ^{4} such that {e_{1}, e_{2}} is a basis for V_{1} and {e_{3}, e_{4}} is a basis for V_{2}. Now the (symmetric) matrix of Q in this basis is of

the form (\begin{matrix}0 & 0 & ∗ & ∗ \\ 0 & 0 & ∗ & ∗ \\ ∗ & ∗ & 0 & 0 \\ ∗ & ∗ & 0 & 0 \end{matrix}). The remaining entries of this matrix are determined up to

proportionality by the 3 linear equations implied by the condition that the restriction of Q to V_{3} is zero.

Proposition 10.2. Let Q be a quadratic form of signature (2, 2). Then ∧^{2}ℝ^{4} decomposes into two invariant subspaces V_{1} and V_{2}.

(a) The subspaces V_{1} and V_{2} are orthogonal with respect to the 6-variable bilinear form Q^{(6)}(v, w) = v ∧ w. The restriction of Q^{(6)} to V_{i} has signature (2, 1).

(b) The pair of subspaces (V_{1}, V_{2}) determines Q up to proportionality.

(c) The map f taking (V_{1}, V_{2}) to Q/proportionality is a rational map defined over ℚ.

(d) If V_{1} is rational and the restriction of Q^{(6)} to V_{1} is split over ℚ then Q = f(V_{1}, V_{1}^{⊥}) is a split form defined over ℚ.

Proof of (a). Follows from Lemma 2.1.

Proof of (b). If V_{1} is given, then V_{2} is determined by (a). This implies that all the isotropic 2-dimensional subspaces of Q are given. Since any null vector of Q is the intersection of 2 isotropic 2-subspaces, we have that the null cone of Q is determined. Hence Q is determined up to proportionality.

Proof of (c). The rationality of f is clear from the construction. The fact that f is defined over ℚ follows from the fact that f is bijective. Note that the pair (V_{1}, V_{1}^{⊥}) may be defined over ℚ even if V_{1} is not rational.

Proof of (d). The restriction of Q^{(6)} to V_{1} is split over ℚ if and only if there exists a basis for V_{1} consisting of null-vectors for Q^{(6)}. Now the statement follows from Lemma 10.1.

Lemma 10.3. Let Q be an irrational quadratic form of signature (2, 2). Then Q has at most 4 rational isotropic subspaces.

Proof. Suppose Q has at least 5 rational isotropic subspaces. Let N denote the light cone of Q^{(6)}. Then for some i, N ∩ V_{i} has at least 3 linearly independent rational points, and thus the lemma follows from Lemma 10.1.

10.2 The number of quasinull subspaces

For a rational subspace L ⊂ ℝ^{4}, let the norm of L denote the length of v^{L} = v_{1} ∧ v_{2} ∈ ∧^{2}ℝ^{4}, where {v_{1}, v_{2}} is any integral basis for L∩ℤ^{4}. The norm is equal to the volume of L/(L ∩ ℤ^{4}).

Theorem 10.4. Suppose Q is not EWAS (see Definition 1.2). Then there exists δ > 0 such that the number of quasinull subspaces of norm between T/2 and T is O(T^{1−δ}).

To prove Theorem 10.4 we work in ∧^{2}ℝ^{4} ≅ ℝ^{6}. Instead of counting quasinull subspaces L ⊂ ℝ^{4}, we count the associated vectors v^{L} ∈ ∧^{2}ℝ^{4}. The following obvious lemma summarizes the properties of this transition.

Lemma 10.5. Given μ_{1} > 0 there exists c > 0 such that for any μ_{1}-quasinull subspace L of norm between T/2 and T, the corresponding primitive vector w = v^{L} ∈ ∧^{2}ℝ^{4} ≅ ℝ^{6} satisfies the following conditions:

(a) Q^{(6)}(w) = 0 (recall Q^{(6)}(w) denotes w ∧ w).

(b) For some i ∈ {1, 2}, ∥π_{i}(w)∥ < c/T (where π_{i} is the orthogonal projection on V_{i}).

(c) T/2 ≤ ∥w∥ ≤ T.

Conversely, the above conditions, with an appropriate choice of c = c(μ_{1}), imply that w corresponds to a 2-dimensional quasinull subspace of ℝ^{4} of norm between T/2 and T.

Hence to prove Theorem 10.4, it suffices to count the number of vectors in ℝ^{6} satisfying (a)-(c) of Lemma 10.5. Without loss of generality we may assume i = 2.

Define

Q^{(3)}(v) = Q^{(6)}(π_{1}(v))

Note that for any vector w satisfying (a)-(c) of Lemma 10.5, we have

|Q^{(3)}(w)| < const.

Let g_{T} ∈ GL(6, ℝ) denote the linear transformation which is the identity on V_{1} and stretches by a factor of T on V_{2}. Note that g_{T} preserves Q^{(3)}. Let Δ_{T} denote the image of ℤ^{6} under g_{T}.

Thus it suffices to estimate the number of primitive vectors w ∈ Δ_{T} satisfying the following conditions:

(a’) |Q^{(3)}(w)| < const.

(b’) ∥π_{2}(w)∥ < const′.

(c’) T/2 ≤ ∥w∥ ≤ T.

Let H_{1} ⊂ SL(6, ℝ) denote the group which acts via the identity on V_{2} and preserves Q^{(3)}. Then H_{1} ≅ SO(2, 1). Let {w_{1}, w_{2}, w_{3}} be a basis for V_{1} in which Q^{(3)}(xw_{1} + yw_{2} + zw_{3}) = 2xz − y^{2}. Let a_{t} ∈ H_{1} be defined by

a_{t}w_{1} = e^{t}w_{1}
a_{t}w_{2} = w_{2}
a_{t}w_{3} = e^{−t}w_{3}.

Let K ≅ SO(2) be the intersection of H_{1} with the group preserving the norm ∥xw_{1} + yw_{2} + zw_{3}∥ = (x^{2} + y^{2} + z^{2})^{1/2}. Then K is a maximal compact subgroup of H_{1}.

Proposition 10.6. There exists a bounded function f supported on a compact set in ℝ^{6} such that the number of vectors in Δ_{T} satisfying (a’)-(c’) is bounded above by

∫_{K} f̂(a_{t}kΔ_{T}) dm(k)

where dm(k) is the normalized Haar measure on K, t = log T, and

f̂(Δ) = ∑_{v∈Δ} f(v).

Proof. See [EMM, §3], and also the paragraph following the statement of Theorem 2.3 in the present paper.

Recall the definitions of the functions α_{i} and α from §2. We note the following:

Lemma 10.7.

(i) α_{6}(Δ_{T}) = T^{−3}.

(ii) for any j, 1 ≤ j ≤ 6, α_{j}(Δ_{T}) ≤ 1.

(iii) There is an absolute constant c such that for any j, 2 ≤ j ≤ 6, α_{j}(Δ_{T}) ≤ cα_{j−1}(Δ_{T}).

(iv) There is a constant C < ∞ such that α_{4}(Δ_{T}) ≤ CT^{−1/3}.

Proof. (i) is immediate from the fact that α_{6}(Λ) = 1 and the definition of Δ_{T}. (ii) and (iii) are also clear from the definitions.

To prove (iv) let W̃ denote the Δ_{T} -rational 4-dimensional subspace such that 1/d_{4}(W̃) = α_{4}(W̃), and let W = g_{T}^{−1}(W̃) denote the corresponding rational subspace. Suppose d_{4}(W̃) ≤ C^{−1}T^{1/3}, hence W̃ has a reduced basis w̃_{1}, … w̃_{4}, where all vectors have length at most C^{−1}T^{1/3}. Let w_{i} = g_{T}^{−1}w̃_{i}. Let S_{T} = {w ∈ ℝ^{4} : ∥w∥ ≤ T and ∥π_{2}(w)∥ ≤ 1/T }. Let B ⊂ W denote the intersection of W with the ball of radius C^{−1}T^{1/3}. By construction, w_{i} ∈ B ∩ S_{T}, and the w_{i} are 4 linearly independent vectors in ℤ^{4}. Then the 4-dimensional volume of B ∩ S_{T} must be bounded below by an absolute constant. But, from the form of S_{T}, the 4-dimensional volume of B ∩ S_{T} is at most an absolute constant times C^{−3}. This is a contradiction if C is sufficiently large.

Lemma 10.8. There exists absolute constants σ > 0, and ρ > 0 such that for every sufficiently small δ > 0 and any T > 2, the following holds: let Q, T, Δ_{T}, g_{T} be as in Lemma 10.7. Suppose α_{3}(Δ_{T}) ≥ T^{−δ}, and let U_{3} denote the Δ_{T} -rational 3dimensional subspace such that 1/d_{3}(U_{3}) = α_{3}(Δ_{T}). Then one of the following holds:

(a) The restriction of Q^{(6)} to g_{T}^{−1}U_{3} is anisotropic over ℚ.

(b) There exists a split integral form Q′, with ∥Q′∥ ≤ T^{σδ}, such that for some 1 ≤ λ ∈ ℝ, ∥Q − \frac{1}{λ}Q′∥ ≤ T^{−ρ}. If w ∈ g_{T}^{−1}U_{3} is any rational point satisfying Q^{(6)}(w) = 0, then w corresponds to rational isotropic subspace of Q′.

Proof. Suppose that Q^{(6)} restricted to g_{T}^{−1}U_{3} is isotropic, hence split over ℚ. Let Q′′ = f(g_{T}^{−1}U_{3}, g_{T}^{−1}U_{3}^{⊥}) denote a quadratic form of determinant ±1 associated with g_{T}^{−1}U_{3} as in Proposition 10.2. Let Q′ be the unique primitive integral form which is a multiple of Q′′. Since g_{T}^{−1}U_{3} has a reduced integral basis of vectors of norm at most T^{δ}, g_{T}^{−1}U_{3}^{⊥} also has an integral basis bounded by a fixed power of T^{δ}. Hence, the coefficents of Q′ are also bounded by a fixed power of T^{δ}, see Lemma 10.1 and Proposition 10.2 (d). Choose λ ∈ ℝ so that Q and \frac{1}{λ}Q′ have the same determinant. By Proposition 10.2 (c), ∥Q − \frac{1}{λ}Q′∥ is bounded by a fixed power of T^{δ−1}. This, for sufficiently small δ < 1, implies that (b) holds.

For 1 ≤ j ≤ 6, consider the representation of H_{1} on ∧^{j}(ℝ^{6}). We have ∧^{j}(ℝ^{6}) = W_{0} ⊕ (⊕_{i=1}^{m_{j}}W_{i}) where H_{1} acts trivially on W_{0}, and acts via the 3-dimensional representaion on each W_{i}, 1 ≤ i ≤ m_{j}. Let p_{i} denote the associated projections.

Let ∥ · ∥^{∗} denote the norm on ℝ^{3} defined in [EMM, (5.23)]. Let ∥ · ∥^{#} denote the norm on ∧^{j}(ℝ^{6}) defined by

∥w∥^{#} = max(∥p_{0}(w)∥, max_{1≤i≤m_{j}} ∥p_{i}(w)∥^{∗})

For a lattice Δ in ℝ^{4} and a Δ-rational subspace M, we define d_{Δ}^{#}(M) = ∥e_{1}∧⋯∧e_{m}∥^{#}, where {e_{1}, …, e_{m}} ⊂ Δ is a basis for M.

We now continue the proof of Theorem 10.4. Let δ be as in Lemma 10.8. We may assume δ < 1/3. Let U_{1} ⊂ U_{2} ⊂ U_{3} ⊂ U_{4} ⊂ U_{5} ⊂ U_{6} ≅ ℝ^{6} be defined as follows: U_{1} is spanned by the shortest vector in Δ_{T}, U_{2} is spanned by U_{0} and the shortest vector in Δ_{T} outside U_{1}, etc. It is a standard fact from reduction theory that there exists a constant C_{6} depending only on the dimension such that

\frac{1}{d_{i}(U_{i})} ≤ α_{i}(Δ_{T}) ≤ C_{6} \frac{1}{d_{i}(U_{i})}

By Lemma 10.7 (iv), d_{4}(U_{4}) > C_{6}^{−1}C^{−1}T^{1/3}. Let k ≥ 0 be the largest integer such that d_{k}(U_{k}) < C_{6}^{−1}C^{−1}T^{δ}. Then, k ≤ 3. For 1 ≤ i ≤ k, and g ∈ SL(6, ℝ) let

α_{i}^{(−)}(gΔ_{T}) = sup \{\frac{1}{d_{gΔ_{T}}^{#} (gL)}| L is a Δ_{T} -rational subspace of dimension i not contained in U_{k} \}

(this is an abuse of notation because α_{i}^{(−)} depends on g as well as gΔ_{T}). For k + 1 ≤ i ≤ 6, set α_{i}^{(−)} = α_{i}. We now claim that for 1 ≤ i ≤ 3,

α_{i}^{(−)}(Δ_{T}) ≤ const · T^{−δ/3}.  (51)

Indeed, if k = 0, (51) is clear. let v denote the shortest vector in U′_{i} which is not contained in U_{k}. Note that d(U′_{i}) ≥ const · ∥v∥ (since the length of every vector in Δ_{T} is bounded from below). Also note that ∥v∥ ≥ const·d(U_{k})^{1/k} (otherwise v would belong to U_{k}). Thus,

C_{6}^{−1}C^{−1}T^{δ} ≤ d(ℝv + U_{k}) ≤ ∥v∥d(U_{k}) ≤ const · ∥v∥^{k+1},

hence ∥v∥ ≥ const · T^{\frac{δ}{k+1}} which implies the result for k = 1, 2. For k = 3, we have by Lemma 10.7 (iv),

C^{−1}T^{1/3} ≤ d(ℝv + U_{k}) ≤ ∥v∥d(U_{k}),

thus ∥v∥ ≥ T^{\frac{1}{3} −δ}. Thus (51) follows in this case as well.

Now as in [EMM, §5] let A_{τ} be the averaging operator defined by

(A_{τ}f)(x) = ∫_{K} f(a_{τ}kx) dm(k)

Then, arguing as in the proof of [EMM, Lemma 5.7] we see that the following inequalities hold with ω = ω(τ):

A_{τ}α_{1}^{(−)} ≤ α_{1}^{(−)} + ω^{2}\sqrt{α_{2}^{(−)}}
A_{τ}α_{2}^{(−)} ≤ α_{2}^{(−)} + ω^{2}\sqrt{α_{1}α_{3}^{(−)}} + ω^{2}\sqrt{α_{4}}
A_{τ}α_{3}^{(−)} ≤ α_{3}^{(−)} + ω^{2}\sqrt{α_{2}α_{4}} + ω^{2}\sqrt{α_{1}α_{5}} + ω^{2}\sqrt{α_{6}}  (52)
A_{τ}α_{4} ≤ α_{4} + ω^{2}\sqrt{α_{2}α_{6}} + ω^{2}\sqrt{α_{3}α_{5}}
A_{τ}α_{5} ≤ α_{5} + ω^{2}\sqrt{α_{4}α_{6}}

In the above, we used the fact that if L is a subspace which is not contained in U_{k}, and M is any other subspace, then L + M is also not contained in U_{k}. (However, L ∩ M may belong to U_{k}).

We also have

A_{τ}α_{1} ≤ α_{1} + ω^{2}\sqrt{α_{2}}
A_{τ}α_{2} ≤ α_{2} + ω^{2}\sqrt{α_{1}α_{3}} + ω^{2}\sqrt{α_{4}}  (53)
A_{τ}α_{3} ≤ α_{3} + ω^{2}\sqrt{α_{2}α_{4}} + ω^{2}\sqrt{α_{1}α_{5}} + ω^{2}\sqrt{α_{6}}

Let

β_{1} = T^{δ/16}α_{1}^{(−)} + α_{1}
β_{2} = T^{δ/8}α_{2}^{(−)} + α_{2}
β_{3} = T^{δ/4}α_{3}^{(−)} + α_{3}  (54)
β_{4} = T^{δ/2}α_{4}
β_{5} = T^{δ}α_{5}
β_{6} = α_{6} = T^{−3}

Now from (52) and (53) we get

A_{τ}β_{1} ≤ β_{1} + ω^{2}\sqrt{β_{2}}
A_{τ}β_{2} ≤ β_{2} + ω^{2}\sqrt{β_{1}β_{3}} + ω^{2}\sqrt{β_{4}}
A_{τ}β_{3} ≤ β_{3} + ω^{2}\sqrt{β_{2}β_{4}} + ω^{2}\sqrt{β_{1}β_{5}} + ω^{2}\sqrt{β_{6}}  (55)
A_{τ}β_{4} ≤ β_{4} + ω^{2}\sqrt{β_{2}β_{6}} + ω^{2}\sqrt{β_{3}β_{5}}
A_{τ}β_{5} ≤ β_{5} + ω^{2}\sqrt{β_{4}β_{6}}

From these inequalities we derive the following:

Claim 10.9. For any ρ > 0 there exists C_{ρ} so that for any sufficiently large T, and 1 ≤ i ≤ 6,

(A_{t}β_{i})(Δ_{T}) ≤ C_{ρ}T^{ρ},

where t = log T.

Proof of claim. This argument is similar to the proofs of Proposition 5.12 and Lemma 5.13 in [EMM]. Let q(i) = i(6 − i). Fix ϵ > 0, and consider the linear combination

β = ∑_{i=1}^{6} ϵ^{q(i)}β_{i}

Then β satisfies the inequality

A_{τ}β ≤ T^{−3} + (1 + 6ϵω^{2})β  (56)

(cf. [EMM, eqns (5.65)-(5.67)]). Let

β̃(h) = ∫_{K} β(hkΔ_{T}) dm(k).

Note that β̃ is spherically symmetric, i.e.

β̃(KhK) = β̃(h), h ∈ H_{1}.  (57)

It follows from the uniform continuity of log β̃ that for a suitable neighborhood V of the identity,

\frac{1}{2}β̃(h) < β̃(uh) < 2β̃(h), h ∈ H_{1}, u ∈ V.  (58)

According to [EMM, 5.11] there exists a neighborhood U of 1 in H_{1} such that a_{τ}Ua_{t} ⊂ KV a_{τ}a_{t}K for any t ≥ 0 and τ ≥ 0. Then we get from (57) and (58) that

(A_{τ} β̃)(a_{t}) = ∫_{K} β̃(a_{t}ka_{τ}) dm(k) ≥ ∫_{U∩K} β̃(a_{τ}ka_{t}) dm(k) > \frac{1}{2}m(U ∩ K)β̃(a_{τ}a_{t}). (59)

Hence, from (56) and (59) we have

β̃(a_{τ}a_{t}) ≤ \frac{2}{m(U ∩ K)} (T^{−3} + (1 + 6ϵω^{2})β̃(a_{t})) .  (60)

Given ρ > 0 we choose τ = τ(ρ) > 1 so that log(4/m(U ∩K))/τ ≤ ρ, and ϵ = ϵ(τ) > 0 so that 1 + 6ϵω^{2} ≤ 2 (recall ω = ω(τ)). We assume that T is large enough so that 2T^{−3}/m(U ∩ K) < 1. Hence we get

β̃(a_{τ}a_{t}) < e^{ρτ} β̃(a_{t}) + 1  (61)

Using induction on n we get from (61) that for n ∈ ℕ,

β̃(a_{nτ}) < (β̃(1) + 1)e^{nτρ},  (62)

where we have used the fact that e^{ρτ} ≥ 4. Since {a_{r} | 0 ≤ r ≤ τ} belongs to V^{i} for some i where V^{1} = V, V^{i} = V V^{i−1}, it follows that for any t > 0,

β̃(a_{t}) < C(β̃(1) + 1)e^{tρ},  (63)

where C = C(τ). Recall that β̃(a_{t}) = (A_{t}β)(Δ_{T}), and β̃(1) = β(1). Note that β(1) is bounded by a constant in view of (54), (51) and Lemma 10.7 (iii),(iv). Now let t = log T. This shows that

(A_{t}β)(Δ_{T}) ≤ C′_{ρ}T^{ρ}

from which the claim follows.

Hence, in view of (54), for all i,

A_{t}α_{i}^{(−)}(Δ_{T}) ≤ C′_{ρ}T^{ρ−δ/16} ≤ C′′T^{−δ/32}

where we chose ρ < δ/32. Now, using Proposition 10.6, we see that the number of primitive w ∈ Δ_{T} satisfying (a’), (b’), (c’) which are not contained in U_{k} is O(T^{1−δ/32}). Equivalently, the number of primitive w ∈ ℤ^{6} satisfying (a),(b),(c) of Lemma 10.5 which are not contained in g_{T}^{−1}U_{k} is O(T^{1−δ/32}). It remains to count the number of primitive points of g_{T}^{−1}U_{k} satisfying (a), (b), (c) of Lemma 10.5.

Now if k = 1, g_{T}^{−1}U_{1} has at most one primitive point. If k = 2, then there may be at most 2 primitive points in g_{T}^{−1}U_{2} satisfying (a) from Lemma 10.5 (there are two points if the restriction of Q^{(6)} to g_{T}^{−1}U_{2} is split, no points otherwise). If k = 3, and Q is not EWAS then for an apropriate choice of δ, the restriction of Q^{(6)} to g_{T}^{−1}U_{3} must be anisotropic by Lemma 10.8. Hence there are no points satisfying (a) in g_{T}^{−1}U_{3}. Thus in all cases, there at most three primitive points in g_{T}^{−1}U_{k} satisfying (a)-(c). Hence, the total number of primitive vectors in ℤ^{6} satisfying (a)-(c) is O(T^{1−δ}).

One corollary of the argument is the following:

Proposition 10.10. Suppose Q is any (possibly rational) quadratic form of signature (2, 2). Then the number of quasinull subspaces of norm between T/2 and T is O(T log T).

In fact, in view of Lemma 10.8 part (b), we also proved the following statements:

Proposition 10.11. There exists an absolute constant ρ > 0 such that the following holds: Suppose Q is any irrational quadratic form of signature (2, 2). Then for every sufficiently small δ > 0 and for every T > 2 one of the following holds:

(a) The number of quasinull subspaces of Q of norm between T/2 and T is O(T^{1−δ}).

(b) There exists a split integral quadratic form Q′ and 1 ≤ λ ∈ ℝ satisfying ∥Q − \frac{1}{λ}Q′∥ ≤ T^{−ρ} such that the number of quasinull subspaces of Q of norm between T/2 and T which are not isotropic subspaces of Q′ is O(T^{1−δ}). The coefficients of Q′ are bounded by a fixed power of T^{δ}.

10.3 Proof of Theorem 1.5

Lemma 10.12. Given a ℤ^{4}-rational quasinull subspace L, we say L is T -degenerate if any ℤ-basis for L ∩ ℤ^{4} contains a vector of norm greater then 2T. Then for any C < ∞, the number of lattice points v contained in T -degenerate quasinull subspaces L with ∥v^{L}∥ ≤ CT such that a < Q(v) < b and ∥v∥ ≤ T is o(T).

Proof. It is convenient to work with the standard form B. As in §2, let g_{Q} be such that Q(v) = B(g_{Q}v), and let Λ = g_{Q}ℤ^{4}. For a Λ-rational subspace L, let v_{L} denote the shortest vector in L ∩ Λ. Let Ω be the image under g_{Q} of the unit ball. Let f, a_{t}, K be as in §2. We know that the number of points in L ∩ Λ ∩ T Ω is bounded by

T^{2} ∑_{v∈L∩Λ} ∫_{K} f(a_{t}kv) dk  (64)

where t = log T. Let

H(R) = {k ∈ K : ∥a_{t}kv_{L}∥ < R}.

Since all vectors in L ∩ Λ ∩ T Ω are integral multiples of v_{L}, the integral in (64) can be restricted to H(R_{f}) ⊂ K where R_{f} = max{∥v∥ : f(v) ≠ 0}. Also if M > 0 is any constant, the number of points in L ∩ Λ ∩ T Ω is also bounded by

M + T^{2} ∑_{v∈L∩Λ} ∫_{H(R_{f} /M)} f(a_{t}kv) dk  (65)

Indeed if k ∉ H(R_{f}/M), i.e ∥a_{t}kv_{L}∥ > R_{f}/M, then at most M integer multiples of v_{L} can be in the support of f. Also,

∑_{v∈L∩Λ} f(a_{t}kv) ≤ constα_{1}(a_{t}kΛ)  (66)

and for k ∈ H(R_{f}/M), α_{1}(a_{t}kΛ) ≥ M/R_{f}. Substituting (66) into (65) and summing over all T -degenerate quasinull subspaces, we get

CMT log T + constT^{2} ∫_{C(M/R_{f})} α_{1}(a_{t}kΛ) dk

where C(M/R_{f}) = {k ∈ K : α_{1}(a_{t}kΛ) ≥ M/R_{f}}, and we have used Proposition 10.10 to bound the number of quasinull subspaces. Pick any 0 < ξ < 1, then the above expression is bounded by

CMT log T + constT^{2} (\frac{R_{f}}{M} )^{ξ} ∫_{C(M/R_{f})} α_{1}(a_{t}kΛ)^{1+ξ} dk

and the integral is bounded independently of T by Theorem 2.5. Since M is arbitrary, this implies that the total contribution of the T -degenerate quasinull subspaces is o(T^{2}).

Proof of Theorem 1.5. In view of Lemma 10.12 we need only estimate the number of points in the non-T -degenerate quasinull subspaces. Let M ≫ 1 be a parameter to be chosen later. Let T ≫ 1 be arbitrary. For a ℤ^{4}-rational subspace L, let v^{L} = v_{1}∧v_{2}, where {v_{1}, v_{2}} is any ℤ-basis for L ∩ ℤ^{4}. We say that a quasinull subspace is short if ∥v^{L}∥ ≤ M, and is long if M ≤ ∥v^{L}∥ ≤ T.

The number of short subspaces is bounded depending on M. Since we are assuming that the restriction of the quadratic form to L is not identically 0, the number of points v in L ∩ Λ such that a < Q(v) < b and ∥v∥ ≤ T is O(T) as T → ∞. Thus, for any choice of M, the contribution of the short subspaces is o(T^{2}).

To estimate the contribution of the long subspaces, divide them into dyadic intervals, S/2 ≤ ∥v^{L}∥ ≤ S. (the shortest interval begins at M; the longest ends at T). By Theorem 10.4, in each dyadic interval there are at most C(δ)S^{1−δ} subspaces; since each subspace is T -non-degenerate, it contributes at most O(T^{2}/S) points. Hence the contribution of each dyadic interval is at most C′(δ)T^{2}/S^{δ}. Letting S = 2^{k}M and summing the resulting geometric series, we see that the total contribution of the long intervals is C′′(δ)T^{2}/M^{δ}.

Since one may choose M arbitrary large, we see that the total contribution is o(T^{2}).

A A proof of Proposition 4.2

Notation: All the constants in this appendix are absolute. We use the notation a ≈ b to indicate that there exist constants 0 < c < C such that c < \frac{a}{b} < C. We write a ≫ b to indicate that there exists a “large” constant κ such that a > κb. A statement of the form “there exists c > 0 such that if a ≫ b then ...” means that “there exist constants c > 0 and κ > 1 such that if a > κb then ...”.

A.1 Transition to the hyperbolic plane

Set G = SL(2, ℝ), K = SO(2), b_{t} = (\begin{matrix}e^{t/2} & 0 \\ 0 & e^{−t/2} \end{matrix}), and B = \{b_{t}\}. On K\G we

consider the right G-invariant metric d(·, ·) normalized so that d(Kb_{t}, K) = t. This identifies K\G with the hyperbolic disk ℍ^{2} of curvature −1. Let p : G → ℍ^{2} be the natural projection. Let e = p(1) denote the origin of ℍ^{2}.

The space of the 3-dimensional representation of G can be considered to be the space S_{2} of 2-by-2 symmetric matrices, with the action given by g · v = gvg^{t}. This action preserves the determinant of v, which is a quadratic form of signature (2, 1).

Lemma A.1. Suppose v ∈ S_{2}. Then, if det v > 0, there exists a point z_{v} ∈ ℍ^{2} and a number λ_{v} > 0 such that

∥g · v∥ = λ_{v}\sqrt{cosh(2d(p(g), z_{v}))} ≈ λ_{v}e^{d(p(g),z_{v})}  (67)

If det v < 0 there exists a geodesic γ_{v} ∈ ℍ^{2} and a number λ_{v} > 0 such that

∥g · v∥ = λ_{v}\sqrt{cosh(2d(p(g), γ_{v}))} ≈ λ_{v}e^{d(p(g),γ_{v})}  (68)

Proof. Let v_{1} = \frac{1}{\sqrt{2}} (\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}) and v_{2} = \frac{1}{\sqrt{2}} (\begin{matrix}1 & 0 \\ 0 & −1 \end{matrix}). Note that for any v ∈ S_{2} with

det v ≠ 0 there exists g_{v} ∈ G such that g_{v} · v = λ_{v}v_{i}, where i = 1 if det v > 0, and i = 2 if det v < 0. Hence it is enough to prove the lemma for v = v_{1} and for v = v_{2}.

If v = v_{1}, then the stabilizer of v is K, and ∥b_{t} · v∥ = \sqrt{(e^{2t} + e^{−2t})/2} = \sqrt{cosh 2t}. Since, G = KBK and the norm ∥ · ∥ is K-invariant, (67) follows. If v = v_{2} then

the stabilizer of v_{2} is the group H = (\begin{matrix}cosh s & sinh s \\ sinh s & cosh s \end{matrix}). Then, as above, ∥b_{t} · v∥ =

\sqrt{(e^{2t} + e^{−2t})/2} = \sqrt{cosh 2t}. Since G = KBH, for any g ∈ G we may write g = kb_{t}h where k ∈ K and h ∈ H; then ∥g · v∥ = \sqrt{cosh 2t}, but since p(B) and p(H) are orthogonal geodesics in ℍ^{2}, t = d(p(b_{t}), e) = d(p(b_{t}), p(H)) = d(p(g), p(H)). This proves (68).

A.2 Some hyperbolic geometry

We now recall some well known lemmas from hyperbolic geometry.

Lemma A.2. Suppose Δ ⊂ ℍ^{2} is an isosoles triangle, with sides a, a and b. Let θ denote the angle opposite b. Suppose b ≫ 1 and θ ≪ 1. Then,

θ ≈ e^{\frac{b}{2} −a} Proof. The hyperbolic law of cosines states that

cosh b = cosh^{2} a − sinh^{2} a cos θ

This can be rewritten as

2 sin^{2} \frac{θ}{2} = \frac{cosh b − 1}{sinh^{2} a}

Now the lemma easily follows.

Lemma A.3. Let Δ be a right-angled triangle in ℍ^{2}. Let t denote the length of the side opposite the right angle. Let θ denote another angle of Δ, and let ρ denote the length of the side opposite θ. Then, if t ≫ 1 and θ ≪ 1,

θ ≈ e^{ρ−t}

Proof. By the hyperbolic law of sines,

\frac{sinh t}{sin \frac{π}{2}} = \frac{sinh ρ}{sin θ}

Hence,

sin θ = \frac{sinh ρ}{sinh t} ≈ e^{ρ−t}

Lemma A.4. Suppose d(p, e) = t, d(q, e) ≥ t. Set p′ = \overline{eq} ∩ S(t, e) where S(t, e) denotes the circle of radius t centered at e. Then,

d(q, p′) + d(p′, p) ≤ d(q, p) ≤ d(q, p′) + d(p′, p) + O(1)

Proof. This follows from the fact that the angle at p′ between \overline{p′q} and \overline{p′p} is greater then π/2.

Lemma A.5. There exists an absolute constant C so that the following holds: Let γ ⊂ ℍ^{2} be a geodesic. Let q_{+} and q_{−} in ∂ℍ^{2} denote the endpoints of γ, and let L_{+} and L_{−} denote the geodesic rays \overline{eq_{+}} and \overline{eq_{−}} respectively. Let r = d(γ, e), and let ℓ_{±}, also geodesic rays, denote the intersection of L_{±} with the complement of the ball B(r, e) (see Figure 1). Then,

hd(γ, ℓ_{+} ∪ ℓ_{−}) < C

where hd denotes the Hausdorff distance.

r e  ℓ_{−} ℓ_{+}  γ q_{+}  q_{−}

Figure 1. Lemma A.5.

Proof. Let q = B(r, e) ∩ γ, p_{±} = B(r, e) ∩ ℓ_{±}. It is clear that hd(γ, ℓ_{+} ∪ ℓ_{−}) = d(q, p_{+}) = d(q, p_{−}). By considering the right-angled triangle (e, q, q_{+}) we see that d(q, L_{+}) = O(1). Let p′_{+} ∈ L_{+} denote the closest point to q, so that d(p′_{+}, q) = O(1). Then, since the path e → p′_{+} → q is “almost a geodesic”, d(e, p′_{+}) = d(e, q)−d(p′_{+}, q)+ O(1) = r + O(1). Since d(e, p_{+}) = r and both p_{+} and p′_{+} lie on L_{+}, d(p_{+}, p′_{+}) = O(1). Hence d(p_{+}, q) ≤ d(p_{+}, p′_{+}) + d(p′_{+}, q) = O(1).

A.3 Proof of Proposition 4.2

For v ∈ ℝ^{3} and t > 0, let

m_{t}(v) = min_{θ} ∥b_{t}k_{θ}v∥.

The properties of the sets {k ∈ K : ∥b_{t}kv∥ < δ} which imply Proposition 4.2 are summarized in the following:

Lemma A.6. Let v ∈ ℝ^{3} be such that max_{k} ∥b_{t}kv∥ ≫ 1. Then, there exist intervals I_{t}^{v,+}(δ), I_{t}^{v,−}(δ) such that the following properties hold:

(a) There exists c > 0 such that for all δ > 0 with max_{k} ∥b_{t}kv∥ ≫ δ ≫ m_{t}(v) we

have

I_{t}^{v,±}(cδ) ⊂ {θ : ∥b_{t}k_{θ}v∥ < δ} ⊂ I_{t}^{v,+}(c^{−1}δ) ∪ I_{t}^{v,−}(c^{−1}δ)  (69)

(b) There exists C < ∞ such that if max_{k} ∥b_{i}kv∥ ≫ δ ≫ m_{t}(v), τ ≫ 0 and

δ ≫ m_{t+τ}(v),

I_{t+τ}^{v,±}(δ) ⊂ Ce^{−τ/2}I_{t}^{v,±}(δ)  (70)

(c) If max_{k} ∥b_{t}kv∥ ≫ η ≥ δ ≫ m_{t}(v), then

\frac{|I_{t}^{v,±}(δ)|}{|I_{t}^{v,±}(η)|} ≪ (\frac{δ}{η} )^{1/2}  (71)

(d) Either for all δ with max_{k} ∥b_{t}kv∥ ≫ δ ≫ m_{t}(v),

|I_{t}^{v,±}(δ)| ≈ e^{−t}m_{t}(v)^{−1/2}δ^{1/2}  (72)

or for all τ ≥ 0,

m_{t+τ}(v) ≥ m_{t}(v).  (73)

Proof of Lemma A.6. Suppose first that det v > 0. We let I_{t}^{v,+}(δ) = I_{t}^{v,−}(δ) = {k ∈ K : ∥b_{t}kv∥ < δ}, where λ_{v} is as in (67). Then (a) of Lemma A.6 is trivially satisfied. In the case det v > 0 we drop the ± from the notation. Let ρ be defined by the equation

\frac{δ}{λ_{v}} =^{def} \sqrt{cosh 2ρ} ≈ e^{ρ}  (74)

Then, in view of (67),

{p(g) ∈ ℍ^{2} : ∥g · v∥ < δ} = B(z_{v}, ρ)

We may identify the K with the unit circle; then I_{t}^{v}(δ) = {k ∈ K : ∥b_{t}k · v∥ < δ} may be identified with the set of angles at which the set S(t, e) ∩ B(z_{v}, ρ) is visible from the origin. Thus the center of the interval I_{t}^{v}(δ) is always the same; it is the angle at which z_{v} is visible from e.

Let r = d(e, z_{v}). In view of (67), m_{t}(v) (considered as a function of t) is decreasing for t ≤ r and increasing for t ≥ r. Also, in view of (67),

m_{t}(v) ≈ λ_{v}e^{|t−r|}.  (75)

Also, max_{k} ∥b_{t}kv∥ ≈ λ_{v}e^{t+r}. Thus, the assumption max_{k} ∥b_{t}kv∥ ≫ δ ≫ m_{t}(v) implies

t + r ≫ ρ ≫ 1 + |t − r|  (76)

Now suppose t ≤ r. Let p ∈ S(t, e) denote a point such that d(p, z_{v}) = ρ, and let p′ = S(t, e) ∩ \overline{ez_{v}} (see Figure 2). Then, by Lemma A.4, d(p, p′) = ρ − r + t + O(1). In view of (76), d(p, p′) ≫ 1. Hence, by Lemma A.2 (with b = d(p, p′) and a = d(e, p) =

d(e, p′) = t), we get |I_{t}^{v}(δ)| ≈ e^{\frac{ρ−r+t}{2} −t}. Hence, using (74) we get

|I_{t}^{v}(δ)| ≈ λ_{v}^{−1/2}e^{−r/2}e^{−t/2}δ^{1/2}  (77)

p′ z_{v} p  e ρt  r

Figure 2. The case det v > 0, t ≤ r.

In view of (75), (77) implies (72); hence (d) holds for this case.

Now suppose t ≥ r. Since, in this range m_{t}(v) is an increasing function of t,

(d) holds. Let q ∈ S(t, e) be a point such that d(q, z_{v}) = ρ, and let p denote the intersection of \overline{eq} with S(r, e) (see Figure 3). Then, by Lemma A.4, d(p, z_{v}) = ρ + t − r + O(1). In view of (76), d(p, z_{v}) ≫ 1. Now, by Lemma A.2 (with b = d(p, z_{v}) and

a = d(p, e) = d(z_{v}, e) = r) we get |I_{t}^{v}(δ)| ≈ e^{\frac{ρ+r−t}{2} −r}. Hence, using (74) we get that (77) still holds when t ≥ r as long as δ ≫ m_{t}(v). Now (b) and (c) follow immediately from (77). This completes the proof of Lemma A.6 for the case det v > 0.

Now suppose det v < 0. Let λ_{v} and γ_{v} be as in (68), and set r = d(γ_{v}, e). In view of (68), for t ≥ r, m_{t}(v) = λ_{v} (since S(t, e) intersects γ_{v}). Hence if t ≥ r then m_{t}(v) is not increasing and (d) is automatically satisfied.

Let ρ be as in (74). We denote by N_{ρ}(γ_{v}) the set {z ∈ ℍ^{2} : d(z, γ) ≤ ρ}. Then, in view of (68), the set {k ∈ K : ∥b_{t}kv∥ ≤ δ} may be identified with the set of angles at which S(t, e) ∩ N_{ρ}(γ_{v}) is visible from the origin e.

Let ℓ_{+} and ℓ_{−} be as in Lemma A.5. For σ ∈ {+, −} set N_{ρ}(ℓ_{σ}) = {z ∈ ℍ^{2} : d(z, ℓ_{σ}) ≤ ρ}. We define I_{t}^{v,σ}(δ) to be the set of angles at which the set S(t, e)∩N_{ρ}(ℓ_{σ}) is visible from the origin. Then, in view of Lemma A.5, (a) of Lemma A.6 holds.

Let p_{σ} denote the endpoint of ℓ_{σ}; then d(p_{σ}, e) = r. Let α denote the geodesic orthogonal to ℓ passing through p_{σ} (see Figure 4). Let t_{c} > r be as in Figure 4; then for t < t_{c}, |I_{t}^{v,σ}| is given by (77). If t > t_{c}, let p be a point on ∂N_{ρ}(ℓ_{σ}) ∩ S(t, e), and let q ≠ p_{σ} denote the closest point on ℓ_{σ} to p. Then the triangle (e, q, p) is a right triangle, and by Lemma A.3, |I_{t}^{v,σ}(δ)| ≈ e^{ρ−t} ≈ λ_{v}^{−1}e^{−t}δ. Hence, as long as

e r  t z_{v} q ρ p

Figure 3. The case det v > 0, t ≥ r.

max_{k} ∥b_{t}kv∥ ≫ δ ≫ m_{t}(v),

|I_{t}^{v,σ}(δ)| ≈ \begin{cases}λ_{v}^{−1/2}e^{−r/2}e^{−t/2}δ^{1/2} & t < t_{c} \\ λ_{v}^{−1}e^{−t}δ & t > t_{c} \end{cases}  (78)

Since |I_{t}^{v,σ}| is a continuous function of t, (b) and (c) of Lemma A.6 follow immediately from (78). To show (d), we note that if t > r then m_{t}(v) = λ_{v} and is thus nonincreasing. If t < r then for z ∈ S(t, e), d(z, ℓ_{σ}) = d(z, p_{σ}) and hence (75) and (77) hold. Thus (72) follows as in the case det v > 0. This completes the proof of Lemma A.6 in the case det v < 0.

The remaining case det v = 0 can be considered as a limiting case of the “ball” case det v > 0; we pass to the limit by sending z_{v_{i}} → ∞ and λ_{v_{i}} → 0 while keeping λ_{v_{i}}e^{d(z_{v_{i}},e)} =^{def} c_{v} constant. The function ∥gv∥ becomes essentially a Buseman function, and its level sets are horocycles. It is easy to show that (77) becomes

|I_{t}^{v}(δ)| ≈ c_{v}^{−1/2}e^{−t/2}δ^{1/2}

and (75) becomes

m_{t}(v) ≈ c_{v}^{1/2}e^{−t}.

Then the assertions for the lemma follow immediately.

Proof of Proposition 4.2. We let the rectangle R_{t}^{L,±,±}(δ_{1}, δ_{2}) be the product of the intervals intervals I_{t}^{π_{1}(v^{L}),±}(δ_{1}) and I_{t}^{π_{2}(v^{L}),±}(δ_{2}). It is clear that parts (a)-(c) of

r  t_{c} p_{σ} α ℓ_{σ}ρ e

Figure 4. The neighborhood of a geodesic ray.

Proposition 4.2 follow immediately from the corresponding assertion of Lemma A.6. To derive (d) of Proposition 4.2 from (d) of Lemma A.6 we may argue as follows: Note that M_{t}(L) = max(m_{t}(π_{1}(v^{L})), m_{t}(π_{2}(v^{L}))). Without loss of generality, we may assume that M_{t}(L) = m_{t}(π_{1}(v^{L})). By Lemma A.6 part (d), either (72) or (73) holds. If (72) holds then (24) holds. If (73) holds, then for any τ > 0, M_{t+τ}(L) ≥ m_{t+τ}(π_{1}(v^{L})) ≥ m_{t}(π_{1}(v^{L})) = M_{t}(L), hence (25) holds. Thus, part (d) of Proposition 4.2 also holds.

B Proof of Lemma 8.2

In this appendix we prove Lemma 8.2. The following lemma describes the shapes of the sets F (η):

Lemma B.1. There exist constants 0 < η_{0} < η_{1} < 1 and κ > 1 depending only on Λ such that the following holds: Let H be either a one-dimensional or a threedimensional Λ-rational subspace, and let v = v^{H}, where v^{H} is defined in §2.

Suppose t > 0. Suppose δ =^{def} min_{k} d_{a_{t}kΛ}(a_{t}kH) < η_{0}. Let λ_{1}^{2} ≤ λ_{2}^{2} be the eigenvalues of v^{t}v (where we identify ℝ^{4}, when dim H = 1 or ∧^{3}ℝ^{4}, when dim H = 3 with M_{2}(ℝ)). Let F^{H}(η) ⊂ ℝ^{2} denote the region {(x, y) : |λ_{1}−λ_{2}xy| < e^{−t}η, and |λ_{2}x| < η and |λ_{2}y| < η}. Then for 8δ < η < η_{1},

F^{H}(κ^{−1}η) ⊂ {(tan(α − α_{0}), tan(β − β_{0})) : ∥a_{t}(k_{α}, k_{β})v∥ < η} ⊂ F^{H}(κη)

for some α_{0} ∈ ℝ, β_{0} ∈ ℝ depending only on v.

Proof of Lemma B.1. We may replace v by (\begin{matrix}λ_{1} & 0 \\ 0 & λ_{2} \end{matrix}), with |λ_{1}| < |λ_{2}|. Then

kv = (\begin{matrix}cos α & sin α \\ − sin α & cos α \end{matrix}) (\begin{matrix}λ_{1} & 0 \\ 0 & λ_{2} \end{matrix}) (\begin{matrix}cos β & sin β \\ − sin β & cos β \end{matrix}) =

= (\begin{matrix}λ_{1} cos α cos β − λ_{2} sin α sin β & λ_{1} cos α sin β + λ_{2} sin α cos β \\ −λ_{1} sin α cos β − λ_{2} cos α sin β & −λ_{1} sin α sin β + λ_{2} cos α cos β \end{matrix}) (79)

Hence,

a_{t}kv = (\begin{matrix}e^{t}(λ_{1} cos α cos β − λ_{2} sin α sin β) & λ_{1} cos α sin β + λ_{2} sin α cos β \\ −λ_{1} sin α cos β − λ_{2} cos α sin β & e^{−t}(−λ_{1} sin α sin β + λ_{2} cos α cos β) \end{matrix})

Note that ∥a_{t}^{−1}∥ = e^{−t}, and ∥v∥ ≥ |λ_{2}|. Hence,

δ = min_{k} ∥a_{t}kv∥ ≥ e^{−t}|λ_{2}|.  (80)

Since |λ_{2}| is comparable to the norm of v, we have |λ_{2}| > c_{0}, where c_{0} depends only on Λ. If det v = 0, then λ_{1} = 0. If δ = min_{k} ∥a_{t}kv∥ < η_{0} then | det v| = |λ_{1}λ_{2}| ≤ η_{0}^{2}. Hence, if we choose η_{0} small enough, we may always assume that |λ_{1}| ≪ |λ_{2}|.

Suppose ∥a_{t}kv∥ < η < η_{1} ≪ 1. Then from the (1, 1) coefficient, | sin α sin β| < 2/|λ_{2}| ≪ 1. From the (1, 2) coefficient, | sin α cos β∥ < 2/|λ_{2}|, and from the (2, 1) coefficient, | cos α sin β| < 2/|λ_{2}|. Hence | sin α| ≪ 1, and | sin β| ≪ 1. Hence α is near 0 or π and β is near 0 or π. After possibly applying the transformation α → α+π, β → β + π (which have the effect of changing the signs of λ_{1} and λ_{2}), we may assume that α and β are near 0. Thus we may assume that | cos α| > 1/2 and | cos β| > 1/2. Let C_{0} = | cos α cos β|^{−1}, and let x = tan α, y = tan β. The inequalities ∥a_{t}kv∥ < η may be rewritten as:

|λ_{1} − λ_{2}xy| < C_{0}e^{−t}η  (81)
| − λ_{1}x − λ_{2}y| < C_{0}η  (82)
|λ_{1}y + λ_{2}x| < C_{0}η  (83)
|λ_{2} − λ_{1}xy| < C_{0}e^{t}η  (84)

where we note that always 1 < C_{0} < 4.

We first note that since |λ_{1}| ≪ |λ_{2}|, the equations (82) and (83) are equivalent to |x| < (const)η/|λ_{2}| and |y| < (const)η/|λ_{2}|. The equation (81) is equivalent to the first condition in the definition of F^{H}(η). These observations imply that if κ is sufficiently large, {k : ∥a_{t}kv∥ < η} ⊂ F^{H}(κη).

To establish the other inclusion F^{H}(κ^{−1}η) ⊂ {k : ∥a_{t}kv∥ < η} suppose k = (k_{α}, k_{β}) ∈ F^{H}(κ^{−1}η). As above, let x = tan α, y = tan β. It is clear that if κ is sufficiently large, then (81), (82) and (83) are satisfied with η/4 in place of η. Also, since η < η_{1} ≪ 1 and |λ_{2}| > c_{0} > 0, it follows from the second and third conditions in the definition of F^{H}(κ^{−1}η) that |x| < 1 and |y| < 1. Then,

|λ_{2} − λ_{1}xy| ≤ |λ_{2}| + |λ_{1}| ≤ 2|λ_{2}| ≤ 2e^{t}δ ≤ C_{0}e^{t}(η/4)

where in the next-to-last estimate we used (80) and in the last estimate we used the condition η > 8δ. Hence, ∥a_{t}kv∥ < η as required. This shows the inclusion F^{H}(κ^{−1}η) ⊂ {k : ∥a_{t}kv∥ < η} which completes the proof of the lemma.

In view of Lemma B.1, the set F (η) is contained in a small neighborhood of the origin. Hence we may assume that the Jacobian of the map (α, β) → (tan α, tan β) is bounded between 1/2 and 2. Lemma 8.2 would follow from the following:

Lemma B.2. Let H be a Λ-rational subspace of dimension 1 or 3. For η > 0 let F^{H}(η) ⊂ ℝ^{2} be as in Lemma B.1. Suppose R_{1} and R_{2} are disjoint rectangles such that for i = 1, 2, |R_{i} ∩ F (η)| ≥ 0.6|R_{i}|, and let λ > 1 be such that 3λR_{1} and 3λR_{2} are still disjoint and of diameter at most π/8. Then, for some i ∈ {1, 2},

|R_{i}| ≤ c_{3}λ^{−1}|(λR_{i}) ∩ F^{H}(2η)|

where c_{3} > 0 depends only on Λ.

Note that F^{H}(η) = F ∩ S(η_{2}), where η_{2} = η/|λ_{2}|, S(η_{2}) denotes the square centered at the origin of side length η_{2} and F is a region between two hyperbolas as in Lemma B.4.

The proof of Lemma B.2 will be carried out in two steps. The following lemma allows us to reduce to the case when R_{i} ⊂ F^{H}(η), i = 1, 2.

Lemma B.3. Let R be a rectangle such that |R∩F^{H}(η)| ≥ 0.6|R|. Then, there exists a rectangle R′ ⊂ R such that R′ ⊂ F^{H}(η) and |R′| > \frac{1}{(100)^{2}} |R|.

Proof of Lemma B.3. After replacing R by R ∩ S(η_{2}) we may assume that R ⊂ S(η_{2}). Consider a partition of R into 100 × 100 equal disjoint subrectangles. Each of the boundary curves of the region F is a monotone function. Clearly, a monotone curve can intersect at most 200 of these subrectangles. Since there are at most four boundary curves, the number of subrectangles intersected by boundary curves is at most 800. If no subractangle is contained in F, then F ∩R is covered by subrectangles intersecting boundary curves, and hence we would have |R ∩ F| < 0.08|R| < 0.6|R|, contradicting the assumption.

Lemma B.4. Let P = [x_{0}−a, x_{0}]×[y_{0}−b, y_{0}] and Q = [x_{0}−A, x_{0}+A]×[y_{0}−B, y_{0}+B] be rectangles. Suppose x_{0} > 0, y_{0} > 0, 0 < a < A, and 0 < b < B. Assume that the origin (0, 0) does not belong to Q. Let ω_{1} = (x_{0} − a)(y_{0} − b) and let ω_{2} = x_{0}y_{0}, and denote F′ = {(x, y) : ω_{1} ≤ xy ≤ ω_{2}}, so that P ⊂ F′. Then,

|Q ∩ F′| ≥ \frac{1}{8} min(a/A, b/B)|Q| = \frac{1}{2} min(aB, Ab)

Proof. After rescaling we may assume that ω_{2} = 1 and ω_{1} < 1. Let α, β be as in Figure 5 (i.e. α is the distance from the midpoint of the top side of Q to the intersection of the top side of Q with the hyperbola xy = 1; we set α equal to half the length of the top side if the hyperbola and the top side do not intersect).

x_{0}  B β a b  A α y_{0}  P Q

Figure 5. The proof of Lemma B.4.

Suppose α > \frac{1}{2}A. For x_{0}−α ≤ u ≤ x_{0} let h(u) denote the length of the intersection of Q∩F′ with the line x = u. We note that h(u) is a decreasing function on [x_{0}−α, x_{0}], and since P ⊂ F′, h(x_{0}) ≥ b. Hence

|Q ∩ F| ≥ ∫_{x_{0}−α}^{x_{0}} h(u) du ≥ αb ≥ \frac{1}{2}Ab

If β > \frac{1}{2}B then a similar argument shows that |Q ∩ F| ≥ \frac{1}{2}bB. Thus the lemma follows from the following:

Claim B.5. Either α ≥ \frac{1}{2}A or β ≥ \frac{1}{2}B.

Proof of Claim B.5 If the hyperbola xy = 1 does not intersect both the top and the right sides of Q, the claim is automatically satisfied. Thus, we may assume that the hyperbola intersects both the top and the right side. Hence,

α = x_{0} − \frac{1}{y_{0} + B}

β = y_{0} − \frac{1}{x_{0} + A}

If B > y_{0} then A < x_{0} (because (0, 0) ∉ Q). Hence, α = x_{0} − 1/(y_{0} + B) ≥ x_{0} − 1/(2y_{0}) = x_{0}/2 ≥ A/2, which proves the claim in this case. Similarly, if A > x_{0} then B < y_{0} (because (0, 0) ∉ Q) and then β = y_{0} − 1/(x_{0} + A) ≥ y_{0} − 1/(2x_{0}) = y_{0}/2 ≥ B/2.

Finally if A < x_{0} and B < y_{0} we have Ay_{0} < 1 and Bx_{0} < 1. Then

αβ = (x_{0} − \frac{1}{y_{0} + B} ) (y_{0} − \frac{1}{x_{0} + A}) = \frac{AB}{(1 + Ay_{0})(1 + Bx_{0})} ≥ \frac{1}{4}AB

Thus either α ≥ A/2 or β ≥ B/2. This completes the proof of the claim and thus of Lemma B.4.

We now complete the proof of Lemma B.2. Let R_{1} and R_{2} be as in Lemma B.2. Since 3λR_{1} and 3λR_{2} are disjoint, we may assume that for some j ∈ {1, 2}, 3λR_{j} does not contain the origin. Denote R_{j} by R, and let R′ ⊂ R be as in Lemma B.3. There is a quadrant containing at least half of R′; without loss of generality we may assume it is the first quadrant. Partition the intersection of R′ with the first quadrant into 3 × 3 equal subrectangles, and let P = [x_{0} − a, x_{0}] × [y_{0} − b, y_{0}] denote the middle subrectangle. Let Q = [x_{0} − A, x_{0} + A] × [y_{0} − B, y_{0} + B] be the biggest rectangle centered around the top right corner of P and is completely contained in λR. Note that the area of Q is at least a fixed fraction of the area of λR. Let F′ be as in Lemma B.4. We observe that F′ ⊂ F. Thus by Lemma B.4,

|λR ∩ F| ≥ |λR ∩ F′| ≥ |Q ∩ F′| ≥ \frac{1}{8} min(a/A, b/B)|Q| ≥ cλ^{−1}|λR| = cλ|R|

Thus, if either λR ∩ F′ ⊂ S(η_{2}) or λR ∩ F′ ⊂ S(2η_{2}), Lemma B.2 is proved. Suppose not; then without loss of generality we may assume that λR ∩ F′ intersects both of the lines y = η_{2} and y = 2η_{2}. Thus, λR contains the points (\frac{1}{2}(ω_{2}/η_{2}), 2η_{2}) and (ω_{2}/η_{2}, η_{2}), (where ω_{2} is as in Lemma B.4). Then, 3λR contains the origin, which is a contradiction. This completes the proof of Lemma B.2.

References

[DM] S.G. Dani and G.A. Margulis. Limit distributions of orbits of unipotent flows and values of quadratic forms. Advances in Soviet Math. 16(1993), 91–137.

[EMM] A. Eskin, G. Margulis, and S. Mozes. Upper bounds and asymptotics in a quantitative version of the Oppenheim conjecture. Ann. of Math. (2) 147(1998), no. 1, 93–141

[Mar1] G. A. Margulis. Discrete Subgroups and Ergodic Theory. In Number theory, trace formulas and discrete subgroups (a symposium in honor of A. Selberg), pages 377–398. Academic Press, Boston, MA, 1989.

[Mar2] G. A. Margulis. Oppenheim conjecture. In Fields medalists lectures, Eds. Atiyah, Iagolnitzer, pages 272–327, World Scientific, 1997.

[Ratner] M. Ratner. On Raghunathan’s measure conjecture. Annals of Math. 134(1991), 545–607.

[Sar] P. Sarnak. Values at integers of binary quadratic forms. To appear in a volume in memory of C. Herz. S. Drury, editor.

[Sch] W. Schmidt. Asymptotic formulae for point lattices of bounded determinant and subspaces of bounded height. Duke Math. J. 35(1968), 327–339.

[V1] J. Vanderkam. Values at integers of homogeneous polynomials. Duke Math. J. 97 (1999), no. 2, 379–412.

[V2] J. Vanderkam. Pair correlation of four-dimensional flat tori. Duke Math. J. 97 (1999), no. 2, 413–438.

[V3] J. Vanderkam. Correlations of eigenvalues on multi-dimensional flat tori. Comm. Math. Phys. 210 (2000), no. 1, 203–223.

Department of Mathematics, University of Chicago, Chicago IL 60637, USA; eskin@math.uchicago.edu

Department of Mathematics, Yale University, New Haven CT, USA; margulis@math.yale.edu

Institute of Mathematics, Hebrew University, Jerusalem 91904, ISRAEL; mozes@math.huji.ac.il