I. Introduction

Gessel and Viennot’s [16, 17] beautiful idea of interpreting tableaux as families of nonintersecting lattice paths is a very useful tool for obtaining determinantal expressions for generating functions for various families of tableaux with a given shape. (Subsequently, Stembridge [41] widened this theory. He showed that generating functions for tableaux where the shape now is allowed to vary can frequently given in form of a Pfaffian.) What is done in this theory, is counting families of nonintersecting lattice paths by weights which depend on the position of the path’s edges in the integer lattice ℤ^{2}. (ℤ stands for the set of integers.) We in turn propose the counting of families of nonintersecting lattice paths by MacMahon’s [32] major index and generalizations of it. The original motivation of doing major counting of nonintersecting lattice paths lay in a problem of Choi and Gouyou-Beauchamps. In [7] they found a nice product formula for the number of tableaux with p odd rows, with the parts being bounded by n, and where the lengths of the rows are bounded by 2r. It was conjectured that there is also a simple product formula for the generating function of this family of tableaux. By a slight modification of Choi and Gouyou-Beauchamps’ idea of proof, which is inspired by Desainte-Catherine and Viennot’s [10] geometric interpretation of a variation of the celebrated Knuth correspondence [24, 6], in this paper it is shown that the computation of the generating function of the above family of tableaux can be solved by counting nonintersecting lattice paths with given starting and final points, which in addition do not cross the diagonal line x = y, by a variation of the major index. So we decided to systematically develop a theory of counting families of nonintersecting lattice paths by major index and generalizations of it, which we call strange major indices (cf. (4.0.4)–(4.0.4)).

This major counting theory is developed quite analogously to the usual theory which encounters edge weights. The only difference is that for the major counting we have to assume that the starting points of the lattice paths lie on a line parallel to x + y = 0. Once this is done, we also obtain determinants involving q-binomials for the major and strange major generating functions for families of nonintersecting lattice paths with given starting and final points. Also here these determinants in special cases can be evaluated to result in simple products. However, the really significant parts of this paper consider nonintersecting paths which are bounded by x = y. This has no counterpart with edge weights. Only the number of such families of nonintersecting lattice paths has been previously considered [17, 10, 7]. Again we find determinantal expressions, though a little bit more complicated. And again, in special cases these determinants can be evaluated to result in nice product expressions.

Applications of this theory do not only include the solution of the above tableaux problem. There are other tableaux problems which also can be treated that way. An interesting application of our major counting is to give an alternative proof for the generating function expression for tableaux with rows of even lengths which do not exceed 2r and where the parts are bounded by n. (Of course this is the p = 0 case of the Choi–Gouyou-Beauchamps problem.) The proofs so far [11, 37, 40] rely on either the theory of symmetric functions or group representation theory. Our proof only uses combinatorial constructions and a simple determinant evaluation. A further byproduct of our major counting results is the computation of the generating function for tableaux with p odd rows, with at most c columns, with only odd parts which lie between 1 and 2n − 1. Equivalently, we thus compute the generating function for symmetric plane partitions with at most n rows, with parts between 1 and c, and with exactly p odd entries on the main diagonal. The p = 0 result was also previously given by Désarménien [11, Théoréme 1.2, second identity], Proctor [37, Theorem 1 case (CYH)], and Stembridge [40, Corollary 4.3 (b)]. Moreover, having solved the Choi–Gouyou-Beauchamps problem, by summing with respect to p and using the q-analog of Kummer’s summation we are able to present a new proof of the Bender–Knuth (ex-)Conjecture [4, p. 50] about the generating function for tableaux with parts between 1 and n, and with at most c columns. Likewise, by summing the generating functions for the above described symmetric plane partitions with respect to p, we provide a new proof of the MacMahon (ex-)Conjecture [32, p. 270] about the generating function for symmetric plane partitions with at most n rows and with parts between 1 and c. Proofs of these conjectures have been given by Andrews [2, 3], Gordon [18, only for the Bender–Knuth conjecture], Macdonald [33, pp. 51–53], and Proctor [36, Propositions 7.2, 7.3]. In addition we prove two theorems (Theorems 9 and 20) about specific generating functions for pairs of tableaux, or, equivalently, for specific sums of squares of Schur functions.

Received by the editor November 20, 1993.

It should be noted that the tableaux and plane partition theorems in this paper could also be derived starting from the summation theorems for Schur functions in Goulden’s paper [19] (cf. Remark (2) at the end of section 4). However, it is possible neither to obtain our theorems for the major and strange major counting of nonintersecting lattice paths from Goulden’s results nor Goulden’s results from our theorems. However, the ideas of the present paper can be suitably modified to even give bijective proofs of Goulden’s results. (Goulden used a linear operator for symmetric functions and some tricky manipulations to derive his theorems.) These bijective proofs will appear in a forthcoming paper [29] together with far-reaching generalizations and further applications of the results of Goulden and of the present paper.

We organize the paper in the following way. In the next section most of the definitions are given. The definition of the strange major indices is deferred to section 4, where they appear for the first time. Besides, in subsection 2.1 it is explained how to represent paths as two-rowed arrays, which will be crucial for our combinatorial constructions in subsections 5.1–5.3. In subsection 2.2, as a preparation, we review two sign-reversing involutions which serve to compute the number of families of nonintersecting lattice paths. Subsequently, in sections 3 and 4 these involutions will be modified in order to be applicable for the major and strange major counting. In section 3 the counting theory for the major index is developed. Here the results about tableaux with even rows (Theorem 10), about tableaux with odd parts (Theorem 11), respectively symmetric plane partitions with prescribed number of odd entries on the main diagonal (Theorem 12), and the MacMahon Conjecture (Theorem 13) are included. In section 4 the counting theory for the strange major index is developed. Here in Theorem 21 the Choi–Gouyou-Beauchamps problem is solved. The new proof of the Bender–Knuth Conjecture (Theorem 22) is found in this subsection. Since the complete details of the means which we need for the proofs in sections 3 and 4 very often are unpleasant and cumbersome, there is a separate section, namely section 5, which contains all auxiliary results and correspondences. Subsection 5.1 provides the basic results about major and strange major generating functions of single lattice paths. Subsection 5.2 contains two correspondences for paths which cross the line x = y. They are “major analogues” for the reflection principle. In subsection 5.3 we give a correspondence for pairs of intersecting lattice paths. It is a “major analogue” for the usual interchanging procedure (cf. [17, 41]). The modified Knuth correspondence together with its geometric interpretation, which was mentioned above, is described in subsection 5.4. The last two subsections are devoted to computational lemmas. A determinant lemma (Lemma 34) of a very general kind is proved in subsection 5.5. Previously, special cases have been successfully used in the computation of plane partition generating functions [25, 26, 28]. Finally, subsection 5.6 provides a multiple summation formula which is of crucial importance in the proofs of Theorems 18 and 19. R. Gustafson and S. Milne pointed out to me that it is a special case of Gustafson’s C_{r} _{6}ψ_{6} summation (or in the old notation: Sp(r) _{6}ψ_{6} summation) [21, Theorem 5.1]. The derivation of (5.6.1) from Gustafson’s C_{r} _{6}ψ_{6} sum can be found in [30]. I have included my original proof of (5.6.1) since it is purely elementary and independent of the deep theory of multiple basic hypergeometric summations and transformations currently under development, mainly by Milne and Gustafson (cf. e.g. [9, 21, 30, 31, 34, 35] and the references cited there). Besides, the ideas in this proof could turn out to be useful for proving other multiple summation formulas. For further comments on the connections with the theory of A_{r} and C_{r} summations (formerly: U(r) and Sp(r) summations) see Remark (3) at the end of section 4. For instance, an interesting feature of Theorem 15 is that it implicitly contains a new A_{r}-type q-Gauß summation formula (identity (4.3.12)).

II. Definitions and Preliminaries

2.1. Definitions and notation. In this paper we always consider lattice paths in the plane consisting of unit horizontal and vertical steps in the positive direction. In the sequel we shall call them paths for short. Any path in a natural way corresponds to a multiset permutation consisting of 1’s and 2’s. Let P be a path from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}). Later we frequently abbreviate the fact that a path P goes from A to E by P : A → E. P may be represented by a pair (A, π), where A is the starting point of P and π = π_{1}π_{2} … π_{E_{1}+E_{2}−A_{1}−A_{2}}, where π_{i} = 1 if the i’th step in the path P is a horizontal one and π_{i} = 2 if the i’th step in the path P is a vertical one. π is a multiset permutation consisting of E_{1} − A_{1} entries of 1 and E_{2} − A_{2} entries of 2. For example, the path P_{0} in Figure 1 is represented by ((1, −1), 221221112122). Of course, this representation of paths is unique. Hence, we may identify each path with its representation.

• _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} • •^{P_{0}} Figure 1

The major index (or “greater index”) of a multiset permutation π = π_{1}π_{2} … π_{n}, π_{i} ∈ ℕ (set of positive integers), is defined by

maj π = ∑_{i=1}^{n−1} i · χ(π_{i} > π_{i+1}) ,

where χ is the usual truth function, χ(A) = 1 if A is true, and χ(A) = 0 otherwise.

Given a path P = (A, π), we extend the major index to P by defining maj P := maj π. For our path in Figure 1 we have maj P_{0} = 2 + 5 + 9 = 16.

By definition each pair 21 that occurs in a multiset permutation π, and only these, makes a contribution to the major index of π. Given a path P = (A, π), the occurence of 21 in π means that a vertical step is followed by a horizontal one. The point which is the end point of this vertical step (and at the same time the starting point of this horizontal step) will be called a North-East corner of the path P. The North-East corners of our path in Figure 1 are (1, 1), (2, 3), and (5, 4). By the above consideration we see that only North-East corners of a path make a contribution to the major index. Besides, the contribution of the North-East corner (a, b) is the number of steps from the starting point of the path to (a, b), or in symbols a + b − A_{1} − A_{2} provided that the starting point is (A_{1}, A_{2}). Therefore North-East corners are of great significance when dealing with the major index of paths. In fact, given the starting and the final point of a path, the North-East corners uniquely determine the path. Suppose that P is a path from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}) and let the North-East corners of P be (a_{1}, b_{1}), (a_{2}, b_{2}), …, (a_{k}, b_{k}), where we assume that the (a_{i}, b_{i}) are ordered from left to right, which is equivalent with A_{1} ≤ a_{1} < a_{2} < ⋯ < a_{k} ≤ E_{1} − 1, and A_{2} + 1 ≤ b_{1} < b_{2} < ⋯ < b_{k} ≤ E_{2}. Then P can be represented by the two-rowed array

\begin{matrix}a_{1} & a_{2} & … & a_{k} \\ b_{1} & b_{2} & … & b_{k} \end{matrix} ,  (2.1.1)

or, if we wish to make the bounds which are caused by the starting and the final point transparent,

\begin{aligned}A_{1} ≤ & a_{1} & a_{2} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & b_{2} & … & b_{k} & ≤ E_{2} \end{aligned} .  (2.1.2)

For a given starting point and a given final point, by definition the empty array is the representation for the only path that has no North-East corner. For the path in our running example we obtain the array representation

{1 2 5 \atop 1 3 4} ,

or with bounds included,

\begin{aligned}1 ≤ & 1 & 2 & 5 & ≤ 5 \\ 0 ≤ & 1 & 3 & 4 & ≤ 6 \end{aligned} .

The notation (2.1.2) will be used with a three-fold meaning. First, we write down special arrays in that way if in addition we want to point out the row bounds. Secondly, we shall speak of two-rowed arrays of the type (2.1.2) by which we will mean arrays of the form (2.1.1) which obey the row bounds in (2.1.2). Thirdly, the notation (2.1.2) will also be used for the set of all arrays of the type (2.1.2). It should always be clear from the context which meaning is intended.

In section 5, also two-rowed arrays with its rows being of unequal length will be considered. But these arrays also will have the property that the rows are strictly increasing. So by convention, whenever in this subsection and in subsections 5.1– 5.3 we speak of two-rowed arrays we mean two-rowed arrays with strictly increasing rows. Also for these arrays we will use a notation of the kind (2.1.2) together with its three-fold meaning just explained. (We remark that in subsection 5.4 we consider two-rowed arrays of different type.) We shall frequently use the short notation (a | b) for two-rowed arrays, where a denotes the sequence (a_{i}) of elements of the first row, and b denotes the sequence (b_{i}) of elements of the second row.

Given a sequence a = a_{1}, a_{2}, …, a_{k} we write ∥a∥ for the sum ∑_{i=0}^{k} a_{i} of all the elements of the sequence. Also for r-tuples η = (η_{1}, …, η_{k}) we shall write ∥η∥ for the sum ∑_{i=1}^{k} η_{i}. In order to relate the major index of a path P with starting point A = (A_{1}, A_{2}) to its array representation (a | b), given by (2.1.1), it is readily verified that

maj P = ∥a∥ + ∥b∥ − k(A_{1} + A_{2}).  (2.1.3)

An r-tuple λ = (λ_{1}, …, λ_{r}) with λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r} is called a partition. The components λ_{i} are called parts of the partition. Let λ be a partition. A tableau τ of shape λ is an array

\begin{matrix}τ_{11} & τ_{12} & . . . . . . . . . . . . . . . & τ_{1λ_{1}} \\ τ_{21} & τ_{22} & . . . . . . . . & τ_{2λ_{2}} \\ ⋮ & . . . . . . . . . & ... \\ τ_{r1} & … & τ_{rλ_{r}} \end{matrix}  (2.1.4)

of positive integers τ_{ij}, 1 ≤ i ≤ r, 1 ≤ j ≤ λ_{i}, such that the rows are weakly and the columns are strictly increasing. The entries of τ are called parts of the tableau. The sum of all the parts of a tableau τ is called the norm, in symbols n(τ), of the tableau. Given a set T of tableaux, the norm generating function for T is defined to be ∑_{τ∈T} q^{n(τ)}. In the sequel, if we speak of the generating function for some set of tableaux we always mean the norm generating function.

A plane partition of shape λ is an array τ of positive integers τ_{ij} of the form (2.1.4) such that the rows and the columns are weakly decreasing. It is called column-strict if the columns are strictly decreasing. The notions part, norm, generating function are used for plane partitions in the same sense as with tableaux.

The Schur function s_{λ}(x_{1}, x_{2}, …) = s_{λ}(x) is a symmetric function (cf. [33, 38, 39]) in the variables x_{1}, x_{2}, … and is combinatorially defined by

s_{λ}(x) = ∑_{τ} ∏ x_{τ_{ij}} ,

where the sum is over all tableaux τ of shape λ and the product is over all parts τ_{ij} of τ. Equivalently, the sum could also run over all column-strict plane partitions τ of shape λ. The vector x of variables can be finite or infinite.

Since we deal with generating functions, a short notation for generating functions is in order. We write

GF (M; w) := ∑_{x∈M} w(x)  (2.1.5)

for the generating function of a set M with respect to a weight function w.

The q-notations which will be used are [α]_{q} = 1 − q^{α}, [n]_{q}! = [1]_{q}[2]_{q} ⋯ [n]_{q}, [0]_{q}! = 1,

(a; q)_{k} = ∏_{j=0}^{k−1}(1 − aq^{j}) , and (a; q)_{0} = 1,

(a; q)_{∞} = ∏_{j=0}^{∞}(1 − aq^{j}) ,

so that in particular [n]_{q}! = (q; q)_{n}, and

[ \begin{matrix}n \\ k \end{matrix} ]_{q} = \{ \begin{matrix}\frac{[n]_{q} · [n − 1]_{q} ⋯ [n − k + 1]_{q}}{[k]_{q}!} & k ≥ 0 \\ 0 & k < 0 \end{matrix} .

The base q in [α]_{q}, [n]_{q}!, (a; q)_{k}, (a; q)_{∞}, and [ \begin{matrix}n \\ k \end{matrix} ]_{q} will in most cases be omitted. Only if the base is different from q it will be explicitely stated.

The basic hypergeometric series _{p+1}ϕ_{p} is defined by

p+1ϕ_{p} [ \begin{matrix}a_{1}, …, a_{p+1} \\ b_{1}, …, b_{p} \end{matrix}; q, z] = ∑_{i=0}^{∞} \frac{(a_{1}; q)_{i} (a_{2}; q)_{i} ⋯ (a_{p+1}; q)_{i}}{(q; q)_{i} (b_{1}; q)_{i} ⋯ (b_{p}; q)_{i}} z^{i}.

An excellent reference book for basic hypergeometric series is [14]. We will frequently refer to it.

2.2. Two involutions and nonintersecting lattice paths. In order to make the arguments in sections 3 to 5 more transparent, it is perhaps appropriate to recapitulate the usual procedure with nonintersecting lattice paths (cf. [16, 17, 41, section 1]), before going into the major counting theory.

Given two lattice points in the integer lattice ℤ^{2}, A = (A_{1}, A_{2}) and E = (E_{1}, E_{2}), we denote the set of all paths from A to E by P (A → E). For the number of those paths we have

|P (A → E)| = \binom{E_{1} + E_{2} − A_{1} − A_{2}}{E_{1} − A_{1}} .  (2.2.1)

As usual, for a set S by |S| we mean the cardinality of S. Let A_{i} = (A_{1}^{(i)}, A_{2}^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points. Set A := (A_{1}, …, A_{r}) and E := (E_{1}, …, E_{r}). The set of all families (P_{1}, …, P_{r}) of lattice paths where P_{i} goes from A_{i} to E_{i}, i = 1, …, r, is denoted by P (A → E). A family of lattice paths is called intersecting if there are two paths in the family which have a point in common, if not the family is called nonintersecting. Two paths which have a point in common will also be called intersecting. The set of families (P_{1}, …, P_{r}) of nonintersecting lattice paths where P_{i} goes from A_{i} to E_{i} is denoted by P (A → E)^{+}, while the set of families (P_{1}, …, P_{r}) of intersecting lattice paths is denoted by P (A → E)^{−}.

Now let A_{i}, E_{i} as above and

A_{1}^{(1)} ≤ A_{1}^{(2)} ≤ ⋯ ≤ A_{1}^{(r)} and A_{2}^{(1)} ≥ A_{2}^{(2)} ≥ ⋯ ≥ A_{2}^{(r)} ,  (2.2.2a)

and

E_{1}^{(1)} ≤ E_{1}^{(2)} ≤ ⋯ ≤ E_{1}^{(r)} and E_{2}^{(1)} ≥ E_{2}^{(2)} ≥ ⋯ ≥ E_{2}^{(r)} .  (2.2.2b)

To count the number |P (A → E)^{+}| of all families (P_{1}, …, P_{r}) of nonintersecting lattice paths, P_{i} : A_{i} → E_{i}, the following procedure is used. Obviously the relation

|P (A → E)^{+}| = |P (A → E)| − |P (A → E)^{−}|  (2.2.3)

holds. Let 𝔖_{r} be the symmetric group of order r. For a permutation σ ∈ 𝔖_{r} define A_{σ} to be the r-tuple (A_{σ(1)}, A_{σ(2)}, …, A_{σ(r)}). In particular, we have A_{id} = A. The trick is to consider the sets P (A_{σ} → E)^{−}, σ ∈ 𝔖_{r}, i.e. one allows permutations of the starting points. We are going to construct a map Ω which acts on intersecting families. Suppose that the family 𝔓 = (P_{1}, …, P_{r}) is an element of P (A_{σ} → E)^{−}. Among all meeting points between any two paths of that family consider the rightmost meeting points, and among these the highest meeting point. To be precise, if (X, Y) is this meeting point just obtained, then for any other meeting point (x, y) we have

x < X, or if x = X then Y ≥ y.  (2.2.4)

Among all pairs of paths which have (X, Y) in common let (P_{I}, P_{J}) be that pair for which (I, J) is minimal with respect to the lexicographical order of pairs of integers. To be precise, if (P_{i}, P_{j}) is another pair of paths with common point (X, Y) then

I < i, or if I = i then J ≤ j.  (2.2.5)

It is a simple matter of fact that by (2.2.4) the point (X, Y) is the last meeting point between P_{I} and P_{J}. Denote this last meeting point by S. Next we decompose P_{I} into P_{I}^{(1)} and P_{I}^{(2)}, where P_{I}^{(1)} is the portion of P_{I} going from A_{I} to S, and P_{I}^{(2)} is the portion of P_{I} between S and E_{I}. Analogously, P_{J} by S is decomposed into P_{J}^{(1)} and P_{J}^{(2)}. Then by concatenation form the new paths P′_{I} = P_{J}^{(1)}P_{I}^{(2)} and P′_{J} = P_{I}^{(1)}P_{J}^{(2)}. In other terms, the initial portions of P_{I} and P_{J} up to their last meeting point are interchanged, the tails are preserved. Thus we obtain the new family Ω(𝔓) = (P_{1}, …, P′_{I}, …, P′_{J}, …, P_{r}). Obviously it is an element of P (A_{σ(IJ)} → E). ((IJ) denotes the transposition which exchanges I and J. σ(IJ) is the composition of σ and (IJ).) Obviously this new family is again intersecting, therefore it is an element of P (A_{σ(IJ)} → E)^{−}. Moreover, since nothing was changed after the point S, S is also the right-most and highest meeting point for the family Ω(𝔓), and I and J are also the minimal indices for intersecting pairs of paths of the family Ω(𝔓). Therefore when applying the interchanging procedure Ω to Ω(𝔓), one arrives at 𝔓 again. Consequently Ω is an involution on

⋃_{σ∈𝔖_{r}} P (A_{σ} → E)^{−} ,  (2.2.6)

and moreover a bijection between the subsets

⋃_{σ even} P (A_{σ} → E)^{−} and ⋃_{σ odd} P (A_{σ} → E)^{−} .

This last fact may be formulated in a different way. On the set

⋃_{σ∈𝔖_{r}} P (A_{σ} → E) ,  (2.2.7)

we introduce the weight function ω by

ω(𝔓) = sgn σ if 𝔓 ∈ P (A_{σ} → E).

Then Ω is a sign reversing involution on the set (2.2.6) with respect to the weight function ω. As usual, Ω being sign-reversing with respect to ω means that ω(Ω(𝔓)) = −ω(𝔓). From this we get

∑_{σ∈𝔖_{r}} sgn σ |P (A_{σ} → E)^{−}| = 0 .  (2.2.8)

By consecutive use of (2.2.3), (2.2.8), and (2.2.1), and by the observation that by (2.2.2) we have

P (A_{σ} → E)^{−} = P (A_{σ} → E) if σ ≠ id,  (2.2.9)

we obtain

|P (A → E)^{+}| = |P (A → E)| − |P (A → E)^{−}|

= |P (A → E)| + ∑_{σ∈𝔖_{r}, σ≠id} sgn σ |P (A_{σ} → E)^{−}|

= ∑_{σ∈𝔖_{r}} sgn σ |P (A_{σ} → E)|

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} |P (A_{σ(i)} → E_{i})|

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} \binom{E_{1}^{(i)} + E_{2}^{(i)} − A_{1}^{(σ(i))} − A_{2}^{(σ(i))}}{E_{1}^{(i)} − A_{1}^{(σ(i))}}

= det_{1≤s,t≤r} ( \binom{E_{1}^{(s)} + E_{2}^{(s)} − A_{1}^{(t)} − A_{2}^{(t)}}{E_{1}^{(s)} − A_{1}^{(t)}} ) .  (2.2.10)

Since the set of edges trivially does not change under the above described interchanging procedure, this procedure also preserves edge weights (by which we mean weights which depend on the position of the path’s edges in ℤ^{2}) and therefore can also be used for the counting of intersecting lattice paths with respect to edge weights (see [17, 41, section 1]). But interchanging portions of paths does not at all take care of the major index or strange major index of the paths. So, in subsections 3.1 and 4.1, the interchanging procedure Ω has to be modified in order to work with the major and the strange major index.

In subsections 3.2 and 4.2 we are going to count families of nonintersecting lattice paths with given starting and final points, which in addition do not cross the line x = y, by major index and strange major index, respectively. What we are going to use in these sections are modifications of the following procedure which yields the number of all those families. Again we are going to construct a sign-reversing involution, which we this time denote by Θ. The idea is to combine the above described interchanging procedure with the reflection principle (see e.g. [8, p. 22]). Recall that the reflection principle serves to count all paths from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}) which do not cross the line x = y. Suppose that A_{1} ≥ A_{2} and E_{1} ≥ E_{2}, i.e. that A and E lie below or on the line x = y. Suppose that we want to count all the paths from A to E which lie below x = y thereby being allowed to touch x = y. If in the sequel we say that “a path lies below the line g” we always mean that it is allowed to touch g, the same holding for “above”. The reflection principle shows that the paths from A to E which cross x = y are in one-to-one correspondence with the paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}). This is seen by reflecting a crossing path’s initial portion up to the last point which lies above x = y in the line x = y − 1. (This differs from the presentation in [8], where the first point is considered.) The number of paths which lie below x = y (and hence do not cross x = y) then is obtained as the difference between the number of all paths from (A_{1}, A_{2}) to (E_{1}, E_{2}) and the number of all paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}). Let us denote the reflection in the line x = y − 1 by ℜ.

As before, let A_{i} = (A_{1}^{(i)}, A_{2}^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points which satisfy (2.2.2). Again, set A := (A_{1}, …, A_{r}) and E := (E_{1}, …, E_{r}). The set of all families (P_{1}, …, P_{r}) of lattice paths, where P_{i} goes from A_{i} to E_{i}, i = 1, …, r, this time for convenience is denoted by Q(A → E). We want to determine the number of all families (P_{1}, …, P_{r}) of nonintersecting lattice paths which do not cross x = y, P_{i} going from A_{i} to E_{i}. The problem only makes sense if all the points A_{i} and E_{i} lie on one side of the line x = y. For the following we assume that the points A_{i}, E_{i} lie below or on the line x = y, i.e.

A_{1}^{(i)} ≥ A_{2}^{(i)} and E_{1}^{(i)} ≥ E_{2}^{(i)}, i = 1, 2, …, r.  (2.2.11)

The set of families (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie below x = y, P_{i} going from A_{i} to E_{i}, is denoted by Q(A → E)^{+}, while the set of families (P_{1}, …, P_{r}) of lattice paths which are either intersecting or contain a path which crosses x = y is denoted by Q(A → E)^{−}.

What we want to determine is the number of families of nonintersecting paths lying below x = y, i.e. we want to determine the number |Q(A → E)^{+}|. Obviously the relation

|Q(A → E)^{+}| = |Q(A → E)| − |Q(A → E)^{−}|  (2.2.12)

holds. Just as we above allowed permutations of the starting points, here we allow permutations and reflections of the starting points. We extend the reflection in the line x = y −1, ℜ, to r-tuples of points in the following way. Let η be an r-tuple consisting of 0’s and 1’s, η ∈ {0, 1}^{r}. By ℜ^{η}A we mean the r-tuple (ℜ^{η_{1}}A_{1}, …, ℜ^{η_{r}}A_{r}), where of course ℜ^{0} means the identity. Now we consider the sets Q(ℜ^{η}A_{σ} → E)^{−}. Suppose that the family 𝔓 = (P_{1}, …, P_{r}) is an element of Q(ℜ^{η}A_{σ} → E)^{−}. Among all meeting points between any two paths of that family and all meeting points of any path with the line x = y − 1, consider the right-most meeting points, and among these the highest meeting point. Again this is understood in the sense of (2.2.4).

Denote this meeting point by S. Now we distinguish between two cases. If S is a common point of two paths of the family 𝔓, then, as before, among the pairs of paths which have S in common, choose (P_{I}, P_{J}) with (I, J) being minimal in the sense of (2.2.5) and apply the above described interchanging procedure, thus obtaining the new family Θ(𝔓) = (P_{1}, …, P′_{I}, …, P′_{J}, …, P_{r}). It is an element of Q(ℜ^{η^{(IJ)}}A_{σ(IJ)} → E)^{−}, where η^{(IJ)} is η with the I’th and J’th coordinate exchanged. To be precise, η_{i}^{(IJ)} = η_{i} for i ≠ I, J, η_{I}^{(IJ)} = η_{J}, and η_{J}^{(IJ)} = η_{I}. On the other hand, if S is not a common point then it must lie on x = y − 1. Among the paths which go through S choose P_{I} with I being minimal. Let the image of P_{I} under application of the reflection principle be denoted by P′′_{I}. Here, “applying the reflection principle to a path P ” is understood to mean the reflection of the initial portion of P up to the last meeting point with x = y − 1 in the line x = y − 1. We then obtain the new family Θ(𝔓) = (P_{1}, …, P′′_{I}, …, P_{r}). This family is an element of Q(ℜ^{η^{(I)}}A_{σ} → E)^{−}, where η^{(I)} is given by η^{(I)} = (η_{1}, …, 1 − η_{I}, …, η_{r}). Just as above, since in both cases nothing was changed after the point S, by application of Θ to Θ(𝔓), one arrives at 𝔓 again. Consequently Θ is an involution on

⋃_{σ∈𝔖_{r}, η∈{0,1}^{r}}Q(ℜ^{η}A_{σ} → E)^{−} .  (2.2.13)

Next on the set

⋃_{σ∈𝔖_{r}, η∈{0,1}^{r}}Q(ℜ^{η}A_{σ} → E) .  (2.2.14)

we introduce the weight ϑ by

ϑ(𝔓) := (−1)^{∥η∥} sgn σ if 𝔓 ∈ Q(ℜ^{η}A_{σ} → E)^{−}.

It is easily verified that Θ is sign-reversing with respect to ϑ. From this we infer

∑_{σ∈𝔖_{r}, η∈{0,1}^{r}}(−1)^{∥η∥} sgn σ |Q(ℜ^{η}A_{σ} → E)^{−}| = 0.  (2.2.15)

By consecutive use of (2.2.12), (2.2.15), and (2.2.1), and by the simple observation that we have

Q(ℜ^{η}A_{σ} → E)^{−} = Q(ℜ^{η}A_{σ} → E) if (σ, η) ≠ (id, 0) ,  (2.2.16)

12 C. Krattenthaler

we obtain

|Q(A → E)^{+}| = |Q(A → E)| − |Q(A → E)^{−}|

= |Q(A → E)| +_{∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} ; (σ,η)≠(id,0)}(−1)^{∥η∥} sgn σ |Q(ℜ^{η}A_{σ} → E)^{−}|

= ∑_{σ∈𝔖_{r}, η∈{0,1}^{r}}(−1)^{∥η∥} sgn σ |Q(ℜ^{η}A_{σ} → E)|

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r}(|P (A_{σ(i)} → E_{i})| − |P (ℜA_{σ(i)} → E_{i})|)

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} (\binom{E_{1}^{(i)} + E_{2}^{(i)} − A_{1}^{(σ(i))} − A_{2}^{(σ(i))}}{E_{1}^{(i)} − A_{1}^{(σ(i))}}

− \binom{E_{1}^{(i)} + E_{2}^{(i)} − A_{1}^{(σ(i))} − A_{2}^{(σ(i))}}{E_{2}^{(i)} − A_{1}^{(σ(i))} − 1})

= det_{1≤s,t≤r} ( \binom{E_{1}^{(s)} + E_{2}^{(s)} − A_{1}^{(t)} − A_{2}^{(t)}}{E_{1}^{(s)} − A_{1}^{(t)}} − \binom{E_{1}^{(s)} + E_{2}^{(s)} − A_{1}^{(t)} − A_{2}^{(t)}}{E_{2}^{(s)} − A_{1}^{(t)} − 1} ) .

(2.2.17)

III. Counting by Major Index

In this section we want to count families 𝔓 = (P_{1}, …, P_{r}) of lattice paths by the major index. To be precise, we want to count families 𝔓 by maj 𝔓 where the major index is extended to families of paths by

maj 𝔓 = ∑_{i=1}^{r} maj P_{r}.  (3.0.1)

3.1. Without restrictions. We precede our first theorem by an auxiliary lemma.

Lemma 1. Let A_{i} = (A_{1}^{(i)}, A_{2}^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} which satisfy (2.2.2). Given a permutation σ ∈ 𝔖_{r}, suppose that 𝔓 = (P_{1}, …, P_{r}) is an intersecting family of lattice paths that is an element of P (A_{σ} → E). Then there exists an index I such that P_{I} and P_{I+1} intersect.

Proof. Let P_{J} and P_{K}, J < K, be two intersecting paths in the family 𝔓 such that the difference K − J is minimal. If K − J = 1 we are done. Now suppose that K − J > 1. Everything becomes clear if Figure 2 is considered.

•  •  • • • ∘ ∘  ∘  ∘_{B_{1}}  _{B_{2}}  ^{E_{J}}  ^{E_{K}}^{A_{σ(J+1)}}_{A_{σ(J+1)}}  _{A_{σ(J+1)}}  ^{E_{J+1}}^{P_{K}}^{P_{J}} Figure 2

The pair of points (B_{1}, B_{2}) is meant to be identical either with (A_{σ(J)}, A_{σ(K)}) or with its reversion (A_{σ(K)}, A_{σ(J)}), depending on whether σ(J) < σ(K) or not. Because of (2.2.2b) the final point E_{K} of P_{K} must be located in the South-East of E_{J}, as well as because of (2.2.2a) the starting point A_{σ(K)} of P_{K} must be located in the South-East or North-West of A_{σ(J)}. Moreover, because of (2.2.2a) the starting point A_{σ(J+1)} of P_{J+1} must be located either in the North-West of B_{1}, or in the South-East of B_{1} and at the same time in the North-West of B_{2}, or in the South-East of B_{2}. The final point E_{J+1} of P_{J+1} must be located in the South-East of E_{J} and in the North-West of E_{K}. In Figure 2 typical positions of A_{σ(J+1)} and E_{J+1} are marked by circles. Obviously the path P_{J+1} has to intersect either P_{J} or P_{K} in order to connect A_{σ(J+1)} and E_{J+1}. This is a contradiction to the minimality of K − J. □

We call paths P_{I}, P_{I+1} with consecutive indices neighbouring paths. Now we are in the position to state our first result.

Theorem 2. Let A_{i} = (A^{(i)} + D, −A^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} with

A^{(1)} < A^{(2)} < ⋯ < A^{(r)} ,  (3.1.1a)

and

E_{1}^{(1)} < E_{1}^{(2)} < ⋯ < E_{1}^{(r)} and E_{2}^{(1)} ≥ E_{2}^{(2)} ≥ ⋯ ≥ E_{2}^{(r)} .  (3.1.1b)

The generating function ∑ q^{maj 𝔓}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where maj 𝔓 is given by (3.0.1), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D \end{matrix} ]) .  (3.1.2)

Remark. In subsection 5.3 it is explained why for major counting we have to require that the starting points of the paths have to lie on a line parallel to x + y = 0.

Proof of the Theorem. We adopt the notations of subsection 2.2. First we define a weight function ω_{1} on the set (2.2.7) by

ω_{1}(𝔓) = sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})}q^{maj 𝔓}  if 𝔓 ∈ P (A_{σ} → E).  (3.1.3)

In terms of ω_{1} the generating function in question is GF (P (A → E)^{+}; ω_{1}), where the symbol GF (M; w) is explained in (2.1.5). Obviously, the equation

GF (P (A → E)^{+}; ω_{1}) = GF (P (A → E); ω_{1}) − GF (P (A → E)^{−}; ω_{1})  (3.1.4)

holds, which is the analogue for (2.2.3).

Similar to subsection 2.2, we are going to construct an involution Ω_{1} on the set (2.2.6) which is sign-reversing with respect to ω_{1}. Let 𝔓 = (P_{1}, …, P_{r}) be an element of P (A_{σ} → E)^{−}. Consider all meeting points between neighbouring paths. That there is at least one is guaranteed by Lemma 1. (This is more restrictive than in subsection 2.2 where we considered the meeting points between any two paths.) Choose the right-most, and among them, the highest meeting point and denote it by S. Among the pairs P_{I}, P_{I+1} which meet in S choose that one with I being minimal. It should be noted that S is the last meeting point between P_{I} and P_{I+1}. Then we define

Ω_{1}(𝔓) := (P_{1}, …, P′_{I}, P′_{I+1}, …, P_{r})  (3.1.5a)

where the paths P′_{I}, P′_{I+1} are given by

(P′_{I}, P′_{I+1}) = \{ \begin{matrix}Ψ((P_{I}, P_{I+1})) & if σ(I) < σ(I + 1) \\ Ψ^{(−1)}((P_{I}, P_{I+1})) & if σ(I) > σ(I + 1) \end{matrix} .  (3.1.5b)

The map Ψ is defined in Proposition 27. It is readily observed that Ω_{1}(𝔓) is an element of P (A_{σ(I,I+1)} → E)^{−}. Besides, since Ψ is constructed just in the manner that it does not change anything after this last meeting point S, application of Ω_{1} to Ω_{1}(𝔓) gives 𝔓 again. Hence, Ω_{1} is an involution on the set (2.2.7).

Next we show that Ω_{1} is sign-reversing with respect to ω_{1}. Namely, by the definitions (3.1.3), (3.1.5), and using (5.3.4), we get

ω_{1}(Ω_{1}(𝔓)) = sgn (σ(I, I + 1)) q^{∑_{i=1}^{r} i(A^{(i)}−A^{((σ(I,I+1))(i))})}q^{maj Ω_{1}(𝔓)}

= − sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})+I(A^{(σ(I))}−A^{(σ(I+1))})+(I+1)(A^{(σ(I+1))}−A^{(σ(I))})}

· q^{maj 𝔓+A^{(σ(I))}−A^{(σ(I+1))}}
= −ω_{1}(𝔓).  (3.1.6)

This implies the identity

∑_{σ∈𝔖_{r}} GF (P (A_{σ} → E)^{−}; ω_{1}) = 0 .  (3.1.7)

Obviously, this is the analogue for (2.2.8).

The rest is routine after having read the first part of subsection 2.2. We only have to mimic the computation (2.2.10). Indeed, by consecutive use of (3.1.5), (3.1.7), (2.2.9), and (5.1.13) (the analogue for (2.2.1)), we obtain

GF (P (A → E)^{+}; ω_{1}) = GF (P (A → E); ω_{1}) − GF (P (A → E)^{−}; ω_{1})

= GF (P (A → E); ω_{1}) + ∑_{σ∈𝔖_{r}, σ≠id} GF (P (A_{σ} → E)^{−}; ω_{1})

= ∑_{σ∈𝔖_{r}} GF (P (A_{σ} → E); ω_{1})

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})}GF (P (A_{σ(i)} → E_{i}); maj)

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})} [ \begin{matrix}E_{1}^{(i)} + E_{2}^{(i)} − D \\ E_{1}^{(i)} − A^{(σ(i))} − D \end{matrix} ]

= det_{1≤s,t≤r} ( q^{s(A^{(s)}−A^{(t)})} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D \end{matrix} ] ) ,  (3.1.8)

which settles (3.1.2). □

It is very unlikely that the determinant in (3.1.2) can be simplified significantly in general. But there is a special case in which Lemma 35 can be applied to obtain a closed form for the determinant.

Theorem 3. Let A_{i} = (A^{(i)} + D, −A^{(i)}) and E_{i} = (E_{1} + i, E_{2} − i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1a) holds. The generating function ∑ q^{maj(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∏_{i=1}^{r} \frac{[E_{1} + E_{2} + i − D − 1]!}{[E_{2} + A^{(i)} − 1]! [E_{1} + r − D − A^{(i)}]!} ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] .  (3.1.9)

Proof. Setting E_{1}^{(i)} = E_{1} + i and E_{2}^{(i)} = E_{2} − i in Theorem 2, we obtain for the desired generating function the expression

det (q^{s(A^{(s)}−A^{(t)})} [ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s − A^{(t)} − D \end{matrix} ]) .

Taking some factors out of the determinant leads to

det (q^{s(A^{(s)}−A^{(t)})} [ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s − A^{(t)} − D \end{matrix} ])

= q^{∑_{i=1}^{r} iA^{(i)}} ∏_{i=1}^{r} \frac{[E_{1} + E_{2} − D]!}{[E_{1} + r − A^{(i)} − D]! [E_{2} + A^{(i)} − 1]!}

× det(q^{−sA^{(t)}}[E_{1} + r − D − A^{(t)}] ⋯ [E_{1} + s + 1 − D − A^{(t)}]

· [E_{2} − s + 1 + A^{(t)}] ⋯ [E_{2} − 1 + A^{(t)}])

= (−1)^{\binom{s}{2}}q^{∑_{i=1}^{r} ((i−1)A^{(i)}+(E_{1}+r−D)+···+(E_{1}+i+1−D))}

× ∏_{i=1}^{r} \frac{[E_{1} + E_{2} − D]!}{[E_{1} + r − D − A^{(i)}]! [E_{2} + A^{(i)} − 1]!}

× det ((q^{−E_{1}−r−D} − q^{−A^{(t)}}) ⋯ (q^{−E_{1}−s−1−D} − q^{−A^{(t)}})

· (q^{E_{2}−s+1} − q^{−A^{(t)}}) ⋯ (q^{E_{2}−1} − q^{−A^{(t)}})).

Now Lemma 35 with X_{t} = −q^{−A^{(t)}}, A_{s} = q^{−E_{1}−s−D}, p_{j−1}(X) = ∏_{i=2}^{j}(q^{E_{2}−i+1} + X) can be applied. After a little bit of manipulation one arrives at (3.1.9). □

3.2. With a diagonal boundary. In this subsection we count nonintersecting families of paths with fixed starting and final points which do not cross the line x = y with respect to the major index. We assume that the starting points lie on the line x + y = D (see the Remark in the previous subsection). Shifts in the direction (v, v) show that in fact there are two classes of locations of the starting points, one if D is even the other if D is odd. But it is more convenient to state our results for generic D.

First we turn to the case that the paths lie below x = y. We use the notations of subsection 2.2. In particular we assume that there is an r-tuple A = (A_{1}, …, A_{r}) of starting points and an r-tuple E = (E_{1}, …, E_{r}) of final points satisfying (2.2.2) and (2.2.11). Let us introduce the following phrases. Given a family 𝔓 = (P_{1}, …, P_{r}) that is an element of Q(ℜ^{η}A_{σ} → E) we say that the path P_{i} is of ordinary type if η_{i} = 0 or, equivalently, if P_{i} starts at A_{σ(i)}. We say that P_{i} is of reflected type if η_{i} = 1 or, equivalently, if P_{i} starts at ℜA_{σ(i)}.

Again we precede the general theorem by an auxilary lemma. There are two statements in the lemma. The first will be needed for the proof of the next theorem, the stronger and more technical second statement will be used in the proof of Theorem 16.

Lemma 4. Let A_{i} = (A_{1}^{(i)}, A_{2}^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} which satisfy (2.2.2) and (2.2.11). Given a permutation σ ∈ 𝔖_{r} and η ∈ {0, 1}^{r}, suppose that 𝔓 = (P_{1}, …, P_{r}) is a family of lattice paths that is an element of Q(ℜ^{η}A_{σ} → E)^{−}. Then either P_{1} meets x = y − 1, or else there exists an index I such that P_{I} and P_{I+1} intersect. Even stronger, either P_{1} meets x = y − 1, or else there exists an index I such that P_{I} and P_{I+1} intersect, where P_{I} is of ordinary type and the paths P_{1}, …, P_{I} are pairwise nonintersecting and lie below x = y.

Proof. If P_{1} crosses x = y, and hence meets x = y − 1, then we are done. Therefore let us assume that P_{1} lies below x = y. Because of (2.2.11) P_{1} must be of ordinary type. Suppose that P_{2} intersects P_{1}. Then P_{1}, P_{2} intersect and P_{1} is of ordinary type and lies below x = y. Hence the theorem would be proved. On the other hand, if P_{2} does not intersect P_{1} then because of (2.2.2b) it must entirely lie below P_{1}. Therefore, because of (2.2.2a), P_{2} must be of ordinary type. Moreover, lying below the path P_{1} which lies below x = y, it cannot reach x = y and hence also lies below x = y. Now we continue considering P_{3}, P_{4}, etc. This procedure must terminate since 𝔓 is intersecting or contains a path which crosses x = y. Thus we obtain an intersecting pair P_{I}, P_{I+1}, where P_{I} is of ordinary type and where P_{1}, …, P_{I} are pairwise nonintersecting and lie below x = y. □

Theorem 5. Let A_{i} = (A^{(i)} + D, −A^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1) and

2A^{(i)} + D ≥ 0 and E_{1}^{(i)} ≥ E_{2}^{(i)}, i = 1, 2, …, r,

hold. The generating function ∑ q^{maj(𝔓)}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where maj 𝔓 is given by (3.0.1), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D \end{matrix} ]

− q^{s(2A^{(t)}+D+1)} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D − 1 \end{matrix} ] )). (3.2.1)

Proof. Again we use the notations of subsection 2.2. On the set (2.2.14) we introduce the weight function ϑ_{1} by

ϑ_{1}(𝔓) = (−1)^{∥η∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}(2A^{(σ(i))}+D+1))}q^{maj 𝔓} if 𝔓 ∈ Q(ℜ^{η}A_{σ} → E). (3.2.2)

The generating function in question is GF (Q(A → E)^{+}; ϑ_{1}). (The symbol GF (M; w) is explained in (2.1.5).) Obviously, the equation

GF (Q(A → E)^{+}; ϑ_{1}) = GF (Q(A → E); ϑ_{1}) − GF (Q(A → E)^{−}; ϑ_{1})  (3.2.3)

holds, which is the analogue of (2.2.12). Similar to subsection 2.2, we are going to construct an involution Θ_{1} on the set (2.2.13) which is sign-reversing with respect to ϑ_{1}. Let 𝔓 = (P_{1}, …, P_{r}) be an element of Q(ℜ^{η}A_{σ} → E)^{−}. Consider all meeting points between neighbouring paths and all meeting points of the path P_{1} with the line x = y − 1. That there is at least one is guaranteed by Lemma 4. Choose the right-most, and, among them, the highest meeting point, and denote it by S.

Now there are two cases. If S is a meeting point between two neighbouring paths, then among the pairs P_{I}, P_{I+1} which meet in S choose that one with I being minimal. As before, it should be noted that S is the last meeting point between P_{I} and P_{I+1}. Then we define

Θ_{1}(𝔓) := (P_{1}, …, P′_{I}, P′_{I+1}, …, P_{r}) ,  (3.2.4a)

where the paths P′_{I}, P′_{I+1} are given by

(P′_{I}, P′_{I+1}) = \{ \begin{matrix}Ψ((P_{I}, P_{I+1})) & if σ(I) < σ(I + 1) \\ Ψ^{−1}((P_{I}, P_{I+1})) & if σ(I) > σ(I + 1) \end{matrix} .  (3.2.4b)

The map Ψ is defined in Proposition 27. It is readily observed that with (3.2.4a,b) the family Θ_{1}(𝔓) is an element of Q(ℜ^{η^{(I,I+1)}}A_{σ(I,I+1)} → E)^{−}, where η^{(I,I+1)} = (η_{1}, …, η_{I+1}, η_{I}, …, η_{r}).

In case that S is no meeting point between any two neighbouring paths, S must be the last meeting point of the path P_{1} with the line x = y − 1. Then we define

Θ_{1}(𝔓) := (P′′_{1}, P_{2}, …, P_{r}),  (3.2.4c)

where the path P′′_{1} is given by

P′′_{1} = \{ \begin{matrix}Φ^{(1)}(P_{1}) & if η_{1} = 0 \\ (Φ^{(1)})^{−1} (P_{1}) & if η_{1} = 1. \end{matrix}  (3.2.4d)

The map Φ^{(1)} is defined in Corollary 25. Obviously with (3.2.4c,d) the family Θ_{1}(𝔓) is an element of Q(ℜ^{η^{(1)}}A_{σ} → E)^{−}, where η^{(1)} = (1 − η_{1}, η_{2}, …, η_{r}).

Since both Ψ and Φ^{(1)} are constructed in a manner that they do not change anything after this last meeting point S, application of Θ_{1} to Θ_{1}(𝔓) gives 𝔓 again. Hence, Θ_{1} is an involution on the set (2.2.13).

We want to establish that Θ_{1} is sign-reversing with respect to ϑ_{1}. Recalling that the starting point of P_{i} is ℜ^{η_{i}}A_{σ(i)}, we observe that it can be written in the form

(A^{(σ(i))} + D, −A^{(σ(i))}) + η_{i}(−2A^{(σ(i))} − D − 1, 2A^{(σ(i))} + D + 1).  (3.2.5)

Now, if Θ_{1}(𝔓) is obtained using (3.2.4a,b), by the definition (3.2.2), and using (5.3.4) and (3.2.5), we get

ϑ_{1}(Θ_{1}(𝔓)) = (−1)^{∥η^{(I,I+1)}∥} sgn (σ(I, I + 1))

· q^{∑{r \atop i=1}(i(A^{(i)}−A^{((σ(I,I+1))(i))})+iη_{i}^{(I,I+1)}(2A^{((σ(I,I+1))(i))}+D+1))}q^{maj Θ_{1}(𝔓)}

= −(−1)^{∥η∥} sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})+I(A^{(σ(I))}−A^{(σ(I+1))})+(I+1)(A^{(σ(I+1))}−A^{(σ(I))})}

· q^{∑_{i=1}^{r} iη_{i}(2A^{(σ(i))}+D+1)−η_{I+1}(2A^{(σ(I+1))}+D+1)+η_{I}(2A^{(σ(I))}+D+1)}
· q^{maj 𝔓+(A^{(σ(I))}−η_{I}(2A^{(σ(I))}+D+1))−(A^{(σ(I+1))}−η_{I+1}(2A^{(σ(I+1))}+D+1))}
= −ϑ_{1}(𝔓).  (3.2.6)

On the other hand, if Θ_{1}(𝔓) is obtained using (3.2.4c,d), by the definition (3.2.2), and using (3.2.5) and (5.2.20), we get

ϑ_{1}(Θ_{1}(𝔓)) = (−1)^{∥η^{(1)}∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}^{(1)}(2A^{(σ(i))}+D+1))}q^{maj Θ_{1}(𝔓)}

= −(−1)^{∥η∥} sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})}q^{∑_{i=1}^{r} iη_{i}(2A^{(σ(i))}+D+1)+(1−2η_{1})(2A^{(σ(1))}+D+1)}

· q^{maj 𝔓−(A^{(σ(1))}+D−η_{1}(2A^{σ(1)}+D+1))+(−A^{(σ(1))}+η_{1}(2A^{σ(1)}+D+1))−1}
= −ϑ_{1}(𝔓).  (3.2.7)

This implies the identity

∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} GF (Q(ℜ^{η}A_{σ} → E)^{−}; ϑ_{1}) = 0 .  (3.2.8)

Obviously, this is the analogue for (2.2.15).

The rest is routine in view of the second part of subsection 2.2. This time we only have to mimic the computation (2.2.17). Indeed, by consecutive use of (3.2.3), (3.2.8), (2.2.16), and (5.1.13), we obtain

GF (Q(A → E)^{+}; ϑ_{1}) = GF (Q(A → E); ϑ_{1}) − GF (Q(A → E)^{−}; ϑ_{1})

= GF (Q(A → E); ϑ_{1}) +_{∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} ; (σ,η)≠(id,0)}GF (Q(ℜ^{η}A_{σ} → E)^{−}; ϑ_{1})

= ∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} GF (Q(ℜ^{η}A_{σ} → E); ϑ_{1})

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})}(GF (P (A_{σ(i)} → E_{i}); maj)

− q^{i(2A^{(σ(i))}+D+1)}GF (P (ℜA_{σ(i)} → E_{i}); maj))

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})}( [ \begin{matrix}E_{1}^{(i)} + E_{2}^{(i)} − D \\ E_{1}^{(i)} − A^{(σ(i))} − D \end{matrix} ]

− q^{i(2A^{(σ(i))}+D+1)} [ \begin{matrix}E_{1}^{(i)} + E_{2}^{(i)} − D \\ E_{2}^{(i)} − A^{(σ(i))} − D − 1 \end{matrix} ] )

= det_{1≤s,t≤r} ( q^{s(A^{(s)}−A^{(t)})} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D \end{matrix} ]

− q^{s(2A^{(t)}+D+1)} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D − 1 \end{matrix} ] ), (3.2.9)

which settles (3.2.1). □

Now we consider the case that the paths lie above the line x = y.

Theorem 6. Let A_{i} = (−A^{(i)}, A^{(i)} + D) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that

A^{(1)} < A^{(2)} < ⋯ < A^{(r)} ,  (3.2.10a)
E_{1}^{(1)} > E_{1}^{(2)} > ⋯ > E_{1}^{(r)} and E_{2}^{(1)} ≤ E_{2}^{(2)} ≤ ⋯ ≤ E_{2}^{(r)}  (3.2.10b)

and

2A^{(i)} + D ≥ 0 and E_{1}^{(i)} ≤ E_{2}^{(i)}, i = 1, 2, …, r,  (3.2.10c)

hold. The generating function ∑ q^{maj(𝔓)}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where maj 𝔓 is given by (3.0.1), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D \end{matrix} ]

− q^{(s−1)(2A^{(t)}+D+1)} [ \begin{matrix}E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D − 1 \end{matrix} ] )). (3.2.11)

Proof. The proof of Theorem 5 can be copied almost word-by-word. We only point out the differences.

Here a weight function ϑ_{2} is introduced by

ϑ_{2}(𝔓) = (−1)^{∥η∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+(i−1)η_{i}(2A^{(σ(i))}+D+1))}q^{maj 𝔓} if 𝔓 ∈ Q(ℜ^{η}A_{σ} → E). (3.2.12)

Note that the only difference with (3.2.2) is the term (i − 1) instead of i before the η_{i}.

What has to be determined, is the generating function GF (Q̄(A → E)^{+}, ϑ_{2}), where Q̄(A → E)^{+} is the set of all families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie above x = y, P_{i} : A_{i} → E_{i}. In order to do this, an involution Θ_{2} is defined in the same way as Θ_{1}, only that (3.2.4d) has to be replaced by

P′′_{1} = Φ^{(2)}(P_{1}),  (3.2.13)

where the map Φ^{(2)} is defined in Proposition 26.

The computations (3.2.6), (3.2.7), (3.2.9) run through quite analogously and can therefore be omitted. The only changes which have to be made are caused by the slightly different weight function ϑ_{2} and that instead of (5.2.20) equation (5.2.22) has to be used. □

There are again special cases where, by use of Lemma 34, the determinants in (3.2.1) and (3.2.11) can be evaluated. The result corresponding to Theorem 5 is given in Theorem 7. For a curious implication of Theorem 7 see Remark (4) at the end of section 4.

Theorem 7. Let A_{i} = (A^{(i)} +D, −A^{(i)}) and E_{i} = (E +i, E +2−i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1a) and

2A^{(i)} + D ≥ 0, i = 1, 2, …, r,

hold. The generating function ∑ q^{maj(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∏_{i=1}^{r} \frac{[2E + 2i − D]!}{[E + r − A^{(i)} − D]! [E + r + 1 + A^{(i)}]!}

× ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ∏_{1≤i≤j≤r}[A^{(i)} + A^{(j)} + D + 1] . (3.2.14)

More generally, if the final points of the lattice paths are E_{i} = (E_{1} + i, E_{2} − i), i = 1, 2, …, r, with E_{1} ≥ E_{2} − 2, then the generating function ∑ q^{maj(𝔓)} for all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{η∈{0,1}^{r}}(−1)^{∥η∥}q^{∑_{i=1}^{r} iη_{i}(2A^{(i)}+D+1)}

∏_{i=1}^{r} \frac{[E_{1} + E_{2} + i − D − 1]!}{[E_{2} + A^{(i)} − η_{i}(2A^{(i)} + D + 1) − 1]! [E_{1} + r − D − A^{(i)} + η_{i}(2A^{(i)} + D + 1)]!}

∏_{1≤i<j≤r}[A^{(j)} − η_{j}(2A^{(j)} + D + 1) − A^{(i)} + η_{i}(2A^{(i)} + D + 1)] (3.2.15)

Proof. First we prove (3.2.14). If the final points of the lattice paths are E_{i} = (E +i, E +2−i), i = 1, 2, …, r, then according to Theorem 5, the generating function in question is

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}2E + 2 − D \\ E + s − A^{(t)} − D \end{matrix} ]

− q^{s(2A^{(t)}+D+1)} [ \begin{matrix}2E + 2 − D \\ E − s − A^{(t)} − D + 1 \end{matrix} ] )). (3.2.16)

Taking some factors out of the determinant yields that this is equal to

q^{∑_{i=1}^{r} iA^{(i)}} ∏_{i=1}^{r} \frac{[2E + 2 − D]!}{[E + r − A^{(i)} − D]! [E + r + A^{(i)} + 1]!}
× det (q^{−sA^{(t)}}[E + r − A^{(t)} − D] ⋯ [E + s + 1 − A^{(t)} − D]
· [E + r + A^{(t)} + 1] ⋯ [E + s + 2 + A^{(t)}]
· ([E + s + 1 + A^{(t)}] ⋯ [E − s + 3 + A^{(t)}]

− q^{s(2A^{(t)}+D+1)}[E + s − A^{(t)} − D] ⋯ [E − s + 2 − A^{(t)} − D])).

(3.2.17)

Setting X_{t} = q^{A^{(t)}+D/2+1/2}, A_{i} = −q^{−E−i+D/2−1/2}, the determinant in (3.2.17) can be written in the form

(−1)^{\binom{r}{2}}q^{2 ∑_{i=1}^{r} ∑_{j=i+1}^{r}(E+j−D/2+1/2)}q^{∑_{i=1}^{r} ∑_{j=−i+2}^{0}(E+j−D/2+1/2)}
det ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t})(A_{r} + 1/X_{t}) ⋯ (A_{s+1} + 1/X_{t})

· q^{s(D/2+1/2)}((1/A_{s} + 1/X_{t}) ⋯ (1/A_{1} + 1/X_{t})(A_{0} + X_{t}) ⋯ (A_{−s+2} + X_{t})

− (1/A_{s} + X_{t}) ⋯ (1/A_{1} + X_{t})(A_{0} + 1/X_{t}) ⋯ (A_{−s+2} + 1/X_{t}))). (3.2.18)

This last determinant is almost in a form suitable for application of Lemma 34. The reader now should consult the very beginning of subsection 5.5 where the notions Laurent polynomial and degree of a Laurent polynomial are explained. Let us define

p_{i−1}(X) := q^{i(D/2+1/2)}( ∏_{j=1}^{i}(1/A_{j} + 1/X) ∏_{j=−i+2}^{0}(A_{j} + X)

− ∏_{j=1}^{i}(1/A_{j} + X) ∏_{j=−i+2}^{0}(A_{j} + 1/X))/ (X − \frac{1}{X} ) .

The denominator X − 1/X in fact is a factor of the numerator of p_{i}, which is seen by setting X = ±1. Therefore X − 1/X cancels out. Hence, the p_{i}’s are Laurent polynomials of degree i. Moreover there obviously holds p_{i}(1/X) = p_{i}(X). Taking X_{t} − 1/X_{t} out of the t-th column of the determinant in (3.2.18), it is seen that Lemma 34 with C = 1 and the above choice of p_{i} can be applied to evaluate this determinant. Substitution of the result into (3.2.17) together with some cumbersome manipulations finally furnishes the expression (3.2.14).

Now we turn to the proof of (3.2.15). If the final points of the lattice paths are E_{i} = (E_{1} + i, E_{2} − i), i = 1, 2, …, r, then according to Theorem 5, the generating function in question is

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s − A^{(t)} − D \end{matrix} ]

− q^{s(2A^{(t)}+D+1)} [ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s + A^{(t)} + 1 \end{matrix} ] )).

Next we use the linearity of the determinant in the columns to expand it as an alternating sum,

∑_{η∈{0,1}^{r}}(−1)^{∥η∥} det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})+η_{t}s(2A^{(t)}+D+1)}

[ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s − A^{(t)} − D + η_{t}(2A^{(t)} + D + 1) \end{matrix} ] ),

or equivalently,

∑_{η∈{0,1}^{r}}(−1)^{∥η∥}q^{∑_{i=1}^{r} iη_{i}(2A^{(i)}+D+1)} det_{1≤s,t≤r} (q^{s(A^{(s)}−η_{s}(2A^{(s)}+D+1)−A^{(t)}+η_{t}(2A^{(t)}+D+1))}

[ \begin{matrix}E_{1} + E_{2} − D \\ E_{1} + s − A^{(t)} − D + η_{t}(2A^{(t)} + D + 1) \end{matrix} ] ).

The determinants in this expression have already been evaluated. In the proof of Theorem 3, only A^{(t)} has to be replaced by A^{(t)} − η_{t}(2A^{(t)} + D + 1). The result is the expression (3.1.9), with this replacement. Thus we obtain (3.2.15). □

The analogous specializations for Theorem 6 yield the following result.

Theorem 8. Let A_{i} = (−A^{(i)}, A^{(i)} +D) and E_{i} = (E +2−i, E +i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that the (3.2.10a) and

2A^{(i)} + D ≥ 0, i = 1, 2, …, r,

hold. The generating function ∑ q^{maj(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r is equal to the expression

q^{\binom{r+1}{2}+r(E−D)−∑_{i=1}^{r} A^{(i)}} ∏_{i=1}^{r} \frac{[2E + 2i − D]!}{[E + r − A^{(i)} − D]! [E + r + 1 + A^{(i)}]!}

× ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ∏_{1≤i≤j≤r}[A^{(i)} + A^{(j)} + D + 1] . (3.2.19)

More generally, if the final points of the lattice paths are E_{i} = (E_{1} − i, E_{2} + i), i = 1, 2, …, r, with E_{1} ≤ E_{2} + 2, then the generating function ∑ q^{maj(𝔓)} for all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{η∈{0,1}^{r}}(−1)^{∥η∥}q^{∑_{i=1}^{r}(i−1)η_{i}(2A^{(i)}+D+1)}

∏_{i=1}^{r} \frac{[E_{1} + E_{2} + i − D − 1]!}{[E_{1} + A^{(i)} − η_{i}(2A^{(i)} + D + 1) − 1]! [E_{2} + r − D − A^{(i)} + η_{i}(2A^{(i)} + D + 1)]!}

∏_{1≤i<j≤r}[A^{(j)} − η_{j}(2A^{(j)} + D + 1) − A^{(i)} + η_{i}(2A^{(i)} + D + 1)] (3.2.20)

Proof. First we prove (3.2.19). According to Theorem 6, the generating function in question is

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}2E + 2 − D \\ E + s − A^{(t)} − D \end{matrix} ]

− q^{(s−1)(2A^{(t)}+D+1)} [ \begin{matrix}2E + 2 − D \\ E − s − A^{(t)} − D + 1 \end{matrix} ] )). (3.2.21)

This is almost the same expression as that in (3.2.16). Hence, the computation proceeds quite analogously to that in the previous proof. Again, setting X_{t} = q^{A^{(t)}+D/2+1/2}, A_{i} = −q^{−E−i+D/2−1/2}, and taking some factors out of the determinant, we deduce that (3.2.21) can be written in the form

q^{∑_{i=1}^{r} iA^{(i)}} ∏_{i=1}^{r} \frac{[2E + 2 − D]!}{[E + r − A^{(i)} − D]! [E + r + A^{(i)} + 1]!}q^{∑_{i=1}^{r}(−A^{(i)}−D/2−1/2)}
(−1)^{\binom{r}{2}}q^{2 ∑_{i=1}^{r} ∑_{j=i+1}^{r}(E+j−D/2+1/2)}q^{∑_{i=1}^{r} ∑_{j=−i+2}^{1}(E+j−D/2+1/2)}
det ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t})(A_{r} + 1/X_{t}) ⋯ (A_{s+1} + 1/X_{t})

· q^{s(D/2+1/2)}((1/A_{s} + 1/X_{t}) ⋯ (1/A_{2} + 1/X_{t})(A_{1} + X_{t}) ⋯ (A_{−s+2} + X_{t})

− (1/A_{s} + X_{t}) ⋯ (1/A_{2} + X_{t})(A_{1} + 1/X_{t}) ⋯ (A_{−s+2} + 1/X_{t}))).

(3.2.22)

Here we have to choose

p_{i−1}(X) := q^{i(D/2+1/2)}( ∏_{j=2}^{i}(1/A_{j} + 1/X) ∏_{j=−i+2}^{1}(A_{j} + X)

− ∏_{j=2}^{i}(1/A_{j} + X) ∏_{j=−i+2}^{1}(A_{j} + 1/X))/ (X − \frac{1}{X} ) .

Also these p_{i}’s are Laurent polynomials of degree i which satisfy p_{i}(1/X) = p_{i}(X). Again, after having taken X_{t} − 1/X_{t} out of the determinant in (3.2.22), Lemma 34 with C = 1 and the above choice of p_{i} is applied to evaluate this determinant. This gives almost the same result as in the previous proof only that some power of q remains, which turns out to be q^{\binom{r+1}{2}+r(E−D)−∑_{i=1}^{r} A^{(i)}}.

In order to prove (3.2.20), we use Theorem 6 with E_{1}^{(i)} = E_{1} − i and E_{2}^{(i)} = E_{2} + i, i = 1, 2, …, r. By (3.2.11) the generating function in question is

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( [ \begin{matrix}E_{1} + E_{2} − D \\ E_{2} + s − A^{(t)} − D \end{matrix} ]

− q^{(s−1)(2A^{(t)}+D+1)} [ \begin{matrix}E_{1} + E_{2} − D \\ E_{2} + s + A^{(t)} + 1 \end{matrix} ] )).

As in the proof of (3.2.15) we use the linearity of the determinant in the columns to expand it as an alternating sum,

∑_{η∈{0,1}^{r}}(−1)^{∥η∥} det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})+η_{t}s(2A^{(t)}+D+1)−η_{t}(2A^{(t)}+D+1)}

[ \begin{matrix}E_{1} + E_{2} − D \\ E_{2} + s − A^{(t)} − D + η_{t}(2A^{(t)} + D + 1) \end{matrix} ] ),

or equivalently,

∑_{η∈{0,1}^{r}}(−1)^{∥η∥}q^{∑_{i=1}^{r}(i−1)η_{i}(2A^{(i)}+D+1)}

det_{1≤s,t≤r} (q^{s(A^{(s)}−η_{s}(2A^{(s)}+D+1)−A^{(t)}+η_{t}(2A^{(t)}+D+1))}

[ \begin{matrix}E_{1} + E_{2} − D \\ E_{2} + s − A^{(t)} − D + η_{t}(2A^{(t)} + D + 1) \end{matrix} ] ).

The determinants in this expression are evaluated by using the computation in the proof of Theorem 3, with E_{1} and E_{2} interchanged and A^{(t)} replaced by A^{(t)}−η_{t}(2A^{(t)}+ D + 1). The result is the expression (3.1.9), with these replacements. Thus we obtain (3.2.20). □

3.3. Tableaux generating functions. As first application we give a nice product formula for a generating function for pairs of tableaux with odd parts.

Theorem 9. The generating function ∑ q^{n(τ_{1})+n(τ_{2})} for pairs (τ_{1}, τ_{2}) of tableaux of identical shape with at most r columns and with only odd parts which lie between 1 and 2n − 1 is given by

∏_{1≤i,j≤n} \frac{[r + i + j − 1]_{q^{2}}}{[i + j − 1]_{q^{2}}} .  (3.3.1)

Equivalently, in terms of Schur functions,

∑_{λ,λ_{1}≤r} s_{λ}^{2}(q^{2n−1}, q^{2n−3}, …, q) = ∏_{1≤i,j≤n} \frac{[r + i + j − 1]_{q^{2}}}{[i + j − 1]_{q^{2}}} .

Proof. By Proposition 29 the pairs of tableaux under consideration via the map Δ_{2} are in one-to-one correspondence with families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths, P_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2), where the North-East corners of the paths have odd coordinates. Moreover the relation (5.4.6) holds. Such families 𝔓 are modified in the following way. While fixing the starting and final points of the paths, shift all North-East corners of the paths of 𝔓 into the direction (−1, 1). More precisely, in each path the (horizontal) step before the first vertical step is turned into a vertical one, and the (vertical) step after the last horizontal step is turned into a horizontal one. Figure 3 should clarify what is meant.

• _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} •• • •• • • • ••  ⟶  _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} • • • • • • • • • • Figure 3

We obtain a new family 𝔓̄ = (P̄_{1}, …, P̄_{r}) of nonintersecting lattice paths consisting of double steps (i.e. only of the steps (2, 0) and (0, 2)), P̄_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2). Besides we obviously have maj 𝔓 = maj 𝔓̄. Therefore, in order to compute the desired generating function for pairs of tableaux, the generating function ∑ q^{maj 𝔓̄} for nonintersecting families 𝔓̄ of the above type has to be considered. But this generating function is evaluated using Theorem 3 with q replaced by q^{2}, A^{(i)} = i − 1, E_{1} = n − 1, and E_{2} = n + 1. With these choices of the parameters from (3.1.9) we obtain the expression

∏_{i=1}^{r} \frac{[2n + i − 1]_{q^{2}}!}{[n + i − 1]_{q^{2}}! [n + r − i]_{q^{2}}!} ∏_{1≤i<j≤r}[j − i]_{q^{2}} .

It is not difficult to convert this expression into (3.3.1). □

Our next application concerns a result about tableaux with even rows. It has been previously obtained by Désarménien [11, Théorème 1.2, first identity], Proctor [37, Theorem 1, case (CYI)], and Stembridge [40, Corollary 4.3 (a)].

Theorem 10. The generating function for tableaux with rows of even length which do not exceed 2r, and with parts between 1 and n, is given by

∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]} .  (3.3.2)

Equivalently, in terms of Schur functions,

∑_{λ, λ_{1}≤r} s_{2λ}(q^{n}, q^{n−1}, …, q) = ∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]} .

Proof. By the map Δ_{3} defined in Proposition 30 the tableaux under consideration uniquely correspond to families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie below the line x = y, P_{i} : (i − 1, −i + 1) → (n + i, n + 2 − i). Moreover the relation (5.4.7) is satisfied. Hence, if in Theorem 7 we set A^{(i)} = i − 1, D = 0, and E = n, we obtain for the generating function in question the expression

∏_{i=1}^{r} \frac{[2n + 2i]!}{[n + r − i + 1]! [n + r + i]!} ∏_{1≤i<j≤r}[j − i] ∏_{1≤i≤j≤r}[i + j − 1].

It is easy to verify that this expression simplifies to (3.3.2). □

Our next theorem refines a result about tableaux with even rows and odd parts that was also previously obtained by Désarménien [11, Théoréme 1.2, second identity], Proctor [37, Theorem 1, case (CYH)], and Stembridge [40, Corollary 4.3 (b)]. Their result is the p = 0 special case of Theorem 11. The c = 2r + 1 case of Theorem 11 was first found by Désarménien [12, Théoréme 2, second identity].

Theorem 11. The generating function for tableaux with p odd rows, with at most c columns, and with only odd parts which lie between 1 and 2n − 1, is given by

q^{p^{2}} \frac{[2r + 2p]_{q^{2}} [r]_{q^{2}}}{[2r + p]_{q^{2}} [r + p]_{q^{2}}} [ \begin{matrix}n \\ p \end{matrix} ]_{q^{2}} \begin{matrix}[ \begin{matrix}n + 2r \\ n \end{matrix} ] & q^{2} \\ [ \begin{matrix}n + 2r + p \\ n \end{matrix} ] & q^{2} \end{matrix}

× ∏_{i=1}^{n} \frac{[r + i]_{q^{2}}}{[i]_{q^{2}}} ∏_{1≤i<j≤n} \frac{[2r + i + j]_{q^{2}}}{[i + j]_{q^{2}}} if c = 2r (3.3.3a)

and

q^{p^{2}} [ \begin{matrix}n \\ p \end{matrix} ]_{q^{2}} ∏_{i=1}^{n} \frac{[r + i]_{q^{2}}}{[i]_{q^{2}}} ∏_{1≤i<j≤n} \frac{[2r + i + j]_{q^{2}}}{[i + j]_{q^{2}}}  if c = 2r + 1.  (3.3.3b)

The formulation of this result in terms of Schur functions is that the sum ∑ s_{λ}(q^{2n−1}, q^{2n−3}, …, q), where the sum is over all partitions λ with exactly p odd parts, and where all the parts do not exceed c, is given by the expressions in (3.3.3).

Proof. We have to distinguish between c being even and c being odd.

First let c = 2r. By Proposition 33 we know that, via the map Δ_{6}, the tableaux under consideration are in one-to-one correspondence with pairs (𝔓, S), where 𝔓 = (P_{1}, …, P_{r}) is a family of nonintersecting lattice paths which lie below the line x = y, P_{i} : (2i−2, −2i+2) → (2n+2i−2, 2n−2i+2), where the North-East corners of all the paths have odd coordinates, with the property that if P_{r} starts with 2h + 1 horizontal steps followed by a vertical step then S is a p-subset of {1, 2, …, 2h − 1}. For a path which starts with H horizontal steps followed by a vertical one, we say that it first turns after H steps. Moreover, Δ_{6} satisfies the relation (5.4.14). We claim that these pairs (𝔓, S) bijectively correspond to families 𝔓̂ = (P̂_{1}, …, P̂_{r}) of nonintersecting lattice paths consisting only of double steps, P̂_{i} : (2i, −2i+2) → (2n+2i, 2n+4−2i), i = 1, …, r − 1, P̂_{r} : (2r + 2p, −2r − 2p + 2) → (2n + 2r, 2n + 4 − 2r), which lie below x = y, such that

maj 𝔓 + ∥S∥ = maj 𝔓̂ + p^{2}.  (3.3.4)

Suppose that this had been done. The generating function ∑_{𝔓̂} q^{maj 𝔓̂} for this last set of families of nonintersecting lattice paths is obtained from Theorem 7 by replacing q by q^{2} and then setting A^{(i)} = i − 1, i = 1, …, r − 1, A^{(r)} = r + p − 1, D = 1, E = n. The resulting expression because of (3.3.4) has to be multiplied by q^{p^{2}}. We thus get

q^{p^{2}} ∏_{i=1}^{r} \frac{[2n + 2i − 1]_{q^{2}}!}{[n + r − i]_{q^{2}}! [n + r + i]_{q^{2}}!} · \frac{[n]_{q^{2}}! [n + 2r]_{q^{2}}!}{[n − p]_{q^{2}}! [n + 2r + p]_{q^{2}}!}

∏_{1≤i<j≤r}[j − i]_{q^{2}} ∏_{i=1}^{r−1} \frac{[r + p − i]_{q^{2}}}{[r − i]_{q^{2}}}

∏_{1≤i≤j≤r}[i + j]_{q^{2}} ∏_{i=1}^{r} \frac{[r + p + i]_{q^{2}}}{[r + i]_{q^{2}}} \frac{[2r + 2p]_{q^{2}}}{[2r + p]_{q^{2}}} ,

which after some manipulation is turned into (3.3.3a).

Thus it remains to construct this bijection. Let (𝔓, S) be a pair as described above. First we modify 𝔓 by performing the same operation on 𝔓 as in the proof of Theorem 9. Namely, while fixing the starting and final points of the paths, shift all North-East corners of the paths of 𝔓 into the direction (−1, 1). More precisely, in each path the (horizontal) step before the first vertical step is turned into a vertical one, and the (vertical) step after the last horizontal step is turned into a horizontal one (cf. Figure 3). We thus obtain a new family 𝔓̄ = (P̄_{1}, …, P̄_{r}) of nonintersecting lattice paths consisting only of double steps, P̄_{i} : (2i−2, −2i+2) → (2n+2i−2, 2n−2i+2), which lie below the line x = y − 2. Besides, P̄_{r} first turns after 2h steps. By shifting the family 𝔓 in the direction (2, 0) and attaching a double step at the end of each path, we obtain the family 𝔓̃ = (P̃_{1}, …, P̃_{r}) of nonintersecting lattice paths consisting only of double steps, P̃_{1} : (2i, −2i + 2) → (2n + 2i, 2n + 4 − 2i), which lie below x = y. Besides, P̃_{r} first turns after 2h steps. We remark that the added double step at the end of each path in fact is superflous since, because of the location of the final points and because of the requirements of being nonintersecting and lying below x = y, each path of 𝔓̃ has to end with a vertical double step. These double steps only are inserted to fit with Theorem 7.

Now we come to the set S. Using (5.1.2) with q replaced by q^{2} it is easily seen that the generating function ∑ q^{∥S∥} for the p-subsets of {1, 3, …, 2h − 1} is given by

∑_{S⊆\{1,3,...,2h−1\}, |S|=p} q^{∥S∥} = q^{p^{2}} [ \begin{matrix}h \\ p \end{matrix} ]_{q^{2}} .  (3.3.5)

On the other hand, by replacing q by q^{2} in (5.1.13) we get that the major generating function for paths from (2r + 2p, −2r − 2p + 2) to (2r + 2h, −2r + 2) consisting only of double steps is given by

∑_{P:(2r+2p,−2r−2p+2)→(2r+2h,−2r+2)} P consists of double stepsq^{maj P} = [ \begin{matrix}h \\ p \end{matrix} ]_{q^{2}} .  (3.3.6)

A comparison of (3.3.6) and (3.3.5) shows that there is a bijection δ_{1} between p-subsets S of {1, 3, …, 2h − 1} and paths P from (2r + 2p, −2r − 2p + 2) to (2r + 2h, −2r + 2), consisting only of double steps, such that the relation

∥S∥ = maj δ_{1}(S) + p^{2}  (3.3.7)

holds. (An explicit bijection could be extracted from [20].) Since P̃_{r} starts at (2r, −2r+2) and first turns after 2h steps, the point (2r+2h, −2r+2) lies on P̃_{r}. Now we form a new path P̂_{r} by joining δ_{1}(S) and the portion of P̃_{r} from (2r + 2h, −2r + 2) to its final point (2n + 2r, 2n + 4 − 2r). Moreover, since the step after the point (2r + 2h, −2r + 2) is a vertical one, we always are able to recover δ_{1}(S), and hence S, from P̂_{r}. Summarizing, starting with the pair (𝔓, S) we obtained the family 𝔓̂ = (P̃_{1}, …, P̃_{r−1}, P̂_{r}). Besides, (3.3.4) holds. This gives the claimed bijection.

Finally, let c = 2r + 1. We claim that the tableaux τ in question are in bijection with pairs (τ_{e}, S), where τ_{e} is a tableau with even rows, with at most 2r columns, and with only odd parts which lie between 1 and 2n − 1, and where S is a p-subset of {1, 3, …, 2n − 1}. This is seen using Knuth’s procedures ROW-DELETE and ROW-INSERT (see subsection 5.4). Given a tableau τ with p odd rows, perform the procedure ROW-DELETE for each of the odd rows, beginning from the lowest and working upward to the highest row. We thus obtain a sequence n_{1}, n_{2}, …, n_{p} of odd integers, the n_{1} being deleted by the first application of ROW-DELETE, etc. Clearly, S := {n_{1}, n_{2}, …, n_{p}} is a subset of {1, 3, …, 2n − 1}. Moreover we are left with a tableau τ_{e} without odd rows. The observation that the integers n_{1}, n_{2}, …, n_{p} are in increasing order shows that by successively inserting n_{p}, n_{p−1}, …, n_{1} into τ_{e} utilizing ROW-INSERT the above procedure could be inverted. Hence it is a bijection which in addition satisfies n(τ) = n(τ_{e}) + ∥S∥. Use of (3.3.3a) with p = 0 and of identity (3.3.5) with h replaced by n establishes (3.3.3b). □

Theorem 11 also has a formulation in terms of symmetric plane partitions. A plane partition τ is called symmetric if τ_{ij} = τ_{ji} for all i, j.

Corollary 12. The generating function for symmetric plane partitions with at most n rows, with parts between 1 and c, and with exactly p odd entries on the main diagonal, is given by the expressions in (3.3.3).

Proof. By Theorem 11 the expressions in (3.3.3) are equal to the sum ∑ s_{λ}(q^{2n−1}, q^{2n−3}, …, q) summed over all partitions λ with exactly p odd parts and λ_{1} ≤ c. Examining the correspondence in [40, p. 497] more closely, it is easily verified that it is also a weight preserving-bijection between symmetric plane partitions with at most n rows, with parts between 1 and c, and with exactly p odd entries on the main diagonal, and column-strict plane partitions with p odd rows, with at most c columns, and with parts being odd and between 1 and 2n − 1. But the generating function for this set of column-strict plane partitions is equal to the above summation of Schur functions. □

Now we are in the position to give a new proof of the MacMahon conjecture about symmetric plane partitions. The first proofs were given by Andrews [2], Macdonald [33, Ex. 16 and 17, pp. 51/52], and Proctor [36, Proposition 7.3].

Theorem 13. The generating function for symmetric plane partitions with at most n rows and with parts between 1 and c is equal to

∏_{i=1}^{n} \frac{[c + 2i − 1]_{q}}{[2i − 1]_{q}} ∏_{1≤i<j≤n} \frac{[c + i + j − 1]_{q^{2}}}{[i + j − 1]_{q^{2}}} .  (3.3.8)

Equivalently, the generating function for tableaux with at most c columns and with only odd parts which lie between 1 and 2n − 1 is also given by (3.3.8). Or in terms of Schur functions: The sum ∑_{λ, λ_{1}≤c} s_{λ}(q^{2n−1}, q^{2n−3}, …, q) is also equal to (3.3.8).

Proof. We have to sum the expressions in (3.3.3) with respect to p. If the sum over all p of the expressions (3.3.3a) is converted into hypergeometric notation (cf. subsection 2.1), it reads

∏_{i=1}^{n} \frac{[r + i]_{q^{2}}}{[i]_{q^{2}}} ∏_{1≤i<j≤n} \frac{[2r + i + j]_{q^{2}}}{[i + j]_{q^{2}}}

×_{6}ϕ_{5}[ \begin{matrix}q^{4r}, q^{2r+2}, −q^{2r+2}, q^{2r}, −q^{2r+1}, q^{−2n} \\ q^{2r}, −q^{2r}, q^{2r+2}, −q^{2r+1}, q^{4r+2n+2} \end{matrix} ; q^{2}, −q^{2n+1}] .

The _{6}ϕ_{5} can be evaluated by using the very well-poised _{6}ϕ_{5}-summation (see [14, Appendix (II.21)])

6ϕ_{5} [ \begin{matrix}a, q\sqrt{a}, −q\sqrt{a}, b, c, q^{−n} \\ \sqrt{a}, −\sqrt{a}, aq/b, aq/c, aq^{n+1} \end{matrix} ; q, \frac{aq^{n+1}}{bc} ] = \frac{(aq)_{n} (aq/bc)_{n}}{(aq/b)_{n} (aq/c)_{n}} .  (3.3.9)

Thus we obtain

∏_{i=1}^{n} \frac{[r + i]_{q^{2}}}{[i]_{q^{2}}} ∏_{1≤i<j≤n} \frac{[2r + i + j]_{q^{2}}}{[i + j]_{q^{2}}}  \frac{(q^{4r+2}; q^{2})_{n} (−q; q^{2})_{n}}{(q^{2r+2}; q^{2})_{n} (−q^{2r+1}; q^{2})_{n}}

which after some manipulation yields (3.3.8) with c = 2r.

On the other hand, the sum over all p of the expressions (3.3.3b) is done by using the q-binomial theorem (see e.g. [1, (3.3.7)])

∑_{k=0}^{N} q^{\binom{k}{2}} [ \begin{matrix}N \\ k \end{matrix} ] z^{k} = (1 + z)(1 + qz) ⋯ (1 + q^{N−1}z)  (3.3.10)

with q replaced by q^{2}, N = n, and z = q. This gives

∏_{i=1}^{n}(1 + q^{2i−1}) ∏_{i=1}^{n} \frac{[r + i]_{q^{2}}}{[i]_{q^{2}}} ∏_{1≤i<j≤n} \frac{[2r + i + j]_{q^{2}}}{[i + j]_{q^{2}}} .

A little bit of manipulation shows that this expression is the same as that in (3.3.8) with c = 2r + 1. □

Remark. It is interesting to note that by the deleting-inserting procedure described at the end of the proof of Theorem 11, the theorem of Désarménien, Proctor, and Stembridge about tableaux with even rows and odd parts (Theorem 10 with p = 0) is equivalent with the MacMahon conjecture (Theorem 13) for c = 2r + 1. So in fact we could say that this result about tableaux with even rows and odd parts is a special case of the MacMahon conjecture. This equivalence was previously observed by Désarménien [12] although he is arguing by the Littlewood–Richardson rule.

IV. Counting by Strange Major Index

In this section we consider two generalizations of the major index. Remember (see subsection 2.1) that the major index of a path can be formulated in terms of contributions of the North-East corners of the path. The idea for our extensions of the major index is that every North-East corner which lies strictly above a fixed horizontal line contributes an extra weight, or every North-East corner which lies to the right-hand side of a fixed vertical line contributes an extra weight. We denote the statistics which depends on the y-coordinates of the North-East corners by ymaj and the statistics which depends on the x-coordinates of the North-East corners by xmaj. Let β be some real number, γ be an integer, and P be a path from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}). For γ ≥ A_{2} we introduce the statistics ymaj_{β;γ} by

ymaj_{β;γ}(P) = maj P + β · |{(p_{1}, p_{2}) : (p_{1}, p_{2}) ∈ NE(P) and p_{2} > γ}|, for γ ≥ A_{2},

(4.0.1)

where NE(P) denotes the set of North-East corners of P. For γ ≥ A_{1} we introduce the statistics xmaj_{β;γ} by

xmaj_{β;γ}(P) = maj P + β · |{(p_{1}, p_{2}) : (p_{1}, p_{2}) ∈ NE(P) and p_{1} ≥ γ}|, for γ ≥ A_{1}.

(4.0.2)

For example for the path P_{0} in our example in Figure 1 we have ymaj_{β;2}(P_{0}) = 16+2β or ymaj_{β;3}(P_{0}) = 16 + β, and xmaj_{β;1}(P_{0}) = 16 + 3β or xmaj_{β;6}(P_{0}) = 16. These two statistics ymaj_{β;γ} and xmaj_{β;γ} in the sequel will be referred to as strange major indices.

The definition (4.0.1) would not be reasonable for γ < A_{2}, and definition (4.0.2) would not be reasonable for γ < A_{1}, since in these cases all North-East corners of P would automatically contribute a β. But the observation (5.1.10), together with the remarks after this identity, suggests a useful extension for ymaj_{β;γ} and xmaj_{β;γ} for these values of γ. We define

ymaj_{β;γ}(P) = xmaj_{β;A_{1}+A_{2}−γ}(P) + β(A_{2} − γ)  for γ < A_{2},  (4.0.3)

and

xmaj_{β;γ}(P) = ymaj_{β;A_{1}+A_{2}−γ}(P) + β(A_{1} − γ)  for γ < A_{1}.  (4.0.4)

From the definitions (4.0.1)–(4.0.4) we infer that for all integers γ there holds the relation

ymaj_{β;γ}(P) = xmaj_{β;A_{1}+A_{2}−γ}(P) + β(A_{2} − γ).  (4.0.5)

Since in section 5 we shall most of the time use the array representation of paths, we also have to present formulas for computing the major indices of a path P : A → E in terms of its array representation

\begin{aligned}A_{1} ≤ & a_{1} & a_{2} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & b_{2} & … & b_{k} & ≤ E_{2} \end{aligned} .

We have

ymaj_{β;γ} P = \{ \begin{matrix}∥a∥ + ∥b∥ + β · |\{t : b_{t} > γ\}| − k(A_{1} + A_{2}) & if γ ≥ A_{2} \\ \begin{matrix}∥a∥ + ∥b∥ + β · |\{t : a_{t} ≥ A_{1} + A_{2} − γ\}| \\ +β(A_{2} − γ) − k(A_{1} + A_{2}) \end{matrix} & if γ < A_{2} \end{matrix}  (4.0.6)

and

xmaj_{β;γ} P = \{ \begin{matrix}∥a∥ + ∥b∥ + β · |\{t : a_{t} ≥ γ\}| − k(A_{1} + A_{2}) & if γ ≥ A_{1} \\ \begin{matrix}∥a∥ + ∥b∥ + β · |\{t : b_{t} > A_{1} + A_{2} − γ\}| \\ +β(A_{1} − γ) − k(A_{1} + A_{2}) \end{matrix} & if γ < A_{1} \end{matrix}  (4.0.7)

These definitions admittedly seem to be rather artificial. The motivation of doing strange major counting is revealed in the proof of Theorem 21, where the Choi/Gouyou-Beauchamps problem which was mentioned in the introduction is solved by transferring it, with the help of Proposition 32, to a strange major counting problem for nonintersecting lattice paths.

Intending to count families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths by strange major index, one could try the most general way to do that, namely to count P_{i} by ymaj_{β_{i};γ_{i}} or xmaj_{β_{i};γ_{i}}, i = 1, 2, …, r. But since we are going to rely on Proposition 27, in view of (5.3.2) and (5.3.3), we see that it can only be possible to analogize the procedures of subsection 2.2 if all the β_{i}’s are equal and the γ_{i}’s are successive integers. This motivates the following extension of the strange major indices to families 𝔓 = (P_{1}, …, P_{r}), P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, satisfying (2.2.2). We set

ymaj_{β;γ}(𝔓) = ∑_{i=1}^{r} ymaj_{β;γ−i+1}(P_{i}),  (4.0.8)

and

xmaj_{β;γ}(𝔓) = ∑_{i=1}^{r} xmaj_{β;γ+i−r}(P_{i}).  (4.0.9)

We remark that the effect of requiring (2.2.2) in these definitions is that in (4.0.8) the “highest” path is counted by ymaj_{β;γ}, while in (4.0.9) the “lowest” path is counted by xmaj_{β;γ}.

4.1. Without restrictions. We proceed in analogy with subsection 3.1. We start with the result for the strange major counting of unrestricted families of nonintersecting lattice paths with given starting and final points. In fact there are two results, one for ymaj and one for xmaj.

Theorem 14. Let A_{i} = (A^{(i)} + D, −A^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1) is satisfied.

If γ is an integer satisfying

D − E_{1}^{(1)} ≤ γ ≤ min_{1≤i≤r}(E_{2}^{(i)} + i − 1)  (4.1.1)

then the generating function ∑ q^{ymaj_{β;γ}(𝔓)}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where ymaj_{β;γ}(𝔓) is given by (4.0.8), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D − j \end{matrix} ] ).

(4.1.2)

If γ is an integer satisfying

max_{1≤i≤r}(D − E_{2}^{(i)} + r − i) ≤ γ ≤ E_{1}^{(1)} + r − 1  (4.1.3)

then the generating function ∑ q^{xmaj_{β;γ}(𝔓)}, summed over the same family of nonintersecting lattice paths, equals

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})} ∑_{j≥0} q^{j(j+β+γ−A^{(t)}+s−r)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} + A^{(t)} − j \end{matrix} ] ).

(4.1.4)

Proof. The proof of Theorem 2 can be copied almost word-by-word. We only point out the differences. For the proof of (4.1.2), in (3.1.3) maj has to be replaced by ymaj_{β;γ}, while for the proof of (4.1.4) it has to be replaced by xmaj_{β;γ}. The computation (3.1.6) then also works because instead of (5.3.4) we can use (5.3.2) or (5.3.3), respectively. Finally, in the computation (3.1.8) maj has to be replaced by ymaj_{β;γ−i+1}, respectively xmaj_{β;γ+i−r}. Subsequently, the generating functions GF (P (A_{σ(i)} → E_{i}); ymaj_{β;γ−i+1}) and GF (P (A_{σ(i)} → E_{i}); xmaj_{β;γ+i−r}) are obtained using (5.1.11) and (5.1.12), respectively, instead of (5.1.13). It should be noted that (4.1.1) and (4.1.3) guarantee that the requirements of Lemma 23 are indeed satisfied. □

In the same special case as that in Theorem 3 we can get rid of the determinant. But, unfortunately, what remains is a multiple sum which evidently cannot be summed into a closed form (see Remark (3) at the end of this section), except of course that for β = 0 we obtain Theorem 3 again.

Theorem 15. Let A_{i} = (A^{(i)} +D, −A^{(i)}) and E_{i} = (E_{1} +i, E_{2} −i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1a) holds.

If γ is an integer satisfying

D − E_{1} − 1 ≤ γ ≤ E_{2} − 1,

then the generating function ∑ q^{ymaj_{β;γ}(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

× \frac{[β + E_{1} + E_{2} + i − D − 1]!}{[β + E_{2} + A^{(i)} + k_{i} − 1]! [E_{1} + r − D − A^{(i)} − k_{i}]!} ∏_{1≤i<j≤r}[A^{(j)} + k_{j} − A^{(i)} − k_{i}]) .

(4.1.5a)

An alternative expression (which usually is computationally better) is

∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}E_{2} − γ − 1 \\ k_{i} \end{matrix} ]

× \frac{[E_{1} − D + γ + i]!}{[E_{1} + r − D − A^{(i)} − k_{i}]! [γ + A^{(i)} + k_{i}]!} ∏_{1≤i<j≤r}[A^{(j)} + k_{j} − A^{(i)} − k_{i}]) .

(4.1.5b)

If γ is an integer satisfying

D − E_{2} + r ≤ γ ≤ E_{1} + r,

then the generating function ∑ q^{xmaj_{β;γ}(𝔓)} for the same set of families 𝔓 of nonintersecting lattice paths equals

∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ−A^{(i)}+i−r)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

× \frac{[β + E_{1} + E_{2} + i − D − 1]!}{[E_{2} + A^{(i)} − k_{i} − 1]! [β + E_{1} + r − D − A^{(i)} + k_{i}]!} ∏_{1≤i<j≤r}[A^{(j)} − k_{j} − A^{(i)} + k_{i}]) .

(4.1.6a)

An alternative expression is

∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ−A^{(i)}+i−r)} [ \begin{matrix}E_{1} − D + r − γ \\ k_{i} \end{matrix} ]

× \frac{[E_{2} + γ − i]!}{[E_{2} + A^{(i)} − k_{i} − 1]! [γ − A^{(i)} + k_{i}]!} ∏_{1≤i<j≤r}[A^{(j)} − k_{j} − A^{(i)} + k_{i}]) .

(4.1.6b)

Proof. For the ymaj_{β;γ}-generating function set E_{1}^{(i)} = E_{1} +i and E_{2}^{(i)} = E_{2} −i in (4.1.2). This provides a determinant with a sum in each entry. The summation indices are independent, but of course it is allowed to make them dependent by setting all the summation indices in the t-th column equal to k_{t}, say. Then we use the linearity in the t-th column to obtain for the ymaj_{β;γ}-generating function the expression

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

× det (q^{s(A^{(s)}+k_{s}−A^{(t)}−k_{t})} [ \begin{matrix}β + E_{1} + E_{2} − D \\ E_{1} + s − D − A^{(t)} − k_{t} \end{matrix} ]) .

The determinant in this last expression has already been evaluated. In the proof of Theorem 3, E_{2} has to be replaced by E_{2} + β and A^{(t)} by A^{(t)} + k_{t}. The result is the expression in (3.1.9), with these replacements. This furnishes (4.1.5a).

The expression (4.1.5b) follows from Gustafson’s [22] recently discovered A_{r}-type extension of Heine’s transformation (5.1.6),

∑_{k_{1},...,k_{r}≥0} (∏_{i=1}^{r} q^{k_{i}(1−i)}Z^{k_{i}} \frac{(A)_{k_{i}} (BX_{i})_{k_{i}}}{(q)_{k_{i}} (CX_{i})_{k_{i}}} ) ∏_{1≤i<j≤r} \frac{1 − \frac{X_{j}}{X_{i}} q^{k_{j}−k_{i}}}{1 − \frac{X_{j}}{X_{i}}}

= ∏_{i=1}^{r} \frac{(Cq^{i−r}/B)_{∞} (BZX_{i})_{∞}}{(Zq^{i−r})_{∞} (CX_{i})_{∞}} ∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(1−i)} (\frac{C}{B} )^{k_{i}}

\frac{(ABZ/C)_{k_{i}} (BX_{i})_{k_{i}}}{(q)_{k_{i}} (BZX_{i})_{k_{i}}} ) ∏_{1≤i<j≤r} \frac{1 − \frac{X_{j}}{X_{i}} q^{k_{j}−k_{i}}}{1 − \frac{X_{j}}{X_{i}}} . (4.1.7)

The proofs of (4.1.6a,b) are similar. □

4.2. With a diagonal boundary. Here we proceed in analogy with subsection 3.2. First we state the strange major counting result for families of nonintersecting lattice paths which lie below the line x = y. There is only a result for ymaj but none for xmaj.

Theorem 16. Let A_{i} = (A^{(i)} + D, −A^{(i)}) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1) and

2A^{(i)} + D ≥ 0 and E_{1}^{(i)} ≥ E_{2}^{(i)}, i = 1, 2, …, r,

hold. Let γ be an integer which satisfies the inequalities

D − E_{1}^{(1)} ≤ γ ≤ min_{1≤i≤r}(E_{2}^{(i)} + i − 1), max_{1≤i≤r}(D − E_{2}^{(i)} − i) ≤ γ ≤ E_{1}^{(1)} − 1, (4.2.1a)
and − A^{(1)} ≤ γ ≤ A^{(1)} + D.  (4.2.1b)

The generating function ∑ q^{ymaj_{β;γ}(𝔓)}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where ymaj_{β;γ}(𝔓) is given by (4.0.8), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D − j \end{matrix} ]

− q^{s(2A^{(t)}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}+s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D − j − 1 \end{matrix} ] )).

(4.2.2)

Proof. We adopt the notations of subsection 2.2. Moreover we use the phrases “paths of reflected type” and “paths of ordinary type” which were introduced at the beginning of subsection 3.2. We follow the basic framework of the proof of Theorem 5. The details however are more delicate.

First we have to define a weight function on the set (2.2.14). We are going to use Proposition 24. Relation (5.2.2) tells us that if a path is of reflected type we have to take xmaj instead of ymaj. This observation motivates to introduce a weight function ϑ_{3} by

ϑ_{3}(𝔓) = (−1)^{∥η∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}(2A^{(σ(i))}+D+β+1))}q^{ymaj_{β;γ}^{(η)}(𝔓)} if 𝔓 ∈ Q(ℜ^{η}A_{σ} → E), (4.2.3)

where ymaj_{β;γ}^{(η)} is given by

ymaj_{β;γ}^{(η)}(𝔓) = ∑_{i=1}^{r} ((1 − η_{i}) ymaj_{β;γ−i+1}(P_{i}) + η_{i}(xmaj_{β;γ+i}(P_{i}))).  (4.2.4)

The effect of (4.2.4) is that P_{i} is counted by ymaj if P_{i} is of ordinary type and by xmaj if P_{i} is of reflected type, just as desired. Since for η = 0 all paths P_{i} are of ordinary type, and the i-th path is counted by ymaj_{β;γ−i+1}, because of (4.0.8) the generating function in question is GF (Q(A → E)^{+}; ϑ_{3}). Obviously the equation

GF (Q(A → E)^{+}; ϑ_{3}) = GF (Q(A → E); ϑ_{3}) − GF (Q(A → E)^{−}; ϑ_{3})  (4.2.5)

holds. This is the extension of (3.2.3).

We are going to construct an involution Θ_{3} on the set in (2.2.13) which is signreversing with respect to ϑ_{3}. Let 𝔓 = (P_{1}, …, P_{r}) be an element of Q(ℜ^{η}A_{σ} → E)^{−}. Consider all meeting points of the path P_{1} with the line x = y − 1 and all meeting points of neighbouring paths P_{I}, P_{I+1} with the restriction that if P_{I} and P_{I+1} are of different type (which means that neither both are of ordinary type nor both are of reflected type), then the paths P_{1}, …, P_{I−1} are pairwise nonintersecting and lie below x = y, and both P_{I} and P_{I+1}, after having met for the last time, do not meet one of the paths P_{1}, …, P_{I−1}. In addition, if P_{I} is of ordinary type and P_{I+1} is of reflected type then σ(I) ≥ I, and if P_{I} is of reflected type and P_{I+1} is of ordinary type then σ(I + 1) ≥ I. Note that, compared to the proof of Theorem 5, here we refine the choice of meeting points between neighbouring paths. That there is at least one meeting point of the above kind is guaranteed by the strong statement of Lemma 4: Either there is a meeting point of P_{1} with x = y − 1, or there is an I such that P_{I} and P_{I+1} intersect and such that P_{I} is of ordinary type. Should P_{I+1} be of reflected type then, also by Lemma 4, P_{1}, …, P_{I} are pairwise nonintersecting and lie below x = y. So in particular, P_{I} does not meet P_{1}, …, P_{I−1} at all. Besides, since due to (3.1.1b) P_{I} lies below P_{1}, …, P_{I−1}, and the final point of P_{I+1} lies in the South-East of the final point of P_{I}, P_{I+1} must stay below P_{I} and hence below P_{1}, …, P_{I−1} after having met P_{I} for the last time. That σ(I) ≥ I holds is seen by observing that (3.1.1a) and the fact that P_{2} lies below P_{1}, P_{3} below P_{2}, …, P_{I} below P_{I−1}, imply that 1 ≤ σ(1) < σ(2) < ⋯ < σ(I).

Now choose the right-most and among them the highest meeting point and denote it by S.

Again two cases have to be considered for the definition of Θ_{3}. If S is a meeting point between two neighbouring paths the definition of Θ_{3} is identical with that of Θ_{1} in (3.2.4a,b). Namely, among the pairs P_{I}, P_{I+1} which meet in S choose that one with I being minimal and set

Θ_{3}(𝔓) := (P_{1}, …, P′_{I}, P′_{I+1}, …, P_{r}) ,  (4.2.6a)

where the paths P′_{I}, P′_{I+1} are given by

(P′_{I}, P′_{I+1}) = \{ \begin{matrix}Ψ((P_{I}, P_{I+1})) & if σ(I) < σ(I + 1) \\ Ψ^{−1}((P_{I}, P_{I+1})) & if σ(I) > σ(I + 1) \end{matrix} .  (4.2.6b)

The map Ψ is defined in Proposition 27. Recall that S is the last meeting point between P_{I} and P_{I+1} and that with (4.2.6a,b) the family Θ_{3}(𝔓) is an element of Q(ℜ^{η^{(I,I+1)}}A_{σ(I,I+1)} → E)^{−}, where η^{(I,I+1)} = (η_{1}, …, η_{I+1}, η_{I}, …, η_{r}).

On the other hand, if S is no meeting point between neighbouring paths of the above type, then we define

Θ_{3}(𝔓) := (P′′_{1}, P_{2}, …, P_{r}),  (4.2.6c)

where the path P′′_{1} is given by

P′′_{1} = \{ \begin{matrix}Φ_{γ}^{(1)}(P_{1}) & if η_{1} = 0 \\ (Φ_{γ}^{(1)})^{−1} (P_{1}) & if η_{1} = 1. \end{matrix}  (4.2.6d)

The map Φ_{γ}^{(1)} is defined in Proposition 24. To justify that P′′_{1} is well-defined, note that because of (3.1.1a) and (4.2.1b) we have

−A^{(σ(I))} ≤ −A^{(1)} ≤ γ ≤ A^{(1)} + D ≤ A^{(σ(I))} + D,

which implies −A^{(σ(I))} ≤ γ ≤ A^{(σ(I))} + D. But (A^{(σ(I))} + D, −A^{(σ(I))}) is the starting point of P_{I} for η_{I} = 0 or, respectively, it is the reflected (reflected in the line x = y − 1) starting point of P_{I} for η_{I} = 1. This says nothing but that in both cases (5.2.1) is satisfied and so Φ_{γ}^{(1)}(P_{1}), respectively (Φ_{γ}^{(1)})^{(−1)}(P_{1}) indeed are well-defined. Obviously with (4.2.6c,d) the family Θ_{3}(𝔓) is an element of Q(ℜ^{η^{(1)}}A_{σ} → E)^{−}, where η^{(1)} = (1 − η_{1}, η_{2}, …, η_{r}).

It is not too difficult to see that Θ_{3} is an involution. There are only two nontrivial cases. One of these two non-trivial cases arises if S is the last meeting point between P_{I} and P_{I+1}, I being minimal, where P_{I} is of ordinary type and P_{I+1} is of reflected type. In this case because of our restricted choice of meeting points the paths P_{1}, …, P_{I−1} must be pairwise nonintersecting and lie below x = y. In addition, the portions of P_{I} and P_{I+1} after their last meeting point S do not meet P_{1}, …, P_{I−1}. By (4.2.6a,b), P′_{I} is of reflected type and P′_{I+1} is of ordinary type. Since Ψ has the property of “changing nothing after S” the paths P′_{I} and P′_{I+1}, after having met in S for the last time, do not meet P_{1}, …, P_{I−1}. Besides, (σ(I, I + 1))(I + 1) = σ(I) ≥ I. Hence S, being a meeting point between P′_{I} and P′_{I+1}, is also under consideration for applying Θ_{3} to Θ_{3}(𝔓) = (P_{1}, …, P′_{I}, P′_{I+1}, …, P_{r}). But then it is the right-most and highest meeting point for Θ_{3}(𝔓). Therefore when applying Θ_{3} to Θ_{3}(𝔓) one obtains 𝔓 again. The case of P_{I} being of reflected type and P_{I+1} being of ordinary type is dealt with similarly.

We want to establish that Θ_{3} is sign-reversing with respect to ϑ_{3}. First assume that Θ_{3}(𝔓) is obtained using (4.2.6a,b). There are four cases depending on P_{I} being of ordinary or reflected type and on P_{I+1} being of ordinary or reflected type. If both P_{I} and P_{I+1} are of ordinary type, i.e. η_{I} = η_{I+1} = 0, from (5.3.2) we get

ymaj_{β;γ−I+1}(P_{I}) + ymaj_{β;γ−I}(P_{I+1})

= ymaj_{β;γ−I+1}(P′_{I}) + ymaj_{β;γ−I}(P′_{I+1}) + A^{(σ(I+1))} − A^{(σ(I))}

and hence

ymaj_{β;γ}^{(η)}(𝔓) = ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) + A^{(σ(I+1))} − A^{(σ(I))}
= ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) + A^{(σ(I+1))} − η_{I+1}(2A^{(σ(I+1))} + D + β + 1)
− A^{(σ(I))} + η_{I}(2A^{(σ(I))} + D + β + 1).

Secondly, if both P_{I} and P_{I+1} are of reflected type, i.e. η_{I} = η_{I+1} = 1, then from (5.3.3) we get

xmaj_{β;γ+I}(P_{I}) + xmaj_{β;γ+I+1}(P_{I+1})

= xmaj_{β;γ+I}(P′_{I}) + xmaj_{β;γ+I+1}(P′_{I+1}) − A^{(σ(I+1))} + A^{(σ(I))}

and hence

ymaj_{β;γ}^{(η)}(𝔓) = ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) − A^{(σ(I+1))} + A^{(σ(I))}
= ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) + A^{(σ(I+1))} − η_{I+1}(2A^{(σ(I+1))} + D + β + 1)
− A^{(σ(I))} + η_{I}(2A^{(σ(I))} + D + β + 1).

Now comes the first delicate case. Let P_{I} be of reflected and P_{I+1} be of ordinary type, i.e. η_{I} = 1 and η_{I+1} = 0. We want to apply (5.3.5) with γ_{1} = γ +I and γ_{2} = γ −I. In order to be allowed to use (5.3.5), according to Proposition 27 it suffices to check that A+D < γ +I, that the last meeting point S of P_{1} and P_{2} is located to the right or on the vertical line x = γ + I, and that −B − 1 ≤ γ − I < −A, where (A + D, −A) is the starting point of P_{I} and (B+D, −B) is the starting point of P_{I+1}. Recall that because of 𝔓 ∈ Q(ℜ^{η}A_{σ} → E)^{−} the starting point of P_{I} is (−A^{(σ(I))}−1, A^{(σ(I))}+D+1), while the starting point of P_{I+1} is (A^{(σ(I+1))} + D, −A^{(σ(I+1))}). Hence the first inequality trivially holds because of (4.2.1b), −A^{(σ(I))} − 1 < −A^{(1)} ≤ γ < γ + I. To continue, let S = (S_{1}, S_{2}). Since S is on P_{I} we must have S_{2} ≥ A^{(σ(I))} + D + 1. It is easy to see that S must lie below or on x = y. From the assumption that P_{1}, …, P_{I−1} are pairwise nonintersecting and that the portions of P_{I} and P_{I+1} after S do not meet P_{1}, …, P_{I−1}, we infer that P_{1} passes through y = S_{2} left-most, that P_{2} passes y = S_{2} to the right of P_{1}, …, P_{I−1} to the right of P_{I−2}, that moreover P_{1}, …, P_{I−1} have no point in common and that S is located strictly to the right of the passage of P_{I−1} through y = S_{2}. Besides, because of lying below x = y, P_{1} leaves y = S_{2} earliest in (S_{2} + 1, S_{2}). Thus we infer S_{1} ≥ S_{2} + I. These observations together with (3.1.1a) and (4.2.1b) yield

γ + I ≤ A^{(1)} + D + I < A^{(σ(I))} + D + I + 1 ≤ S_{2} + I ≤ S_{1},

which says nothing but that S is located to the right of x = γ + I. Besides, because of σ(I + 1) ≥ I (this is due to the assumption of P_{I} and P_{I+1} being of different type; cf. the paragraph after (4.2.5)), (3.1.1a), and (4.2.1b) we have

−A^{(σ(I+1))} − 1 ≤ −A^{(I)} − 1 ≤ −A^{(1)} − I ≤ γ − I ≤ γ ≤ A^{(1)} + D < A^{(σ(I))} + D + 1,

which in particular implies −A^{(σ(I+1))} − 1 ≤ γ − I ≤ A^{(σ(I))} + D + 1, which is just the second inequality required to hold in order to be allowed to apply (5.3.5). Hence we may indeed apply (5.3.5) to get

xmaj_{β;γ+I}(P_{I}) + ymaj_{β;γ−I}(P_{I+1})

= ymaj_{β;γ−I+1}(P′_{I}) + xmaj_{β;γ+I+1}(P′_{I+1}) + A^{(σ(I+1))} + A^{(σ(I))} + D + 1 + β,

and hence

ymaj_{β;γ}^{(η)}(𝔓) = ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) + A^{(σ(I+1))} + A^{(σ(I))} + D + 1 + β
= ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) + A^{(σ(I+1))} − η_{I+1}(2A^{(σ(I+1))} + D + β + 1)
− A^{(σ(I))} + η_{I}(2A^{(σ(I))} + D + β + 1).

Finally, let P_{I} be of ordinary type and P_{I+1} be of reflected type, i.e. η_{I} = 0 and η_{I+1} = 1. Again we want to apply (5.3.5) with γ_{1} = γ +I and γ_{2} = γ −I, but this time in the reverse direction, meaning that we choose (Q_{1}, Q_{2}) = (P_{I}, P_{I+1}). The starting point of P_{I} now is (A^{(σ(I))}+D, −A^{(σ(I))}), and that of P_{I+1} is (−A^{(σ(I+1))}−1, A^{(σ(I+1))}+D+1). Therefore in Proposition 27 we have to choose A = −A^{(σ(I+1))}−D−1 and B = A^{(σ(I))}. The first inequality required for (5.3.5) is easily shown by again using (3.1.1a) and (4.2.1b), −A^{(σ(I+1))} − 1 < −A^{(1)} ≤ γ ≤ γ + I. Now, as before let S = (S_{1}, S_{2}). Since S is on P_{I+1} we must have S_{2} ≥ A^{(σ(I+1))} + D + 1. In the same manner as before we conclude S_{1} ≥ S_{2} + I. Combining these observations with (3.1.1a) and (4.2.1b) gives

γ + I ≤ A^{(1)} + D + I < A^{(σ(I+1))} + D + I + 1 ≤ S_{2} + I ≤ S_{1},

which implies that S is located to the right of x = γ +I. Besides, because of σ(I) ≥ I (cf. the paragraph after (4.2.5)), (3.1.1a), and (4.2.1b) we have

−A^{(σ(I))} − 1 ≤ −A^{(I)} − 1 ≤ −A^{(1)} − I ≤ γ − I ≤ γ ≤ A^{(1)} + D < A^{(σ(I+1))} + D + 1,

which in particular implies −A^{(σ(I))} − 1 ≤ γ − I ≤ A^{(σ(I+1))} + D + 1, which is the second inequality required for (5.3.5). Hence we may apply (5.3.5) to obtain

xmaj_{β;γ+I}(P′_{I}) + ymaj_{β;γ−I}(P′_{I+1})

= ymaj_{β;γ−I+1}(P_{I}) + xmaj_{β;γ+I+1}(P_{I+1}) + A^{(σ(I))} + A^{(σ(I+1))} + D + 1 + β,

and hence

ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) = ymaj_{β;γ}^{(η)}(𝔓) + A^{(σ(I))} + A^{(σ(I+1))} + D + 1 + β
= ymaj_{β;γ}^{(η)}(𝔓) − A^{(σ(I+1))} + η_{I+1}(2A^{(σ(I+1))} + D + β + 1)
+ A^{(σ(I))} − η_{I}(2A^{(σ(I))} + D + β + 1).

Summarizing, in all four cases we have

ymaj_{β;γ}^{(η^{(I,I+1)})}(Θ_{3}(𝔓)) = ymaj_{β;γ}^{(η)}(𝔓) + A^{(σ(I))} − η_{I}(2A^{(σ(I))} + D + β + 1)

− A^{(σ(I+1))} + η_{I+1}(2A^{(σ(I+1))} + D + β + 1). (4.2.7)

Now assume that Θ_{3}(𝔓) is obtained by (4.2.6a,b). Using (4.2.7) we modify (3.2.6) to get

ϑ_{3}(Θ_{3}(𝔓)) = (−1)^{∥η^{(I,I+1)}∥} sgn (σ(I, I + 1))
· q^{∑{r \atop i=1}(i(A^{(i)}−A^{((σ(I,I+1))(i))})+iη_{i}^{(I,I+1)}(2A^{((σ(I,I+1))(i))}+D+β+1))}q^{ymaj_{β;γ}^{(η^{(I,I+1)})} Θ_{3}(𝔓)}

= −(−1)^{∥η∥} sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})+I(A^{(σ(I))}−A^{(σ(I+1))})+(I+1)(A^{(σ(I+1))}−A^{(σ(I))})}

· q^{∑_{i=1}^{r} iη_{i}(2A^{(σ(i))}+D+β+1)−η_{I+1}(2A^{(σ(I+1))}+D+β+1)+η_{I}(2A^{(σ(I))}+D+β+1)}
· q^{ymaj_{β;γ}^{(η)} 𝔓+(A^{(σ(I))}−η_{I}(2A^{(σ(I))}+D+β+1))−(A^{(σ(I+1))}−η_{I+1}(2A^{(σ(I+1))}+D+β+1))}
= −ϑ_{3}(𝔓).  (4.2.8)

Next we assume that Θ_{3}(𝔓) is obtained using (4.2.6c,d). If η_{1} = 0, using (3.2.5) and (5.2.2), we get

xmaj_{β;γ+1}(Φ_{γ}^{(1)}(P_{1})) = ymaj_{β;γ}(P_{1}) − (A^{(σ(1))} + D) − A^{(σ(1))} − 1 − β.

Consequently, by the definitions of ϑ_{3} (cf. (4.2.3)) and of ymaj_{β;γ}^{(η)} (cf. (4.2.4)) we obtain

ϑ_{3}(Θ_{3}(𝔓))
= (−1)^{∥η^{(1)}∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}^{(1)}(2A^{(σ(i))}+D+β+1))}q^{ymaj_{β;γ}^{(η^{(1)})} Θ_{3}(𝔓)}

= −(−1)^{∥η∥} sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})}q^{∑_{i=1}^{r} iη_{i}(2A^{(σ(i))}+D+β+1)+(2A^{(σ(1))}+D+β+1)}

· q^{ymaj_{β;γ}^{(η)} 𝔓−(A^{(σ(1))}+D)−A^{(σ(1))}−1−β}
= −ϑ_{3}(𝔓).  (4.2.9)

Likewise, if η_{1} = 1 by (3.2.5) and (5.2.2) we have

ymaj_{β;γ}((Φ_{γ}^{(1)})^{(−1)}(P_{1})) = xmaj_{β;γ+1}(P_{1}) + A^{(σ(1))} + D + A^{(σ(1))} + 1 + β.

Consequently, by the definitions (4.2.3) and (4.2.4) we obtain

ϑ_{3}(Θ_{3}(𝔓))

= (−1)^{∥η^{(1)}∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}^{(1)}(2A^{(σ(i))}+D+β+1))}q^{ymaj_{β;γ}^{(η^{(1)})} Θ_{3}(𝔓)} = −(−1)^{∥η∥} sgn σ q^{∑_{i=1}^{r} i(A^{(i)}−A^{(σ(i))})}q^{∑_{i=1}^{r} iη_{i}(2A^{(σ(i))}+D+β+1)−(2A^{(σ(1))}+D+β+1)}

· q^{ymaj_{β;γ}^{(η)} 𝔓+A^{(σ(1))}+D+A^{(σ(1))}+1+β}
= −ϑ_{3}(𝔓).  (4.2.10)

Now, since by (4.2.8), (4.2.9), (4.2.10) we finally settled that Θ_{3} is sign-reversing with respect to ϑ_{3}, we may infer

∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} GF (Q(ℜ^{η}A_{σ} → E)^{−}; ϑ_{3}) = 0 .  (4.2.11)

All that remains, is to analogize the computation (3.2.9). Indeed, by consecutive use of (4.2.5), (4.2.11), (2.2.16), (5.1.11) and (5.1.12),

GF (Q(A → E)^{+}; ϑ_{3}) = GF (Q(A → E); ϑ_{3}) − GF (Q(A → E)^{−}; ϑ_{3})

= GF (Q(A → E); ϑ_{3}) +_{∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} ; (σ,η)≠(id,0)}GF (Q(ℜ^{η}A_{σ} → E)^{−}; ϑ_{3})

= ∑_{σ∈𝔖_{r}, η∈{0,1}^{r}} GF (Q(ℜ^{η}A_{σ} → E); ϑ_{3})

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})}(GF (P (A_{σ(i)} → E_{i}); ymaj_{β;γ−i+1})

− q^{i(2A^{(σ(i))}+D+β+1)}GF (P (ℜA_{σ(i)} → E_{i}); xmaj_{β;γ+i}))

= ∑_{σ∈𝔖_{r}} sgn σ ∏_{i=1}^{r} q^{i(A^{(i)}−A^{(σ(i))})}

× ( ∑_{j≥0} q^{j(j+β+γ−i+1+A^{(σ(i))})} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(i)} + E_{2}^{(i)} − D \\ E_{1}^{(i)} − A^{(σ(i))} − D − j \end{matrix} ]

− q^{i(2A^{(σ(i))}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+i+A^{(σ(i))}+1)}

[ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(i)} + E_{2}^{(i)} − D \\ E_{2}^{(i)} − A^{(σ(i))} − D − j − 1 \end{matrix} ] )

= det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D − j \end{matrix} ]

− q^{s(2A^{(t)}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}+s+1)}

[ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D − j − 1 \end{matrix} ] )).

Note that because of the restrictions (4.2.1a), the requirements being necessary for being allowed to use (5.1.11) and (5.1.12) are always satisfied. The last line is identical with (4.2.2). □

If the paths lie above the line x = y, there is a result for the xmaj.

Theorem 17. Let A_{i} = (−A^{(i)}, A^{(i)} + D) and E_{i} = (E_{1}^{(i)}, E_{2}^{(i)}), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.2.10) holds. Let γ be an integer satisfying

max_{1≤i≤r}(D − E_{2}^{(i)} + i − 1) ≤ γ ≤ min_{1≤i≤r}(E_{1}^{(i)} + i − 1),

max_{1≤i≤r}(D − E_{1}^{(i)} − i + 2) ≤ γ ≤ min_{1≤i≤r}(E_{2}^{(i)} − i + 2),

and − A^{(1)} ≤ γ ≤ A^{(1)} + D. (4.2.12)

The generating function ∑ q^{xmaj_{β;γ}(𝔓)}, where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, and where xmaj_{β;γ} 𝔓 is given by (4.0.9) (see the Remark below), is equal to the expression

det_{1≤s,t≤r} (q^{s(A^{(s)}−A^{(t)})}( ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{2}^{(s)} − A^{(t)} − D − j \end{matrix} ]

− q^{(s−1)(2A^{(t)}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}+s−1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1}^{(s)} + E_{2}^{(s)} − D \\ E_{1}^{(s)} − A^{(t)} − D − j − 1 \end{matrix} ] )).

(4.2.13)

Remark. Because of (3.2.10a,b) the paths P_{1}, …, P_{r} are ordered “from bottom to top”. It is important to observe that since we made the definition (4.0.9) for xmaj_{β;γ} of a family of paths, by requiring (2.2.2), dependent on the order of the paths, here in fact we have

xmaj_{β;γ}(𝔓) = ∑_{i=1}^{r} xmaj_{β;γ−i+1}(P_{i}).  (4.2.14)

Sketch of Proof. We follow the line of the proof of Theorem 16. The main difference is that instead of Φ_{γ}^{(1)} we have to use Φ^{(2)}, which is defined in Proposition 26. Of course, everywhere the roles of ymaj and xmaj have to be exchanged. We only remark that the weight function to be used is

ϑ_{4}(𝔓) = (−1)^{∥η∥} sgn σ q^{∑{r \atop i=1}(i(A^{(i)}−A^{(σ(i))})+iη_{i}(2A^{(σ(i))}+D+β+1))}q^{xmaj_{β;γ}^{(η)} 𝔓} if 𝔓 ∈ Q(ℜ^{η}A_{σ} → E), (4.2.15)

where xmaj_{β;γ}^{(η)} is given by

xmaj_{β;γ}^{(η)}(𝔓) = ∑_{i=1}^{r} ((1 − η_{i}) xmaj_{β;γ−i+1}(P_{i}) + η_{i}(ymaj_{β;γ+i−2}(P_{i}))). (4.2.16)

What has to be determined, is the generating function GF (Q̄(A → E)^{+}, ϑ_{4}), where Q̄(A → E)^{+} is the set of all families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie above x = y, P_{i} : A_{i} → E_{i}. This is done quite analogously as in the proof of Theorem 16 and is therefore left to the reader. □

Next we consider cases in which (4.2.2) or (4.2.13) can be simplified. With the special final points of Theorem 7, we can get rid of the determinant in (4.2.2) thus obtaining the multiple sum (4.2.17) and for the more general case the multiple sum (4.2.19). If β = 1 and γ = (D −1)/2, with the help of Proposition 36 the sum (4.2.17) can even be evaluated to result in a nice product. Since γ and D are integers, the relation γ = (D − 1)/2 implies that in this case D has to be odd. There do not seem to be other special cases where (4.2.17) or (4.2.19) can be evaluated. In particular, for r = 1 none of the known basic hypergeometric summation theorems can be applied to (4.2.17) except for β = 1 and γ = (D − 1)/2. So it is very unlikely that (4.2.17) can be summed for r > 1 in other cases.

Theorem 18. Let A_{i} = (A^{(i)}+D, −A^{(i)}) and E_{i} = (E+i, E+2−i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.1.1a) and

2A^{(i)} + D ≥ 0, i = 1, 2, …, r,

hold. If γ is an integer satisfying

D − E − 1 ≤ γ ≤ E and − A^{(1)} ≤ γ ≤ A^{(1)} + D,

then the generating function ∑ q^{ymaj_{β;γ}(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

\frac{[β + 2E + 2i − D]!}{[E + r − A^{(i)} − k_{i} − D]! [β + E + r + 1 + A^{(i)} + k_{i}]!}

∏_{1≤i<j≤r}[A^{(j)} + k_{j} − A^{(i)} − k_{i}] ∏_{1≤i≤j≤r}[A^{(i)} + k_{i} + A^{(j)} + k_{j} + D + β + 1] . (4.2.17)

In particular, if D is odd, say D = 2D′ + 1, and if −A^{(1)} ≤ D′ ≤ E, the generating function ∑ q^{ymaj_{1;D′} (𝔓)} over the same set of nonintersecting families 𝔓 is given by

∏_{i=1}^{r} \frac{[2E + 2i − 2D′]!}{[E + r − A^{(i)} − 2D′ − 1]! [E + r + 1 + A^{(i)}]!} ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}]

× ∏_{1≤i<j≤r}[A^{(i)} + A^{(j)} + 2D′ + 2] ∏_{i=1}^{r} \frac{[A^{(i)} + D′ + 1]}{[E + i − D′]} . (4.2.18)

More generally, if D is again generic and if the final points of the lattice paths are E_{i} = (E_{1} + i, E_{2} − i), i = 1, 2, …, r, with E_{1} ≥ E_{2} − 2, then the generating function ∑ q^{ymaj_{β;γ}(𝔓)} for all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie below the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{η∈\{0,1\}^{r}}(−1)^{∥η∥} ∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} (q^{iη_{i}(2A^{(i)}+2k_{i}+D+β+1)+k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

\frac{[β + E_{1} + E_{2} + i − D − 1]!}{[β + E_{2} + A^{(i)} + k_{i} − η_{i}(2A^{(i)} + 2k_{i} + D + β + 1) − 1]!})

\begin{matrix}∏_{1≤i<j≤r} & [A^{(j)} + k_{j} − η_{j}(2A^{(j)} + 2k_{j} + D + β + 1) − A^{(i)} − k_{i} + η_{i}(2A^{(i)} + 2k_{i} + D + β + 1)] \\ ∏_{i=1}^{r} & [E_{1} + r − D − A^{(i)} − k_{i} + η_{i}(2A^{(i)} + 2k_{i} + D + β + 1)]! \end{matrix}

(4.2.19)

Proof. In order to confirm (4.2.17), in Theorem 16 set E_{1}^{(i)} = E + i and E_{2}^{(i)} = E + 2 − i to obtain for the desired generating function the expression

det (q^{s(A^{(s)}−A^{(t)})}( ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + 2E + 2 − D \\ E + s − A^{(t)} − D − j \end{matrix} ]

− q^{s(2A^{(t)}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}+s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + 2E + 2 − D \\ E − s − A^{(t)} − D − j + 1 \end{matrix} ] )).

As before in the proof of Theorem 15, we denote the summation indices in the t-th column by k_{t} and then use the linearity in the t-th column to obtain that the last expression equals

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

det (q^{s(A^{(s)}+k_{s}−A^{(t)}−k_{t})}( [ \begin{matrix}β + 2E + 2 − D \\ E + s − A^{(t)} − k_{t} − D \end{matrix} ]

− q^{s(2A^{(t)}+2k_{t}+D+β+1)} [ \begin{matrix}β + 2E + 2 − D \\ E − s − A^{(t)} − k_{t} − D + 1 \end{matrix} ] )).

The determinant in this last expression has already been evaluated. This is seen by replacing E by E + β, D by D + β, A^{(t)} by A^{(t)} + k_{t} in the proof of Theorem 7. The result is the expression (3.2.14), with these replacements. This immediately settles (4.2.17).

In order to establish (4.2.18), in (4.2.17) set β = 1, D = 2D′ + 1, γ = D′. Use hypergeometric notation to obtain for the desired generating function the expression

∏_{i=1}^{r} \frac{[2E + 2i − 2D′]!}{[E + r − A^{(i)} − 2D′ − 1]! [E + r + 2 + A^{(i)}]!}

× ∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(1+E−D′+r−i)} \frac{(q^{A^{(i)}+2D′+1−E−r})_{k_{i}}}{(q^{E+r+3+A^{(i)}})_{k_{i}}}

∏_{1≤i<j≤r}(1 − q^{A^{(j)}+k_{j}−A^{(i)}−k_{i}}) ∏_{1≤i≤j≤r}(1 − q^{A^{(i)}+k_{i}+A^{(j)}+k_{j}+2D′+3}) . (4.2.20)

By setting m_{i} = q^{A^{(i)}+D′+3/2}, i = 1, 2, …, r, and A = q^{D′−E−r−1/2}, this can be written in the form

∏_{i=1}^{r} \frac{[2E + 2i − 2D′]!}{[E + r − A^{(i)} − 2D′ − 1]! [E + r + 2 + A^{(i)}]!}

× ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ∏_{1≤i≤j≤r}[A^{(i)} + A^{(j)} + 2D′ + 3]

× ∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} (\frac{\sqrt{q}}{q^{i}A})^{k_{i}} ∏_{i=1}^{r} \frac{(m_{i}A)_{k_{i}}}{(qm_{i}/A)_{k_{i}}}

× ∏_{1≤i<j≤r} \frac{1 − \frac{m_{j}}{m_{i}} q^{k_{j}−k_{i}}}{1 − \frac{m_{j}}{m_{i}}} ∏_{1≤i≤j≤r} \frac{1 − m_{i}m_{j}q^{k_{i}+k_{j}}}{1 − m_{i}m_{j}} .

The r-fold sum is evaluated using (5.6.1). Thus we obtain

∏_{i=1}^{r} \frac{[2E + 2i − 2D′]!}{[E + r − A^{(i)} − 2D′ − 1]! [E + r + 2 + A^{(i)}]!}

× ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ∏_{1≤i≤j≤r}[A^{(i)} + A^{(j)} + 2D′ + 3]

× ∏_{1≤i<j≤r} \frac{[A^{(i)} + A^{(j)} + 2D′ + 2]}{[A^{(i)} + A^{(j)} + 2D′ + 3]} ∏_{i=1}^{r} (\frac{[A^{(i)} + D′ + 1]}{[2A^{(i)} + 2D′ + 3]} \frac{[E + A^{(i)} + r + 2]}{[E + r − D′ − i + 1]}) ,

which simplifies to (4.2.18).

The proof of (4.2.19) proceeds in analogy with the proof of (3.2.15) and is therefore omitted. □

The analogue of Theorem 18 which corresponds to Theorem 17 is the following result. A closed form is obtained for E_{1}^{(i)} = E + 2 − i, E_{2}^{(i)} = E + i, β = 1 and γ = (D + 1)/2, which forces D to be odd here, too. Again, obtaining a closed form for other choices of the parameters seems to be very unlikely.

Theorem 19. Let A_{i} = (−A^{(i)}, A^{(i)}+D) and E_{i} = (E+2−i, E+i), i = 1, 2, …, r, be lattice points in the integer lattice ℤ^{2} such that (3.2.10a) and

2A^{(i)} + D ≥ 0, i = 1, 2, …, r,

hold. If γ is an integer satisfying

D − E ≤ γ ≤ E + 1 and − A^{(1)} ≤ γ ≤ A^{(1)} + D,

then the generating function ∑ q^{xmaj_{β;γ}(𝔓)} where the sum is over all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r is equal to the expression

q^{\binom{r+1}{2}+r(E−D)−∑_{i=1}^{r} A^{(i)}} ∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i)}

[ \begin{matrix}−β \\ k_{i} \end{matrix} ] \frac{[β + 2E + 2i − D]!}{[E + r − A^{(i)} − k_{i} − D]! [β + E + r + 1 + A^{(i)} + k_{i}]!}

∏_{1≤i<j≤r}[A^{(j)} + k_{j} − A^{(i)} − k_{i}] ∏_{1≤i≤j≤r}[A^{(i)} + k_{i} + A^{(j)} + k_{j} + D + β + 1] . (4.2.21)

In particular, if D is odd, say D = 2D′ + 1, and if −A^{(1)} ≤ D′ ≤ E, the generating function ∑ q^{xmaj_{1;D′+1}(𝔓)} over the same set of nonintersecting families 𝔓 is given by

q^{\binom{r+1}{2}+r(E−D)−∑_{i=1}^{r} A^{(i)}} ∏_{i=1}^{r} \frac{[2E + 2i − 2D′]!}{[E + r − A^{(i)} − 2D′ − 1]! [E + r + 1 + A^{(i)}]!}

× ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ∏_{1≤i<j≤r}[A^{(i)} + A^{(j)} + 2D′ + 2] ∏_{i=1}^{r} \frac{[A^{(i)} + D′ + 1]}{[E + i − D′]} . (4.2.22)

More generally, if D is again generic and if the final points of the lattice paths are E_{i} = (E_{1} − i, E_{2} + i), i = 1, 2, …, r, with E_{1} ≤ E_{2} + 2, then the generating function ∑ q^{xmaj_{β;γ}(𝔓)} for all nonintersecting families 𝔓 = (P_{1}, …, P_{r}) of lattice paths which lie above the line x = y, P_{i} : A_{i} → E_{i}, i = 1, 2, …, r, is equal to the expression

∑_{η∈\{0,1\}^{r}}(−1)^{∥η∥} ∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} (q^{(i−1)η_{i}(2A^{(i)}+2k_{i}+D+β+1)+k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

\frac{[β + E_{1} + E_{2} + i − D − 1]!}{[β + E_{1} + A^{(i)} + k_{i} − η_{i}(2A^{(i)} + 2k_{i} + D + β + 1) − 1]!})

\begin{matrix}∏_{1≤i<j≤r} & [A^{(j)} + k_{j} − η_{j}(2A^{(j)} + 2k_{j} + D + β + 1) − A^{(i)} − k_{i} + η_{i}(2A^{(i)} + 2k_{i} + D + β + 1)] \\ ∏_{i=1}^{r} & [E_{2} + r − D − A^{(i)} − k_{i} + η_{i}(2A^{(i)} + 2k_{i} + D + β + 1)]! \end{matrix}

(4.2.23)

Proof. In order to confirm (4.2.21), in Theorem 17 set E_{1}^{(i)} = E + 2 − i and E_{2}^{(i)} = E + i. Thus we obtain for the desired generating function the expression

det (q^{s(A^{(s)}−A^{(t)})}( ∑_{j≥0} q^{j(j+β+γ+A^{(t)}−s+1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + 2E + 2 − D \\ E + s − A^{(t)} − D − j \end{matrix} ]

−q^{(s−1)(2A^{(t)}+D+β+1)} ∑_{j≥0} q^{j(j+β+γ+A^{(t)}+s−1)} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + 2E + 2 − D \\ E − s − A^{(t)} − D − j + 1 \end{matrix} ] )).

Again the summation indices in the t-th column are denoted by k_{t}. Next we use the linearity in the t-th column to turn the last expression into

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+γ+A^{(i)}−i+1)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

det (q^{s(A^{(s)}+k_{s}−A^{(t)}−k_{t})}( [ \begin{matrix}β + 2E + 2 − D \\ E + s − A^{(t)} − k_{t} − D \end{matrix} ]

− q^{(s−1)(2A^{(t)}+2k_{t}+D+β+1)} [ \begin{matrix}β + 2E + 2 − D \\ E − s − A^{(t)} − k_{t} − D + 1 \end{matrix} ] )).

The determinant in this last expression has already been evaluated. This is seen by replacing E by E + β, D by D + β, A^{(t)} by A^{(t)} + k_{t} in the proof of Theorem 8. The result is the expression (3.2.19), with these replacements. This furnishes (4.2.21) after having performed a little cancellation.

In order to establish (4.2.22), in (4.2.21) set β = 1, D = 2D′ + 1, γ = D′ + 1. Then it is seen that this just yields the expression (4.2.20) times some power of q. Hence the result is (4.2.18) times this power of q.

The proof of (4.2.23) proceeds in analogy with the proof of (3.2.20) and is therefore omitted. □

4.3. Tableaux generating functions. As a warm-up we provide the analogue of Theorem 9 for the case that the parts are not restricted to be odd. Unfortunately it is not as nice because it is not possible to evaluate the multiple sum which is obtained from (4.1.5).

Theorem 20. The generating function ∑ q^{n(τ_{1})+n(τ_{2})} for pairs (τ_{1}, τ_{2}) of tableaux of identical shape consisting of at most r columns and with parts between 1 and n is given by

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r}(−1)^{k_{i}}q^{\binom{k_{i}+1}{2}} \frac{[2n + i]!}{[n + i + k_{i}]! [n + r − i − k_{i}]!} ∏_{1≤i<j≤r}[j+k_{j} −i−k_{i}] . (4.3.1)

In terms of Schur functions: The sum ∑_{λ,λ_{1}≤r} s_{λ}^{2}(q^{n}, q^{n−1}, …, q) equals the expression (4.3.1).

Proof. By Proposition 28 we know that the pairs of tableaux in question are in one-to-one correspondence with families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths, P_{i} : (i − 1, −i + 1) → (n + i − 1, n − i + 1), where in addition the relation (5.4.1) is satisfied. Therefore Theorem 15 can be applied with A^{(i)} = i − 1, E_{1} = n − 1, E_{2} = n + 1, β = 1, γ = 0. Taking these choices in (4.1.5), leads to (4.3.1). □

Finally we are in the position to attack the Choi/Gouyou-Beauchamps problem.

Theorem 21. The generating function for tableaux with p odd rows, with at most c columns, and with parts between 1 and n is given by

q^{\binom{p+1}{2}} \frac{[2r]}{[2r + p]} [ \begin{matrix}n \\ p \end{matrix} ] \begin{matrix}[ & \begin{matrix}n + 2r \\ n \end{matrix} & ] \\ [ & \begin{matrix}n + 2r + p \\ n \end{matrix} & ] \end{matrix} ∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]}  if c = 2r  (4.3.2a)

and

q^{\binom{p+1}{2}} [ \begin{matrix}n \\ p \end{matrix} ] ∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]}  if c = 2r + 1.  (4.3.2b)

Equivalently, the sum ∑ s_{λ}(q^{n}, q^{n−1}, …, q), where the sum is over all partitions λ with p odd parts, where each part does not exceed c, equals the expressions (4.3.2).

Proof. First we consider the case c = 2r . By Proposition 32 we know that by means of the map Δ_{5} the tableaux under consideration are in one-to-one correspondence with pairs (𝔓, S), where 𝔓 = (P_{1}, …, P_{r}) is a family of nonintersecting lattice paths which lie below the line x = y, P_{i} : (i−1, −i+1) → (n+i, n+2−i), i = 1, …, r, with the property that if P_{r} starts with h horizontal steps followed by a vertical step then S is a p-subset of {1, 2, …, h − 1}. Moreover, Δ_{5} satisfies the relation (5.4.13). We claim that these pairs (𝔓, S) bijectively correspond to families 𝔓̂ = (P̂_{1}, …, P̂_{r}) of nonintersecting lattice paths which lie below x = y, P̂_{i} : (i, −i+1) → (n+i, n+2−i), i = 1, 2, …, r − 1, P̂_{r} : (r + p, −r − p + 1) → (n + r, n + 2 − r), such that

maj 𝔓 + ∥S∥ = ymaj_{1;0}(𝔓̂) + \binom{p + 1}{2}.  (4.3.3)

Suppose that this has been done. The generating function ∑_{𝔓̂} q^{ymaj_{1;0}(𝔓̂)} for this last set of families 𝔓̂ of nonintersecting lattice paths is computed by setting A^{(i)} = i − 1, i = 1, …, r − 1, A^{(r)} = r + p − 1, E = n, D′ = 0, in (4.2.18). The resulting expression because of (4.3.3) and (5.4.13) has to be multiplied by \binom{p+1}{2}. We thus obtain

q^{\binom{p+1}{2}} ∏_{i=1}^{r} \frac{[2n + 2i]!}{[n + r − i]! [n + r + i]!} · \frac{[n]! [n + 2r]!}{[n − p]! [n + 2r + p]!}

× ∏_{1≤i<j≤r}[j − i] · ∏_{i=1}^{r−1} \frac{[r + p − i]}{[r − i]}

× ∏_{1≤i<j≤r}[i + j] · ∏_{i=1}^{r−1} \frac{[r + p + i]}{[r + i]} · ∏_{i=1}^{r} \frac{[i]}{[n + i]} · \frac{[r + p]}{[r]} ,

which after some manipulation simplifies to (4.3.2a).

The construction of the claimed bijection follows the idea of converting the set S into a path which subsequently is attached to the r-th path P_{r} that was already applied in the proof of Theorem 11. But first we slightly modify 𝔓. A moment’s thought confirms that due to the property of lying below x = y and being nonintersecting, all the paths have to start with a horizontal step. These first steps are deleted from the paths, thus obtaining the new family 𝔓̄ = (P̄_{1}, …, P̄_{r}) of nonintersecting lattice paths which lie below x = y, but P̄_{i} : (i, −i + 1) → (n + i, n + 2 − i), i = 1, …, r. Moreover, if P̄_{r} first turns after h − 1 steps then S is a p-subset of {1, 2, …, h − 1}. Recall that we say that a path P first turns after H steps if P starts with H horizontal steps followed by a vertical step. The crucial observation is that remembering the definitions (4.0.1) and (4.0.8) of ymaj for paths and families of paths, we have

maj 𝔓 = ymaj_{1;0}(𝔓̄).  (4.3.4)

Now we consider the set S. Using (5.1.2) we get that the generating function ∑ q^{∥S∥} for the p-subsets of {1, 2, …, h − 1} is given by

∑_{S⊆\{1,2,...,h−1\}, |S|=p} q^{∥S∥} = q^{\binom{p+1}{2}} [ \begin{matrix}h − 1 \\ p \end{matrix} ] .  (4.3.5)

On the other hand, by (5.1.13) the major generating function for paths from (r + p, −r − p + 1) to (r + h − 1, −r + 1) is given by

∑_{P:(r+p,−r−p+1)→(r+h−1,−r+1)} q^{maj P} = [ \begin{matrix}h − 1 \\ p \end{matrix} ] .  (4.3.6)

Comparing (4.3.5) and (4.3.6) we infer that there is a bijection δ_{2} between p-subsets S of {1, 2, …, h − 1} and paths P from (r + p, −r − p + 1) to (r + h − 1, −r + 1), such that the relation

∥S∥ = maj δ_{2}(S) + \binom{p + 1}{2}  (4.3.7)

holds. (An explicit bijection could be extracted from [20].) Since P̄_{r} starts at (r, −r + 1) and first turns after h − 1 steps, the point (r + h − 1, −r + 1) lies on P̄_{r}. Now we form a new path P̂_{r} by joining δ_{2}(S) and the portion of P̄_{r} from (r + h − 1, −r + 1) to its final point (n + r, n + 2 − r). Since in P̄_{r} the step after (r + h − 1, −r + 1) is a vertical one, we always are able to recover δ_{2}(S), and hence S, given P̂_{r}. Now we define 𝔓̂ := (P̄_{1}, …, P̄_{r−1}, P̂_{r}). Summarizing, starting with the pair (𝔓, S) we obtained the family 𝔓̂ = (P̂_{1}, …, P̂_{r−1}, P̂_{r}), P̂_{i} : (i, −i + 1) → (n + i, n + 2 − i), i = 1, 2, …, r − 1, P̂_{r} : (r + p, −r − p + 1) → (n + r, n + 2 − r). Luckily, this attaching of δ_{2}(S) to P̄_{r} in order to form P̂_{r}, perfectly fits into ymaj-counting. Namely we have

ymaj_{1;−r+1}(P̄_{r}) + maj δ_{2}(S) = ymaj_{1;−r+1} P̂_{r} .  (4.3.8)

Combining (4.3.4), (4.3.7), (4.3.8) with the definition (4.0.8) of the ymaj_{1;0} of a family of paths, we arrive at (4.3.3). Hence we have found the asserted bijection.

Finally let c = 2r + 1. We claim that the tableaux in question are in bijection with pairs (τ_{e}, S), where τ_{e} is a tableau with even rows, with at most 2r columns, and with parts between 1 and n, and where S is a p-subset of {1, 2, …, n}. This is established by Knuth’s procedures ROW-DELETE and ROW-INSERT, which are explained in subsection 5.4, just in the same manner as described in the c = 2r + 1-part of the proof of Theorem 11. Moreover this bijection has the property n(τ) = n(τ_{e}) + ∥S∥. Use of Theorem 10 and of identity (4.3.5) with h replaced by n + 1 leads to (4.3.2b). □

Choi and Gouyou-Beauchamps [7] obtained the special case q = 1 of Theorem 21. The p = 0 special case has already been the contents of Theorem 10. The c = 2r + 1 case of Theorem 21 was first found by Désarménien [12, Théoréme 2, first identity].

By summing with respect to p we obtain a new proof for the Bender–Knuth conjecture about the generating function for column-strict plane partitions with at most c columns and with parts between 1 and n. This conjecture was proved by Andrews [3], Gordon [18], Macdonald [33, Ex. 19, p. 53], and Proctor [36, Proposition 7.2]. We give an equivalent formulation in terms of tableaux.

Theorem 22. The generating function for tableaux with at most c columns and with parts between 1 and n equals

∏_{1≤i≤j≤n} \frac{[c + i + j − 1]}{[i + j − 1]} .  (4.3.9)

Equivalently, in terms of Schur functions

∑_{λ,λ_{1}≤c} s_{λ}(q^{n}, q^{n−1}, …, q) = ∏_{1≤i≤j≤n} \frac{[c + i + j − 1]}{[i + j − 1]} .

Proof. We have to sum the expressions (4.3.2) with respect to p. If the sum over all p of the expressions (4.3.2a) is converted into basic hypergeometric notation (cf. subsection 2.1), it reads

∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]}_{2}ϕ_{1}[ \begin{matrix}q^{2r}, q^{−n} \\ q^{2r+n+1} \end{matrix} ; q, −q^{n+1}] .

The _{2}ϕ_{1} can be computed using the q-analogue of Kummer’s summation (see [14, Appendix (II.9)]),

2ϕ_{1}[ \begin{matrix}a, b \\ aq/b \end{matrix}; q, −q/b] = \frac{(−q; q)_{∞} (aq; q^{2})_{∞} (aq^{2}/b^{2}; q^{2})_{∞}}{(−q/b; q)_{∞} (aq/b; q)_{∞}} .

This yields

∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]} · \frac{(−q; q)_{∞} (q^{2r+1}; q^{2})_{∞} (q^{2r+2n+2}; q^{2})_{∞}}{(−q^{n+1}; q)_{∞} (q^{2r+n+1}; q)_{∞}} ,

which after some manipulation results into (4.3.9) with c replaced by 2r.

On the other hand, the sum over all p of the expressions (4.3.2b) is obtained using the q-binomial theorem (3.3.10) with N = n and z = q. This gives

∏_{i=1}^{n}(1 + q^{i}) ∏_{1≤i≤j≤n} \frac{[2r + i + j]}{[i + j]} ,

which readily simplifies to (4.3.9) with c replaced by 2r + 1. □

Remarks. (1) Similar to the remark after Theorem 13, it should be observed that via the deleting-inserting procedure sketched at the end of the proof of Theorem 21, the theorem of Désarménien, Proctor, and Stembridge about tableaux with even rows (Theorem 10) is equivalent with the Bender–Knuth conjecture for c = 2r + 1. This justifies saying that in fact this result about tableaux with even rows is a special case of the Bender–Knuth conjecture. This equivalence was also first discovered by Désarménien [12], however again arguing by the Littlewood–Richardson rule.

(2) It was remarked in the Introduction that the tableaux and plane partition theorems of sections 3 and 4 could also be derived starting with Goulden’s paper [19]. Using a linear operator for symmetric functions and some tricky manipulations, Goulden derives determinant formulas for specific summations of Schur functions [19, Theorems 2.4 and 2.6] and for a summation of products of Schur functions [19, section 3]. (The latter result was previously obtained by Gessel [15, Theorem 16].) Indeed, the determinants that we used to prove our Theorems 9–11, 20, 21 could be also obtained by first applying the homomorphism for symmetric functions that interchanges the roles of the complete homogenous symmetric functions and the elementary symmetric functions (cf. [33, p. 14]) to both sides of Goulden’s theorems (to be precise: for Theorems 9 and 20 start with the summation theorem of section 3 in [19], for Theorems 10, 11, 21 start with Theorems 2.4 and 2.6 of [19]), then setting x_{i} = q^{i}, and finally using the Heine transformation (5.1.6) to transform each of the resulting q-binomial sums in the entries of the determinants. The application of the Heine transformation already parallels the computations in this paper where this Heine transformation is used to obtain Lemma 23, from which the entries of our determinants actually are extracted. The remaining steps to arrive at Theorems 9–11, 20, 21 would be almost the same as in this paper. However, as remarked in the introduction, neither do our results about major and strange major counting of nonintersecting lattice paths follow from Goulden’s results nor the other way round. A forthcoming paper [29] will be devoted to combinatorially proving Goulden’s results by a modification of the ideas of the present paper. This future paper will also contain the derivations which were indicated above, and generalizations and further applications of the results of both Goulden’s and the present paper.

(3) The C_{r} summation (5.6.1) is crucial in the proof of Theorems 18 and 19. In fact, A_{r}- and C_{r}-type summations pop up in this section at several places (e.g. in (4.1.5), (4.1.6), (4.2.17), (4.2.21)). For an exact definition of A_{r} and C_{r} basic hypergeometric sums the reader should consult e.g. [31, 34]. But to have a rough understanding it is enough to know that A_{r} sums can be identified if they somehow look like

∑_{k_{1},...,k_{r}≥0} … ∏_{1≤i<j≤r} \frac{1 − \frac{m_{j}}{m_{i}} q^{k_{j}−k_{i}}}{1 − \frac{m_{j}}{m_{i}}} ,  (4.3.10)

while C_{r} sums are identified if they somehow look like

∑_{k_{1},...,k_{r}≥0} … ∏_{1≤i<j≤r} \frac{1 − \frac{m_{j}}{m_{i}} q^{k_{j}−k_{i}}}{1 − \frac{m_{j}}{m_{i}}} ∏_{1≤i≤j≤r} \frac{1 − m_{i}m_{j}q^{k_{i}+k_{j}}}{1 − m_{i}m_{j}} .  (4.3.11)

The reader should in the obvious way associate {m_{j}/m_{i} : 1 ≤ i < j ≤ r} with the positive roots of the root system A_{r−1}, and {m_{j}/m_{i} : 1 ≤ i < j ≤ r} ∪ {m_{j}m_{i} : 1 ≤ i ≤ j ≤ r} with the positive roots of the root system C_{r} (cf. [5, 23] for information about root systems). Besides, the expressions … in (4.3.10) and (4.3.11) consist of q-factorials where the parameters occur “in multiples of the m_{i}”. To understand this vague statement consider (5.6.1). The parameters of the numerator q-factorials, m_{i}A, as well as the parameters of the denominator q-factorials, qm_{i}/A, are obviously multiples of the m_{i}’s. However, when transforming (4.1.5), (4.1.6), (4.2.17), or (4.2.21) into hypergeometric notation the term (q^{β})_{k_{i}} appears in the summand. Obviously this violates this “multiple” rule unless β = 1, where these terms cancel with the denominator terms (q)_{k_{i}}. (Remember that indeed before applying the C_{r} summation (5.6.1) we had to set β = 1!) So for generic β the sums (4.1.5), (4.1.6), (4.2.17), and (4.2.21) do not fit into the classical A_{r} and C_{r} patterns. I chose the name A_{r}-type, respectively C_{r}-type sum for them, since terms of the type (4.3.10), respectively (4.3.11) appear in the summands and at least some of the parameters obey the “multiple” rule (while others do not; see (4.3.12) where the parameters BX_{i} and CX_{i} obey the “multiple” rule, while the A’s in (A)_{k_{i}} do not; observe that in (4.3.12) the X_{i}’s are in the role of the above m_{i}’s).

A very striking phenomenon is the following, for which a convincing theoretical explanation would be very welcome: In Theorem 15 we obtain the A_{r}-type sums (4.1.5) and (4.1.6). These were obtained from the determinants (4.1.2) and (4.1.4) in Theorem 14. A prominent constituent in the proof of Theorem 14 has been the symmetric group 𝔖_{r} (which acted on the starting points of the families of lattice paths), or, in other terms, the Weyl group of the root system A_{r}! Quite analogously, in Theorems 18 and 19 we obtained the C_{r}-type sums (4.2.17) and (4.2.21). These resulted from the determinants (4.2.2) and (4.2.13) in Theorems 16, respectively 17. The group that acted on the starting points of the families of lattice paths in the proofs of Theorems 16 and 17 has actually been the hyperoctahedral group of order r, or, in other terms, the Weyl group for C_{r}!

Even more interestingly, a closer look at Theorem 15 reveals that we implicitly proved a new A_{r}-type q-Gauß summation formula. It reads

∑_{k_{1},...,k_{r}≥0} (∏_{i=1}^{r} q^{k_{i}(1−i)} (\frac{C}{AB} )^{k_{i}} \frac{(A)_{k_{i}} (BX_{i})_{k_{i}}}{(q)_{k_{i}} (CX_{i})_{k_{i}}} ) ∏_{1≤i<j≤r} \frac{1 − \frac{X_{j}}{X_{i}} q^{k_{j}−k_{i}}}{1 − \frac{X_{j}}{X_{i}}}

= ∏_{i=1}^{r} \frac{(\frac{C}{B} q^{i−r})_{∞} (\frac{C}{A} X_{i})_{∞}}{(CX_{i})_{∞} (\frac{C}{AB} q^{i−r})_{∞}} , (4.3.12)

provided that none of the denominators vanish, |q| < 1, and |C/AB| < |q^{r−1}|. Obviously for r = 1 identity (4.3.12) agrees with the classical q-Gauß summation formula [14, Appendix (II.8)]. The derivation of (4.3.12) proceeds as follows. In Theorem 15 choose γ = E_{2} − 1. Since the y-coordinate of E_{i}, the final point of the i-th path of the families of nonintersecting lattice paths under consideration, equals E_{2} − i, for each family 𝔓 = (P_{1}, …, P_{r}) of lattice paths, P_{i} : A_{i} → E_{i}, i = 1, …, r, by (4.0.1), (4.0.8), and (3.0.1) we have

ymaj_{β;E_{2}−1}(𝔓) = ∑_{i=1}^{r} ymaj_{β;E_{2}−i} P_{i} = ∑_{i=1}^{r} maj P_{i} = maj 𝔓.

Therefore the expression (4.1.5) for γ = E_{2} − 1 equals the major generating function for the same set of families of nonintersecting lattice paths. But this generating function is also computed in Theorem 3. Hence we infer

∑_{k_{1},...,k_{r}≥0} ( ∏_{i=1}^{r} q^{k_{i}(k_{i}+β+E_{2}+A^{(i)}−i)} [ \begin{matrix}−β \\ k_{i} \end{matrix} ]

× \frac{[β + E_{1} + E_{2} + i − 1]!}{[β + E_{2} + A^{(i)} + k_{i} − 1]! [E_{1} + r − A^{(i)} − k_{i}]!} ∏_{1≤i<j≤r}[A^{(j)} + k_{j} − A^{(i)} − k_{i}])

= ∏_{i=1}^{r} \frac{[E_{1} + E_{2} + i − 1]!}{[E_{2} + A^{(i)} − 1]! [E_{1} + r − A^{(i)}]!} ∏_{1≤i<j≤r}[A^{(j)} − A^{(i)}] ,

or in hypergeometric notation,

∑_{k_{1},...,k_{r}≥0} (∏_{i=1}^{r} q^{k_{i}(E_{1}+E_{2}+r−i+1)} \frac{(q^{β})_{k_{i}} (q^{−E_{1}−r+A^{(i)}})_{k_{i}}}{(q)_{k_{i}} (q^{(β+E_{2}+A^{(i)})})_{k_{i}}} )

× ∏_{1≤i<j≤r} \frac{1 − q^{A^{(j)}+k_{j}−A^{(i)}−k_{i}}}{1 − q^{A^{(j)}−A^{(i)}}} = ∏_{i=1}^{r} \frac{(q^{β+E_{1}+E_{2}+i})_{∞} (q^{E_{2}+A^{(i)}})_{∞}}{(q^{β+E_{2}+A^{(i)}})_{∞} (q^{E_{1}+E_{2}+i})_{∞}} . (4.3.13)

Performing the relabelling C := q^{β+E_{2}}, B := q^{−E_{1}−r}, A := q^{β}, X_{i} := q^{A^{(i)}}, because of A^{(r)} ≤ E_{1} +r (otherwise there would not exist any family of paths and so Theorem 15 would not make any sense), and hence (using (3.1.1a)) also A^{(i)} ≤ E_{1} + r for all i = 1, …, r, we see that (4.3.13) is a terminating special case of (4.3.12). But then using a standard argument, (4.3.12) holds for any terminating special case, in particular for A = q^{−n}, where n is a nonnegative integer. Next observe that for |q| < 1 both sides of (4.3.12) are analytic functions in z := 1/A on the circle {z : |z| < |Bq^{r−1}/C|}. Besides they agree for z = q^{n}, n a nonnegative integer, and, because of |q| < 1, the sequence (q^{n}) converges. By a well-known theorem in complex analysis we thus infer that both sides of (4.3.12) agree as analytical functions in z, which settles our claim. Using the classical q-Gauß _{2}ϕ_{1} summation, Heine’s transformation, simple properties of Schur functions, and induction on r, Gustafson [22] found a direct proof of (4.3.12). Later he even found A_{r}-type extensions of Heine’s _{2}ϕ_{1}-transformation formulas (cf. [14, Appendix (III.1–3)]) which contain (4.3.12) as special case. The second of these transformations, (4.1.7), we already used.

But in general, as was remarked before the statement of Theorem 15, the sum in (4.1.5) very probably cannot be evaluated to result in a closed form. This observation stems from the fact that for r = 1 the sum in (4.1.5) in hypergeometric notation is

[ \begin{matrix}β + E_{1} + E_{2} \\ E_{1} + 1 − A^{(1)} \end{matrix} ] _{2}ϕ_{1}[ \begin{matrix}q^{β}, q^{−E_{1}−1+A^{(1)}} \\ q^{β+E_{2}+A^{(1)}} \end{matrix} ; q, q^{γ+E_{1}+2}] ,  (4.3.14)

to which none of the known basic hypergeometric summations can be applied except for γ = E_{2}−1 and γ = −E_{1}−1, where the q-Vandermonde summations [14, Appendix (II.7), respectively (II.6)] yield closed forms. (The case γ = E_{2} − 1 has already been discussed and turned out to be combinatorially trivial. The case γ = −E_{1} − 1 is also combinatorially trivial since ymaj_{β;−E_{1}−1} also agrees with the major index of the families of paths under consideration. The summation that would result by comparing (4.1.5) and (3.1.9) in this case would also be a terminating special case of (4.3.12).)

(4) The formulas (3.2.14) and (3.2.15) with E_{1} = E and E_{2} = E + 2 must agree. Hence we obtain a curious identity which, after replacing q^{A^{(i)}+D/2+1/2} by X_{i} and q^{E−D/2+1/2} by E, can be rewritten in the form

∑_{ε∈\{−1,1\}^{r}} sgn ε ∏_{i=1}^{r} X_{i}^{(1−ε_{i})i}(qEX_{i}^{ε_{i}})_{r} ∏_{1≤i<j≤r} (1 − \frac{X_{j}^{ε_{j}}}{X_{i}^{ε_{i}}} )

= ∏_{i=1}^{r}(E^{2}q^{i+1})_{i−1} ∏_{1≤i<j≤r} (1 − \frac{X_{j}}{X_{i}} ) ∏_{1≤i≤j≤r}(1 − X_{i}X_{j}) . (4.3.15)

By sgn ε we mean the product ∏_{i=1}^{r} ε_{i}. This identity relates the alternating sum of 2^{r} expressions of A_{r}-type with a C_{r}-type expression.

V. Detailed Proofs and Auxiliary Results

5.1. The major index and strange major index generating functions for lattice paths without restrictions. In this section the strange major generating functions for unrestricted lattice paths with fixed starting and final point are computed. Although it is well-known, the result for the major generating function is derived as a special case of the strange major result.

Recall that in subsection 2.1 we gave a one-to-one correspondence beetween lattice paths from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}) and two-rowed arrays which are strictly increasing in the rows and of the following type

\begin{aligned}A_{1} ≤ & a_{1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & … & b_{k} & ≤ E_{2} \end{aligned} ,  (5.1.1)

where k is some nonnegative integer (cf. (2.1.1) and (2.1.2)). Since all the arrays which will appear in the sequel will be row-strict, we will assume that if we speak of arrays we always mean them to be strictly increasing in the rows. By definition the empty array is the only array for k = 0.

Let P_{K}[N_{1}, N_{2}] denote the set of K-element subsets of {N_{1}, N_{1} + 1, …, N_{2}}. For a set S of integers we write ∥S∥ for the sum of all the elements of S. Then it is well-known (this is seen e.g. equating coefficients in [1, (3.3.6)]) that

∑_{S∈P_{K}[0,N−1]} q^{∥S∥} = q^{\binom{K}{2}} [ \begin{matrix}N \\ K \end{matrix} ] .  (5.1.2)

This identity enables us to find expressions for the xmaj- and ymaj-generating functions for lattice paths from A to E by using the above correspondence.

Let A and E be as above, A ≤ E, and A_{2} ≤ γ ≤ E_{2}. Then using (4.0.6) and (5.1.2) we obtain for ymaj_{β;γ}

∑_{P:A→E}q^{ymaj_{β;γ}(P)}

= ∑_{k≥0}_{∑_{a∈P_{k}[A_{1},E_{1}−1]} ; b∈P_{k}[A_{2}+1,E_{2}]}q^{∥a∥+∥b∥+β·|{t:b_{t}>γ}|−k(A_{1}+A_{2})}  (5.1.3)

= ∑_{k≥0} ( ∑_{a∈P_{k}[A_{1},E_{1}−1]} q^{∥a∥−kA_{1}}) ∑_{i+j=k} ( ∑_{b′∈P_{i}[A_{2}+1,γ]} q^{∥b′∥−iA_{2}})

× ( ∑_{b′′∈P_{j}[γ+1,E_{2}]} q^{∥b′′∥+jβ−jA_{2}})

= ∑_{k≥0} q^{\binom{k}{2}} [ \begin{matrix}E_{1} − A_{1} \\ k \end{matrix} ] ∑_{i+j=k} q^{\binom{i+1}{2}} [ \begin{matrix}γ − A_{2} \\ i \end{matrix} ] q^{j(β+γ−A_{2})+\binom{j+1}{2}} [ \begin{matrix}E_{2} − γ \\ j \end{matrix} ]

by dividing the set b into two arrays b′ and b′′ where b′ collects all elements of b being smaller or equal γ and where b′′ collects those being greater than γ. Next we replace i by k − j and interchange the order of summations. This transforms the last sum into

∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}E_{2} − γ \\ j \end{matrix} ] ∑_{k≥0} q^{k(k−j)} [ \begin{matrix}E_{1} − A_{1} \\ k \end{matrix} ] [ \begin{matrix}γ − A_{2} \\ k − j \end{matrix} ] .

The inner sum can be evaluated by the q-Vandermonde summation (see e.g. [1, (3.3.10)]), which we restate in the form

∑_{K≥0} q^{K(K−H+M)} [ \begin{matrix}N \\ K \end{matrix} ] [ \begin{matrix}M \\ H − K \end{matrix} ] = [ \begin{matrix}N + M \\ H \end{matrix} ] .  (5.1.4)

Utilizing [ \begin{matrix}γ−A_{2} \\ k−j \end{matrix} ] = [ \begin{matrix}γ−A_{2} \\ γ−A_{2}+j−k \end{matrix} ] and (5.1.4) with N = E_{1} − A_{1}, M = γ − A_{2},

H = γ − A_{2} + j, we get

∑_{P:A→E}q^{ymaj_{β;γ}(P)}

= ∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}E_{2} − γ \\ j \end{matrix} ] [ \begin{matrix}E_{1} + γ − A_{1} − A_{2} \\ j + γ − A_{2} \end{matrix} ]

= ∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}E_{2} − γ \\ j \end{matrix} ] [ \begin{matrix}E_{1} + γ − A_{1} − A_{2} \\ E_{1} − A_{1} − j \end{matrix} ] ,  (5.1.5)

provided A_{2} ≤ γ ≤ E_{2}.

There is an alternative expression for the ymaj-generating function which is obtained using Heine’s basic hypergeometric transformation formula (cf. [14, Appendix (III.2)])

2ϕ_{1}[ \begin{matrix}a, b \\ c \end{matrix}; q, z] = \frac{(c/b; q)_{∞}(bz; q)_{∞}}{(c; q)_{∞}(z; q)_{∞}}_{2}ϕ_{1}[ \begin{matrix}abz/c, b \\ bz \end{matrix}; q, \frac{c}{b}] .  (5.1.6)

First we have to write the sum in the last line of (5.1.5) in hypergeometric notation (cf. subsection 2.1),

∑_{P:A→E} q^{ymaj_{β;γ}(P)} = \frac{(q^{E_{1}−A_{1}+1})_{γ−A_{2}}}{(q)_{γ−A_{2}}} _{2}ϕ_{1}[ \begin{matrix}q^{γ−E_{2}}, q^{A_{1}−E_{1}} \\ q^{γ−A_{2}+1} \end{matrix} ; q, q^{1+β+E_{1}+E_{2}−A_{1}−A_{2}}] .

Now (5.1.6) can be applied to get

∑_{P:A→E} q^{ymaj_{β;γ}(P)} = \frac{(q^{E_{1}−A_{1}+1})_{γ−A_{2}} (q^{γ+E_{1}−A_{1}−A_{2}+1})_{∞} (q^{1+β+E_{2}−A_{2}})_{∞}}{(q)_{γ−A_{2}} (q^{γ−A_{2}+1})_{∞} (q^{1+β+E_{1}+E_{2}−A_{1}−A_{2}})_{∞}}

2ϕ_{1}[ \begin{matrix}q^{β}, q^{A_{1}−E_{1}} \\ q^{1+β+E_{2}−A_{2}} \end{matrix} ; q, q^{1+γ+E_{1}−A_{1}−A_{2}}] ,

and finally, after simplifying and rewriting this in q-binomial notation,

∑_{P:A→E} q^{ymaj_{β;γ}(P)} = ∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1} + E_{2} − A_{1} − A_{2} \\ E_{1} − A_{1} − j \end{matrix} ] , (5.1.7)

provided A_{2} ≤ γ ≤ E_{2}.

Now we turn to the xmaj-generating function. Let A_{1} ≤ γ ≤ E_{1}. By (4.0.7) we have

∑_{P:A→E}q^{xmaj_{β;γ}(P)}

= ∑_{k≥0}_{∑_{a∈P_{k}[A_{1},E_{1}−1]} ; b∈P_{k}[A_{2}+1,E_{2}]}q^{∥a∥+∥b∥+β·|{t:a_{t}≥γ}|−k(A_{1}+A_{2})}

= ∑_{k≥0}_{∑_{a′∈P_{k}[A_{1}+1,E_{1}]} ; b′∈P_{k}[A_{2},E_{2}−1]}q^{∥a′∥+∥b′∥+β·|{t:a′_{t}>γ}|−k(A_{1}+A_{2})} ,

where a′_{i} := a_{i} + 1 and b′_{i} = b_{i} − 1, i = 1, …, k. Comparing the last line with (5.1.3), one observes that to obtain expressions for the xmaj-generating function in the previous computations for the ymaj only the roles of A_{1} and A_{2}, and those of E_{1} and E_{2} have to be exchanged. Therefore, from (5.1.5) we get

∑_{P:A→E} q^{xmaj_{β;γ}(P)} = ∑_{j≥0} q^{j(j+β+γ−A_{1})} [ \begin{matrix}E_{1} − γ \\ j \end{matrix} ] [ \begin{matrix}E_{2} + γ − A_{1} − A_{2} \\ E_{2} − A_{2} − j \end{matrix} ] , (5.1.8)

and from (5.1.7) we obtain

∑_{P:A→E} q^{xmaj_{β;γ}(P)} = ∑_{j≥0} q^{j(j+β+γ−A_{1})} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1} + E_{2} − A_{1} − A_{2} \\ E_{2} − A_{2} − j \end{matrix} ] , (5.1.9)

both identities holding provided A_{1} ≤ γ ≤ E_{1}.

There is an important connection between the ymaj- and the xmaj-generating function. This is revealed by combining (5.1.5) and (5.1.8). Let A_{1}+A_{2}−E_{1} ≤ γ < A_{2} and in (5.1.8) replace γ by A_{1} + A_{2} − γ. Then shift the summation index by replacing j by j + γ − A_{2}:

∑_{P:A→E}q^{xmaj_{β;A_{1}+A_{2}−γ}(P)} = ∑_{j≥A_{2}−γ} q^{(j+γ−A_{2})(j+β)} [ \begin{matrix}E_{1} + γ − A_{1} − A_{2} \\ j + γ − A_{2} \end{matrix} ] [ \begin{matrix}E_{2} − γ \\ E_{2} − γ − j \end{matrix} ]

= q^{β(γ−A_{2})} ∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}E_{2} − γ \\ j \end{matrix} ] [ \begin{matrix}E_{1} + γ − A_{1} − A_{2} \\ E_{1} − A_{1} − j \end{matrix} ] .  (5.1.10)

Comparing this last line with (5.1.5) leads to the surprising discovery that the sum in (5.1.5), and hence also the sum in (5.1.7), for γ ≥ A_{2} is equal to the ymaj_{β;γ}- generating function for paths from A to E, while for γ < A_{2} both sums are, save for the factor q^{β(γ−A_{2})}, equal to the xmaj_{β;A_{1}+A_{2}−γ}-generating function for the same family of paths. Similarly, replacing γ by A_{1} +A_{2} −γ in the computation (5.1.10) and in (5.1.5) shows that the sum in (5.1.8), and hence also the sum in (5.1.9), for γ ≥ A_{1} is equal to the xmaj_{β;γ}-generating function for paths from A to E, while for γ < A_{1} both sums are, save to the factor q^{β(γ−A_{1})}, equal to the ymaj_{β;A_{1}+A_{2}−γ}-generating function for the same family of paths. It is these observations that justify the seemingly artificial extension (4.0.3) of ymaj_{β;γ} to γ < A_{2} and the likewise seemingly artificial extension (4.0.4) of xmaj_{β;γ} to γ < A_{1}.

The next Lemma summarizes results about ymaj- and xmaj-generating functions for unrestricted paths. Most of them have already been proved.

Lemma 23. Let A = (A_{1}, A_{2}) and E = (E_{1}, E_{2}) be two lattice points in the integer lattice ℤ^{2}.

For A_{1} + A_{2} − E_{1} ≤ γ ≤ E_{2} the ymaj_{β;γ}-generating function for the paths from A to E is given by

∑_{P:A→E} q^{ymaj_{β;γ}(P)} = ∑_{j≥0} q^{j(j+β+γ−A_{2})} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1} + E_{2} − A_{1} − A_{2} \\ E_{1} − A_{1} − j \end{matrix} ] . (5.1.11)

For A_{1} + A_{2} − E_{2} ≤ γ ≤ E_{1} the xmaj_{β;γ}-generating function for the paths from A to E is given by

∑_{P:A→E} q^{xmaj_{β;γ}(P)} = ∑_{j≥0} q^{j(j+β+γ−A_{1})} [ \begin{matrix}−β \\ j \end{matrix} ] [ \begin{matrix}β + E_{1} + E_{2} − A_{1} − A_{2} \\ E_{2} − A_{2} − j \end{matrix} ] . (5.1.12)

In particular, the maj-generating function is given by

∑_{P:A→E} q^{maj P} = [ \begin{matrix}E_{1} + E_{2} − A_{1} − A_{2} \\ E_{1} − A_{1} . \end{matrix} ]  (5.1.13)

Remark. Of course (5.1.13) is a well-known result of MacMahon ([32, p. 206], cf. [1, Theorem 3.1]).

Proof of the Lemma. First let A ≤ E. That (5.1.11) holds for A_{2} ≤ γ ≤ E_{2} has been shown above when establishing (5.1.7) via (5.1.5). In case that A_{1} + A_{2} − E_{1} ≤ γ < A_{2}, by (4.0.3) ymaj_{β;γ} is equal to xmaj_{β;A_{1}+A_{2}−γ} +β(A_{2} − γ). Because of A_{1} < A_{1} +A_{2} −γ ≤ E_{1}, use of (5.1.10) and the transformation that leads from (5.1.5) to (5.1.7) shows that (5.1.11) also holds in this case. The arguments for (5.1.12) are similar. (5.1.13) can be easily derived from (5.1.11) or (5.1.12) by setting β = 0 and applying the q-Vandermonde summation (5.1.4).

What remains is to show that (5.1.11)–(5.1.13) also hold for A ≰ E. Since in this case there are no paths between A and E, we have to show that the right-hand sides in (5.1.11)–(5.1.13) equal zero. This is trivial for (5.1.13). Next we turn to (5.1.11). If E_{1} < A_{1} all the summands in (5.1.11) are zero because of the second q-binomial coefficient. Now suppose that E_{1} ≥ A_{1} and E_{2} < A_{2}. By a careful examination of the derivation of (5.1.7) from (5.1.5) by means of Heine’s transformation formula, it is seen that the right-hand side of (5.1.11) agrees with the right-hand side of (5.1.5) for all choices of the parameters provided that γ ≥ A_{2} and provided that E_{1} − A_{1} is a nonnegative integer. Since because of the second requirement the summations are finite, both, (5.1.5) and (5.1.11), can be written as polynomials in q^{γ}. There are infinitely many values of γ (and therefore of q^{γ}) where (5.1.5) and (5.1.11) agree, hence they agree as polynomials in q^{γ}. This implies that they agree for all γ. But the summands in (5.1.5) for γ ≤ E_{2} are non-zero only if j ≤ E_{2} − γ and j ≥ A_{2} − γ.

Supposing that such a j exists we would get A_{2} − γ ≤ E_{2} − γ in contradiction to the assumption E_{2} < A_{2}. Therefore the sum in (5.1.5) equals zero for E_{1} ≥ A_{1} and E_{2} < A_{2}, hence also the sum in (5.1.11) vanishes in this case, as desired. All possibilities of A ≰ E are now covered. In all cases (5.1.11) holds.

The arguments for showing that the sum in (5.1.12) vanishes if A ≰ E are similar. □

5.2. A correspondence for lattice paths that cross the line x = y. We wish to handle the strange major counting of paths from A = (A_{1}, A_{2}) to E = (E_{1}, E_{2}) which do not cross the line x = y. Recall that ordinary counting of these paths is done by the reflection principle (see the second part of subsection 2.2). Suppose that A_{1} ≥ A_{2} and E_{1} ≥ E_{2}, i.e. that A and E lie below or maybe on the line x = y. Remember that when saying that a path “lies below x = y” we allow the path to touch x = y. The reflection principle shows that the paths from A to E which cross x = y are in one-to-one correspondence with the paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}).

To undertake the strange major counting, the reflection has to be replaced by a bijection having similar properties, because we cannot use the reflection itself since, when reflecting a part of a path, nothing can be said about the major index or the strange major index of its image. We construct two fundamental bijections for crossing paths, one for the case A_{1} ≥ A_{2}, E_{1} ≥ E_{2}, and one for the case A_{1} ≤ A_{2}, E_{1} ≤ E_{2}. The first bijection is the analogue for the reflection procedure described in subsection 2.2. It maps paths P from (A_{1}, A_{2}) to (E_{1}, E_{2}) which cross x = y onto paths P′ from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}) in a way that only the initial portion till the last point lying above x = y is changed. These properties are shared with the reflection procedure. In addition, for this mapping there holds a relation between ymaj_{β;γ}(P) and xmaj_{β;γ+1}(P′).

Proposition 24. First fundamental bijection for crossing paths. Let A = (A_{1}, A_{2}) and E = (E_{1}, E_{2}) be two lattice points in the integer lattice ℤ^{2} with A_{1} ≥ A_{2} and E_{1} ≥ E_{2}. For every integer γ satisfying

A_{2} ≤ γ ≤ A_{1}  (5.2.1)

there is a bijection Φ_{γ}^{(1)} between paths P from A to E which cross the line x = y and the paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}), such that the last meeting point of P with x = y − 1 is identical with the last meeting point of Φ_{γ}^{(1)}(P) with x = y − 1, such that the path’s portions after this meeting point are identical, and

ymaj_{β;γ}(P) = xmaj_{β;γ+1}(Φ_{γ}^{(1)}(P)) + A_{1} − A_{2} + 1 + β.  (5.2.2)

Remark. The relevance of the assumption γ ≤ A_{1} lies in the fact that it allows a path from A to E to cross x = y only before having crossed the horizontal line y = γ. After having crossed x = y for the first time, the path must have at least one North-East corner before returning to x = y. This North-East corner, because of lying above y = γ, contributes a β to the statistics ymaj_{β;γ}. This explains the β in (5.2.2).

Proof of the Proposition. First we consider the case γ ≥ E_{2}. It is clear that for each point (p_{1}, p_{2}) of a path P from A to E we then have p_{1} ≥ A_{1} ≥ γ ≥ E_{2} ≥ p_{2}. This simply means that P cannot cross the line x = y but stays below everywhere only being allowed to touch it. Therefore the set of crossing paths is empty as well as the set of paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}) because of A_{1} ≥ E_{2}. This implies that the Proposition in this case trivially holds.

Now let γ < E_{2}. The construction of Φ_{γ}^{(1)} is done in four steps. For the reader who is not interested in all the details, we outline a short-cut through the following construction. The mapping is essentially explained by the lines (5.2.3a,b), (5.2.4), (5.2.10)–(5.2.15), and the first passage of the fourth step. An example is added at the end of the proof.

Suppose that P is a path from A to E which crosses x = y.

First step. We map P onto its array representation (cf. (2.1.2)). This gives a bijection between crossing paths P and arrays (a | b) = (a_{1}, …, a_{k} | b_{1}, …, b_{k}),

paths P : (A_{1}, A_{2}) → (E_{1}, E_{2}) which cross x = y  (5.2.3a)
↕
\begin{aligned}A_{1} ≤ & a_{1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & … & b_{k} & ≤ E_{2} \end{aligned} , there exists an i, 1 ≤ i ≤ k with a_{i} < b_{i}

(5.2.3b)

which (because of (4.0.6)) satisfies

ymaj_{β;γ}(P) = ∥a∥ + ∥b∥ + β · |{t : b_{t} > γ}| − k(A_{1} + A_{2}).  (5.2.3c)

The condition required for the arrays in (5.2.3b) is clear from the construction of the array (which was described in subsection 2.1) and the fact that a path’s North-East corner (a_{i}, b_{i}) which lies above x = y satisfies a_{i} < b_{i}.

Second step. Here we use a modification of a correspondence first introduced in [26, section 4]. Given an array (a | b) of the type (5.2.3b) let I be maximal with a_{I} < b_{I}. We map (a | b) onto another two-rowed array by

\begin{matrix}a_{1} & . . . . . . . . . & a_{I} & a_{I+1} & … & a_{k} \\ b_{1} & … & b_{I−1} & b_{I} & . . . . . . . . . . & b_{k} \end{matrix}

⟶ \begin{matrix}b_{1} & . . . . . . . & b_{I−1} & a_{I+1} & … & a_{k} \\ a_{1} & a_{2} & a_{3} & … & a_{I} & b_{I} & . . . . . . . . . . & b_{k} \end{matrix} . (5.2.4)

(The vertical and horizontal lines in (5.2.4) are drawn only to support the orientation of the reader. They visually separate the parts of the arrays that are changed from the parts that remain unchanged.) More precisely, the new array (ā | b̄) = (ā_{2}, …, ā_{k} | b̄_{0}, …, b̄_{k}) is given by

ā_{i} = \{ \begin{matrix}b_{i−1} & 2 ≤ i ≤ I \\ a_{i} & I < i ≤ k \end{matrix}

and

b̄_{i} = \{ \begin{matrix}a_{i+1} & 0 ≤ i < I \\ b_{i} & I ≤ i ≤ k \end{matrix} .

Obviously the new array (ā | b̄) is strictly increasing in both rows. Moreover, I is maximal with ā_{I} < b̄_{I}. Therefore the inverse of the mapping (5.2.4) simply is

\begin{matrix}ā_{2} & . . . . . . . . . & ā_{I} & ā_{I+1} & … & ā_{k} \\ b̄_{0} & b̄_{1} & b̄_{2} & … & b̄_{I−1} & b̄_{I} & . . . . . . . . . . & b̄_{k} \end{matrix}

⟶ \begin{matrix}b̄_{0} & . . . . . . . & b̄_{I−1} & ā_{I+1} & … & ā_{k} \\ ā_{2} & … & ā_{I} & b̄_{I} & . . . . . . . . . . & b̄_{k} \end{matrix} ,

where I is maximal with ā_{I} < b̄_{I}. If there is no such I then I = 1. From this it follows that the mapping (5.2.4) is a bijection between the following two families of arrays

\begin{aligned}A_{1} ≤ & a_{1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & … & b_{k} & ≤ E_{2} \end{aligned} , there is an i with a_{i} < b_{i}  (5.2.5a)

⟷ \begin{aligned}A_{2} + 1 ≤ & & & ā_{2} & … & ā_{k} & ≤ E_{1} − 1 \\ A_{1} ≤ & b̄_{0} & b̄_{1} & b̄_{2} & … & b̄_{k} & ≤ E_{2} \end{aligned} (5.2.5b)

which satisfies

∥a∥ + ∥b∥ + β · |{t : b_{t} > γ}| = ∥ā∥ + ∥b̄∥ + β · |{t : ā_{t} > γ}| + β.  (5.2.5c)

The last identity is due to the assumption γ ≤ A_{1}. Namely, for an array (a | b) of the type (5.2.3b) and I being maximal with a_{I} < b_{I}, we have γ ≤ A_{1} ≤ a_{I} < b_{I}. Therefore whenever b_{i} ≤ γ, there holds i < I. Let L be maximal with b_{L} ≤ γ and let (ā | b̄) the array which by (5.2.4) corresponds to (a | b). Then by means of (5.2.4) there holds

|{t : b_{t} > γ}| = k − L

and

|{t : ā_{t} > γ}| = |{t : ā_{t} > γ and 2 ≤ t ≤ I}|
+ |{t : ā_{t} > γ and I < t ≤ k}|
= |{t : b_{t−1} > γ and 2 ≤ t ≤ I}|
+ |{t : a_{t} > γ and I < t ≤ k}|
= (I − L − 1) + (k − I)
= k − L − 1,

which settles (5.2.5c).

An auxiliary correspondence. Before being able to perform the next step we need an auxiliary correspondence. We claim that there is a bijection between the following two families of arrays,

\begin{aligned}1 ≤ & & & c_{2} & … & c_{I−1} & c_{I} & ≤ M − 1 \\ 0 ≤ & d_{0} & d_{1} & d_{2} & … & d_{I−1} & & ≤ N − 1 \end{aligned}  (5.2.6a)

⟷ \begin{aligned}0 ≤ & c̄_{1} & … & c̄_{J−1} & c̄_{J} & ≤ M \\ 1 ≤ & d̄_{1} & … & d̄_{J−1} & & ≤ N − 2 \end{aligned}

(5.2.6b)

such that

∥c∥ + ∥d∥ + β · |\{t : c_{t} > \overline{γ}\}| = ∥c̄∥ + ∥d̄∥ + β · |\{t : c̄_{t} > \overline{γ} + 1\}|  (5.2.6c)

for all integers \overline{γ} satisfying 0 ≤ \overline{γ} ≤ M − 1. The easiest way to see this, is to show that the corresponding generating functions are equal. For arrays of the type (5.2.6a) we obtain

∑_{I≥1} _{∑_{c∈P_{I−1}[1,M−1]} ; d∈P_{I}[0,N−1]}q^{∥c∥+∥d∥+β·|\{t:c_{t}>\overline{γ}\}|}  (5.2.7a)

= ∑_{I≥1} ( ∑_{d∈P_{I}[0,N−1]} q^{∥d∥})

× ∑_{i+j=I−1} ( ∑_{c′∈P_{i}[1,\overline{γ}]} q^{∥c′∥})( ∑_{c′′∈P_{j}[\overline{γ}+1,M−1]} q^{∥c′′∥+β·|\{t:c′′_{t}>\overline{γ}\}|})

= ∑_{I≥1} q^{\binom{I}{2}} [ \begin{matrix}N \\ I \end{matrix} ] ∑_{i+j=I−1} q^{\binom{i+1}{2}} [ \overline{γ} ; i ] q^{j(β+\overline{γ})+\binom{j+1}{2}} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ]

= ∑_{j≥0} q^{j(j+β+\overline{γ}+1)} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ] ∑_{I≥1} q^{I(I−1−j)} [ \begin{matrix}N \\ I \end{matrix} ] [ \overline{γ} ; I − 1 − j ] (5.2.7b)

= ∑_{j≥0} q^{j(j+β+\overline{γ}+1)} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ] [ \begin{matrix}N + \overline{γ} \\ N − 1 − j \end{matrix} ] ,  (5.2.7c)

where in the last step we used the q-Vandermonde summation (5.1.4). On the other hand for arrays of the type (5.2.6b) we get

∑_{J≥1}_{∑_{c̄∈P_{J} [0,M]} ; d̄∈P_{J−1}[1,N−2]}q^{∥c̄∥+∥d̄∥+β·|\{t:c̄_{t}>\overline{γ}+1\}|}  (5.2.8a)

= ∑_{J≥1} ( ∑_{d∈P_{J−1}[1,N−2]} q^{∥d∥})

× ∑_{i+j=J} ( ∑_{c′∈P_{i}[0,\overline{γ}+1]} q^{∥c′∥})( ∑_{c′′∈P_{j}[\overline{γ}+2,M]} q^{∥c′′∥+β·|\{t:c′′_{t}>\overline{γ}+1\}|})

= ∑_{J≥1} q^{\binom{J}{2}} [ \begin{matrix}N − 2 \\ J − 1 \end{matrix} ] ∑_{i+j=J} q^{\binom{i}{2}} [ \begin{matrix}\overline{γ} + 2 \\ i \end{matrix} ] q^{j(β+\overline{γ}+2)+\binom{j}{2}} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ]

= ∑_{j≥0} q^{j(j+β+\overline{γ}+1)} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ] ∑_{J≥1} q^{(J−1)(J−j)} [ \begin{matrix}N − 2 \\ J − 1 \end{matrix} ] [ \begin{matrix}\overline{γ} + 2 \\ J − j \end{matrix} ] (5.2.8b)

= ∑_{j≥0} q^{j(j+β+\overline{γ}+1)} [ \begin{matrix}M − \overline{γ} − 1 \\ j \end{matrix} ] [ \begin{matrix}N + \overline{γ} \\ N − 1 − j \end{matrix} ] ,  (5.2.8c)

by renewed application of the q-Vandermonde summation. Since (5.2.7c) and (5.2.8c) obviously are identical, the generating functions (5.2.7a) and (5.2.8a) agree. Hence, there must be a bijection satisfying (5.2.6).

This is only an existence proof which does not please any true combinatorialist. But the computations (5.2.7) and (5.2.8) tell us how to explicitely construct such a bijection. For sake of completeness we give a description of it below. We leave it to the reader to verify that it is well-defined and satisfies the desired properties (5.2.6).

Let (c | d) be an array of the type (5.2.6a). To illustrate the following we choose a running example, say M = 11, N = 10, \overline{γ} = 7, and the array (c_{ex} | d_{ex}) given by

\begin{aligned}1 ≤ & 1 4 5 7 8 & ≤ 10 \\ 0 ≤ & 1 3 6 7 8 9 & ≤ 9 \end{aligned} ,

or more precisely c_{ex} = {1, 4, 5, 7, 8} and d_{ex} = {1, 3, 6, 7, 8, 9}.

First c is split up into the initial part c′ and the final part c′′, where c′ contains all elements of c which are smaller or equal \overline{γ}, and c′′ contains the rest. In c′ add N to each element to obtain c′′′. In our example we have c′_{ex} = {1, 4, 5, 7}, c′′_{ex} = {8}, and c′′′_{ex} = {11, 14, 15, 17}. Let d′ denote the complement of d with respect to {0, 1, …, N − 1}. In d′ replace each element e by N − e, thus obtaining d′′. In our example we have d′_{ex} = {0, 2, 4, 5} and d′′_{ex} = {5, 6, 8, 10}. Now form the union of d′′ and c′′′ and denote it by u. In our example this is {5, 6, 8, 10, 11, 14, 15, 17}.

Next u is split up into d̄′′ and c̄′′′ where d̄′′ contains all elements of u being smaller or equal \overline{γ} + 1 and c̄′′′ the rest. In our example we have d̄′′_{ex} = \{5, 6, 8\} and c̄′′′_{ex} = {10, 11, 14, 15, 17}. Now in d̄′′ we replace each element e by N − 1 − e, thus obtaining d̄′. The complement of d̄′ with respect to {1, 2, …, N − 2} is denoted by d̄. On the other hand we subtract N − 1 from each element of c̄′′′ and thus get c̄′. The set c̄′′ is formed by adding 1 to each element of c′′. In our example we get d̄′_{ex} = {1, 3, 4}, d̄_{ex} = {2, 5, 6, 7, 8}, c̄′_{ex} = {1, 2, 5, 6, 8}, and c̄′′_{ex} = {9}. Finally we arrive at the array (c̄ | d̄) by defining c̄ to be the union of c̄′ and c̄′′.

In our example this gives the correspondence

\begin{aligned}1 ≤ & 1 4 5 7 8 & ≤ 10 \\ 0 ≤ & 1 3 6 7 8 9 & ≤ 9 \end{aligned} ⟷ \begin{aligned}0 ≤ & 1 2 5 6 8 9 & ≤ 11 \\ 1 ≤ & 2 5 6 7 8 & ≤ 8 \end{aligned} .  (5.2.9)

It is a routine verification that this above defined mapping is a bijection between the sets of arrays in (5.2.6a,b) which satisfies (5.2.6c). As mentioned above, the motivation of this construction lies in the computations (5.2.7) and (5.2.8). What we did was to interpret each step in these computations combinatorially. Starting at (5.2.7a) first c is split up into c′ and c′′. Then for the moment we forget about c′′ and consider only d and c′ which corresponds to the reordering of summations in (5.2.7b). In order to have a combinatorial version of the next step, the q-Vandermonde summation, we first had to transform d into d′′. Roughly spoken this corresponds to transforming [ \begin{matrix}N \\ I \end{matrix} ] to [ \begin{matrix}N \\ N−I \end{matrix} ] in (5.2.7b). Now we merged d′′ and c′′′ (which has been a shifted version of c′) together, what could be called the combinatorial q-Vandermonde summation. Now we arrived at (5.2.7c) and therefore are able to continue at (5.2.8c). It should be clear that splitting up u, etc., has been nothing but working backward from (5.2.8c) to (5.2.8a).

Third step. Recall that by the first and second step we mapped a path P from A to E which crosses x = y to an array of the type (5.2.5b). Besides, we denoted by I the maximal index for which ā_{I} < b̄_{I}. Given an array of the type (5.2.5b) and its correlated index I,

\begin{aligned}A_{2} + 1 ≤ & ā_{2} & . . . . . . . . . & ā_{I} & ā_{I+1} & … & ā_{k} & ≤ E_{1} − 1 \\ A_{1} ≤ & b̄_{0} & b̄_{1} & b̄_{2} & … & b̄_{I−1} & b̄_{I} & . . . . . . . . . . & b̄_{k} & ≤ E_{2} \end{aligned}  (5.2.10)

for the moment we only consider the subarray

\begin{aligned}& & ā_{2} & … & ā_{I−1} & ā_{I} \\ b̄_{0} & b̄_{1} & b̄_{2} & … & b̄_{I−1} \end{aligned} ,  (5.2.11)

or with bounds included (note the upper bounds!)

\begin{matrix}A_{2} + 1 ≤  ā_{2} … ā_{I−1} ā_{I} ≤ b̄_{I} − 1 \\ A_{1} ≤ b̄_{0} b̄_{1} b̄_{2} … b̄_{I−1}  ≤ b̄_{I} − 1 \end{matrix} .

(Similar to (5.2.4), the vertical and horizontal lines in (5.2.10) are drawn only to help the reader in separating the subarray from the rest.) We subtract A_{2} from each entry of the first row and A_{1} from each entry of the second row of this subarray. It is clear that the resulting array is of the type (5.2.6a) with the choices M = b̄_{I} − A_{2}, N = b̄_{I} − A_{1}, γ̄ = γ − A_{2},

\begin{aligned}1 ≤ & ā_{2} − A_{2} … ā_{I−1} − A_{2} ā_{I} − A_{2} & ≤ b̄_{I} − A_{2} − 1 \\ 0 ≤ & b̄_{0} − A_{1} b̄_{1} − A_{1} b̄_{2} − A_{1} … b̄_{I−1} − A_{1} & ≤ b̄_{I} − A_{1} − 1 \end{aligned} .

(5.2.12)

Because of the assumption γ ≥ A_{2} and because of γ < b_{I} and b_{I} = b̄_{I} (this results from the second step) the difference γ − A_{2} indeed satisfies 0 ≤ γ − A_{2} ≤ b̄_{I} − A_{2} − 1. Therefore by (5.2.6a,b) the array (5.2.12) uniquely corresponds to an array of the type

\begin{aligned}0 ≤ & ã_{1} & … & ã_{J−1} & ã_{J} & ≤ b̄_{I} − A_{2} \\ 1 ≤ & b̃_{1} & … & b̃_{J−1} & & ≤ b̄_{I} − A_{1} − 2 \end{aligned} .  (5.2.13)

Next we add A_{2} − 1 to each entry in the first row and A_{1} + 1 to each entry in the second row to obtain the array

\begin{aligned}A_{2} − 1 ≤ & ã_{1} + A_{2} − 1 & … & ã_{J−1} + A_{2} − 1 & ã_{J} + A_{2} − 1 & ≤ b̄_{I} − 1 \\ A_{1} + 2 ≤ & b̃_{1} + A_{1} + 1 & … & b̃_{J−1} + A_{1} + 1 & & ≤ b̄_{I} − 1 \end{aligned} . (5.2.14)

Finally this array can be put together with the rest of (5.2.10),

\begin{aligned}A_{2} − 1 ≤ & ã_{1} + A_{2} − 1 & . . . . . . . . . . . . . . . . . . & ã_{J} + A_{2} − 1 & ā_{I+1} & … & ā_{k} & ≤ E_{1} − 1 \\ A_{1} + 2 ≤ & b̃_{1} + A_{1} + 1 & … & b̃_{J−1} + A_{1} + 1 & b̄_{I} & . . . . . . . . . . & b̄_{k} & ≤ E_{2} \end{aligned} ,

(5.2.15)

thus obtaining an array of the type

\begin{aligned}A_{2} − 1 ≤ & â_{1} & … & â_{m} & ≤ E_{1} − 1 \\ A_{1} + 2 ≤ & b̂_{1} & … & b̂_{m} & ≤ E_{2} \end{aligned} .  (5.2.16)

How are the weights of (5.2.10) and (5.2.16) related? Assuming that (â | b̂) is defined by (5.2.15), by means of (5.2.6c) and (5.2.10)–(5.2.15) we have

∥ā∥+∥b̄∥ + β · |{t : ā_{t} > γ}|
= ∥â∥ + ∥b̂∥ + β · |{t : â_{t} ≥ γ + 1}|
+ (I − 1)A_{2} + IA_{1} − J(A_{2} − 1) − (J − 1)(A_{1} + 1)
= ∥â∥ + ∥b̂∥ + β · |{t : â_{t} ≥ γ + 1}| + (I − J)(A_{1} + A_{2}) + A_{1} − A_{2} + 1.

Subtraction of k(A_{1} + A_{2}) then gives

∥ā∥ + ∥b̄∥ + β · |{t : ā_{t} > γ}| − k(A_{1} + A_{2})

= ∥â∥ + ∥b̂∥ + β · |{t : â_{t} ≥ γ + 1}| − (k − I + J)(A_{1} + A_{2}) + A_{1} − A_{2} + 1. (5.2.17)

Fourth step. Obviously, arrays of the type (5.2.16) are in one-to-one correspondence with paths P̂ from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}), and besides there holds

xmaj_{β;γ+1}(P̂) = ∥â∥ + ∥b̂∥ + β · |{t : â_{t} ≥ γ + 1}| − m(A_{2} − 1 + A_{1} + 1). (5.2.18)

Now we are able to define Φ_{γ}^{(1)}. For any path P from A to E which crosses x = y perform the steps 1–4. The image of P under application of Φ_{γ}^{(1)} by definition is the resulting path P̂,

Φ_{γ}^{(1)}(P) := P̂.

Combining (5.2.3c), (5.2.5c), (5.2.17), and (5.2.18), we obtain the desired property (5.2.2). Here it is important to notice that the row lengths of the array in (5.2.15) equal k − I + J.

Since all steps which led to the construction of Φ_{γ}^{(1)} of a crossing path P are invertible, Φ_{γ}^{(1)} is a bijection.

The only thing which is still open is the property about the last meeting point with the line x = y −1. Recall that in the second step I was maximal with a_{I} < b_{I}. Having in mind the connection between a path P and its corresponding array representation (a | b), we see that the last meeting point of P with x = y−1 is (b_{I} −1, b_{I}). Obviously, the construction of Φ_{γ}^{(1)}(P) has the property that the last k − I elements in the first row of the array in (5.2.3b) (i.e. the elements a_{I+1}, …, a_{k}) are identical with the last k − I elements in the first row of the array in (5.2.15), the same being valid for the last k − I + 1 elements in the second rows (i.e. the elements b_{I}, …, b_{k}). Besides by construction there holds ã_{J} + A_{2} − 1 < b̄_{I} = b_{I} in (5.2.15). Hence, the last meeting point of Φ_{γ}^{(1)}(P) with x = y − 1 is (b_{I} − 1, b_{I}), too, and the path portions of P and Φ_{γ}^{(1)}(P) after this last meeting point are identical.

This completes the proof of the Proposition.

To illustrate the action of Φ_{γ}^{(1)}, consider the path P_{ex} = ((−1, −2), 1211222111212212122211211112212) with starting point A_{ex} = (−1, −2) and final point E_{ex} = (15, 13). It is the lower path in Figure 4. Choose γ = 5. In the first step P_{ex} is mapped to its array representation (a^{(ex)} | b^{(ex)}) given by

\begin{aligned}−1 ≤ & 0 2 5 6 7 8 10 14 & ≤ 14 \\ −1 ≤ & −1 2 3 5 6 9 10 12 & ≤ 13 \end{aligned} .

The index I which is maximal with respect to a_{I}^{(ex)} < b_{I}^{(ex)} is I = 6. Hence, in the second step by (5.2.4) the array (a^{(ex)} | b^{(ex)}) is mapped to the array

\begin{aligned}−1 ≤ & −1 2 3 5 6 10 14 & ≤ 14 \\ −1 ≤ & 0 2 5 6 7 8 9 10 12 & ≤ 13 \end{aligned} .  (5.2.19)

To perform the third step, we first have to restrict our attention to the subarray

\begin{aligned}−1 ≤ & −1 2 3 5 6 & ≤ 8 \\ −1 ≤ & 0 2 5 6 7 8 & ≤ 8 \end{aligned} .

By virtue of (5.2.12) we obtain the array

\begin{aligned}1 ≤ & 1 4 5 7 8 & ≤ 10 \\ 0 ≤ & 1 3 6 7 8 9 & ≤ 9 \end{aligned} .

Now we have to apply the auxiliary correspondence to this array. Luckily it just agrees with the left-hand array of (5.2.9) and therefore corresponds to

\begin{aligned}0 ≤ & 1 2 5 6 8 9 & ≤ 11 \\ 1 ≤ & 2 5 6 7 8 & ≤ 8 \end{aligned} ,

the right-hand array in (5.2.9). By (5.2.14), from this array we obtain

\begin{aligned}−3 ≤ & −2 −1 2 3 5 6 & ≤ 8 \\ 1 ≤ & 2 5 6 7 8 & ≤ 8 \end{aligned} .

By (5.2.15), this is glued together with the rest of (5.2.19) to obtain

\begin{aligned}−3 ≤ & −2 −1 2 3 5 6 10 14 & ≤ 14 \\ 1 ≤ & 2 5 6 7 8 9 10 12 & ≤ 13 \end{aligned} ,

which is the array representation of the second path in Figure 4. The last meeting point of P_{ex} with x = y − 1 is S_{ex} = (8, 9), it is the same as that of Φ_{5}^{(1)}(P_{ex}). Besides these two paths obviously are identical after S_{ex}, as was claimed.

[figure: •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; ∘; •; •; •; •; P_{ex}; Φ_{5}^{(1)}(P_{ex}); A_{ex}; ℜA_{ex}; E_{ex}; S_{ex}; ^{x = 6  y = 5}]

Figure 4

By definition (4.0.1), we have

ymaj_{β;5}(P_{ex}) = 2 + 7 + 11 + 14 + 16 + 20 + 23 + 29 + 4β = 122 + 4β,

and by (4.0.2) we have

xmaj_{β;6}(Φ_{5}^{(1)}(P_{ex})) = 3 + 7 + 11 + 13 + 16 + 18 + 23 + 29 + 3β = 120 + 3β,

in accordance with (5.2.2). □

Setting β = 0 in Proposition 24, we obtain as a corollary a bijection for the ordinary major index.

Corollary 25. Let A = (A_{1}, A_{2}) and E = (E_{1}, E_{2}) be two lattice points in the integer lattice ℤ^{2} with A_{1} ≥ A_{2} and E_{1} ≥ E_{2}. There is a bijection Φ^{(1)} between paths P from A to E which cross the line x = y and the paths from (A_{2} − 1, A_{1} + 1) to (E_{1}, E_{2}) such that the last meeting point of P with x = y − 1 is identical with the last meeting point of Φ^{(1)}(P) with x = y − 1, such that the path’s portions after this meeting point are identical, and

maj P = maj Φ^{(1)}(P) + A_{1} − A_{2} + 1.  (5.2.20)

Proof. Set β = 0 and define Φ^{(1)} := Φ_{A_{2}}^{(1)}. □

A simpler way to define a bijection satisfying the properties of the above Corollary has been previously given in [27, Proof of (5.36)] by extending an idea of Fürlinger and Hofbauer [13, Lemma on p. 255]. But this bijection is only applicable for the major index and is not extendable in order to be applicable with the strange major index.

The next Proposition provides an analogue of Proposition 24 in the case that the starting point and the final point of the paths both lie above the line x = y.

Proposition 26. Second fundamental bijection for crossing paths. Let A = (A_{1}, A_{2}) and E = (E_{1}, E_{2}) be two lattice points in the integer lattice ℤ^{2} with A_{1} ≤ A_{2} and E_{1} ≤ E_{2}. There is a bijection Φ^{(2)} between paths P from A to E which cross the line x = y and the paths from (A_{2} +1, A_{1} −1) to (E_{1}, E_{2}), such that the last meeting point of P with x = y + 1 is identical with the last meeting point of Φ^{(2)}(P) with x = y + 1, such that the path’s portions after this meeting point are identical, and which satisfies

xmaj_{β;γ}(P) = ymaj_{β;γ−1}(Φ^{(2)}(P))  (5.2.21)

for every integer γ with A_{1} ≤ γ ≤ A_{2}, and also

maj P = maj Φ^{(2)}(P).  (5.2.22)

Remark. Quite similarly to the Remark after Proposition 24, it should be observed that the relevance of the assumption γ ≤ A_{2} lies in the fact that this allows a path from A to E to cross x = y only after before having crossed the vertical line x = γ. Again this will be important in the proof.

Proof of the Proposition. The procedure is quite similar to that in the proof of Proposition 24. It is even simpler since this auxiliary correspondence and therefore also the third step can be avoided. So it will not be necessary to go into all details, a description without comments should suffice. For a better understanding of the following the reader might look up the example at the end of the proof.

First we consider the case γ ≥ E_{1}. It is clear that for each point (p_{1}, p_{2}) of a path P from A to E we then have p_{2} ≥ A_{2} ≥ γ ≥ E_{1} ≥ p_{1}. This simply means that P cannot cross the line x = y but stays below everywhere only being allowed to touch it. Therefore the set of crossing paths is empty as well as the set of paths from (A_{2} + 1, A_{1} − 1) to (E_{1}, E_{2}) because of A_{2} ≥ E_{1}. This implies that the Proposition in this case trivially holds.

Now let γ < E_{1}. The construction of Φ^{(2)} this time is done in three steps. Suppose that P is a path from A to E which crosses x = y.

First step. Use the array representation (2.1.2) of paths to see that this provides a bijection between the following sets,

paths P : (A_{1}, A_{2}) → (E_{1}, E_{2}) which cross x = y  (5.2.23a)
↕
\begin{aligned}A_{1} ≤ & a_{1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & … & b_{k} & ≤ E_{2} \end{aligned} , there exists an i, 1 ≤ i ≤ k − 1 with a_{i+1} > b_{i}

(5.2.23b) which (because of (4.0.7)) satisfies

xmaj_{β;γ}(P) = ∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ γ}| − k(A_{1} + A_{2}).  (5.2.23c)

Second step. For an array (a | b) of the type (5.2.23b) let I be maximal with a_{I+1} > b_{I}. We map (a | b) to another two-rowed array (ā | b̄) by

\begin{aligned}A_{1} ≤ & a_{1} & … & a_{I} & a_{I+1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{2} + 1 ≤ & b_{1} & … & b_{I} & b_{I+1} & … & b_{k} & ≤ E_{2} \end{aligned}  (5.2.24a)

⟷ \begin{aligned}A_{2} + 1 ≤ & b_{1} & … & b_{I} & a_{I+1} & … & a_{k} & ≤ E_{1} − 1 \\ A_{1} ≤ & a_{1} & … & a_{I} & b_{I+1} & … & b_{k} & ≤ E_{2} \end{aligned}

(5.2.24b)

(Again the vertival lines in (5.2.24) are drawn only to show which parts of the arrays are changed and which not.) More precisely, the new array (ā | b̄) = (ā_{1}, …, ā_{k} | b̄_{1}, …, b̄_{k}) is given by

ā_{i} = \{ \begin{matrix}b_{i} & 1 ≤ i ≤ I \\ a_{i} & I < i ≤ k \end{matrix}

and

b̄_{i} = \{ \begin{matrix}a_{i} & 1 ≤ i ≤ I \\ b_{i} & I < i ≤ k \end{matrix} .

This correspondence has the property

∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ γ}| = ∥ā∥ + ∥b̄∥ + β · |{t : b̄_{t} ≥ γ}|.  (5.2.24c)

Third step. Each array (ā | b̄) of the type (5.2.24b) uniquely corresponds to a path P̄ from (A_{2} + 1, A_{1} − 1) to (E_{1}, E_{2}). Besides there holds

ymaj_{β;γ−1}(P̄) = ∥ā∥ + ∥b̄∥ + β · |{t : b̄_{t} > γ − 1}| − k(A_{1} + A_{2}).  (5.2.25)

Now we are able to define Φ^{(2)}. For any path P from A to E which crosses x = y perform the steps 1–3. The image of P under application of Φ^{(2)} by definition is the resulting path P̄,

Φ^{(2)}(P) = P̄.

Combining (5.2.23c), (5.2.24c), and (5.2.25), one obtains the desired property (5.2.21). Evidently, (5.2.22) results from (5.2.21) by setting β = 0.

The fact that the last k − I elements in the rows of the arrays in (5.2.24a) are identical with the corresponding last k − I elements in the array in (5.2.24b), implies that the last meeting point of P with x = y + 1 as well as the last meeting point of Φ^{(2)}(P) with x = y + 1 is (b_{I} + 1, b_{I}), and that the portions of P and Φ^{(2)}(P) after this last meeting point are identical.

Finally we give an example to illustrate the action of Φ^{(2)}. Consider the path P_{ex} = ((−3, −1), 1121122111212212212211) in Figure 5, having the starting point A_{ex} = (−3, −1) and the final point E_{ex} = (9, 9). (We have marked the North-East corners of the two paths in Figure 5 by thicker dots. So the paths should be easily distinguished.) In the first step P_{ex} is mapped to its array representation (a^{(ex)} | b^{(ex)}) given by

\begin{aligned}−3 ≤ & −1 1 4 5 6 7 & ≤ 8 \\ 0 ≤ & 0 2 3 5 7 9 & ≤ 9 \end{aligned} .

The index I which is maximal with respect to a_{I+1}^{(ex)} > b_{I}^{(ex)} is I = 4. So in the second step, by (5.2.24) the array (a^{(ex)} | b^{(ex)}) is mapped to the array

\begin{aligned}0 ≤ & 0 2 3 5 6 7 & ≤ 8 \\ −3 ≤ & −1 1 4 5 7 9 & ≤ 9 \end{aligned} ,

which is the array representation of the second path in Figure 5. S_{ex} is the last meeting point of P_{ex} and Φ^{(2)}(P_{ex}) with the line x = y + 1.

[figure: •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; ∘; •; •; •; •; •; •; •; •; •; •; •; •; •; ℜA_{ex}; A_{ex}; E_{ex}; Φ^{(2)}(P_{ex}); P_{ex}; S_{ex}]

Figure 5

It is left to the reader to check that (5.2.21) always holds. □

Also for this case, a simpler bijection which satisfies (5.2.22) only has been previously given in [27, Proof of (5.37)]. It is also not extendable in order to be applicable with the strange major index.

5.3. A correspondence for pairs of intersecting lattice paths. Let A = (A_{1}, A_{2}), B = (B_{1}, B_{2}), E = (E_{1}, E_{2}), and F = (F_{1}, F_{2}) be lattice points in ℤ^{2} such that A_{1} ≤ B_{1}, A_{2} ≥ B_{2}, E_{1} ≤ F_{1}, E_{2} ≥ F_{2}. Recall [16, 17, 41] (see subsection 2.2) that the key for obtaining determinantal expressions for generating functions for families of nonintersecting lattice paths where the weight depends on the position of the edges of the paths in the lattice ℤ^{2} lies in the observation that the generating function for pairs (P_{1}, P_{2}) of intersecting paths, P_{1} : A → E, P_{2} : B → F, is equal to the generating function of arbitrary pairs (Q_{1}, Q_{2}) of paths, Q_{1} : B → E, Q_{2} : A → F. (Recall that paths are called intersecting if they have points in common, and nonintersecting if not.) This has already been explained in subsection 2.2. But we are not interested in edge weights but in weighing families of paths by the major index or the strange major indices, respectively. So what we need is a correspondence with similar properties, which in addition should take care of the major index or the strange major indices. The familiar interchanging procedure described in subsection 2.2 obviously does not.

The first observation is that there cannot be any useful result unless (A_{1}, A_{2}) and (B_{1}, B_{2}) lie on a line parallel to the anti-diagonal x + y = 0. After we have proved the following Proposition we shall try to give something like an “explanation” of this phenomenon. Here we want to content ourselves with an example. Take A_{0} = (0, 0), E_{0} = (2, 3), and B_{0} = (1, −2), F_{0} = (3, 1). The maj-generating function for intersecting pairs of paths (P_{1}, P_{2}), P_{1} : A_{0} → E_{0}, P_{2} : B_{0} → F_{0}, is given by

∑_{P_{1}:A_{0}→E_{0}} ; P_{2}:B_{0}→F_{0} P_{1},P_{2} intersectq^{maj P_{1}+maj P_{2}} = q^{2} + 2q^{3} + 4q^{4} + 4q^{5} + 5q^{6} + 4q^{7} + 2q^{8} + q^{9} + q^{10}. (5.3.1)

Suppose that there would be some bijection taking care of the major index, such that the above intersecting pairs (P_{1}, P_{2}) of paths would uniquely correspond to pairs (Q_{1}, Q_{2}) of paths, Q_{1} : B_{0} → E_{0}, Q_{2} : A_{0} → F_{0}. Then the generating function (5.3.1) had to equal

∑_{Q_{1}:B_{0}→E_{0}} ; Q_{2}:A_{0}→F_{0}q^{stat_{1} Q_{1}+stat_{2} Q_{2}} = ( ∑_{Q_{1}:B_{0}→E_{0}} q^{stat_{1} Q_{1}})( ∑_{Q_{2}:A_{0}→F_{0}} q^{stat_{2} Q_{2}}),

where stat_{1} and stat_{2} denote some major-like statistics. This means that the polynomial on the right-hand side should factor into two polynomials, one being the sum of 6 monomials of the form q^{i}, the second being the sum of 4 monomials of the form q^{i}, i ∈ {0, 1, …, 10}. (There are 6 paths from B_{0} = (1, −2) to E_{0} = (2, 3), and 4 paths from A_{0} = (0, 0) to F_{0} = (3, 1).) But the polynomial in (5.3.1) does not factor at all!

Therefore we will restrict ourselves to A and B lying on a line parallel to x+y = 0. This means that in the sequel we will always assume that there are some integers A, B, D such that A = (A + D, −A) and B = (B + D, −B). For this choice of the initial points of the paths it is indeed possible to find a strange major analogue (and hence also a major analogue) for the interchanging procedure of subsection 2.2. This analogue also maps pairs (P_{1}, P_{2}) of intersecting paths, P_{1} : A → E, P_{2} : B → F, onto arbitrary pairs (Q_{1}, Q_{2}) of paths, Q_{1} : B → E, Q_{2} : A → F, and it also preserves the path’s portions after the last meeting point.

Proposition 27. Fundamental bijection for pairs of intersecting paths. Let A = (A + D, −A), B = (B + D, −B), E = (E_{1}, E_{2}), and F = (F_{1}, F_{2}) be lattice points in ℤ^{2} with A < B, E_{1} < F_{1}, E_{2} ≥ F_{2}. There is a bijection Ψ between pairs (P_{1}, P_{2}), P_{1} : A → E, P_{2} : B → F, of intersecting paths and pairs (Q_{1}, Q_{2}), Q_{1} : B → E, Q_{2} : A → F, of paths such that the portions of P_{1} and Q_{1} after the last meeting point of P_{1} and P_{2} are identical, including this last meeting point, and such that the portions of P_{2} and Q_{2} after this last meeting point, including the last meeting point, are identical. Moreover, if (Q_{1}, Q_{2}) = Ψ((P_{1}, P_{2})) then the equations

ymaj_{β;γ+1}(P_{1}) + ymaj_{β;γ}(P_{2}) = ymaj_{β;γ+1}(Q_{1}) + ymaj_{β;γ}(Q_{2}) + B − A (5.3.2)

and

xmaj_{β;γ}(P_{1}) + xmaj_{β;γ+1}(P_{2}) = xmaj_{β;γ}(Q_{1}) + xmaj_{β;γ+1}(Q_{2}) + B − A, (5.3.3)

hold for all integers γ. In particular, there holds

maj P_{1} + maj P_{2} = maj Q_{1} + maj Q_{2} + B − A.  (5.3.4)

Besides, if A + D < γ_{1} ≤ B + D and −B − 1 ≤ γ_{2} < −A, the equation

xmaj_{β;γ_{1}}(P_{1})+ymaj_{β;γ_{2}}(P_{2}) = ymaj_{β;γ_{2}+1}(Q_{1})+xmaj_{β;γ_{1}+1}(Q_{2})+B−A+β (5.3.5)

holds. Identity (5.3.5) also holds if the inequality γ_{1} ≤ B + D is replaced by the requirement that the last meeting point of P_{1} and P_{2} is located to the right or on the vertical line x = γ_{1}.

Proof. Once the equation (5.3.2) is settled, the equations (5.3.3) and (5.3.4) immediately follow. For (5.3.3) this is seen using the relation (4.0.5) between ymaj and xmaj, while for deriving (5.3.4) we only have to set β = 0 in (5.3.2).

The construction of Ψ consists of four steps. As in the proof of Proposition 24, for the reader who is not interested in all the details we outline a short-cut through the following construction. The map Ψ is essentially explained by the lines (5.3.6a–e), (5.3.10), (5.3.11), (5.3.20), (5.3.21), and the first passage of the fourth step of the proof. For illustration, an example is carried out at the end of the proof.

First step. Again we utilize the array representation of paths. Let (P_{1}, P_{2}) be a pair of intersecting paths, P_{1} : A → E, P_{2} : B → F. P_{1} and P_{2} are in one-to-one correspondence with two-rowed arrays (a | b) and (c | d) of the types

\begin{aligned}A + D ≤ & a_{1} & … & a_{k} & ≤ E_{1} − 1 \\ −A + 1 ≤ & b_{1} & … & b_{k} & ≤ E_{2} \end{aligned}  (5.3.6a)

and

\begin{aligned}B + D ≤ & c_{1} & … & c_{l} & ≤ F_{1} − 1 \\ −B + 1 ≤ & d_{1} & … & d_{l} & ≤ F_{2} \end{aligned} ,  (5.3.6b)

respectively.

Let S be the last meeting point of P_{1} and P_{2}. By definition set a_{k+1} := E_{1}, b_{0} := −A, b_{k+1} := E_{2} + 1, and c_{l+1} := F_{1}. (Note that the thereby augmented sequences a, b, c remain strictly increasing.)

• • •^{S}^{(c_{j}, d_{j})} _{(a_{i}, b_{i−1})}  ^{P_{1}}  ^{P_{2}} Figure 6

Considering the East-North corner (a_{i}, b_{i−1}) in P_{1} immediately preceding S (and being allowed to be equal to S) and the North-East corner (c_{j}, d_{j}) in P_{2} immediately preceding S (and being allowed to be equal to S), we get the inequalities (cf. Figure 6)

c_{j} ≤ a_{i} < c_{j+1},  (5.3.6c)
b_{i−1} ≤ d_{j} < b_{i},  (5.3.6d)

where

1 ≤ i ≤ k + 1, 1 ≤ j ≤ l.  (5.3.6e)

Of course, k, l, a_{i}, b_{i}, c_{j}, d_{j}, etc., refer to the array representations (5.3.6a,b) of P_{1} and P_{2}. It now becomes apparent that the above assignments for a_{k+1}, b_{0}, b_{k+1}, c_{l+1} are needed for the inequalities (5.3.6c,d) to make sense for i = 1, i = k + 1, or j = l. Vice versa, if (5.3.6c,d,e) is satisfied then (a_{i}, d_{j}) is a meeting point between P_{1} and P_{2}. Besides, a meeting point S is the last meeting point between P_{1} and P_{2} if and only if i and j are maximal with respect to the property (5.3.6c,d,e).

Summarizing, the existence of i, j satisfying (5.3.6c,d,e) characterize the array representations of intersecting pairs of paths. Thus we have a bijection between intersecting pairs (P_{1}, P_{2}) of paths, P_{1} : A → E, P_{2} : B → F, and pairs ((a | b), (c | d)) of arrays of the types (5.3.6a) and (5.3.6b), respectively, for which there exist integers i, j which satisfy (5.3.6c,d,e). In the sequel we call such a pair of arrays a pair of arrays of the type (5.3.6).

We finally want to show (5.3.2). So it is necessary to relate ymaj_{β;γ+1}(P_{1}) + ymaj_{β;γ}(P_{2}) to weights of the corresponding array representations (5.3.6a,b). Because of (4.0.6), according to the choice of γ, we have to distinguish between three cases:

Case A. γ ≥ −A − 1. Here we have

ymaj_{β;γ+1}(P_{1}) + ymaj_{β;γ}(P_{2}) = ∥a∥ + ∥b∥ + β · |{t : b_{t} > γ + 1}| + ∥c∥ + ∥d∥ + β · |{t : d_{t} > γ}| − (k + l)D. (5.3.7a)

Case B. −A − 1 > γ ≥ −B. Here we have

ymaj_{β;γ+1}(P_{1}) + ymaj_{β;γ}(P_{2}) = ∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ D − γ − 1}| + ∥c∥ + ∥d∥ + β · |{t : d_{t} > γ}| − (k + l)D + (−A − γ − 1)β. (5.3.7b)

Case C. −B > γ. Here we have

ymaj_{β;γ+1}(P_{1}) + ymaj_{β;γ}(P_{2}) = ∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ D − γ − 1}| + ∥c∥ + ∥d∥ + β · |{t : c_{t} ≥ D − γ}| − (k + l)D + (−A − γ − 1)β + (−B − γ)β. (5.3.7c)

To show (5.3.5) it will be necessary to use the following identity

xmaj_{β;γ_{1}}(P_{1}) + ymaj_{β;γ_{2}}(P_{2}) = ∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ γ_{1}}| + ∥c∥ + ∥d∥ + β · |{t : d_{t} > γ_{2}}| − (k + l)D, (5.3.8a)

which in view of (4.0.6) and (4.0.7) certainly holds if A + D < γ_{1} and −B ≤ γ_{2}. For γ_{2} = −B − 1 we have

xmaj_{β;γ_{1}}(P_{1}) + ymaj_{β;−B−1}(P_{2}) = ∥a∥ + ∥b∥ + β · |{t : a_{t} ≥ γ_{1}}| + ∥c∥ + ∥d∥ + β · |{t : c_{t} ≥ B + D + 1}| − (k + l)D + β. (5.3.8b)

Second step. This is the crucial step in the construction of Ψ.

Let I and J be maximal with

c_{J} ≤ a_{I} < c_{J+1},  (5.3.9a)
b_{I−1} ≤ d_{J} < b_{I},  (5.3.9b)

and

1 ≤ I ≤ k + 1, 1 ≤ J ≤ l.  (5.3.9c)

We map a pair ((a | b), (c | d)) of arrays of the type (5.3.6) to the pair ((ā | b̄), (c̄ | d̄)) of arrays, where (ā | b̄) = (ā_{1}, …, ā_{k−I+J+1} | b̄_{2}, …, b̄_{k−I+J+1}) is given by

\begin{aligned}c_{1} − 1 & c_{2} − 1 & … & c_{J} − 1 & a_{I} & … & a_{k} \\ & d_{1} + 1 & … & d_{J−1} + 1 & b_{I} & … & b_{k} \end{aligned}  (5.3.10a)

and (c̄ | d̄) = (c̄_{2}, …, c̄_{l−J+I} | d̄_{1}, …, d̄_{l−J+I}) is given by

\begin{aligned}& a_{1} + 1 & . . . . . . . . . . . . . & a_{I−1} + 1 & c_{J+1} & … & c_{l} \\ b_{1} − 1 & b_{2} − 1 & … & b_{I−1} − 1 & d_{J} & d_{J+1} & … & d_{l} \end{aligned} .  (5.3.10b)

We claim that this mapping is invertible. More precisely, we are going to establish that the mapping (5.3.10) is a bijection between pairs ((a | b), (c | d)) of arrays of the type (5.3.6) and pairs ((ā | b̄), (c̄ | d̄)) of arrays of the types

\begin{aligned}B + D − 1 ≤ & ā_{1} & . . . . . . & āk̄ & ≤ E_{1} − 1 \\ − B + 2 ≤ & b̄_{2} & … & b̄k̄ & ≤ E_{2} \end{aligned}  (5.3.11a)

and

\begin{aligned}A + D + 1 ≤ & c̄_{2} & … & c̄l̄ & ≤ F_{1} − 1 \\ − A ≤ & d̄_{1} & . . . . . . & d̄l̄ & ≤ F_{2} \end{aligned} ,  (5.3.11b)

respectively. That the bounds in (5.3.11) are the right ones, is trivial.

To establish that (5.3.10) is invertible we construct the inverse mapping. Suppose that ((ā | b̄), (c̄ | d̄)) is a pair of arrays of the type (5.3.11). By definition set ā_{k̄+1} := E_{1}, b̄_{1} := −B + 1, b̄_{k̄+1} := E_{2} + 1, c̄_{1} := A + D, and c̄_{l̄+1} := F_{1}. (Again note that these assignments guarantee the thereby augmented sequences ā, b̄, c̄ to remain strictly increasing.)

Let Ī and J̄ be maximal with

c̄J̄ ≤ āĪ < c̄_{J̄+1},  (5.3.12a)
b̄_{Ī−1} ≤ d̄J̄ < b̄Ī,  (5.3.12b)

and

2 ≤ Ī ≤ k̄ + 1, 1 ≤ J̄ ≤ l̄.  (5.3.12c)

The assignments for ā_{k̄+1}, b̄_{1}, etc., have been necessary in order that the inequalities (5.3.12a,b) make sense for Ī = 2, Ī = k̄ + 1, J̄ = 1, or J̄ = l̄. We map a pair ((ā | b̄), (c̄ | d̄)) of arrays of the type (5.3.11) to the arrays

\begin{matrix}c̄_{2} − 1 & … & c̄J̄ − 1 & āĪ & … & āk̄ \\ d̄_{1} + 1 & … & d̄_{J̄−1} & b̄Ī & … & b̄k̄ \end{matrix}  (5.3.13a)

and

\begin{matrix}ā_{1} + 1 & . . . . . . . . . . . . . & ā_{Ī−1} + 1 & c̄_{J̄+1} & … & c̄l̄ \\ b̄_{2} − 1 & … & b̄_{Ī−1} − 1 & d̄J̄ & d̄_{J̄+1} & … & d̄l̄ \end{matrix} .  (5.3.13b)

The mapping (5.3.13) will be shown to be the inverse of (5.3.10).

The first thing to have to be established, is that the map (5.3.13) is always welldefined. This is the case if and only if there exist Ī and J̄ which are maximal with respect to (5.3.12). So we have to show that there is at least one pair of integers i, j, with

c̄_{j} ≤ ā_{i} < c̄_{j+1},  (5.3.14a)
b̄_{i−1} ≤ d̄_{j} < b̄_{i},  (5.3.14b)

and

2 ≤ i ≤ k̄ + 1, 1 ≤ j ≤ l̄.  (5.3.14c)

Let U = {2, …, k̄ + 1} be the range of the index i and V = {1, …, l̄} be the range of the index j. Next, two mappings f and g, f : U → V, g : V → U, are defined. For i ∈ U denote by f(i) the uniquely determined integer that satisfies

c̄_{f(i)} ≤ ā_{i} < c̄_{f(i)+1}.  (5.3.15)

Likewise, for j ∈ V denote by g(j) the uniquely determined integer that satisfies

b̄_{g(j)−1} ≤ d̄_{j} < b̄_{g(j)}.  (5.3.16)

We have to confirm that f and g are well-defined for all i ∈ U, and that g is welldefined for all j ∈ V. What has to be established is the existence of integers f(i) satisfying (5.3.15) and the existence of integers g(j) satisfying (5.3.16). Once this has been done the uniqueness is clear from the fact that the sequences ā, b̄, c̄, d̄ are in strictly increasing order.

So let i ∈ U. We have B + D − 1 ≤ ā_{1} < ā_{i} ≤ ā_{k̄+1} = E_{1}. Because of the assumptions A < B and E_{1} < F_{1}, we conclude that A + D ≤ ā_{i} < F_{1}, or in terms of c̄_{1} and c̄_{l̄+1}, c̄_{1} ≤ ā_{i} < c̄_{l̄+1}. Clearly this settles the existence of an integer f(i) satisfying (5.3.15).

Similarly, let j ∈ V. We have −A ≤ d̄_{j} ≤ F_{2}. Because of the assumptions A < B and E_{2} ≥ F_{2}, we get −B + 1 ≤ d̄_{j} < E_{2} + 1. This inequality, written in terms of b̄_{1} and b̄_{k̄+1}, reads b̄_{1} ≤ d̄_{j} < b̄_{k̄+1}, which confirms the existence of an integer g(j) satisfying (5.3.16).

It is a simple matter of fact that f and g are monotone functions. To be precise, if i_{1} ≤ i_{2} then f(i_{1}) ≤ f(i_{2}), and if j_{1} ≤ j_{2} then g(j_{1}) ≤ g(j_{2}). This is due to the strictly increasing order of ā, b̄, c̄, and d̄. Now start with 2 ∈ U and alternatively apply f and g. To be precise, we form the sequence s_{1}, s_{2}, s_{3}, …, where s_{1} = 2, s_{2t} = f(s_{2t−1}), and s_{2t+1} = g(s_{2t}), t = 1, 2, . . .. Then we consider the subsequences s_{1}, s_{3}, s_{5}, … and s_{2}, s_{4}, s_{6}, . . .. Obviously, because of the monotonicity of both f and g these subsequences are weakly increasing. In addition they are infinite and bounded sequences, hence both must eventually become stationary. Denote by ī the limit element of the first sequence, and by j̄ the limit element of the second sequence. By definition of the sequences we have f(ī) = j̄ and g(j̄) = ī which by means of (5.3.15) and (5.3.16) says that c̄j̄ ≤ āī < c̄_{j̄+1} and b̄_{ī−1} ≤ d̄j̄ < b̄ī. Hence we have found a pair of indices which satisfies (5.3.14).

Now that we have convinced ourselves that the mapping (5.3.13) is well-defined, it remains to show that it is indeed the inverse of (5.3.10). As before, let ((a | b), (c | d)) be a pair of arrays of the type (5.3.6) and I, J be maximal with respect to (5.3.9). Denote the array in (5.3.10a) by (ā | b̄) and the array in (5.3.10b) by (c̄ | d̄). (Note that k̄ = k − I + J + 1 and l̄ = l − J + I.) Let Ī, J̄ be maximal with respect to (5.3.12). All we have to show is that Ī = J + 1 and J̄ = I since then from (5.3.10) it is obvious that by applying (5.3.13) to the pair ((ā | b̄), (c̄ | d̄)) we obtain the pair ((a | b), (c | d)) again.

Also from (5.3.9) and (5.3.10) it is clear that (5.3.12) holds if Ī is replaced by J +1 and J̄ by I. Hence the actual values of Ī, J̄ because of their maximality must be greater or equal. To be precisely, the inequalities Ī ≥ J + 1 and J̄ ≥ I must hold. We distinguish between three cases.

Case 1. Ī > J + 1 and J̄ > I. Under these assumptions by (5.3.10) we have āĪ = a_{Ī−J+I−1}, b̄Ī = b_{Ī−J+I−1}, b̄_{Ī−1} = b_{Ī−J+I−2}, c̄J̄ = c_{J̄−I+J}, c̄_{J̄+1} = c_{J̄−I+J+1}, and d̄J̄ = d_{J̄−I+J}. Setting I_{0} := Ī − J + I − 1 and J_{0} := J̄ − I + J, the inequalities (5.3.12a,b) read c_{J_{0}} ≤ a_{I_{0}} < c_{J_{0}+1} and b_{I_{0}−1} ≤ d_{J_{0}} < b_{I_{0}}, respectively. Besides, because of J + 1 < Ī ≤ k̄ + 1 = k − I + J + 2 we get I < I_{0} ≤ k + 1, and because of I < J̄ ≤ l̄= l − J + I we get J < J_{0} ≤ l. This contradicts the maximality of I and J with respect to (5.3.9).

Case 2. Ī > J + 1 and J̄ = I. Here by (5.3.10) we have c̄_{J̄+1} = c_{J+1}, d̄J̄ = d_{J}, and āĪ, b̄Ī, b̄_{Ī−1} as in Case 1. Setting I_{0} := Ī− J + I − 1 and J_{0} := J from the second inequality in (5.3.12a) we get a_{I_{0}} < c_{J_{0}+1}, while (5.3.12b) reads b_{I_{0}−1} ≤ d_{J_{0}} < b_{I_{0}}. Again, because of J + 1 < Ī ≤ k̄ + 1 = k − I + J + 2 we have I < I_{0} ≤ k + 1. In particular, from this and (5.3.9a) we conclude that c_{J} ≤ a_{I} < a_{I_{0}}. This implies c_{J_{0}} < a_{I_{0}} < c_{J_{0}+1}. This is a contradiction to the maximality of I and J with respect to (5.3.9).

Case 3. Ī = J + 1 and J̄ > I. Here by (5.3.10) we have āĪ = a_{I}, b̄Ī = b_{I}, and c̄J̄, c̄_{J̄+1}, d̄J̄ as in Case 1. Setting I_{0} := I and J_{0} := J̄− I + J, the inequality (5.3.12a) reads c_{J_{0}} ≤ a_{I_{0}} < c_{J_{0}+1}, while from the second inequality in (5.3.12b) we derive d_{J_{0}} < b_{I_{0}}. Because of I < J̄ ≤ l̄= l − J + I we have J < J_{0} ≤ l. In particular, from this and (5.3.9b) we deduce b_{I−1} ≤ d_{J} < d_{J_{0}}. This implies b_{I_{0}−1} < d_{J_{0}} < b_{I_{0}}. Again we obtain a contradiction to the maximality of I and J with respect to (5.3.9).

This completes the proof that (5.3.10) is a bijection between pairs of arrays of the type (5.3.6) and pairs of arrays of the type (5.3.11).

Finally we need to say something about weight properties. Again, let ((a | b), (c | d)) be a pair of arrays of the type (5.3.6) and denote by ((ā | b̄), (c̄ | d̄)) its image under application of (5.3.10). The right-hand sides of (5.3.7a,b,c) serve as a guide which expressions have to be considered. To begin with, a short computation shows that

∥a∥ + ∥b∥ + ∥c∥ + ∥d∥ − (k + l)D = ∥ā∥ + ∥b̄∥ + ∥c̄∥ + ∥d̄∥ − (k̄ + l̄)D + D + 1. (5.3.17)

To count contributed β’s, even five cases have to be distinguished. Since the sense of the following manipulations will only become clear later, maybe for the moment the reader should skip the following case study and come back to it before reading the passage after (5.3.21), when the study of weight properties is continued.

Case A1. γ ≥ −A. Having in mind (5.3.7a) we obtain

β · |{t : b_{t} > γ + 1}| + β · |{t : d_{t} > γ}|
= β · |{t : b_{t} > γ + 1 and t ≥ I}| + β · |{t : b_{t} − 1 > γ and t < I}|
+ β · |{t : d_{t} > γ and t ≥ J}| + β · |{t : d_{t} + 1 > γ + 1 and t < J}|
= β · |{t : b̄_{t} > γ + 1}| + β · |{t : d̄_{t} > γ}|.  (5.3.18a)

Case A2. γ = −A − 1. Again having in mind (5.3.7a) we get

β · |{t : b_{t} > −A}| + β · |{t : d_{t} > −A − 1}|
= β · |{t : b_{t} > −A and t ≥ I}| + β · |{t : b_{t} > −A and t < I}|

+ β · |{t : d_{t} > −A − 1 and t ≥ J}| + β · |{t : d_{t} > −A − 1 and t < J}|.

Since because of (5.3.6a) the relations b_{t} > −A and a_{t} ≥ A+D hold for all t = 1, …, k, in the above equation the condition b_{t} > −A may freely be substituted by a_{t} + 1 ≥ A + D + 1. Another observation is that by (5.3.9b) we have −A = b_{0} ≤ b_{I−1} ≤ d_{J}. On the other hand (5.3.9a) gives A + D ≤ a_{I} < c_{J+1}. These facts imply

|{t : d_{t} > −A − 1 and t ≥ J}| = l − J + 1
= |{t : c_{t} ≥ A + D + 1 and t ≥ J + 1}| + 1.

Putting everything together, we obtain

β · |{t : b_{t} > −A}| + β · |{t : d_{t} > −A − 1}|

= β · |{t : b_{t} > −A and t ≥ I}| + β · |{t : a_{t} + 1 ≥ A + D + 1 and t < I}|

+ β · |{t : c_{t} ≥ A + D + 1 and t ≥ J + 1}| + β
+ β · |{t : d_{t} + 1 > −A and t < J}|
= β · |{t : b̄_{t} > −A}| + β · |{t : c̄_{t} ≥ A + D + 1}| + β.  (5.3.18b)

Case B. −A − 1 > γ ≥ −B. This time having in mind (5.3.7b), we have

β · |{t : a_{t} ≥ D − γ − 1}| + β · |{t : d_{t} > γ}|

= β · |{t : a_{t} ≥ D − γ − 1 and t ≥ I}| + β · |{t : a_{t} ≥ D − γ − 1 and t < I}|

+ β · |{t : d_{t} > γ and t ≥ J}| + β · |{t : d_{t} > γ and t < J}|.

Use of γ ≥ −B and of (5.3.9a) gives D − γ ≤ B + D ≤ c_{J} ≤ a_{I}. On the other hand obviously we have γ + 1 < −A ≤ b_{I}. These facts imply

|{t : a_{t} ≥ D − γ − 1 and t ≥ I}| = k − I + 1
= |{t : b_{t} > γ + 1 and t ≥ I}|.

Moreover, by −A − 1 > γ and (5.3.9b) we get γ < −A = b_{0} ≤ b_{I−1} ≤ d_{J}, and by γ ≥ −B we get D − γ ≤ B + D ≤ c_{J}. These facts imply

|{t : d_{t} > γ and t ≥ J}| = l − J + 1
= |{t : c_{t} ≥ D − γ and t ≥ J + 1}| + 1.

Putting everything together, we obtain

β · |{t : a_{t} ≥ D − γ − 1}| + β · |{t : d_{t} > γ}|
= β · |{t : b_{t} > γ + 1 and t ≥ I}| + β · |{t : a_{t} + 1 ≥ D − γ and t < I}|
+ β · |{t : c_{t} ≥ D − γ and t ≥ J + 1}| + β
+ β · |{t : d_{t} + 1 > γ + 1 and t < J}|
= β · |{t : b̄_{t} > γ + 1}| + β · |{t : c̄_{t} ≥ D − γ}| + β.  (5.3.18c)

Case C1. γ = −B − 1. Having in mind (5.3.7c), we get

β · |{t : a_{t} ≥ B + D}| + β · |{t : c_{t} ≥ B + D + 1}|
= β · |{t : a_{t} ≥ B + D and t ≥ I}| + β · |{t : a_{t} ≥ B + D and t < I}|
+ β · |{t : c_{t} ≥ B + D + 1 and t ≥ J + 1}|
+ β · |{t : c_{t} ≥ B + D + 1 and t < J + 1}|.

By (5.3.9a) we have B + D ≤ c_{J} ≤ a_{I}, while by (5.3.9b) we have −B < d_{J} < b_{I}. This yields

|{t : a_{t} ≥ B + D and t ≥ I}| = k − I + 1
= |{t : b_{t} > −B and t ≥ I}|.

Moreover, by definition for all t there hold c_{t} ≥ B + D and d_{t} > −B. Hence

|{t : c_{t} ≥ B + D + 1 and t < J + 1}| = J − χ(c_{1} = B + D) = |{t : d_{t} + 1 > −B and t < J}| + 1 − χ(c_{1} = B + D),

where, as in section 2, χ is the usual truth function. Putting everything together, we obtain

β · |{t : a_{t} ≥ B + D}| + β · |{t : c_{t} ≥ B + D + 1}|

= β · |{t : b_{t} > −B and t ≥ I}| + β · |{t : a_{t} + 1 ≥ B + D + 1 and t < I}|

+ β · |{t : c_{t} ≥ B + D + 1 and t ≥ J + 1}|
+ β · |{t : d_{t} + 1 > −B and t < J}| + (1 − χ(c_{1} = B + D)) · β

= β · |{t : b̄_{t} > −B}| + β · |{t : c̄_{t} ≥ B + D + 1}| + χ(c_{1} > B + D) · β. (5.3.18d)

Case C2. γ < −B − 1. Once more, having in mind (5.3.7c), we get

β · |{t : a_{t} ≥ D − γ − 1}| + β · |{t : c_{t} ≥ D − γ}|

= β · |{t : a_{t} ≥ D − γ − 1 and t ≥ I}| + β · |{t : a_{t} + 1 ≥ D − γ and t < I}|

+ β · |{t : c_{t} ≥ D − γ and t ≥ J + 1}|
+ β · |{t : c_{t} − 1 ≥ D − γ − 1 and t < J + 1}|
= β · |{t : ā_{t} ≥ D − γ − 1}| + β · |{t : c̄_{t} ≥ D − γ}|.  (5.3.18e)

In order to establish (5.3.5), we need the following considerations. For γ_{2} ≥ −B, having in mind (5.3.8a), we have

β · |{t : a_{t} ≥ γ_{1}}| + β · |{t : d_{t} > γ_{2}}|
= β · |{t : a_{t} ≥ γ_{1} and t ≥ I}| + β · |{t : a_{t} ≥ γ_{1} and t < I}|
+ β · |{t : d_{t} > γ_{2} and t ≥ J}| + β · |{t : d_{t} > γ_{2} and t < J}|.

If γ_{1} ≤ B + D then using (5.3.9a) we have γ_{1} ≤ B + D ≤ c_{J} ≤ a_{I}. On the other hand, if the last meeting point between P_{1} and P_{2} is located to the right or on the line x = γ_{1}, we directly infer γ_{1} ≤ a_{I}. (Recall that this last meeting point actually is (a_{I}, d_{J}).) Using this, (5.3.9a,b), and the assumption γ_{2} < −A, we get the inequalities γ_{1} ≤ a_{I} < c_{J+1}, γ_{2} + 1 ≤ −A < b_{I}, and γ_{2} < −A ≤ b_{I−1} ≤ d_{J}. These facts imply

|{t : a_{t} ≥ γ_{1} and t ≥ I}| = k − I + 1
= |{t : b_{t} > γ_{2} + 1 and t ≥ I}|

and

|{t : d_{t} > γ_{2} and t ≥ J}| = l − J + 1
= |{t : c_{t} ≥ γ_{1} + 1 and t ≥ J + 1}| + 1.

Putting everything together, we obtain

β · |{t : a_{t} ≥ γ_{1}}| + β · |{t : d_{t} > γ_{2}}|
= β · |{t : b_{t} > γ_{2} + 1 and t ≥ I}| + β · |{t : a_{t} + 1 ≥ γ_{1} + 1 and t < I}|
+ β · |{t : c_{t} ≥ γ_{1} + 1 and t ≥ J + 1}| + β
+ β · |{t : d_{t} + 1 > γ_{2} + 1 and t < J}|
= β · |{t : b̄_{t} > γ_{2} + 1}| + β · |{t : c̄_{t} ≥ γ_{1} + 1}| + β.  (5.3.19a)

On the other hand, for γ_{2} = −B − 1, we have to consider the right-hand side of (5.3.8b). We have

β · |{t : a_{t} ≥ γ_{1}}| + β · |{t : c_{t} ≥ B + D + 1}|
= β · |{t : a_{t} ≥ γ_{1} and t ≥ I}| + β · |{t : a_{t} ≥ γ_{1} and t < I}|
+ β · |{t : c_{t} ≥ B + D + 1 and t ≥ J + 1}|
+ β · |{t : c_{t} ≥ B + D + 1 and t < J + 1}|.

Examining the above considerations for the case γ_{2} ≥ −B, we discover that the inequality γ_{1} ≥ −B in fact has never been used. So also here there hold γ_{1} ≤ a_{I} < c_{J+1} and −B < b_{I}. Besides we have

|{t : c_{t} ≥ B + D + 1 and t < J + 1}| = J − χ(c_{1} = B + D) = |{t : d_{t} + 1 > −B and t < J}| + 1 − χ(c_{1} = B + D).

Putting everything together, we obtain

β · |{t : a_{t} ≥ γ_{1}}| + β · |{t : c_{t} ≥ B + D + 1}|
= β · |{t : b_{t} > −B and t ≥ I}| + β · |{t : a_{t} + 1 ≥ γ_{1} + 1 and t < I}|
+ β · |{t : c_{t} ≥ γ_{1} + 1 and t ≥ J + 1}|
+ β · |{t : d_{t} + 1 > −B and t < J}| + (1 − χ(c_{1} = B + D)) · β
= β · |{t : b̄_{t} > −B}| + β · |{t : c̄_{t} ≥ γ_{1} + 1}| + χ(c_{1} > B + D) · β.(5.3.19b)

Third step. So far we have bijectively mapped intersecting pairs (P_{1}, P_{2}) of paths onto pairs ((ā | b̄), (c̄ | d̄)) of arrays of the type (5.3.11). To interpret these arrays as paths again, we have to modify them such that both become arrays consisting of rows with equal length. For (ā | b̄) of the type (5.3.11a) we distinguish between the cases ā_{1} = B + D − 1 and ā_{1} > B + D − 1. In the first case ā_{1} is deleted from the array (ā | b̄), in the second case the element −B + 1 is inserted as first element in the second row. This yields a new array (â | b̂) which comes out of (ā | b̄) by

(â | b̂) = \{ \begin{matrix}\begin{matrix}ā_{2} & … & āk̄ \\ b̄_{2} & … & b̄k̄ \end{matrix} & ā_{1} = B + D − 1 \\ \begin{matrix}ā_{1} & ā_{2} & … & āk̄ \\ −B + 1 & b̄_{2} & … & b̄k̄ \end{matrix} & ā_{1} > B + D − 1 . \end{matrix}  (5.3.20a)

Similarly, we define a new array (ĉ | d̂) which comes out of (c̄ | d̄) by

(ĉ | d̂) = \{ \begin{matrix}\begin{matrix}c̄_{2} & … & c̄l̄ \\ d̄_{2} & … & d̄l̄ \end{matrix} & d̄_{1} = −A \\ \begin{matrix}A + D & c̄_{2} & … & c̄l̄ \\ d̄_{1} & d̄_{2} & … & d̄l̄ \end{matrix} & d̄_{1} > −A . \end{matrix}  (5.3.20b)

It is clear that (5.3.20) defines a bijection between pairs ((ā | b̄), (c̄ | d̄)) of the type (5.3.11) and pairs ((â | b̂), (ĉ | d̂)) of the types

\begin{aligned}B + D ≤ & â_{1} & … & âk̂ & ≤ E_{1} − 1 \\ −B + 1 ≤ & b̂_{1} & … & b̂k̂ & ≤ E_{2} \end{aligned}  (5.3.21a)

and

\begin{aligned}A + D ≤ & ĉ_{1} & … & ĉl̂ & ≤ F_{1} − 1 \\ −A + 1 ≤ & d̂_{1} & … & d̂l̂ & ≤ F_{2} \end{aligned} ,  (5.3.21b)

respectively.

Turning to the weights of the arrays, first observe that in all four cases of (5.3.20) the following identity holds,

∥ā∥ + ∥b̄∥ + ∥c̄∥ + ∥d̄∥ − (k̄ + l̄)D

= ∥â∥ + ∥b̂∥ + ∥ĉ∥ + ∥d̂∥ − (k̂ + l̂)D + B − A − D − 1. (5.3.22)

Besides, it is readily seen that (5.3.18a,b,c,e) also hold if ā, b̄, c̄, d̄ are replaced by â, b̂, ĉ, d̂, respectively. In the exceptional case C1 (i.e. γ = −B − 1) we have

|{t : b̂_{t} > −B}| = k̄ − 1 + χ(ā_{1} > B + D − 1)
= |{t : b̄_{t} > −B}| + χ(c_{1} > B + D)  (5.3.23)

by remembering (5.3.10a). Combining this with (5.3.18d) gives

β · |{t : a_{t} ≥ B + D}| + β · |{t : c_{t} ≥ B + D + 1}|

= β · |{t : b̂_{t} > −B}| + β · |{t : ĉ_{t} ≥ B + D + 1}|. (5.3.24)

In order to finally prove (5.3.5), we observe that for γ_{2} ≥ −B the identity (5.3.19a) also holds if b̄, c̄ are replaced by b̂, ĉ, respectively. For γ_{2} = −B − 1 however, we may use (5.3.23) to get from (5.3.19b)

β · |{t : a_{t} ≥ γ_{1}}| + β · |{t : c_{t} ≥ B + D + 1}|

β · |{t : b̂_{t} > −B}| + β · |{t : ĉ_{t} ≥ γ_{1} + 1}|. (5.3.25)

Fourth step. Obviously, pairs ((â | b̂), (ĉ | d̂)) of arrays of the type (5.3.21) uniquely correspond to pairs (Q_{1}, Q_{2}) of paths, Q_{1} : B → E, Q_{2} : A → F such that the following weight properties hold:

Case A’. γ ≥ −A. Here we have

ymaj_{β;γ+1}(Q_{1}) + ymaj_{β;γ}(Q_{2}) = ∥â∥ + ∥b̂∥ + β · |{t : b̂_{t} > γ + 1}|

+ ∥ĉ∥ + ∥d̂∥ + β · |{t : d̂_{t} > γ}| − (k̂ + l̂)D. (5.3.26a)

Case B’. −A > γ ≥ −B − 1. Here we have

ymaj_{β;γ+1}(Q_{1}) + ymaj_{β;γ}(Q_{2}) = ∥â∥ + ∥b̂∥ + β · |{t : b̂_{t} > γ + 1}|

+ ∥ĉ∥ + ∥d̂∥ + β · |{t : ĉ_{t} ≥ D − γ}| − (k̂ + l̂)D + (−A − γ)β. (5.3.26b)

Case C’. −B − 1 > γ. Here we have

ymaj_{β;γ+1}(Q_{1}) + ymaj_{β;γ}(Q_{2}) = ∥â∥ + ∥b̂∥ + β · |{t : â_{t} ≥ D − γ − 1}|

+ ∥ĉ∥ + ∥d̂∥ + β · |{t : ĉ_{t} ≥ D − γ}| − (k̂ + l̂)D + (−A − γ)β + (−B − γ − 1)β. (5.3.26c)

What concerns the right-hand side of (5.3.5), we have

ymaj_{β;γ_{2}+1}(Q_{1}) + xmaj_{β;γ_{1}+1}(Q_{2}) = ∥â∥ + ∥b̂∥ + β · |{t : b̂_{t} > γ_{2} + 1}|

+ ∥ĉ∥ + ∥d̂∥ + β · |{t : ĉ_{t} ≥ γ_{1} + 1}| − (k̂ + l̂)D, (5.3.27)

provided that A + D < γ_{1} and −B − 1 ≤ γ_{2}.

Now we are in the position to define Ψ. For any intersecting pair (P_{1}, P_{2}) of paths, P_{1} : A → E, P_{2} : B → F, perform the steps 1–4. The image of (P_{1}, P_{2}) under application of Ψ by definition is the resulting pair (Q_{1}, Q_{2}),

Ψ((P_{1}, P_{2})) := (Q_{1}, Q_{2}).

It has already been shown that Ψ is a bijection. Considering all cases for γ, a combination of (5.3.7a,b,c), (5.3.17), (5.3.18a,b,c,e), respectively (5.3.24), (5.3.22), (5.3.26a,b,c), together with the remark that (5.3.18a,b,c,e) also hold if ā, b̄, c̄, d̄ are replaced by â, b̂, ĉ, d̂, respectively, by a little bit of manipulation finally shows, that (5.3.2), and hence also (5.3.3) and (5.3.4), indeed hold. To establish (5.3.5) we have to combine (5.3.8a,b), (5.3.17), (5.3.19a), respectively (5.3.25), (5.3.22), (5.3.27).

Finally we turn to the statement in the Proposition concerning the last meeting point. Remember that in the second step I and J have been maximal with respect to (5.3.9) which had the meaning that (a_{I}, d_{J}) is the last meeting point between P_{1} and P_{2}, where a_{I} and d_{J} refer to the array representations (5.3.6a,b) of P_{1} and P_{2}. On the other hand, (â | b̂) and (ĉ | d̂), defined via (5.3.10) and (5.3.20), were the array representations of Q_{1} and Q_{2}, respectively. A short look at (5.3.10) and (5.3.20) confirms that Q_{1} and Q_{2} were constructed just in such a manner that (a_{I}, d_{J}) is the last meeting point of Q_{1} and Q_{2}, too. Besides, the last k − I + 1 elements in the corresponding rows of (5.3.6a) and (5.3.10a) are identical, as well as the last l − J elements in the corresponding rows of (5.3.6b) and (5.3.10b). Therefore the tails of P_{1} and Q_{1} and the tails of P_{2} and Q_{2} beginning from this last meeting point are identical.

The proof of the Proposition now is complete.

Again, we provide an example. Consider the paths P_{1}^{ex} : A_{ex} → E_{ex} and P_{2}^{ex} : B_{ex} → F_{ex} in Figure 7, P_{1}^{ex} = ((−1, 0), 1121112122112212), P_{2}^{ex} = ((1, −2), 122212211 121121121), A_{ex} = (−1, 0), E_{ex} = (8, 7), B_{ex} = (1, −2), F_{ex} = (11, 6). The array representations of (P_{1}^{ex}, P_{2}^{ex}) which are formed in the first step are

( \begin{matrix}−1 ≤ & 1 4 5 7 & ≤ 7 \\ 1 ≤ & 1 2 4 6 & ≤ 7 \end{matrix} , \begin{matrix}1 ≤ & 2 3 6 8 10 & ≤ 10 \\ −1 ≤ & 1 3 4 5 6 & ≤ 6 \end{matrix} ) .

The integers I, J which are maximal with respect to (5.3.9) are given by I = 4 and J = 3. They correspond to the last meeting point S_{ex} = (a_{4}^{ex}, d_{3}^{ex}) = (7, 4) of P_{1}^{ex} and P_{2}^{ex}. Hence, by (5.3.10) this pair of arrays is mapped in the second step to the pair

( \begin{matrix}0 ≤ & 1 2 5 7 & ≤ 7 \\ 0 ≤ & 2 4 6 & ≤ 7 \end{matrix} , \begin{matrix}0 ≤ & 2 5 6 8 10 & ≤ 10 \\ 0 ≤ & 0 1 3 4 5 6 & ≤ 6 \end{matrix} ) .

Next (5.3.20) is applied to obtain from this pair the pair of arrays

( \begin{matrix}1 ≤ & 1 2 5 7 & ≤ 7 \\ −1 ≤ & −1 2 4 6 & ≤ 7 \end{matrix} , \begin{matrix}−1 ≤ & 2 5 6 8 10 & ≤ 10 \\ 1 ≤ & 1 3 4 5 6 & ≤ 6 \end{matrix} ) .

These two arrays are the array representations for the paths Q_{1}^{ex} and Q_{2}^{ex} in Figure 7. Q_{1}^{ex} and Q_{2}^{ex} also meet in S_{ex} = (7, 4) for the last time, and the path portions after this last meeting point are identical with the corresponding portions of P_{1}^{ex} and P_{2}^{ex}.

• _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} •  • •  • • A_{ex} B_{ex}  E_{ex} F_{ex} P_{1}^{ex} P_{1}^{ex}  P_{2}^{ex} P_{2}^{ex}  S_{ex}  ⟷  _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} •  • •  • • A_{ex} B_{ex}  E_{ex} F_{ex} Q_{1}^{ex} Q_{1}^{ex} Q{ex \atop 2}  Q_{2}^{ex} S_{ex}

Figure 7

To illustrate (5.3.2), first choose γ = 0. By (4.0.1) we have ymaj_{β;1}(P_{1}^{ex}) + ymaj_{β;0}(P_{2}^{ex}) = 34 + 3β + 53 + 5β = 87 + 8β and ymaj_{β;1}(Q_{1}^{ex}) + ymaj_{β;0}(Q_{2}^{ex}) = 30 + 3β + 55 + 5β = 85 + 8β in accordance with (5.3.2). For γ = −2 by (4.0.1)– (4.0.3) we have ymaj_{β;−1}(P_{1}^{ex}) + ymaj_{β;−2}(P_{2}^{ex}) = 34 + 4β + β + 53 + 5β = 87 + 10β and ymaj_{β;−1}(Q_{1}^{ex}) + ymaj_{β;−2}(Q_{2}^{ex}) = 30 + 3β + 55 + 5β + 2β = 85 + 10β again in accordance with (5.3.2). It is left to the reader to try other cases. □

Remark. We promised to try to give an explanation why nothing reasonable can be expected unless the starting points A and B of the paths P_{1} and P_{2} lie on a line parallel to x + y = 0. Remember that the aim has been to find some correspondence between intersecting pairs (P_{1}, P_{2}) of paths P_{1} : A → E, P_{2} : B → F, and arbitrary pairs (Q_{1}, Q_{2}) of paths Q_{1} : B → E, Q_{2} : A → F, such that the tails of P_{1} and Q_{1}, and the tails of P_{2} and Q_{2} after the last meeting point between P_{1} and P_{2}, which in addition is the same for Q_{1} and Q_{2}, are identical. We are basically counting by the major index. Consider any North-East corner in P_{1}, say, which comes after this last meeting point. This corner contributes the number of steps from A to this corner to the major index or strange major index. If the desired correspondence would obey the above described property, then this North-East corner would be also a North-East corner for Q_{1}. But if B, the starting point of Q_{1}, did not lie on the same line parallel to x + y = 0 as A does, then the number of steps from the starting point of Q_{1} to this corner would be completely different. Of course the same is true with North-East corners of P_{2} and Q_{2}. Since the number of North-East corners which occur after this last meeting point is not predictable, the difference which would be caused if A and B would not lie on a line parallel to x + y = 0 could not be handled properly.

5.4. Correspondences between tableaux and nonintersecting lattice paths. Here we explain the connection between the major and strange major counting of nonintersecting lattice paths and the computation of generating functions for tableaux. The ideas which are used are the celebrated Knuth correspondences [24], one of Burge’s [6, p. 22] modifications of it, the geometric interpretations of Knuth’s and Burge’s correspondences due to Viennot [42] and Desainte-Catherine and Viennot [10] and a refinement of Choi and Gouyou–Beauchamps [7]. Though our presentation sometimes differs from the presentations in the cited papers, in fact all the statements of this subsection are, at least implicitly, in these papers. Therefore the proofs for the following Propositions will be kept short and cursory. What could be called new is the observation that the norm of tableaux carries over into the major index or strange major index of nonintersecting lattice paths.

The main tools of the Knuth correspondences and its variants are algorithms which insert a positive integer into a tableaux or delete an integer from a tableaux, respectively.

We start with the algorithms ROW-INSERT and ROW-DELETE. Given a tableau τ and a positive integer x the algorithm ROW-INSERT inserts x into τ by the following procedure.

Algorithm ROW-INSERT [24, p. 712].

(1) Compare x with the elements of the first row of τ. While reading from left to right, replace the first element which is larger than x by x. If there is no larger element then add x as last element to the first row.

(2) If an element was “bumped” in the previous step, insert it into the second row following the same rule. Repeat this with the subsequent rows until the element to be inserted can be placed at the end of a (possibly empty) row.

(3) Denote the tableaux which results from this inserting procedure by ROW-INSERT(τ, x).

Given the output τ̄ = ROW-INSERT(τ, x) of such an inserting procedure, we are able to recover τ and x if we know the number r of the row of τ̄ where ROW-INSERT stopped. This is accomplished by the algorithm ROW-DELETE. Let τ̄ be a tableau and r a positive integer.

Algorithm ROW-DELETE [24, p. 713].

(1) Take the last element of the r-th row of τ̄. Use it to bump and replace the right-most element of the (r − 1)-th row that is smaller.

(2) Take the bumped element and repeat this procedure in the (r − 2)-th row, etc.

(3) The element which is finally bumped in the first row is denoted by x, the remaining tableau by τ. The output of ROW-DELETE(τ̄, r) is (τ, x).

Dually, there are column insertion and column deletion. The difference is that now we insert into the columns and that everywhere strict order is replaced by weak order.

Algorithm COLUMN-INSERT [24, p. 720].

(1) Compare x with the elements of the first column of τ. While reading from top to bottom, replace the first element which is greater or equal to x by x. If there is no such element then add x as last element to the first column.

(2) If an element was “bumped” in the previous step, insert it into the second column following the same rule. Repeat this with the subsequent columns until the element to be inserted can be placed at the bottom of a (possibly empty) column.

(3) Denote the tableaux which results from this inserting procedure by COLUMN-INSERT(τ, x).

Also with column-insertion, given the number r of the column of the tableau τ̄ = COLUMN-INSERT(τ, x) where COLUMN-INSERT stopped, we are able to recover τ and x, this time using COLUMN-DELETE.

Algorithm COLUMN-DELETE [24, p. 720].

(1) Take the last element of the r-th column of τ̄. Use it to bump and replace the down-most element of the (r − 1)-th column that is smaller or equal.

(2) Take the bumped element and repeat this procedure in the (r −2)-th column, etc.

(3) The element which is finally bumped in the first column is denoted by x, the remaining tableau by τ. The output of COLUMN-DELETE(τ̄, r) is (τ, x).

We begin with two correspondences for pairs of tableaux.

Proposition 28. There exists a bijection Δ_{1} between pairs (τ_{1}, τ_{2}) of tableaux of identical shape, with at most r columns and with parts between 1 and n, and families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths, P_{i} : (i − 1, −i + 1) → (n + i − 1, n − i + 1), i = 1, …, r, such that

ymaj_{1,0} Δ_{1}(τ) = n(τ_{1}) + n(τ_{2}).  (5.4.1)

(The extension of the strange major index ymaj to families of lattice paths is introduced in (4.0.8).)

Proof. First we give a bijection between pairs (τ_{1}, τ_{2}) of tableaux of identical shape and two-line arrays (u | v),

( \begin{matrix}u_{1} & u_{2} & … & u_{N} \\ v_{1} & v_{2} & … & v_{N} \end{matrix} )  (5.4.2)

of positive integers which satisfy

i ≤ j ⇒ either u_{i} < u_{j}
or u_{i} = u_{j} and v_{i} ≥ v_{j}.  (5.4.3)

As an example of an array satisfying (5.4.3) consider the two-line array in Figure 8. Given an array (u | v) satisfying (5.4.3) we form a sequence (τ_{1}^{(i)}, τ_{2}^{(i)})_{i=0,...,N} of pairs of tableaux. We start with τ_{1}^{(0)} = τ_{2}^{(0)} = ∅ (∅ denotes the empty tableau). Then for i = 1, …, N we set τ_{1}^{(i)} = COLUMN-INSERT(τ_{1}^{(i−1)}, v_{i}), and, if r is the number of the column of τ_{1}(i) where COLUMN-INSERT stopped while inserting v_{i} into τ_{1}^{(i−1)}, we place u_{i} at the bottom of the r-th column of τ_{2}^{(i−1)} thus obtaining τ_{2}(i). Finally we set (τ_{1}, τ_{2}) = (τ_{1}^{(N)}, τ_{2}^{(N)}). In Figure 8 the reader finds an example of a two-line array and a pair of tableaux which correspond to each other by this procedure.

It can be seen that this mapping is a bijection between pairs (τ_{1}, τ_{2}) of tableaux of the same shape, with parts between 1 and n, and with exactly r_{0} columns, and two-line arrays (u | v) satisfying (5.4.3) which consist of integers between 1 and n and where the longest (weakly) decreasing subsequence of v, that is a sequence

v_{i_{1}} ≥ v_{i_{2}} ≥ ⋯ ≥ v_{i_{j}} with i_{1} ≤ i_{2} ≤ ⋯ ≤ i_{j}, with j maximal,  (5.4.4)

has length r_{0}. The non-obvious part is the equality between the number of columns of the tableaux and the maximal length of a decreasing subsequence of the bottom row of the two-line array. Since this statement is only implicit in existing papers [24, 6, sec. 3] we sketch a proof at the end of the proof of this Proposition. For later use we denote the mapping from a two-line array to a pair of tableaux by K_{C}. In the tableaux in Figure 8 the length of the first row is 3, and longest decreasing subsequences of the bottom row of the two-line array in Figure 8 for instance are (4, 4, 2), (6, 5, 3), (6, 6, 6), all being of length 3.

Example with n = 7, r_{0} = 3, r = 4

( \begin{matrix}2 & 4 & 4 \\ 3 & 5 & 6 \\ 6 & 6 \\ 7 \end{matrix} , \begin{matrix}1 & 1 & 3 \\ 2 & 3 & 4 \\ 5 & 5 \\ 6 \end{matrix} )_{1. pair of tableaux} ⟷ ( \begin{matrix}1 1 2 3 3 4 5 5 6 \\ 4 4 6 5 2 3 6 6 7 \end{matrix} )_{2. two-line array} ⟷ _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} • • • • • • • • • ∘  ∘^{(2)}  ^{(2)}_{3. multiset of points}

⟷  _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} • • • ∘  ∘^{• •  •}^{• • •}  ^{⟷}  _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}_{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•} _{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•}^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} ^{•} • • • ∘  ∘ ∘ ∘  ∘ ∘ ∘  ∘ • •  • • • •

4. light and shadow  5. family of nonintersecting lattice paths

Figure 8

Next the array (u | v) is viewed geometrically by interpreting (u_{i}, v_{i}), i = 1, …, N, as points (with multiplicity) in the integer lattice ℤ^{2}. Besides we add the points (1, 0) and (n + 1, n) to this multiset of points. In the third picture of Figure 8 these two points are indicated as circles. The symbol (2) being located above the points (1, 4) and (5, 6) indicates that these points have multiplicity 2.

Now Viennot’s light and shadow procedure [42, 10] is applied. We suppose that there is a light being located in the North-West of the multiset of points. The shadow of a point (x, y) is defined to be the set of points (x′, y′) ∈ ℝ^{2} with x ≤ x′ ≤ n+1 and 0 ≤ y′ ≤ y. We consider the (North-West) border of the union of the shadows of all the points of the multiset. This border is a lattice path from (1, 0) to (n + 1, n). The North-East corners of this path are elements of the given multiset. Now we remove one copy of each of these North-East corners from the multiset. Then the light and shadow procedure is repeated, as long as there are points left. A moment’s thought shows that if in a two-line array (u | v) the maximal length of a (weakly) decreasing subsequence of v is r_{0}, then by the light and shadow procedure we obtain exactly r_{0} lattice paths. By construction, these lattice paths may meet but do not cross each other. Item 4 in Figure 8 displays the lattice paths which in our example are obtained by this procedure.

Next we shift the i-th path in the direction (i − 2, −i + 1), i = 1, …, r_{0}, thus obtaining a family (P_{1}, …, P_{r_{0}}) of nonintersecting lattice paths, P_{i} : (i − 1, −i + 1) → (n + i − 1, n − i + 1), i = 1, …, r_{0}. Finally we add the paths P_{r_{0}+1}, …, P_{r}, P_{i} : (i − 1, −i + 1) → (n + i − 1, n − i + 1), i = r_{0} + 1, …, r, all of which start with n horizontal steps followed by n vertical steps. In our example the resulting family of nonintersecting lattice paths is item 5 of Figure 8.

To verify (5.4.1), observe that in the correspondence between pairs of tableaux and two-line arrays the elements are preserved. Therefore the sum n(τ_{1}) + n(τ_{2}) of the norms of the original tableaux equals the sum of all the coordinates of the North-East corners of the paths which are obtained by the light and shadow procedure (item 4 in Figure 8). The contribution of a North-East corner (x, y) to ymaj_{1,0}(P) of a path P which starts in (1, 0) is just the number of steps from (1, 0) to (x, y) plus 1, that is (x + y − 1) + 1 = x + y. Hence n(τ_{1}) + n(τ_{2}) also equals ∑ ymaj_{1,0} P, where the sum is over all paths P which are obtained by the light and shadow procedure. The final shifting and adding of paths converts this sum into ∑_{i=1}^{r} ymaj_{1,−i+1} P_{i}, P_{i} being the i-th path of the resulting family of nonintersecting paths. Comparison with (4.0.8) yields (5.4.1). For the pair of tableaux (τ_{1}^{ex}, τ_{2}^{ex}), say, in Figure 8 we have n(τ_{1}^{ex}) + n(τ_{2}^{ex}) = 43 + 30 = 73. For the resulting family 𝔓^{ex} = (P_{1}^{ex}, P_{2}^{ex}, P_{3}^{ex}, P_{4}^{ex}) of nonintersecting lattice paths we have

ymaj_{1,0}(𝔓^{ex}) = ymaj_{1,0}(P_{1}^{ex}) + ymaj_{1,−1}(P_{2}^{ex}) + ymaj_{1,−2}(P_{3}^{ex}) + ymaj_{1,−3}(P_{4}^{ex})
= 26 + 24 + 23 + 0 = 73,

in accordance with (5.4.1).

As promised, at the end we sketch a proof that the mapping K_{C} that was defined above and maps a two-line array (u | v) to a pair of tableaux (τ_{1}, τ_{2}) has the property that the longest decreasing subsequence of v (see (5.4.4)) has length r_{0} if and only if the number of columns in τ_{1} and τ_{2} is r_{0}. This can be seen in several different ways. We decide to prove it by means of Viennot’s [42, 38, sec. 3.8] geometric interpretation of the Knuth correspondence since we shall also need it at another place. Actually, in [42, 38, sec. 3.8] only the Robinson–Schensted correspondence is considered. But it is easy to see that everything carries over to the Knuth correspondence.

Assuming that the reader is already familiar with either [42] or [38, sec. 3.8], we briefly recall Viennot’s construction. Let (u | v) be a two-line array of the form (5.4.2) such that

i ≤ j ⇒ either u_{i} < u_{j}
or u_{i} = u_{j} and v_{i} ≤ v_{j}.  (5.4.5)

Note that this differs from (5.4.3) in the behaviour of v_{i}’s beneath equal u_{i}’s. Suppose that with this two-line array we do the same as under the mapping K_{C}, but using ROW-INSERT instead of COLUMN-INSERT. Let us denote this mapping by K_{R} and the resulting pair of tableaux by (P_{R}, Q_{R}). Again (u | v) is viewed as a multiset of points in ℤ^{2}, only that now (1, 0) and (n + 1, n) are not added. Next we assume a light to be located in the South-West of the multiset of points. By light and shadow we obtain a set of lattice paths, each having exactly one infinite horizontal and one infinite vertical ray. Viennot’s theorem says that we can read off the first row of P_{R} by the y-coordinates of the horizontal rays. Likewise, we can read off the first row of Q_{R} by the x-coordinates of the vertical rays. In order to get the second rows, we take the “skeletons” of the paths (see [42, 38, sec. 3.8]), apply light and shadow and read the y-coordinates of the horizontal rays and the x-coordinates of the vertical rays, as before. This procedure is iterated to obtain the remaining rows of P_{R} and Q_{R}. Simplifying, we say ‘with a light in the South-West the rows of P_{R} can be read off from the right and the rows of Q_{R} can be read off from the top’. This terminology clearly is motivated by the picture of the construction.

So far we considered ROW-INSERTion. What we need is the analogue of Viennot’s construction for COLUMN-INSERTion. Let (u | v) be a two-line array, ordered according to (5.4.3). Let (P_{C}, Q_{C}) be the pair of tableaux that results from (u | v) by applying K_{C}. Again interpret (u | v) as a multiset of points in ℤ^{2}. Our claim is that with a light located in the South-East the rows of P_{C} can be read off from the left, and that with a light in the North-West the rows of Q_{C} can be read off from the bottom. This is the geometric interpretation of the COLUMN-Knuth correspondence K_{C}. It does not seem to have been recorded before. It is easily checked for our example in Figure 8 (items 1 and 4).

First we turn to the reading of P_{C}. It is known that if the bottom row v of the twoline array by repeated COLUMN-INSERT is mapped to a tableau we would obtain the same as when mapping v^{r} := v_{N}v_{N−1} … v_{1}, the reversion of v, to a tableau by using repeated ROW-INSERT. This is the contents of e.g. [38, Theorem 4.8.8; the equation should read T (π̌^{r}) = T′(π̌), where T′ denotes application of R − S − K′, the repeated COLUMN-INSERTion]. Hence, if we apply K_{R} to (ũ | v^{r}), where v^{r} is as before and ũ is defined by ũ_{i} = U − u_{N−i+1}, i = 1, 2, …, N, with U larger than all u_{i}’s, the resulting tableau P_{R} would be the same as the tableau P_{C} which is obtained by applying K_{C} to (u | v). Note that geometrically the multiset of points corresponding to (ũ | v^{r}) results from the multiset of points corresponding to (u | v) by a reflection in the vertical line x = U/2. Also, if (u | v) is ordered according to (5.4.3) then (ũ | v^{r}) is ordered according to (5.4.5). This guarantees that application of K_{R} to (ũ | v^{r}) is indeed allowed. But we know how to read the rows of P_{R}: With a light in the South-West of (ũ | v^{r}) (viewed as multiset of points) the rows of P_{R} are read from the right. Since P_{R} (corresponding to (ũ | v^{r})) is the same as P_{C} (corresponding to (u | v)), and since (u | v) results from (ũ | v^{r}) by a reflection in the vertical line x = U/2, we know how to read P_{C}: The light in the South-West under the reflection in the vertical line becomes a light in the South-East, instead of reading from the right after the reflection we are reading from the left. This proves the claim about the reading of P_{C}.

In order to prove the claim about the reading of Q_{C}, we recall another standard property of the COLUMN-Knuth correspondence K_{C}: If (u | v) by K_{C} is mapped to (P_{C}, Q_{C}) then the transpose (u | v)^{t} by K_{C} is mapped to (Q_{C}, P_{C}). The transpose (u | v)^{t} is defined by interchanging the top and bottom row of (u | v) and then reordering the pairs \binom{v_{i}}{u_{i}} according to (5.4.3). A proof of this fact is sketched in [6, p. 21]. Note that, geometrically, the multiset of points corresponding to (u | v)^{t} results from the multiset of points corresponding to (u | v) by a reflection in the diagonal x = y. We have already proved that the first tableau, P_{C}, that is obtained by K_{C} is read with a light in the South-East from the left. Hence, Q_{C} can be read off (u | v)^{t} with a light in the South-East from the left. After reflecting back we see that Q_{C} can be read off (u | v) with a light in the North-West from the bottom.

Now we are prepared to attack our original problem. Let (u | v) be a two-line array ordered according to (5.4.3). It is easy to see that the longest (weakly) decreasing subsequence of v has length r_{0} if and only if with a light in the South-East (or in the North-West) light and shadow would produce exactly r_{0} lattice paths. But, by our geometric interpretation of K_{C}, this means that in the pair of tableaux (τ_{1}, τ_{2}) that corresponds to (u | v) under application of K_{C} the first rows of τ_{1} and τ_{2} have length r_{0}. □

Quite analogously, we obtain a result for pairs of tableaux with odd entries.

Proposition 29. There exists a bijection Δ_{2} between pairs (τ_{1}, τ_{2}) of tableaux of identical shape, with at most r columns and with only odd parts which lie between 1 and 2n − 1, and families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths, P_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2), i = 1, …, r, where the North-East corners of the paths have odd coordinates, such that

maj Δ_{2}(τ) = n(τ_{1}) + n(τ_{2}).  (5.4.6)

(The extension of the major index to families of lattice paths is introduced in (3.0.1).)

Proof. We do almost the same as in the previous proof. First, as described above, we map a pair (τ_{1}, τ_{2}) of tableaux of the type under consideration onto its corresponding two-line array. This array is interpreted geometrically as multiset of points in ℤ^{2} as before, but now we add the points (0, 0) and (2n, 2n) as the future starting and final points of the paths which are subsequently constructed by the light and shadow procedure. Next the i-th path is shifted in the direction (2i − 2, −2i + 2). If r_{0} is the number of columns in τ_{1} (and hence also in τ_{2}), then by now we obtained r_{0} pairwise nonintersecting paths, P_{1}, …, P_{r_{0}}. Finally we add the paths P_{r_{0}+1}, …, P_{r}, P_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2), i = r_{0} + 1, …, r, all of which start with 2n horizontal steps followed by 2n vertical steps.

The asserted properties of this bijection are readily verified. □

Now we turn to correspondences for single tableaux. They are based on Burge’s correspondence [6, p. 22].

Proposition 30. There exists a bijection Δ_{3} between tableaux τ with rows of even lengths, with at most 2r columns, and with parts between 1 and n, and families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie below the line x = y, P_{i} : (i − 1, −i + 1) → (n + i, n + 2 − i), i = 1, …, r, such that

maj Δ_{3}(τ) = n(τ).  (5.4.7)

Moreover Δ_{3} has the property that if the first row of τ has length 2r and if the last entry in the first row of τ is h then the r-th path of the family Δ_{3}(τ) starts with h horizontal steps followed by a vertical step. If the length of the first row of τ is less than 2r then the r-th path of Δ_{3}(τ) starts with n + 1 horizontal steps (of course followed by n + 1 vertical steps).

Proof. We follow Desainte–Catherine and Viennot [10], but we present the tableaux in English notation.

First we set up a bijection between tableaux τ with even rows and two-line arrays (u | v) of the form (5.4.2) which satisfy (5.4.3), and in addition satisfy

u_{i} ≥ v_{i} for i = 1, 2 …, N.  (5.4.8)

Given such an array (u | v) we form the sequence (τ^{(i)})_{i=0,...,N} of tableaux, where τ^{(0)} is the empty tableau and τ^{(i)} is obtained from τ^{(i−1)} and the pair (u_{i}, v_{i}) in the following way. Insert v_{i} into τ^{(i−1)} using COLUMN-INSERT. Then add u_{i} at the end of the row of COLUMN-INSERT(τ^{(i−1)}, v_{i}) where COLUMN-INSERT stopped. The resulting tableau is τ^{(i)}, i = 1, …, N. Finally we set τ = τ^{(N)}. The tableau and the two-line array in Figure 9 correspond to each other by this procedure.

It can be seen that this mapping is a bijection between tableaux with even rows, with exactly 2r_{0} columns, and with parts between 1 and n, and two-line arrays (5.4.2) satisfying (5.4.3) and (5.4.8) where the longest (weakly) decreasing subsequence of v, i.e. a subsequence satisfying (5.4.4), equals r_{0}. Again, the non-obvious part is the correspondence between the number of columns of the tableau and the maximal length of a decreasing subsequence of the bottom row of the two-line array. Also this statement has not been proved explicitly, therefore we sketch a proof at the end of the proof of this Proposition. For later use we denote the mapping from a two-line array to a tableau with even rows by K_{E}. In the tableau in Figure 9 the length of the first row is 6, and the longest decreasing subsequences of the two-line array in Figure 9 are (3, 3, 1) and (3, 3, 2), both being of length 3.

Again the array (u, v) is interpreted as multiset of points (u_{i}, v_{i}) in ℤ^{2}. Because of (5.4.8) all the points are located below or on the diagonal x = y. This time we add the points (0, 0) and (n + 1, n + 1). Once more the light and shadow procedure is performed, yielding a set of paths, all of which running from (0, 0) to (n + 1, n + 1), which do not cross each other and lie below x = y. If the maximal length of a (weakly) decreasing subsequence of v is r_{0}, then by the light and shadow procedure we obtain exactly r_{0} paths. Shifting the i-th path in the direction (i − 1, −i + 1) gives a family (P_{1}, …, P_{r_{0}}) of nonintersecting lattice paths which lie below x = y, P_{i} : (i−1, −i+1) → (n+i, n−i+2). Finally the paths P_{r_{0}+1}, …, P_{r}, P_{i} : (i−1, −i+ 1) → (n + i, n − i + 2), are added, each of them starting with n + 1 horizontal steps followed by n + 1 vertical steps. This sets up the desired bijection. Relation (5.4.7) can be readily verified using similar arguments to those in the proof of Proposition 28. For our tableau τ^{ex}, say, in Figure 9 we have n(τ^{ex}) = 33, while for the corresponding family 𝔓^{ex} = (P_{1}^{ex}, P_{2}^{ex}, P_{3}^{ex}) of nonintersecting lattice paths in Figure 9 we have maj P_{1}^{ex} + maj P_{2}^{ex} + maj P_{3}^{ex} = 17 + 6 + 10 = 33, in accordance with (5.4.7).

Example with n = 5, r_{0} = r = 3, h = 3

1 1 2 3 3 3 2 3 3 4 4 4_{1. tableau with even rows} ⟷ ( \begin{matrix}2 3 3 3 4 4 \\ 1 3 3 1 4 2 \end{matrix} )_{2. two-line array} ⟷ [figure: •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; ∘; ∘; (2)]_{3. multiset of points}

[figure: ⟷; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; ∘; ∘; ^{• • •}; ^{• •}; ⟷; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; •; ∘; ∘; ∘; ∘; ∘; ∘; •; •; •]

4. light and shadow  5. family of nonintersecting lattice paths

Figure 9

Finally we have to confirm the last two assertments of the Proposition. In case that the tableau has less than 2r columns, by the light and shadow procedure less than r paths would be obtained. Hence P_{r} would be one of the added paths starting with n + 1 horizontal steps followed by n + 1 vertical steps. This proves the last assertment. To establish the next-to-last assertment, observe that if P_{r} starts with h steps followed by a vertical step there is a v̄ such that the point (h, v̄) is the first North-East corner of the r-th path of the family of paths that is obtained by the light and shadow procedure. So the corresponding two-line array looks like

( \begin{matrix}u_{1} & … & u_{k−1} & h & u_{k+1} & … & u_{N} \\ v_{1} & … & v_{k−1} & v̄ & v_{k+1} & … & v_{N} \end{matrix} ) .  (5.4.9)

Besides, v̄ is the last element in a decreasing subsequence (v_{i_{1}}, …, v_{i_{r−1}}, v̄) of v of maximal length. Also, all the maximal decreasing subsequences of the sequence

(v_{1}, …, v_{k−1})  (5.4.10)

have length r − 1. Now recall that in order to obtain the tableau τ that corresponds to the array (5.4.9), we have to form the sequence τ^{(0)}, τ^{(1)}, …, τ^{(N)} of tableaux by using the array (5.4.9) and this insertion procedure. Since all the maximal decreasing subsequences of the sequence (5.4.10) have length r − 1, the length of the first row of τ^{(k−1)} is 2r−2. Next we have to COLUMN-INSERT v̄ into τ^{(k−1)}. Since v̄ belongs to a maximal decreasing subsequence of length r in the bottom row of (5.4.9), COLUMN-INSERT(τ^{(k−1)}, v̄) will stop at the end of the first row of τ^{(k−1)}. Subsequently, h is added at the end of the first row of COLUMN-INSERT(τ^{(k−1)}, v̄) to obtain τ^{(k)}. Everything will be established if we show that h cannot be bumped during the remaining insertions. But this is obvious, because if h is bumped a new column would be created. Thus the resulting tableau τ would consist of more than 2r columns, a contradiction to our assumption.

As promised, at the end we sketch a proof that the mapping K_{E} that was defined above and maps a two-line array (u | v) which satisfies (5.4.3) and (5.4.8) to a tableau τ with even rows has the property that the longest decreasing subsequence of v (see (5.4.4)) has length r_{0} if and only if the number of columns of τ is 2r_{0}. What we need is the following relation between the maps K_{E} and K_{C} (the latter appeared in the proof of Proposition 28) which was stated in [6, sec. 3] but was not proved there: There holds the relation

K_{C}((u | v) ∪ (v | u)) = (K_{E}((u | v)), K_{E}((u | v))).  (5.4.11)

Here, the symbol (u | v) ∪ (v | u) denotes the “union” of (u | v) and (v | u). This is formed by concatenating (u | v) and (v | u) and reordering the concatenation according to (5.4.3). For example, the union of \binom{2 3 3 3}{1 3 3 1} and \binom{1 3 3 1}{2 3 3 3} is \binom{1 1 2 3 3 3 3 3}{3 2 1 3 3 3 3 1}. Note that, when viewing (u | v) and (v | u) as multisets of points in ℤ^{2}, the union (u | v) ∪ (v | u), as a multiset of points, indeed equals the multiset union of (u | v) and (v | u). More precisely, (u | v) ∪ (v | u) is the union of the multiset of points corresponding to (u | v) and its reflection in the diagonal x = y. Thus (u | v) ∪ (v | u), as multiset of points, is symmetrical and all pairs of the form \binom{w}{w} occur in (u | v)∪(v | u) with an even multiplicity.

If we suppose that the relation (5.4.11) between the maps K_{C} and K_{E} holds then the above claim is easily proved: Obviously the maximal length of a decreasing subsequence of v, the bottom row of (u | v), is r_{0} if and only if the maximal length of a decreasing subsequence of the bottom row of the union (u | v)∪(v | u) is 2r_{0}. (This is best seen geometrically.) On the other hand, in the proof of Proposition 28 we have shown that the maximal length of a decreasing subsequence of (u | v) ∪ (v | u) has length 2r_{0} if and only if the tableaux that result from (u | v)∪(v | u) under application of K_{C} have exactly 2r_{0} columns. Thus, by (5.4.11) our claim is established.

What remains is to prove (5.4.11). Let (u | v) be a two-line array satisfying (5.4.3) and (5.4.8), and let τ be the tableau that results from (u | v) under application of K_{E}. Since (u | v) ∪ (v | u) as a multiset of points is symmetric, application of K_{C} to (u | v) ∪ (v | u) yields a pair of identical tableaux. This is a standard property of K_{C} (cf. [6, p. 21]). Therefore we only have to prove that the first tableau equals τ = K_{E}((u | v)). It is also true that application of K_{C} to (u | v) ∪ (v | u) yields a pair of (identical) tableaux with only even rows since pairs of the form \binom{w}{w} always occur with an even multiplicity in (u | v) ∪ (v | u). Again this is standard (cf. [6, p. 21]).

We perform an induction on the length N of the two-line array (u | v). The relation (5.4.11) is trivially true for N = 0. Let ū denote the sequence u_{1}u_{2} … u_{N−1}, and similarly for v̄. Suppose that (5.4.11) has already been proved for all two-line arrays of length N − 1. Using the geometric interpretation of K_{C} that was developed in the proof of Proposition 28, this means that the tableau τ̄ that results from (ū | v̄) by application of K_{E} can be read off (ū | v̄) ∪ (v̄ | ū) with a light in the South-East from the left. To obtain τ we have to insert the pair (u_{N}, v_{N}) into the tableau τ̄ in the way described above. That is, first v_{N} is COLUMN-INSERTed into τ̄ and then u_{N} is placed at the end of the row where the algorithm stopped. Geometrically, COLUMN-INSERTion of v_{N} into τ̄ means to add the point (u_{N}, v_{N}) to the multiset of points (ū | v̄) ∪ (v̄ | ū) and with a light in the South-East read from the left. Now u_{N} is placed at the end of the row where the COLUMN-INSERTion stopped. Thus τ is obtained. We have to show that this is the same as reading the multiset of points corresponding to (u | v) ∪ (v | u) with a light in the South-East from the left. The multiset of points corresponding to (u | v) ∪ (v | u) is obtained from the multiset of points corresponding to (ū | v̄) ∪ (v̄ | ū) ∪ (u_{N} | v_{N}) by adding the point (v_{N}, u_{N}). It is easy to see that the effect that is caused by the addition of (v_{N}, u_{N}) when reading with a light in the South-East from the left will be the placement of u_{N} at the end of a row. This is due to the fact that, because of the order (5.4.3), the point (v_{N}, u_{N}) is among the highest points of (u | v) ∪ (v | u), viewed as a multiset of points, and among all those it is the left-most. Now, the tableau, say τ′, that is read off (ū | v̄) ∪ (v̄ | ū) ∪ (u_{N} | v_{N}) is a tableau whose shape differs from the shape of τ̄, a tableau with only even rows, just in one cell. Hence, τ′ has exactly one row of odd length. And τ′ differs from the reading of (u | v) ∪ (v | u) by the placement of u_{N} at the end of a row of τ′. The result of the placement of u_{N} in τ′ will again be a tableau with only even rows since, as was remarked above, pairs of the form \binom{w}{w} occur with an even multiplicity in (u | v) ∪ (v | u). But because there is exactly one odd row in τ′ this must be the row where u_{N} is added. Besides this is the row where the COLUMN-INSERTion of v_{N} into τ̄ stopped. Thus the reading of (u | v) ∪ (v | u) and the tableau τ agree. □

If we apply the same arguing to tableaux with only odd entries we obtain an analogous result.

Proposition 31. There exists a bijection Δ_{4} between tableaux τ with rows of even lengths, with at most 2r columns, and with only odd parts which lie between 1 and 2n − 1, and families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie below the line x = y, P_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2), i = 1, …, r, where the North-East corners of the paths have odd coordinates, such that

maj Δ_{4}(τ) = n(τ).  (5.4.12)

Moreover Δ_{4} has the property that if the first row of τ has the length 2r and if the last entry in the first row of τ is 2h + 1 then the r-th path of the family Δ_{4}(τ) starts with 2h + 1 horizontal steps followed by a vertical step. If the length of the first row of τ is less than 2r then the r-th path of Δ_{4}(τ) starts with 2n horizontal steps (of course followed by 2n vertical steps). □

Using the idea of Choi and Gouyou–Beauchamps, Proposition 30 can be extended to tableaux with a prescribed number of odd rows.

Proposition 32. There exists a bijection Δ_{5} between tableaux τ with p odd rows, with at most 2r columns, and with parts between 1 and n, and pairs (𝔓, S), where 𝔓 = (P_{1}, …, P_{r}) is a family of nonintersecting lattice paths which lie below the line x = y, P_{i} : (i − 1, −i + 1) → (n + i, n + 2 − i), i = 1, …, r, with the property that if P_{r} starts with h horizontal steps followed by a vertical step then S is a p-subset of {1, 2, …, h − 1}. (By convention, if h = 1 the set {1, 2, …, h − 1} is meant to be the empty set.) Moreover, Δ_{5} satisfies

maj 𝔓 + ∥S∥ = n(τ)  if (𝔓, S) = Δ_{5}(τ).  □  (5.4.13)

Proof. Let τ be a tableau of the considered type.

First we consider the case that τ has exactly 2r columns. We apply the algorithm ROW-DELETE to each of the odd rows of τ, beginning with the lowest and working up to the highest. We thus obtain a sequence n_{1}, n_{2}, …, n_{p} of integers, the n_{1} being deleted by the first application of ROW-DELETE, etc. Besides we are left with a tableaux τ_{e} with only even rows. Clearly, the last element of the first row of τ_{e} is greater than n_{i} for all i = 1, …, p. Denote this last element of τ_{e} by h. By the last consideration {n_{1}, n_{2}, …, n_{p}} is a subset of {1, 2 …, h − 1}. The observation that the integers n_{1}, n_{2}, …, n_{p} are in increasing order shows that by successively inserting n_{p}, n_{p−1}, …, n_{1} into τ_{e} utilizing ROW-INSERT, the above procedure could be inverted. Hence it is a bijection between tableaux with p odd rows, with exactly 2r columns, and with parts between 1 and n, and pairs (τ_{e}, S), where τ_{e} is a tableau with even rows, with exactly 2r columns, and with parts between 1 and n, with the property that if the last element in the first row of τ_{e} is h, then S is a p-subset of {1, 2, …, h − 1}. By Proposition 30 these tableaux are in one-to-one correspondence with families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths which lie below x = y, P_{i} : (i − 1, −i + 1) → (n + i, n + 2 − i), where the path P_{r} starts with h horizontal steps followed by a vertical one. Note that in the above considerations h, since being an element of τ_{e}, can at most be n. What is missing is the case h = n + 1.

Now let the length of the first row of τ be smaller than 2r. We apply the same method. Use ROW-DELETE repeatedly to get rid of the odd rows of τ. We thus obtain a pair (τ_{e}, S) where τ_{e} is a tableau with properties as before but with less than 2r columns, and where S is a p-subset of {1, 2, …, n}. But tableaux τ_{e} with less than 2r columns by Proposition 30 uniquely correspond to families 𝔓 = (P_{1}, …, P_{r}) of nonintersecting lattice paths with the properties as before, but where P_{r} starts with n + 1 horizontal steps followed by n + 1 vertical steps. This supplies us with the case h = n + 1.

Combining both cases we have a bijection between tableaux τ and pairs (𝔓, S) of families 𝔓 of nonintersecting lattice paths and p-subsets S as desired. In addition, from (5.4.7) and the fact that S consists of some elements of τ, we derive (5.4.13). □

Again the same can be done for tableaux with odd parts.

Proposition 33. There exists a bijection Δ_{6} between tableaux τ with p odd rows, with at most 2r columns, and with only odd parts which lie between 1 and 2n−1, and pairs (𝔓, S), where 𝔓 = (P_{1}, …, P_{r}) is a family of nonintersecting lattice paths which lie below the line x = y, P_{i} : (2i − 2, −2i + 2) → (2n + 2i − 2, 2n − 2i + 2), i = 1, …, r, where the North-East corners of the paths have odd coordinates, with the property that if P_{r} starts with 2h + 1 horizontal steps followed by a vertical step then S is a p-subset of {1, 3, …, 2h − 1}. (By convention, if h = 0 the set {1, 3, …, 2h − 1} is meant to be the empty set.) Moreover, Δ_{6} satisfies

maj 𝔓 + ∥S∥ = n(τ)  if (𝔓, S) = Δ_{6}(τ).  (5.4.14)

5.5. A determinant lemma. Let the degree of a Laurent polynomial p(X) = ∑_{i=M}^{N} a_{i}x^{i}, M, N ∈ ℤ, a_{i} ∈ ℝ and a_{N} ≠ 0, be defined by deg p := N.

Lemma 34. Let X_{1}, X_{2}, …, X_{r}, A_{2}, A_{3}, …, A_{r}, C be indeterminates. If p_{0}, p_{1}, …, p_{r−1} are Laurent polynomials with deg p_{j} ≤ j and p_{j}(C/X) = p_{j}(X) for j = 0, 1, …, r − 1, then

det_{1≤s,t≤r} ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t})(A_{r} + C/X_{t}) ⋯ (A_{s+1} + C/X_{t}) · p_{s−1}(X_{t}))

= ∏_{1≤i<j≤r}(X_{i} − X_{j})(1 − C/X_{i}X_{j}) ∏_{i=1}^{r} A_{i}^{i−1} ∏_{i=1}^{r} p_{i−1}(−A_{i}) , (5.5.1)

with the convention that empty products (like (A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t}) for s = r) are equal to 1. (The indeterminate A_{1}, which occurs at the right-hand side of (5.5.1), in fact is superflous since it occurs in the argument of a constant polynomial.)

Proof. First we reduce this determinant to a simpler one. Because of deg p_{s−1} ≤ s − 1 and p_{s−1}(C/X) = p_{s−1}(X) there must exist λ_{i}^{(s)} ∈ ℝ, i = 1, …, s − 1, such that

p_{s−1}(X) + ∑_{i=1}^{s−1} λ_{i}^{(s)}(A_{s} + X) ⋯ (A_{i+1} + X)(A_{s} + C/X) ⋯ (A_{i+1} + C/X) = constant.

The value of this constant must be p_{s−1}(−A_{s}). This is seen setting X = −A_{s}. Hence we obtain that there are λ_{i}^{(s)} ∈ ℝ such that

p_{s−1}(X)+∑_{i=1}^{s−1} λ_{i}^{(s)}(A_{s} +X) ⋯ (A_{i+1} +X)(A_{s} +C/X) ⋯ (A_{i+1} +C/X) = p_{s−1}(−A_{s}) .

(5.5.2)

Now for j = 2, 3, …, r (in that order) we perform the following row operations: Add λ_{i}^{(j)}/p_{i−1}(−A_{i}) times the i-th row, i = 1, 2, …, j − 1, to the j-th row. Then by using (5.5.2) it is seen that the t-th entry in the j-th row becomes

(A_{r} + X_{t}) ⋯ (A_{j+1} + X_{t})(A_{r} + C/X_{t}) ⋯ (A_{j+1} + C/X_{t}) · p_{j−1}(−A_{j}) .

This allows us to take p_{j−1}(−A_{j}) out of the j-th row.

After having completed these operations we obtain that the determinant in (5.5.1) is equal to

∏_{i=1}^{r} p_{i−1}(−A_{i}) det_{1≤s,t≤r} ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t})(A_{r} + C/X_{t}) ⋯ (A_{s+1} + C/X_{t})) .

Therefore it remains to prove

det_{1≤s,t≤r} ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t})(A_{r} + C/X_{t}) ⋯ (A_{s+1} + C/X_{t}))

= ∏_{1≤i<j≤r}(X_{i} − X_{j})(1 − C/X_{i}X_{j}) ∏_{i=1}^{r} A_{i}^{i−1} (5.5.3)

in order to establish (5.5.1). The determinant on the left-hand side of (5.5.3) is a polynomial in X_{1}, …, X_{r} of degree 3\binom{r}{2} divided by ∏_{i} X_{i}^{r−1}. Since, for 1 ≤ t_{1} < t_{2} ≤ r, the determinant is zero for X_{t_{1}} = X_{t_{2}} or X_{t_{1}} = C/X_{t_{2}}, the left-hand side is equal to

p(X_{1}, …, X_{r}) ∏_{i<j}(X_{i} − X_{j})(X_{i}X_{j} − C)/ ∏_{i} X_{i}^{r−1} ,

where p(X_{1}, …, X_{r}) is some polynomial in the X_{i}’s. Comparing the degrees yields that p is a constant, which is computed by comparing coefficients of ∏_{i} X_{i}^{r−i} on both sides of (5.5.3). □

If in the previous lemma we let C → 0, we get the following corollary.

Lemma 35. Let X_{1}, X_{2}, …, X_{r}, A_{2}, A_{3}, …, A_{r} be indeterminates. If p_{0}, p_{1}, …, p_{r−1} are polynomials with deg p_{j} ≤ j for j = 0, 1, …, r − 1, then

det_{1≤s,t≤r} ((A_{r} + X_{t}) ⋯ (A_{s+1} + X_{t}) · p_{s−1}(X_{t}))

= ∏_{1≤i<j≤r}(X_{i} − X_{j}) ∏_{i=1}^{r} p_{i−1}(−A_{i}) . (5.5.4)

Remarks. (1) Note that in Lemma 35 we definitely mean polynomials and not Laurent polynomials.

(2) In [25, 26, 28] the author has used some determinant lemmas to obtain closed forms for generating functions for several families of plane partitions. In fact, all of these determinant lemmas are special cases of Lemma 34. In particular, Lemma 2.2 of [26] comes out of Lemma 35 by setting

p_{j−1}(X) = ∏_{i=1}^{j−1}(B_{i+1} + X).

Lemma 7 of [28] is the special case p_{j} ≡ 1, j = 0, 1, …, r − 1, of Lemma 34.

5.6. A multiple basic hypergeometric summation.

Proposition 36. For r ≥ 1 there holds the summation formula

∑_{k_{1},...,k_{r}≥0} ∏_{i=1}^{r} (\frac{\sqrt{q}}{q^{i}A})^{k_{i}} ∏_{i=1}^{r} \frac{(m_{i}A)_{k_{i}}}{(qm_{i}/A)_{k_{i}}}

× ∏_{1≤i<j≤r} \frac{1 − \frac{m_{j}}{m_{i}} q^{k_{j}−k_{i}}}{1 − \frac{m_{j}}{m_{i}}} ∏_{1≤i≤j≤r} \frac{1 − m_{i}m_{j}q^{k_{i}+k_{j}}}{1 − m_{i}m_{j}}

= ∏_{1≤i<j≤r} \frac{1 − m_{i}m_{j}/q}{1 − m_{i}m_{j}} ∏_{i=1}^{r} \frac{(1 − m_{i}/\sqrt{q})(1 − m_{i}/A)}{(1 − m_{i}^{2})(1 − \sqrt{q}/q^{i}A)} , (5.6.1)

provided that there exist nonnegative integers n_{i} with n_{1} > n_{2} > ⋯ > n_{r} such that m_{i}A = q^{−n_{i}} for all i = 1, 2, …, r.

Remark. I decided to formulate the Proposition in terms of the m_{i}’s rather than the n_{i}’s in order to keep the notation as short as possible. However, in the following the reader should never forget that the m_{i}’s in fact disguise the n_{i}’s via m_{i} = q^{−n_{i}}/A.

Proof. It seems to be convenient to first give a rough sketch of what we are going to do. We establish (5.6.1) using a double induction. The main induction is on r. For r = 1 equation (5.6.1) can be written in basic hypergeometric notation (cf. subsection 2.1):

6ϕ_{5} [ \begin{matrix}m_{1}^{2}, qm_{1}, −qm_{1}, q, \sqrt{q}m_{1}, m_{1}A \\ m_{1}, −m_{1}, m_{1}^{2}, \sqrt{q}m_{1}, qm_{1}/A \end{matrix}; q, \frac{1}{\sqrt{q}A}] = \frac{(1 − m_{1}/\sqrt{q})(1 − m_{1}/A)}{(1 − m_{1}^{2})(1 − 1/\sqrt{q}A)} . (5.6.2)

This is just a special case of the very well-poised _{6}ϕ_{5}-summation (3.3.9). (Recall that m_{1}A = q^{−n_{1}} by assumption.) We remark that actually we could start our induction with r = 0, thereby avoiding the use of the very well-poised_{6}ϕ_{5}-summation. However, it is useful to know what (5.6.1) looks like for r = 1, the simplest non-trivial value for r.

Now suppose that (5.6.1) is valid for r − 1. To do the induction step, i.e. to settle (5.6.1) for r, provided that it is valid for r − 1, we use induction on n_{1} + n_{2} + ⋯ + n_{r}. First we establish a recurrence (identity (5.6.6)) for the left-hand side of (5.6.1). In the second step we shall show that the right-hand side obeys the same recurrence. Hence everything will be done if the validity of (5.6.1) has been checked for the initial values (n_{1}, n_{2}, …, n_{r−1}, 0), n_{1} > n_{2} > ⋯ > n_{r−1} > 0. Only for this last step it will be necessary to use the induction hypothesis with respect to r.

Let us introduce the following notations: Write S_{r}(A; m_{1}, m_{2}, …, m_{r}) = S_{r}(A; m) for the left-hand side and P_{r}(A; m_{1}, m_{2}, …, m_{r}) = P_{r}(A; m) for the right-hand side of (5.6.1). Moreover we denote the summand in S_{r}(A; m) by E_{r}(A; m_{1}, …, m_{r}, k_{1}, …, k_{r}) = E_{r}(A; m, k) so that S_{r}(A; m) = ∑_{k≥0} E_{r}(A; m, k). The set of summation indices, {k_{1}, k_{2}, …, k_{r}} is denoted by Ω. For a subset T of {1, 2, …, r} let Ω_{T} be the set {k_{i} : i ∈ T } of corresponding summation indices. For any subset B of Ω the symbol ∑_{B≥1} E_{r}(A; m, k) means ∑′E_{r}(A; m, k), where the sum ∑′ is taken over k_{i} ≥ 1 for all i ∈ B and over k_{i} ≥ 0 for the remaining k′_{i}s.

Since the sum in (5.6.1) by assumption is finite, i.e. consists only of a finite number of terms, the inclusion-exclusion principle can be applied to obtain that with our notations

1 = ∑_{T ⊆{1,...,r}}(−1)^{|T |} ∑_{Ω_{T} ≥1} E_{r}(A; m, k) .  (5.6.3)

The 1 on the left-hand side is the term E_{r}(A; m, 0).

Next for a given T ⊆ {1, …, r}, we express ∑_{Ω_{T} ≥1} E_{r}(A; m, k) in terms of S_{r}(A; m). This is done using the q-augmentation operators ε_{1}, …, ε_{r} which are defined by

(ε_{i}f)(m_{1}, …, m_{i}, …, m_{r}) = f(m_{1}, …, qm_{i}, …, m_{r})

for rational functions f(m_{1}, …, m_{r}) and i = 1, …, r. If n_{r} ≥ 1 and qm_{t} ≠ m_{t+1} for all t, we get that

∑_{Ω_{T} ≥1}E_{r}(A; m, k)  (5.6.4a)

= (∏_{l∈T} \frac{\sqrt{q}}{q^{l}A} \frac{(1 − m_{l}A)}{(1 − qm_{l}/A)}) \frac{[( ∏_{l∈T} ε_{l})( ∏_{1≤i<j≤r}(1 − m_{j}/m_{i}) ∏_{1≤i≤j≤r}(1 − m_{i}m_{j}))]}{∏_{1≤i<j≤r}(1 − m_{j}/m_{i}) ∏_{1≤i≤j≤r}(1 − m_{i}m_{j})}

(5.6.4b)

× ( ∏_{l∈T} ε_{l}) ∑_{k_{1},...,k_{r}≥0} E_{r}(A; m, k) .  (5.6.4c)

On the other hand, suppose that n_{r} ≥ 1 and qm_{t} = m_{t+1} holds for a fixed t. Then ε_{t}E_{r}(A; m, k) would not be defined due to the denominator term 1 − m_{t+1}/m_{t} in E_{r}(A; m, k). Therefore, at first sight, if t ∈ T but (t + 1) ∉ T the line (5.6.4c) would not make sense. But in this case we can show that ∑_{Ω_{T} ≥1} E_{r}(A; m, k) = 0. This

implies that

∑_{Ω_{T} ≥1} E_{r}(A; m, k) = (1 − m_{t+1}/qm_{t})R(A; m) ,

where R(A; m) is some rational function with the property that lim_{m_{t+1}→qm_{t}} R(A; m)

exists. In fact, the factor (1 − m_{t+1}/qm_{t}) = ( ∏_{l∈T} ε_{l}) (1 − m_{t+1}/m_{t}) appears in the

numerator of (5.6.4b). So the limit m_{t+1} → qm_{t} of the rest of (5.6.4b,c) exists, hence also lim_{m_{t+1}→qm_{t}} ( ∏_{l∈T} ε_{l}) ∑_{Ω_{T} ≥1} E_{r}(A; m, k) exists. This shows that (5.6.4) also holds

for qm_{t} = m_{t+1}, t ∈ T, (t + 1) ∉ T, since both sides of (5.6.4) equal zero in this case. To confirm that for qm_{t} = m_{t+1}, t ∈ T, (t + 1) ∉ T the sum (5.6.4a) indeed vanishes, we write

∑_{Ω_{T} ≥1}E_{r}(A; m, k) = \frac{\sqrt{q}}{q^{t}A} \frac{(1 − m_{t}A)}{(1 − qm_{t}/A)}  (5.6.5a)

× ∑_{Ω_{T′} ≥1} \frac{[ε_{t} ∏_{i=1}^{r} (\frac{\sqrt{q}}{q^{j}A})^{k_{i}} \frac{(m_{i}A)_{k_{i}}}{(qm_{i}/A)_{k_{i}}} ∏_{1≤i<j≤r} (1− \frac{m_{j}}{m_{i}} q^{k_{i}−k_{j}}) ∏_{1≤i≤j≤r} (1−m_{i}m_{j}q^{k_{i}+k_{j}})]}{∏_{1≤i<j≤r}(1 − m_{j}/m_{i}) ∏_{1≤i≤j≤r}(1 − m_{i}m_{j})}

(5.6.5b)

where T′ = T \{t}. To abbreviate the clumsy expressions, for the moment denote the summand in (5.6.5b) by F (k_{t}, k_{t+1}). The dependence on the other parameters does not concern us for the following considerations. If k_{t} = k_{t+1} the expression F (k_{t}, k_{t+1}) = F (k_{t}, k_{t}) vanishes because of the numerator term (1 − \frac{m_{t+1}}{qm_{t}} q^{k_{t+1}−k_{t}}). If k_{t+1} ≠ k_{t} the identity

( \frac{\sqrt{q}}{q^{t}A} )^{k_{t}} ( \frac{\sqrt{q}}{q^{t+1}A} )^{k_{t+1}} \frac{1 − \frac{m_{t+1}}{qm_{t}} q^{k_{t+1}−k_{t}}}{1 − \frac{m_{t+1}}{m_{t}}} = −( \frac{\sqrt{q}}{q^{t}A} )^{k_{t+1}} ( \frac{\sqrt{q}}{q^{t+1}A} )^{k_{t}} \frac{1 − \frac{m_{t+1}}{qm_{t}} q^{k_{t}−k_{t+1}}}{1 − \frac{m_{t+1}}{m_{t}}}

shows that F (k_{t}, k_{t+1}) = −F (k_{t+1}, k_{t}). (Recall that at the moment we are assuming qm_{t} = m_{t+1}. Hence we have m_{t+1}/qm_{t} = 1.) Therefore ∑_{Ω_{T} ≥1} E_{r}(A; m, k) is zero whenever there is a t with qm_{t} = m_{t+1}, t ∈ T, and (t + 1) ∉ T.

Plugging (5.6.4) into (5.6.3) yields the identity

1 = ∑_{T ⊆{1,...,r}}(−1)^{|T |}L_{T} (A; m) · ( ∏_{l∈T} ε_{l})S_{r}(A; m) ,  (5.6.6)

where L_{T} (A; m) is the expression in (5.6.4b). The previous considerations guarantee that (5.6.6) holds for all A, m_{1}, …, m_{r} such that n_{1} > n_{2} > ⋯ > n_{r} ≥ 1.

Next it is shown that the right-hand side P (A; m) of (5.6.1) also satisfies (5.6.6). This is done by a verification. In (5.6.6) replace S(A; m) by P (A; m). After some manipulation this leads to

1 = ( ∏_{1≤i<j≤r} \frac{1 − m_{i}m_{j}/q}{1 − m_{i}m_{j}} ) (∏_{i=1}^{r} \frac{(1 − m_{i}/\sqrt{q})(1 − m_{i}/A)}{(1 − m_{i}^{2})(1 − \sqrt{q}/q^{i}A)})

× ∑_{T ⊆\{1,...,r\}}(−1)^{|T |} ∏_{j∈T} (\frac{\sqrt{q}}{q^{j}A} \frac{1 − m_{j}A}{1 − m_{j}/A})

× \frac{[( ∏_{l∈T} ε_{l})( ∏_{1≤i<j≤r}(1 − m_{j}/m_{i})(1 − m_{i}m_{j}/q) ∏_{i=1}^{r}(1 − m_{i}/\sqrt{q}))]}{∏_{1≤i<j≤r}(1 − m_{j}/m_{i})(1 − m_{i}m_{j}/q) ∏_{i=1}^{r}(1 − m_{i}/\sqrt{q})}. (5.6.7)

Multiplication of both sides of (5.6.7) by the denominator of the right-hand side gives

∏_{1≤i<j≤r}((1 − m_{i}m_{j})(m_{i} − m_{j})) ∏_{i=1}^{r} ((1 − m_{i}^{2})(1 − \sqrt{q}/q^{i}A))

= ∑_{T ⊆\{1,...,r\}}(−1)^{|T |} ∏_{j∈T} (\frac{\sqrt{q}}{q^{j}A} \frac{(1 − m_{j}A)}{(1 − m_{j}/A)}) ∏_{i=1}^{r} (m_{i}^{r−i}(1 − m_{i}/A))

× [( ∏_{l∈T} ε_{l})( ∏_{1≤i<j≤r}(1 − m_{j}/m_{i})(1 − m_{i}m_{j}/q) ∏_{i=1}^{r}(1 − m_{i}/\sqrt{q}))]

=: ∑_{T ⊆{1,...,r}} Expr(T) .  (5.6.8)

The right-hand side of (5.6.8) is a polynomial in m_{1}, m_{2}, …, m_{r} of total degree 2r + 3\binom{r}{2}. We are going to show that the right-hand side vanishes for m_{s} = m_{t}, for m_{s} = 1/m_{t}, and for m_{t} = ±1. Suppose that this is done. It follows that

∏_{1≤i<j≤r} ((1 − m_{i}m_{j})(m_{i} − m_{j})) ∏_{i=1}^{r}(1 − m_{i}^{2})  (5.6.9)

must be a factor of the right-hand side. This factor also has total degree 2r + 3\binom{r}{2} in m_{1}, m_{2}, …, m_{r}. Therefore the right-hand side of (5.6.8) is the expression in (5.6.9) times some expression which is independent of m_{1}, …, m_{r}. To get this expression, look at the coefficient of ∏_{i=1}^{r} m_{i}^{r−i} in the right-hand side of (5.6.8). It is

∑_{T ⊆\{1,...,r\}}(−1)^{|T |} ∏_{j∈T} \frac{\sqrt{q}}{q^{j}A} = ∏_{i=1}^{r}(1 − \sqrt{q}/q^{i}A) ,

just as desired.

Let m_{t} = ±1 for a fixed t. Moreover, fix a subset T_{1} of {1, 2, …, r} with t ∉ T_{1} and write T_{2} := T_{1} ∪ {t}. Then it is seen that

Expr(T_{1})|_{m_{t}=1} = − Expr(T_{2})|_{m_{t}=1} ,

and the same holding if 1 is replaced by −1. (Expr(T) was defined in (5.6.8) as the summand of the sum on the right-hand side. Expr(T)|_{m_{t}=1} means the evaluation of Expr(T) at m_{t} = 1.) Namely for m_{t} = 1,

Expr(T_{2})|_{m_{t}=1} = −(−1)^{|T_{1}|} \frac{\sqrt{q}}{q^{t}A}( ∏_{j∈T_{1}} \frac{\sqrt{q}}{q^{j}A} \frac{(1 − m_{j}A)}{(1 − m_{j}/A)}) · \frac{1 − A}{1 − 1/A}

× ( ∏_{i=1}^{r} m_{i}^{r−i}(1 − m_{i}/A))|_{m_{t}=1} × [( ∏_{l∈T_{1}} ε_{l})( ∏_{i=1}^{r}(1 − m_{i}/\sqrt{q}))

∏_{1≤i<j≤r}(1 − m_{i}/m_{j})(1 − m_{i}m_{j}/q)

\frac{1 − \sqrt{q}}{1 − 1/\sqrt{q}} ∏_{i=1}^{t−1} \frac{1 − q/m_{i}}{1 − 1/m_{i}} ∏_{i=t+1}^{r} \frac{1 − m_{i}/q}{1 − m_{i}}_{∏_{i=1}^{r} ; i≠t} \frac{1 − m_{i}}{1 − m_{i}/q} ]|_{m_{t}=1}

= − Expr(T_{1})|_{m_{t}=1} .

A similar computation can be done for m_{t} = −1. Hence for t ∉ T_{1} each two terms Expr(T_{1})|_{m_{t}=1} and Expr(T_{1} ∪ {t})|_{m_{t}=1} cancel in the sum on the right-hand side of (5.6.8) when it is evaluated at m_{t} = 1, the same being true for m_{t} = −1. Therefore it must be zero for m_{t} = ±1.

Now we turn to the case m_{s} = m_{t} for fixed s and t, 1 ≤ s < t ≤ r. This obviously only makes sense for r ≥ 2. Consider a set S_{1} ⊆ {1, …, r} with s, t ∉ S_{1}. Let us set S_{2} := S_{1} ∪ {s}, S_{3} := S_{1} ∪ {t}, and S_{4} := S_{1} ∪ {s, t}. We have

Expr(S_{2})|_{m_{s}=m_{t}} = (−1)^{|S_{3}|}q^{t−s} ∏_{j∈S_{3}} (\frac{\sqrt{q}}{q^{j}A} \frac{(1 − m_{j}A)}{1 − m_{j}/A)}) ∏_{i=1}^{r} (m_{i}^{r−i}(1 − m_{i}/A))|_{m_{s}=m_{t}}

(5.6.10a)

× [( ∏_{l∈S_{1}} ε_{l}) ∏_{1≤i<j≤r} ((1 − m_{j}/m_{i})(1 − m_{i}m_{j}/q)) ∏_{i=1}^{r}(1 − m_{i}/\sqrt{q}) (5.6.10b)

( ∏_{i=1}^{s−1} \frac{1 − qm_{s}/m_{i}}{1 − m_{s}/m_{i}}_{∏_{i=s+1}^{r} ; i≠t} \frac{1 − m_{i}/qm_{s}}{1 − m_{i}/m_{s}} )\frac{1 − m_{t}/qm_{s}}{1 − m_{t}/m_{s}}  (5.6.10c)

( ∏_{i=1}^{r} \frac{1 − m_{i}m_{s}}{1 − m_{i}m_{s}/q} )\frac{1 − m_{s}\sqrt{q}}{1 − m_{s}/\sqrt{q}} ]|_{m_{s}=m_{t}} .  (5.6.10d)

(The term (1 − m_{t}/m_{s}) in the denominator of (5.6.10c) seems to be a problem. But it is not since in fact it cancels out with a term of (5.6.10b).) The expression

∏_{i=s+1}^{r} ; i≠t \frac{1−m_{i}/qm_{s}}{1−m_{i}/m_{s}} has to be split up into

∏_{i=s+1}^{t−1} \frac{1 − m_{i}/qm_{s}}{1 − m_{i}/m_{s}} ∏_{i=t+1}^{r} \frac{1 − m_{i}/qm_{s}}{1 − m_{i}/m_{s}} .

A little bit of manipulation of the first product turns this into

q^{s−t+1} ∏_{i=s+1}^{t−1} \frac{1 − qm_{s}/m_{i}}{1 − m_{s}/m_{i}} ∏_{i=t+1}^{r} \frac{1 − m_{i}/qm_{s}}{1 − m_{i}/m_{s}} .

Moreover, we have

(1 − m_{t}/qm_{s})|_{m_{s}=m_{t}} = −\frac{1}{q}(1 − qm_{t}/m_{s})|_{m_{s}=m_{t}} .

Taking these manipulations into account, we finally obtain

Expr(S_{2})|_{m_{s}=m_{t}} = q^{t−s} · q^{s−t+1} · (−\frac{1}{q}) · Expr(S_{3})
= − Expr(S_{3}) .

Since obviously Expr(S_{1})|_{m_{s}=m_{t}} = Expr(S_{4})|_{m_{s}=m_{t}} = 0, this means that for s, t ∉ S_{1} each four terms Expr(S_{1})|_{m_{s}=m_{t}}, Expr(S_{1} ∪ {s})|_{m_{s}=m_{t}}, Expr(S_{1} ∪ {t})|_{m_{s}=m_{t}}, Expr(S_{1} ∪ {s, t})|_{m_{s}=m_{t}} cancel in the sum on the right-hand side of (5.6.8) when it is evaluated at m_{s} = m_{t}. Therefore it must be zero for m_{s} = m_{t}.

The arguments for m_{s} = 1/m_{t} are similar. This time one obtains

Expr(S_{2})|_{m_{s}=1/m_{t}} = Expr(S_{3})|_{m_{s}=1/m_{t}} = 0

and

Expr(S_{1})|_{m_{s}=1/m_{t}} = − Expr(S_{4})|_{m_{s}=1/m_{t}}.

Summarizing, we have shown that both sides of (5.6.1) satisfy the functional relation (5.6.6). To be precise, for all n_{1} > n_{2} > ⋯ > n_{r} ≥ 1 we proved that

1 = ∑_{T ⊆{1,...,r}}(−1)^{|T |}L_{T} (A; q^{−n}/A) · ( ∏_{l∈T} E_{l}^{−1})S_{r}(A; q^{−n}/A)  (5.6.11a)

and

1 = ∑_{T ⊆{1,...,r}}(−1)^{|T |}L_{T} (A; q^{−n}/A) · ( ∏_{l∈T} E_{l}^{−1})P_{r}(A; q^{−n}/A) ,  (5.6.11b)

where q^{−n}/A is a short-hand notation for q^{−n_{1}}/A, q^{−n_{2}}/A, …, q^{−n_{r}}/A. E_{l} is the usual shift operator with respect to n_{l},

(E_{l}f)(m_{1}, …, m_{l}, …, m_{r}) = f(m_{1}, …, m_{l} + 1, …, m_{r}) .

Besides, the expressions of the form S_{r}(A; q^{−n̂}/A) and P_{r}(A; q^{−n̂}/A) which are involved in the non-zero terms of (5.6.11) all satisfy n̂_{1} > n̂_{2} > ⋯ > n̂_{r} ≥ 0. Therefore induction on n_{1} + n_{2} + ⋯ + n_{r} can be performed, as soon as the validity of (5.6.1) for the initial values (n_{1}, n_{2}, …, n_{r−1}, 0), n_{1} > ⋯ > n_{r−1} > 0 is checked.

Let n_{r} = 0, which is equivalent to m_{r} = 1/A. It is clear that in this case there are nonvanishing terms in the sum at the left-hand side of (5.6.1) only if k_{r} = 0. Hence,

S_{r}(A;m_{1}, …, m_{r−1}, 1/A)

= ∑_{k_{1},...,k_{r−1}≥0} ∏_{i=1}^{r−1} (\frac{\sqrt{q}}{q^{i}A})^{k_{i}} ∏_{i=1}^{r−1} \frac{(m_{i}A)_{k_{i}}}{(qm_{i}/A)_{k_{i}}}

× ∏_{1≤i<j≤r−1} \frac{1 − \frac{m_{j}}{m_{i}} q^{k_{j}−k_{i}}}{1 − \frac{m_{j}}{m_{i}}} ∏_{1≤i≤j≤r−1} \frac{1 − m_{i}m_{j}q^{k_{i}+k_{j}}}{1 − m_{i}m_{j}}

× ∏_{i=1}^{r−1} \frac{1 − q^{−k_{i}}/m_{i}A}{1 − 1/m_{i}A} \frac{1 − m_{i}q^{k_{i}}/A}{1 − m_{i}/A}

= S_{r−1}(qA; m_{1}, …, m_{r−1})
= P_{r−1}(qA; m_{1}, …, m_{r−1})
= P_{r}(A; m_{1}, …, m_{r−1}, 1/A) .

The next to the last step used the induction hypothesis with respect to r, the last step is an easy exercise.

The proof of the Proposition now is complete. □

References

1. G. E. Andrews, The Theory of Partitions, Encyclopedia of Mathematics and its Applications, Vol. 2, Addison–Wesley, Reading, 1976.

2. G. E. Andrews, Plane partitions I: The MacMahon conjecture, Studies in foundations and combinatorics, G.-C. Rota ed., Adv. in Math. Suppl. Studies, Vol. 1, 1978, pp. 131–150.

3. G. E. Andrews, Plane partitions II: The equivalence of the Bender–Knuth and MacMahon conjectures, Pacific J. Math. 72 (1977), 283–291.

4. E. A. Bender, D. E. Knuth, Enumeration of plane partitions, J. Combin. Theory A 13 (1972), 40–54.

5. N. Bourbaki, Groupes et algèbres de Lie, Chapitres IV, V, VI, Éléments de Mathématiques, Fasc. 34, Hermann, Paris, 1968.

6. W. H. Burge, Four correspondences between graphs and generalized Young tableaux, J. Combin. Theory A 17 (1974), 12–30.

7. S. H. Choi, D. Gouyou–Beauchamps, Enumération de tableaux de Young semi-standard, Actes de 3eme Colloque de Series Formelles et Combinatoire Algebrique, Bordeaux 1991, M. Delest, G. Jacob, P. Leroux, Eds., LabRI, Université Bordeaux I, 1991, pp. 229–243.

8. L. Comtet, Advanced Combinatorics, D. Reidel, Dordrecht, Holland, 1974.

9. R. Y. Denis, R. A. Gustafson, An SU(n) q-beta integral transformation and multiple hypergeometric series identities, SIAM J. Math. Anal. 23 (1992), 552–561.

10. M. Desainte–Catherine, G. Viennot, Enumeration of certain Young tableaux with bounded height, Combinatoire énumérative, G. Labelle, P. Leroux, Eds., Springer–Verlag, Berlin, Heidelberg, New York, 1986, pp. 58–67.

11. J. Désarménien, La démonstration des identités de Gordon et MacMahon et de deux identités nouvelles, Actes de 15^{e} Séminaire Lotharingien, I. R. M. A. Strasbourg, 1987, pp. 39–49.

12. J. Désarménien, Une généralisation des fomules de Gordon et de MacMahon, Comptes Rendus Acad. Sci. Paris, Série I 309 (1989), 269–272.

13. J. Fürlinger, J. Hofbauer, q-Catalan numbers, J. Combin. Theory A 40 (1985), 248–264.

14. G. Gasper, M. Rahman, Basic hypergeometric series, Encyclopedia of Mathematics And Its Applications 35, Cambridge University Press, Cambridge, 1990.

15. I. M. Gessel, Symmetric functions and P -recusiveness, J. Combin. Theory A 53 (1990), 257–285.

16. I. M. Gessel, G. Viennot, Binomial determinants, paths, and hook length formulae, Adv. in Math. 58 (1985), 300–321.

17. I. M. Gessel, G. Viennot, Determinants, paths, and plane partitions, preprint.

18. B. Gordon, A proof of the Bender–Knuth conjecture, Pacific J. Math. 108 (1983), 99–113.

19. I. P. Goulden, A linear operator for symmetric functions and tableaux in a strip with given trace, Discrete Math. 99 (1992), 69–77.

20. S. Guha, S. Padmanabhan, A new derivation of the generating function for the major index, Discrete Math. 81 (1990), 211–215.

21. R. A. Gustafson, The Macdonald identities for affine root systems of classical type and hypergeometric series very-well-poised on semisimple Lie algebras, Ramanujan International Symposium on Analysis (Dec. 26th to 28th, 1987, Pune, India), N. K. Thakare, ed., 1989, pp. 187–224.

22. R. A. Gustafson, private communication.

23. J. E. Humphreys, Introduction to Lie algebras and representation theory, Springer–Verlag, Berlin, Heidelberg, New York, 1972.

24. D. E. Knuth, Permutations, matrices, and generalized Young tableaux, Pacific J. Math. 34 (1970), 709–727.

25. C. Krattenthaler, Enumeration of lattice paths and generating functions for skew plane partitions, Manuscripta Math. 63 (1989), 129–156.

26. C. Krattenthaler, Generating functions for plane partitions of a given shape, Manuscripta Math. 69 (1990), 173–202.

27. C. Krattenthaler, Counting lattice paths with a linear boundary, Part 2: q-ballot and q-Catalan numbers, Sitz.ber. d. ÖAW, Math-naturwiss. Klasse 198 (1989), 171–199.

28. C. Krattenthaler, Generating functions for shifted plane partitions, J. Statist. Plann. Inference 34 (1993), 197–208.

29. C. Krattenthaler, Non-crossing two-rowed arrays, in preparation; a preliminary report appeared in: Proc. of the 5th Conference on Formal Power Series and Algebraic Combinatorics, Florence, 1993 (A. Barlotti, M. Delest, R. Pinzani, eds.), D.S.I., Università di Firenze, pp. 301–314.

30. G. M. Lilly, S. C. Milne, The C_{l} Bailey transform and Bailey Lemma, Constructive Approximation 9 (1993), 473–500.

31. G. M. Lilly, S. C. Milne, Consequences of the A_{l} and C_{l} Bailey Transform and Bailey Lemma, Proceedings of the 4th Colloque de Series Formelles et Combinatoire Algebrique, Montreal 1992, P. Leroux, C. Reutenauer, eds., LaCIM, Université Québec à Montréal, 1992, pp. 297–312.

32. P. A. MacMahon, Combinatory Analysis, Vol. 2, (Cambridge University Press, Cambridge, 1916; reprinted by Chelsea, New York, 1960).

33. I. G. Macdonald, Symmetric Functions and Hall Polynomials, Oxford University Press, New York/London, 1979.

34. S. C. Milne, The multidimensional _{1}ψ_{1} sum and Macdonald identities for A_{l}^{(1)}, Theta Functions Bourdoin 1987, L. Ehrenpreis and R. C. Gunning, eds., Prooceedings of Symposia in Pure Mathematics, Vol. 49, Part 2, 1989, pp. 323–359.

35. S. C. Milne, Classical partition functions and the U(n + 1) Rogers–Selberg identity, Discrete Math. 99 (1992), 199–246.

36. R. A. Proctor, Bruhat lattices, plane partitions generating functions, and minuscule representations, Europ. J. Combin. 5 (1984), 331–350.

37. R. A. Proctor, New symmetric plane partition identities from invariant theory work of DeConcini and Procesi, Europ. J. Combin. 11 (1990), 289–300.

38. B. E. Sagan, The symmetric group, Wadsworth & Brooks/Cole, Pacific Grove, California, 1991.

39. R. P. Stanley, Theory and applications of plane partitions: Part 1,2, Stud. Appl. Math 50 (1971), 167–188, 259–279.

40. J. Stembridge, Hall–Littlewood functions, plane partitions and the Rogers–Ramanujan identities, Trans. Amer. Math. Soc. 319 (1990), 469–498.

41. J. Stembridge, Nonintersecting lattice paths, pfaffians and plane partitions, Adv. Math. 83 (1990), 96–131.

42. G. Viennot, Une forme géométrique de la correspondance de Robinson–Schensted, Combinatoire et Représentation du Groupe Symétrique, D. Foata ed., Lecture Notes in Math., Vol. 579, Springer–Verlag, New York, 1977, pp. 29–58.