The number of rhombus tilings of a “punctured”

hexagon and the minor summation formula

S. Okada^{†} and C. Krattenthaler^{†}

Graduate School of Polymathematics, Nagoya University Furo-cho, Chikusa-ku, Nagoya 464-01, Japan. email: okada@math.nagoya-u.ac.jp

Institut für Mathematik der Universität Wien, Strudlhofgasse 4, A-1090 Wien, Austria. e-mail: KRATT@Pap.Univie.Ac.At

             WWW: http://radon.mat.univie.ac.at/People/kratt

Abstract. We compute the number of all rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed, provided a, b, c have the same parity. The result is B(⌈ \frac{a}{2} ⌉, ⌈ \frac{b}{2} ⌉, ⌈ \frac{c}{2} ⌉)B(⌈ \frac{a+1}{2} ⌉, ⌊ \frac{b}{2} ⌋, ⌈ \frac{c}{2} ⌉)B(⌈ \frac{a}{2} ⌉, ⌈ \frac{b+1}{2} ⌉, ⌊ \frac{c}{2} ⌋)B(⌊ \frac{a}{2} ⌋, ⌈ \frac{b}{2} ⌉, ⌈ \frac{c+1}{2} ⌉), where B(α, β, γ) is the number of plane partitions inside the α × β × γ box. The proof uses nonintersecting lattice paths and a new identity for Schur functions, which is proved by means of the minor summation formula of Ishikawa and Wakayama. A symmetric generalization of this identity is stated as a conjecture.

1. Introduction. In recent years, the enumeration of rhombus tilings of various regions has attracted a lot of interest and was intensively studied, mainly because of the observation (see [7]) that the problem of enumerating all rhombus tilings of a hexagon with sides a, b, c, a, b, c (see Figure 1; throughout the paper by a rhombus we always mean a rhombus with side lengths 1 and angles of 60^{∘} and 120^{∘}) is another way of stating the problem of counting all plane partitions inside an a × b × c box. The latter problem was solved long ago by MacMahon [9, Sec. 429, q → 1, proof in Sec. 494]. Therefore:

The number of all rhombus tilings of a hexagon with sides a, b, c, a, b, c equals

B(a, b, c) = ∏_{i=1}^{a} ∏_{j=1}^{b} ∏_{k=1}^{c} \frac{i + j + k − 1}{i + j + k − 2} .  (1.1)

1991 Mathematics Subject Classification. Primary 05A15; Secondary 05A17 05A19 05B45 05E05 52C20.

Key words and phrases. rhombus tilings, lozenge tilings, plane partitions, nonintersecting lattice paths, Schur functions.

†Research supported in part by the MSRI, Berkeley.

a. A hexagon with sides a, b, c, a, b, c,_{where a = 3, b = 4, c = 5}  _{b. A rhombus tiling of a hexagon}_{with sides a, b, c, a, b, c} }c } } c{  }  } a  b a b

Figure 1

(The form of the expression is due to Macdonald.)

In his preprint [11], Propp proposes several variations of this enumeration problem, one of which (Problem 2) asks for the enumeration of all rhombus tilings of a hexagon with sides n, n + 1, n, n + 1, n, n + 1, of which the central triangle is removed (a “punctured hexagon”). At this point, this may seem to be a somewhat artificial problem. But sure enough, soon after, an even more general problem, namely the problem of enumerating all rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed (see Figure 2), occured in work of Kuperberg related to the Penrose impossible triangle [private communication].

The purpose of this paper is to solve this enumeration problem. (We want to mention that Ciucu [14] has an independent solution of the b = c case of the problem, which builds upon his matchings factorization theorem [1].) Our result is the following:

Theorem 1. Let a, b, c be positive integers, all of the same parity. Then the number of all rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed, equals

B (⌈\frac{a}{2}⌉ , ⌈\frac{b}{2}⌉ , ⌈\frac{c}{2}⌉) B (⌈\frac{a + 1}{2}⌉ , ⌊\frac{b}{2}⌋ , ⌈\frac{c}{2}⌉)

× B (⌈\frac{a}{2}⌉ , ⌈\frac{b + 1}{2}⌉ , ⌊\frac{c}{2}⌋) B (⌊\frac{a}{2}⌋ , ⌈\frac{b}{2}⌉ , ⌈\frac{c + 1}{2}⌉) , (1.2)

A rhombus tiling of a “punctured” hexagon_{with sides a, b + 1, c, a + 1, b, c + 1, where a = 3, b = 5, c = 5} }c } }_{c + 1{}  }  } a  b + 1 a + 1 b

Figure 2

where B(α, β, γ) is the number of all plane partitions inside the α × β × γ box, which is given by (1.1).

We prove this Theorem by first converting the tiling problem into an enumeration problem for nonintersecting lattice paths (see Section 2), deriving a certain summation for the desired number (see Section 3, Proposition 2), and then evaluating the sum by proving (in Section 4) an actually much more general identity for Schur functions (Theorem 3). The summation that we are interested in then follows immediately by setting all variables equal to 1. In order to prove this Schur function identity, which we do in Section 5, we make essential use of the minor summation formula of Ishikawa and Wakayama [4] (see Theorem 7). In fact, it appears that a symmetric generalization of this Schur function identity (Conjecture 5 in Section 4) holds. However, so far we were not able to establish this identity. Also in Section 4, we add a further enumeration result, Theorem 4, on rhombus tilings of a “punctured” hexagon, which follows from a different specialization of Theorem 3.

In conclusion of the Introduction, we wish to point out a few interesting features of our result in Theorem 1.

(1) First of all, it is another application of the powerful minor summation formula of Ishikawa and Wakayama. For other applications see [5, 6, 10].

(2) A very striking fact is that this problem, in the formulation of nonintersecting lattice paths, is the first instance of existence of a closed form enumeration, despite the fact that the starting points of the paths are not in the “right” order. That is, the “compatability” condition for the starting and end points in the main theorem on nonintersecting lattice paths [3, Cor. 2; 13, Theorem 1.2] is violated (compare Figure 3.c) and the main theorem does therefore not apply. (The “compatability” condition is the requirement that whenever we consider starting points A_{i}, A_{j}, with i < j, and end points E_{k}, E_{l}, with k < l, then any path from A_{i} to E_{l} has to touch any path from A_{j} to E_{k}.) This is the reason that we had to resort to something else, which turned out to be the minor summation formula.

(3) The result (1.2) is in a very appealing combinatorial form. It is natural to ask for a bijective proof of the formula, i.e., for setting up a one-to-one correspondence between the rhombus tilings in question and a quadruple of plane partitions as described by the product in (1.2). However, this seems to be a very difficult problem. In particular, how does one split a rhombus tiling of a hexagon into four objects?

(4) In the course of our investigations, we discovered the symmetric generalization, Conjecture 5, of Theorem 3 which we mentioned before. Although the minor summation formula still applies, we were not able to carry out the subsequent steps, i.e., to find the appropriate generalizations of Lemmas 9 and 10.

2. From rhombus tilings to nonintersecting lattice paths. In this section we translate our problem of enumerating rhombus tilings of a “punctured” hexagon into the language of nonintersecting lattice paths. Throughout this section we assume that a, b, c are positive integers, all of the same parity.

We use a slight (but obvious) modification of the standard translation from rhombus tilings of a hexagon to nonintersecting lattice paths. Given a rhombus tiling of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed (see Figure 2), we mark the centres of the a edges along the side of length a, we mark the centres of the a+1 edges along the side of length a+1, and we mark the centre of the edge of the removed triangle that is parallel to the sides of respective lengths a and a + 1 (see Figure 3.a; the marked points are indicated by circles). Starting from each of the marked points on the side of length a, we form a lattice path by connecting the marked point with the centre of the edge opposite to it, the latter centre with the centre of the edge opposite to it, etc. (see Figure 3.a,b; the paths are indicated by broken lines). For convenience, we deform the picture so that the slanted edges of the paths become horizontal (see Figure 3.c). Thus we obtain a family of lattice paths in the integer lattice ℤ^{2}, consisting of horizontal unit steps in the positive direction and vertical unit steps in the negative direction, with the property that no two of the lattice paths have a point in common. In the sequel, whenever we use the term “lattice path” we mean a lattice path consisting of horizontal unit steps in the positive direction and vertical unit steps in the negative direction. As is usually done, we call a family of lattice paths with the property that no two of the lattice paths have a point in common nonintersecting.

It is easy to see that this correspondence sets up a bijection between rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed, and families (P_{1}, P_{2}, …, P_{a+1}) of nonintersecting lattice paths, where for i = 1, 2 …, a the path P_{i} runs from A_{i} = (i−1, c+i) to one of the points E_{j} = (b+j−1, j−1), j = 1, 2, …, a+1, and

a. The paths associated to a rhombus tiling  b. The paths, isolated

•  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•} ∘ ∘ ∘  ∘ ∘ ∘ ∘ ∘^{P_{1}} ^{P_{2}}  ^{P_{3}}^{P_{4}}^{A_{1}} ^{A_{2}} ^{A_{3}}  ^{A_{4}}  _{E_{1}} ^{E_{2}} ^{E_{3}} ^{E_{4}}

c. After deformation: A family of nonintersecting lattice paths Figure 3

where P_{a+1} runs from A_{a+1} = ((a+b)/2, (a+c)/2) to one of the points E_{j} = (b+j−1, j−1), j = 1, 2, …, a + 1. It is the latter enumeration problem that we are going to solve in the subsequent sections.

As mentioned in the Introduction, this enumeration problem for nonintersecting lattice paths is unusual, as the starting points are not lined up in the “right” way (in this case this would mean from bottom-left to top-right), the (a + 1)-st starting point being located in the middle of the region. So, it is not fixed which starting point is connected to which end point. In our example of Figure 3.c, A_{1} is connected with E_{1}, A_{2} is connected with E_{2}, A_{3} is connected with E_{4}, and the “exceptional” starting point A_{4} is connected with E_{3}. But there are other possibilities. In general, for any k, 1 ≤ k ≤ a + 1, it is possible that A_{1} is connected with E_{1}, …, A_{k} is connected with E_{k}, A_{k+1} is connected with E_{k+2}, …, A_{a} is connected with E_{a+1}, and the “exceptional” starting point A_{a+1} is connected with E_{k+1}. To obtain the solution of our enumeration problem, we have to find the number of all nonintersecting lattice paths in each case and then form the sum over all k.

3. From nonintersecting lattice paths to Schur functions. The aim of this section is to describe, starting from the interpretation of our problem in terms of nonintersecting lattice paths that was derived in the previous section, how to derive an expression for the number of rhombus tilings that we are interested in. This expression, displayed in (3.4), features (specialized) Schur functions.

Usually, when enumerating nonintersecting lattice paths, one obtains a determinant by means of the main theorem on nonintersecting lattice paths [3, Cor. 2; 13, Theorem 1.2]. However, as mentioned in the Introduction, the arrangement of the starting points in our problem, as described in the previous section, does not allow a direct application of this theorem.

•  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}  _{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•}  ^{•} ∘ ∘ ∘  ∘ ∘ ∘ ∘ ∘ • • • •  • • • •^{P_{1}} ^{P_{2}}  ^{P_{3}}^{P_{4}} Figure 4

In order to access the problem, as formulated at the end of the previous section, we set it up as follows. Each of the lattice paths P_{i}, i = 1, 2, …, a, has to pass through a lattice point on the diagonal line x−y = (b−c)/2, the central point A_{a+1} = ((a+b)/2, (a+c)/2) (which is the starting point for path P_{a+1}) excluded (see Figure 4; the points on the diagonal where the paths may pass through are indicated by bold dots). So, as in the previous section, let k be an integer between 1 and a + 1. For any fixed choice of points M_{1}, …, M_{k}, M_{k+2}, …, M_{a+1} on the diagonal x − y = (b − c)/2, such that M_{1} is to the left of M_{2}, …, M_{k} is to the left of the central point A_{a+1}, A_{a+1} is to the left of M_{k+2}, …, M_{a} is to the left of M_{a+1}, the number of families (P_{1}, P_{2}, …, P_{a+1}) of nonintersecting lattice paths, where for i = 1, 2 …, k the path P_{i} runs from A_{i} through M_{i} to E_{i}, where for i = k + 1, k + 2, …, a the path P_{i} runs from A_{i} through M_{i+1} to E_{i+1}, and where P_{a+1} runs from A_{a+1} to E_{k+1}, is easily computed using the main theorem on nonintersecting lattice paths. For convenience, let M_{k+1} = A_{a+1}. Then this number equals

det_{1≤i,j≤a} (|P(A_{i} → M_{j+χ(j≥k+1)})|) · det_{1≤i,j≤a+1} (|P(M_{i} → E_{j})|) ,  (3.1)

where |P(A → B)| denotes the number of all lattice paths from A to B, and where χ is the usual truth function, χ(A) = 1 if A is true and χ(A) = 0 otherwise.

It is now no difficulty to observe that the expression (3.1) can be rewritten using (specialized) Schur functions. For information on Schur functions and related definitions we refer the reader to Chapter I of Macdonald’s classical book [8]. There are many ways to express a Schur function. The one we need here is the Nägelsbach–Kostka formula (the “dual” Jacobi-Trudi identity; see [8, Ch. I, (3.5)]). Let λ be a partition with largest part at most m. Then the Schur function s_{λ}(x_{1}, x_{2}, …, x_{n}) is given by

s_{λ}(x_{1}, x_{2}, …, x_{n}) = det_{1≤i,j≤m} (e_{(^{t}λ)_{i}−i+j}(x_{1}, x_{2}, …, x_{n})),

where e_{s}(x_{1}, …, x_{n}) := ∑_{1≤i_{1}<···<i_{s}≤n} x_{i_{1}} ⋯ x_{i_{s}} is the elementary symmetric function of order s in x_{1}, …, x_{n}, and where ^{t}λ denotes the partition conjugate to λ. We write briefly X_{n} for the set of variables {x_{1}, x_{2}, …, x_{n}}. In particular, the symbol s_{λ}(X_{n}) will be short for s_{λ}(x_{1}, x_{2}, …, x_{n}).

Now suppose that M_{ℓ} = ((a + b)/2 + i_{ℓ}, (a + c)/2 + i_{ℓ}), ℓ = 1, 2, …, a + 1. In particular, since M_{k+1} = A_{a+1} = ((a + b)/2, (a + c)/2), we have i_{k+1} = 0. The ordering of the points M_{ℓ} from left to right as ℓ increases implies −(a + b)/2 ≤ i_{1} < ⋯ < i_{k} < i_{k+1} = 0 < i_{k+2} < ⋯ < i_{a+1} ≤ (a + b)/2. Then it is straight-forward to see that, using the above notation, the expression (3.1) equals

s_{λ}(X_{(b+c+2)/2}) · s_{μ}(X_{(b+c)/2})|_{x_{1}=x_{2}=···=x_{(b+c+2)/2}=1} ,  (3.2)

where λ and μ are the partitions whose conjugates ^{t}λ and ^{t}μ are given by

\begin{aligned}^{t}λ_{h} & = \frac{b − a}{2} + i_{a+1−h+χ(a+1−h≥k+1)} + h & for h = 1, …, a, \\ ^{t}μ_{h} & = \frac{b − a}{2} − i_{h} + h − 1 & for h = 1, …, a + 1. \end{aligned}  (3.3)

(In view of the geometric interpretation of the Nägelsbach–Kostka formula in terms of nonintersecting lattice paths, see [2, Sec. 4, Fig. 8; 12, Sec. 4.5], this could also be read off directly from the lattice path picture of our setup, as exemplified by Figure 4.)

According to the preceding considerations, what we would like to do in order to find the total number of all rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed, is to sum the products (3.2) of Schur functions over all possible choices of λ and μ. That is, we want to find the sum

∑_{(λ,μ)} (s_{λ}(X_{(b+c+2)/2}) · s_{μ}(X_{(b+c)/2}))|_{x_{1}=x_{2}=···=x_{(b+c+2)/2}=1} ,

where (λ, μ) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some k and i_{1}, i_{2}, …, i_{a+1} as above.

Summarizing, we have shown the following.

Proposition 2. Let a, b, c be positive integers, all of the same parity. Then the number of all rhombus tilings of a hexagon with sides a, b + 1, c, a + 1, b, c + 1, of which the central triangle is removed, equals

∑_{(λ,μ)} (s_{λ}(X_{(b+c+2)/2}) · s_{μ}(X_{(b+c)/2}))|_{x_{1}=x_{2}=···=x_{(b+c+2)/2}=1} ,  (3.4)

where (λ, μ) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some k with 1 ≤ k ≤ a + 1, and for some i_{1}, i_{2}, …, i_{a+1} with −(a + b)/2 ≤ i_{1} < ⋯ < i_{k} < i_{k+1} = 0 < i_{k+2} < ⋯ < i_{a+1} ≤ (a + b)/2.

4. The main theorem and its implications. As a matter of fact, even the unspecialized sum that appears in (3.4) can be evaluated in closed form. This is the subject of the subsequent Theorem 3, which is the main theorem of our paper. Theorem 1 of the Introduction then follows immediately. Before we proceed to the proof of Theorem 3 in Section 5, in this section we formulate a conjectured generalization of Theorem 3 as Conjecture 5, and we complement the enumeration result in Theorem 1 by a further result (Theorem 4) on the enumeration of rhombus tilings of a “punctured” hexagon, which follows also from Theorem 3.

Theorem 3. Let a, b, n be positive integers, where a and b are of the same parity. Then

∑_{(λ,μ)} s_{λ}(X_{n+1}) · s_{μ}(X_{n})

= s_{(⌈(a+1)/2⌉^{⌊b/2⌋})}(X_{n}) · s_{(⌈a/2⌉^{⌈b/2⌉})}(X_{n}) · s_{(⌈a/2⌉^{⌈b/2⌉})}(X_{n+1}) · s_{(⌊a/2⌋^{⌈(b+1)/2⌉})}(X_{n+1}), (4.1)

where (λ, μ) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some k with 1 ≤ k ≤ a + 1, and for some i_{1}, i_{2}, …, i_{a+1} with −(a + b)/2 ≤ i_{1} < ⋯ < i_{k} < i_{k+1} = 0 < i_{k+2} < ⋯ < i_{a+1} ≤ (a + b)/2.

Theorem 1 follows immediately from Proposition 2 and the n = (b + c)/2 special case of Theorem 3 by using the well-known fact (see [8, Ch. I, Sec. 5, Ex. 13.(b), q → 1]) that the evaluation of a Schur function s_{(A^{B})}(X_{n}) of rectangular shape at x_{1} = x_{2} = ⋯ = x_{n} = 1 counts the number of all plane partitions inside the A × B × (n − B) box.

A different specialization of Theorem 3 leads to an enumeration result for rhombus tilings of a “punctured” hexagon with sides a, b + 1, c, a + 1, b, c + 1 where not all of a, b, c are of the same parity. Namely, on setting x_{1} = x_{2} = ⋯ = x_{n} = 1, x_{n+1} = 0, n = (b + c)/2 in Theorem 3, and on finally replacing c by c + 1, we obtain the following result by performing the analogous translations that lead from rhombus tilings to nonintersecting lattice paths and finally to specialized Schur functions.

A “punctured” hexagon with sides a, b + 1, c, a + 1, b, c + 1,_{c being of parity different from the parity of a and b} }c } }_{c + 1{}  }  } a  b + 1 a + 1 b

Figure 5

Theorem 4. Let a, b, c be positive integers, a and b of the same parity, c of different parity. From the hexagon with sides a, b + 1, c, a + 1, b, c + 1 the triangle is removed, whose side parallel to the sides of lengths a and a + 1 lies on the line that is equidistant to these sides, whose side parallel to the sides of lengths b and b + 1 lies on the line that is by two “units” closer to the side of length b + 1 than to the side of length b, and whose side parallel to the sides of lengths c and c + 1 lies on the line that is by one “unit” closer to the side of length c + 1 than to the side of length c, see Figure 5. Then the number of all rhombus tilings of this “punctured” hexagon with sides a, b + 1, c, a + 1, b, c + 1 equals

B (⌊\frac{a + 2}{2}⌋ , ⌊\frac{b}{2}⌋ , ⌊\frac{c + 2}{2}⌋) B (⌊\frac{a + 1}{2}⌋ , ⌊\frac{b + 1}{2}⌋ , ⌊\frac{c + 1}{2}⌋)^{2} B (⌊\frac{a}{2}⌋ , ⌊\frac{b + 2}{2}⌋ , ⌊\frac{c}{2}⌋) ,

(4.2)

where B(α, β, γ) is the number of all plane partitions inside the α × β × γ box, which is given by (1.1).

Even if Theorem 3 suffices to prove Theorem 1 (and Theorem 4), it is certainly of interest that apparently there holds a symmetric generalization of Theorem 3. We have overwhelming evidence through computer computations that actually the following is true.

Conjecture 5. Let a, b, n be positive integers, where a and b are of the same parity. Then

∑_{(λ,μ)} s_{λ}(X_{n+2}) · s_{μ}(X_{n}) = s_{(⌈(a+1)/2⌉^{⌊b/2⌋})}(X_{n}) · s_{(⌈a/2⌉^{⌈b/2⌉})}(X_{n} ∪ {x_{n+1}})

× s_{(⌈a/2⌉^{⌈b/2⌉})}(X_{n} ∪ {x_{n+2}}) · s_{(⌊a/2⌋^{⌈(b+1)/2⌉})}(X_{n+2}), (4.3)

where (λ, μ) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some k with 1 ≤ k ≤ a + 1, and for some i_{1}, i_{2}, …, i_{a+1} with −(a + b)/2 ≤ i_{1} < ⋯ < i_{k} < i_{k+1} = 0 < i_{k+2} < ⋯ < i_{a+1} ≤ (a + b)/2.

We remark that the specialization n = (b + c)/2, x_{1} = x_{2} = ⋯ = x_{(b+c+4)/2} = 1 of Conjecture 5 leads, up to parameter permutation, again to Theorem 4, thus providing further evidence for the truth of the Conjecture.

5. The minor summation formula and proof of the main theorem. This final section is devoted to a proof of Theorem 3. It makes essential use of the minor summation formula of Ishikawa and Wakayama. An outline of the proof is as follows. First, the minor summation formula is used to convert the sum on the left-hand side of (4.1) into a Pfaffian. This Pfaffian can be easily reduced to a determinant. In Lemma 9 it is seen that this determinant factors into a product of two Pfaffians. Finally, Lemma 10 shows that each of these Pfaffians is, basically, a product of two Schur functions, so that the final form of the result, as given by the right-hand side of (4.1), follows from a simple computation, by which we conclude this section.

To begin with, let us recall the minor summation formula due to Ishikawa and Wakayama [4, Theorem 2].

Theorem 7. Let n, p, q be integers such that n + q is even and 0 ≤ n − q ≤ p. Let G be any n × p matrix, H be any n × q matrix, and A = (a_{ij})_{1≤i,j≤p} be any skew-symmetric matrix. Then we have

∑_{K} Pf (A_{K}^{K}) det (G_{K} ⋮ H) = (−1)^{q(q−1)/2} Pf ( \begin{matrix}G A ^{t}G & H \\ −^{t}H & 0 \end{matrix} ) ,  (5.1)

where K runs over all (n − q)-element subsets of [1, p], A_{K}^{K} is the skew-symmetric matrix obtained by picking the rows and columns indexed by K and G_{K} is the sub-matrix of G consisting of the columns corresponding to K.

In order to apply this formula, we have to first describe the pairs (λ, μ) of partitions over which the sum in (4.1) is taken directly. This is done in Lemma 8 below. The reader should be reminded that the description that the formulation of Theorem 3 gives is in terms of the conjugates of λ and μ (compare (3.3)), which is not suitable for application of the minor summation formula (5.1).

For convenience, let R(a, b) be the set of all pairs (λ, μ) of partitions satisfying (3.3), for some k with 1 ≤ k ≤ a + 1, and for some i_{1}, i_{2}, …, i_{a+1} with −(a + b)/2 ≤ i_{1} < ⋯ < i_{k} < i_{k+1} = 0 < i_{k+2} < ⋯ < i_{a+1} ≤ (a + b)/2.

First note that l(λ) ≤ b + 1 and l(μ) ≤ b for (λ, μ) ∈ R(a, b).

Lemma 8. If we define subsets J = {j_{1} < ⋯ < j_{b+1}} and J′ = {j′_{1} < ⋯ < j′_{b}} by the relations

j_{h} = λ_{b+2−h} + h − 1 for h = 1, …, b + 1,
j′_{h} = μ_{b+1−h} + h − 1 for h = 1, …, b,

then we have (1) (a + b)/2 ∈ J.

(2) J′ = {a + b − j : j ∈ J, j ≠ (a + b)/2}.

Proof. (1) Since i_{k+2} ≥ 1 and i_{k} ≤ −1, we have

tλ_{a−k} = \frac{b − a}{2} + i_{k+2} + a − k ≥ \frac{a + b}{2} − k + 1,
tλ_{a−k+1} = \frac{b − a}{2} + i_{k} + a − k + 1 ≤ \frac{a + b}{2} − k.

Hence we obtain λ_{(a+b)/2−k+1} = a − k, which means j_{(b−a)/2+k+1} = (a + b)/2. (2) It is enough to show that

λ_{b+2−h} + μ_{h} = a + 1 for h = 1, …, \frac{b − a}{2} + k,

λ_{b+2−h} + μ_{h−1} = a for h = \frac{b − a}{2} + k + 2, …, b + 1.

By (3.3), the parts of λ and μ can be expressed in terms of the i_{j}’s:

λ_{b+2−h} = # \{j : i_{a+1−j+χ(a+1−j≥k+1)} ≥ \frac{a + b}{2} + 2 − h − j\} ,
μ_{h} = # \{j : i_{j} ≤ \frac{b − a}{2} − 1 − h + j\} .

If h ≤ (b − a)/2 + k, then i_{k} < (a + b)/2 + 2 − h − (a + 1 − k) and so we have λ_{b+2−h} ≤ a − k. This implies

λ_{b+2−h} = # \{j : i_{j} ≥ \frac{b − a}{2} − h + j\}

and λ_{b+2−h} + μ_{h} = a + 1. Similarly, since μ_{h−1} ≤ k for h ≥ (b − a)/2 + k + 2, we have

λ_{b+2−h} = # \{j : i_{j+χ(j≥k+1)} ≥ \frac{b − a}{2} − h + j + 1\} ,
μ_{h−1} = # \{j : i_{j+χ(j≥k+1)} ≤ \frac{b − a}{2} − h + j\} ,

and λ_{b+2−h} + μ_{h−1} = a. □

Now we describe our choices of matrices G, H and A for our application of the minor summation formula in Theorem 7. For a subset I = {i_{1} < ⋯ < i_{p}} of nonnegative integers, let M_{I}(X_{n}) be the n × p matrix with (k, l) entry x_{k}^{i_{l}}. We define three subsets P, Q, R as follows:

P = \{n − b, n − b + 1, …, n − b + \frac{a + b}{2} − 1, n − b + \frac{a + b}{2} + 1, …, n − b + (a + b)\} ,
Q = \{0, 1, …, n − b − 1, n − b + \frac{a + b}{2}\} ,
R = {0, 1, …, n − b − 1}.

Then let the matrices G and H be given by

G = ( \begin{matrix}M_{P} (X_{n+1}) & 0 \\ 0 & M_{P} (X_{n}) \end{matrix} ) ,
H = ( \begin{matrix}M_{Q}(X_{n+1}) & 0 \\ 0 & M_{R}(X_{n}) \end{matrix} ) .

Furthermore, we define

Γ = \{0, 1, …, \frac{a + b}{2} − 1, \frac{a + b}{2} + 1, …, a + b\} .

Let A be the skew-symmetric matrix whose rows and columns are indexed by the set

Γ ∪ \overline{Γ} = Γ ∪ \{\overline{k} : k ∈ Γ\}

and whose nonzero entries are given by

a_{k,\overline{a+b−k}} = \{ \begin{matrix}1 & if 1 ≤ k ≤ (a + b)/2 − 1 \\ −1 & if (a + b)/2 + 1 ≤ k ≤ a + b. \end{matrix}

We apply Theorem 7 to these matrices G, H and A.

For a 2b-element subset K of Γ ∪ \overline{Γ}, the sub-Pfaffian Pf(A_{K}^{K}) is easily computed. This sub-Pfaffian vanishes unless the number of the unbarred elements in K is equal to that of the barred elements in K. If K = \{j_{1}, …, j_{b}, \overline{j′_{1}}, …, \overline{j′_{b}}\}, then we have

Pf(A_{K}^{K}) = \{ \begin{matrix}(−1)^{#\{h:j_{h}≥(a+b)/2+1\}} & if j_{h} + j′_{b+1−h} = a + b \\ 0 & otherwise. \end{matrix}

Now recall the bideterminantal expression for Schur functions (see [8, Ch. I, (3.1)]),

s_{λ}(X_{n}) = s_{λ}(x_{1}, x_{2}, …, x_{n}) = \frac{det_{1≤i,j≤n}(x_{j}^{λ_{i}+n−i})}{det_{1≤i,j≤n}(x_{j}^{n−i})} = \frac{det_{1≤i,j≤n}(x_{j}^{λ_{i}+n−i})}{Δ(X_{n})}.  (5.2)

Then for a subset K as above, we have

det(G_{K} ⋮ H) = (−1)^{2b(n−b)+b+#{h:j_{h}≥(a+b)/2+1}+b(n−b)}s_{λ}(X_{n+1})s_{μ}(X_{n})Δ(X_{n+1})Δ(X_{n})
= (−1)^{bn+#{h:j_{h}≥(a+b)/2+1}}s_{λ}(X_{n+1})s_{μ}(X_{n})Δ(X_{n+1})Δ(X_{n}),

where λ (resp. μ) is the partition corresponding to the subset J = {j_{1}, …, j_{b}, (a + b)/2} (resp. J′ = {a + b − j_{1}, …, a + b − j_{b}}) and Δ(X_{n}) = ∏_{1≤i<j≤n}(x_{j} − x_{i}). Therefore, if we apply the minor summation formula in Theorem 7, we obtain

∑_{(λ,μ)∈R(a,b)} s_{λ}(X_{n+1})s_{μ}(X_{n}) = \frac{(−1)^{bn}}{Δ(X_{n+1})Δ(X_{n})} ∑_{K⊂Γ∪\overline{Γ},|K|=2b} Pf(A_{K}^{K}) det(G_{K} ⋮ H)

= \frac{(−1)^{bn+(2n−2b+1)(2n−2b)/2}}{Δ(X_{n+1})Δ(X_{n})} Pf ( \begin{matrix}G A ^{t}G & H \\ −^{t}H & 0 \end{matrix} )

= \frac{(−1)^{bn+n−b}}{Δ(X_{n+1})Δ(X_{n})} Pf ( \begin{matrix}G A ^{t}G & H \\ −^{t}H & 0 \end{matrix} ) .  (5.3)

If we write A = ( \begin{matrix}0 & B \\ B & 0 \end{matrix} ), then we have

( \begin{matrix}G A ^{t}G & H \\ −^{t}H & 0 \end{matrix} )

= ( \begin{matrix}0 & M_{P} (X_{n+1}) B ^{t}M_{P} (X_{n}) & M_{Q}(X_{n+1}) & 0 \\ −M_{P} (X_{n}) B ^{t}M_{P} (X_{n+1}) & 0 & 0 & M_{R}(X_{n}) \\ −^{t}M_{Q}(X_{n+1}) & 0 & 0 & 0 \\ 0 & −^{t}M_{R}(X_{n}) & 0 & 0 \end{matrix} ) .

By permuting rows and columns by the permutation

( \begin{matrix}1 & ⋯ & n + 1 & n + 2 & ⋯ & 3n − b + 2 & 3n − b + 3 & ⋯ & 4n − 2b + 2 \\ 1 & ⋯ & n + 1 & 2n − b + 2 & ⋯ & 4n − 2b + 2 & n + 2 & ⋯ & 2n − b + 1 \end{matrix} ) ,

we have

Pf ( \begin{matrix}G A ^{t}G & H \\ −^{t}H & 0 \end{matrix} )

= (−1)^{(2n−b+1)(n−b)}

× Pf ( \begin{matrix}0 & 0 & M_{P} (X_{n+1}) B ^{t}M_{P} (X_{n}) & M_{Q}(X_{n+1}) \\ 0 & 0 & −^{t}M_{R}(X_{n}) & 0 \\ −M_{P} (X_{n}) B ^{t}M_{P} (X_{n+1}) & M_{R}(X_{n}) & 0 & 0 \\ −^{t}M_{Q}(X_{n+1}) & 0 & 0 & 0 \end{matrix} )

= (−1)^{bn−n+(2n−b+1)(2n−b)/2} det ( \begin{matrix}M_{P} (X_{n+1}) B ^{t}M_{P} (X_{n}) & M_{Q}(X_{n+1}) \\ −^{t}M_{R}(X_{n}) & 0 \end{matrix} )

= (−1)^{bn+b(b−1)/2+(n−b)} det ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & M_{\{n−b+(a+b)/2\}}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 & 0 \\ N(x_{n+1}, X_{n}) & M_{R}(x_{n+1}) & x_{n+1}^{n−b+(a+b)/2} \end{matrix} ) (5.4)

where

N(X_{m}, Y_{n}) = M_{P} (X_{m}) B ^{t}M_{P} (Y_{n}).

By direct computation, the (i, j)-entry of N(X_{m}, Y_{n}) is equal to

\frac{x_{i}^{n−b}y_{j}^{n−b}(y_{j}^{(a+b)/2} − x_{i}^{(a+b)/2})(y_{j}^{(a+b)/2+1} − x_{i}^{(a+b)/2+1})}{y_{j} − x_{i}}.

Here, the last determinant in (5.4) can be decomposed into the product of two Pfaffians by using the following lemma.

Lemma 9. Let A be an n × n skew-symmetric matrix, b = ^{t}(b_{1}, …, b_{n}) and c = t(c_{1}, …, c_{n}) be column vectors, and d a scalar. Then the determinant of the (n+1)×(n+1) matrix

Ã = ( \begin{matrix}A & b \\ −^{t}c & d \end{matrix} )

decomposes into the product of two Pfaffians as follows:

(1) If n is even, then

det Ã = − Pf(A) Pf ( \begin{matrix}A & b & c \\ −^{t}b & 0 & −d \\ −^{t}c & d & 0 \end{matrix} ) .  (5.5)

(2) If n is odd, then

det Ã = Pf ( \begin{matrix}A & b \\ −^{t}b & 0 \end{matrix} ) Pf ( \begin{matrix}A & c \\ −^{t}c & 0 \end{matrix} ) .  (5.6)

Proof. Expanding along the last column and the bottom row, we see that

det Ã = ∑_{i,j=1}^{n}(−1)^{n+1+i−2}b_{i} · (−1)^{n+j−2}(−c_{j}) · det (Aĵ ; î) + d det(A)

= ∑_{i,j=1}^{n}(−1)^{i+j}b_{i}c_{j} det (Aĵ ; î) + d det(A),

where Aĵ ; î denotes the matrix obtained from A by deleting the i-th row and the j-th column.

First suppose that n is even. Then, since Aî ; î (resp. A) is a skew-symmetric matrix of odd

(resp. even) degree, det (Aî ; î) = 0 (resp. det(A) = Pf(A)^{2}). By induction hypothesis, we see that, if i < j, then

det (Aĵ ; î) = (−1)^{(n−i−1)+(n−j)} det ( \begin{matrix}& & & a_{1i} \\ & A_{î,ĵ}^{î,ĵ} & & ⋮ \\ & & & a_{ni} \\ a_{j1} & ⋯ & a_{jn} & a_{ji} \end{matrix} )

= (−1)^{i+j−1} · (−1) Pf (A_{î,ĵ}^{î,ĵ}) Pf ( \begin{matrix}& & & a_{1i} & a_{1j} \\ & A_{î,ĵ}^{î,ĵ} & & ⋮ & ⋮ \\ & & & a_{ni} & a_{nj} \\ a_{i1} & ⋯ & a_{in} & 0 & a_{ij} \\ a_{j1} & ⋯ & a_{jn} & a_{ji} & 0 \end{matrix} )

= (−1)^{i+j} · (−1)^{(n−i−1)+(n−j)} Pf (A_{î,ĵ}^{î,ĵ}) Pf(A)

= − Pf (A_{î,ĵ}^{î,ĵ}) Pf(A).

Similarly, if i > j, then we have

det (Aĵ ; î) = Pf (A_{î,ĵ}^{î,ĵ}) Pf(A).

Hence we have

det Ã

= Pf(A) ( ∑_{1≤i<j≤n}(−1)^{i+j+1}b_{i}c_{j} Pf (A_{î,ĵ}^{î,ĵ}) + ∑_{1≤i<j≤n}(−1)^{i+j}b_{i}c_{j} Pf (A_{î,ĵ}^{î,ĵ}) + d Pf(A)) .

By comparing this expression with the expansion of the second Pfaffian in (5.5) along the last two columns,

Pf ( \begin{matrix}A & b & c \\ −^{t}b & 0 & −d \\ −^{t}c & d & 0 \end{matrix} )

= ∑_{1≤i<j≤n}(−1)^{j−1}c_{j} · (−1)^{i−1}b_{i} Pf (A_{î,ĵ}^{î,ĵ}) + ∑_{1≤j<i≤n}(−1)^{j−1}c_{j} · (−1)^{i−2}b_{i} Pf (A_{î,ĵ}^{î,ĵ})

+ (−d) Pf(A). we obtain the desired formula.

Next suppose that n is odd. Then det A = 0. By induction hypothesis, we see that, if i < j,

det (Aĵ ; î) = (−1)^{(n−i−1)+(n−j)} det ( \begin{matrix}& & & a_{1i} \\ & A_{î,ĵ}^{î,ĵ} & & ⋮ \\ & & & a_{ni} \\ a_{j1} & ⋯ & a_{jn} & a_{ji} \end{matrix} )

= (−1)^{i+j−1} Pf ( \begin{matrix}& & & a_{1i} \\ & A_{î,ĵ}^{î,ĵ} & & ⋮ \\ & & & a_{ni} \\ a_{i1} & ⋯ & a_{in} & 0 \end{matrix} ) Pf ( \begin{matrix}& & & a_{1j} \\ & A_{î,ĵ}^{î,ĵ} & & ⋮ \\ & & & a_{nj} \\ a_{j1} & ⋯ & a_{jn} & 0 \end{matrix} )

= (−1)^{i+j−1} · (−1)^{n−i−1} Pf (Aî ; î) · (−1)^{n−j} Pf (Aĵ ; ĵ)

= Pf (Aî ; î) Pf (Aĵ ; ĵ) .

Similarly, for i > j, we have

det (Aĵ ; î) = Pf (Aî ; î) Pf (Aĵ ; ĵ) .

Comparison of this with the expansion of the Pfaffians in (5.6) along the last columns completes the proof. □

Now we apply this lemma. If b is even, then we have

det ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & M_{\{n−b+(a+b)/2\}}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 & 0 \\ N(x_{n+1}, X_{n}) & M_{R}(x_{n+1}) & x_{n+1}^{n−b+(a+b)/2} \end{matrix} )

= − Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 \end{matrix} )

× Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & M_{\{n−b+(a+b)/2\}}(X_{n}) & N(X_{n}, x_{n+1}) \\ −^{t}M_{R}(X_{n}) & 0 & 0 & −^{t}M_{R}(x_{n+1}) \\ −^{t}M_{\{n−b+(a+b)/2\}}(X_{n}) & 0 & 0 & −x_{n+1}^{n−b+(a+b)/2} \\ N(x_{n+1}, X_{n}) & M_{R}(x_{n+1}) & x_{n+1}^{n−b+(a+b)/2} & 0 \end{matrix} )

= − Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 \end{matrix} )

× (−1)^{n−b+1} Pf ( \begin{matrix}N(X_{n+1}, X_{n+1}) & M_{Q}(X_{n+1}) \\ −^{t}M_{Q}(X_{n+1}) & 0 \end{matrix} )  (5.7)

If b is odd, then we have

det ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & M_{\{n−b+(a+b)/2\}}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 & 0 \\ N(x_{n+1}, X_{n}) & M_{R}(x_{n+1}) & x_{n+1}^{n−b+(a+b)/2} \end{matrix} )

= Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & M_{\{n−b+(a+b)/2\}}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 & 0 \\ −^{t}M_{\{n−b+(a+b)/2\}}(X_{n}) & 0 & 0 \end{matrix} )

× Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) & N(X_{n}, x_{n+1}) \\ −^{t}M_{R}(X_{n}) & 0 & −^{t}M_{R}(x_{n+1}) \\ N(x_{n+1}, X_{n}) & M_{R}(x_{n+1}) & 0 \end{matrix} )

= Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{Q}(X_{n}) \\ −^{t}M_{Q}(X_{n}) & 0 \end{matrix} )

× (−1)^{n−b} Pf ( \begin{matrix}N(X_{n+1}, X_{n+1}) & M_{R}(X_{n+1}) \\ −^{t}M_{R}(X_{n+1}) & 0 \end{matrix} )  (5.8)

The four Pfaffians in (5.7) and (5.8) are evaluated by the following Lemma. Lemma 10. (1) If a and b is even, then

Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{R}(X_{n}) \\ −^{t}M_{R}(X_{n}) & 0 \end{matrix} )

= \frac{(−1)^{b(n−b)+(n−b)(n−b−1)/2}}{Δ(X_{n})}

× det (M_{[0,n−b/2−1]}(X_{n}) ⋮ M_{[n+a/2−b/2,n+a/2−1]}(X_{n})) × det (M_{[0,n−b/2−1]}(X_{n}) ⋮ M_{[n+a/2−b/2+1,n+a/2]}(X_{n})) ,

Pf ( \begin{matrix}N(X_{n+1}, X_{n+1}) & M_{Q}(X_{n+1}) \\ −^{t}M_{Q}(X_{n+1}) & 0 \end{matrix} )

= \frac{(−1)^{b(n−b)+b/2+(n−b+1)(n−b)/2}}{Δ(X_{n+1})}

× det (M_{[0,n−b/2]}(X_{n+1}) ⋮ M_{[n+a/2−b/2+1,n+a/2]}(X_{n+1})) × det (M_{[0,n−b/2−1]}(X_{n+1}) ⋮ M_{[n+a/2−b/2,n+a/2]}(X_{n+1})) .

(2) If a and b is odd, then

Pf ( \begin{matrix}N(X_{n}, X_{n}) & M_{Q}(X_{n}) \\ −^{t}M_{Q}(X_{n}) & 0 \end{matrix} )

= \frac{(−1)^{(b−1)(n−b)+(b−1)/2+(n−b+1)(n−b)/2}}{Δ(X_{n})}

× det (M_{[0,n−(b−1)/2−1]}(X_{n}) ⋮ M_{[n+(a+1)/2−(b−1)/2,n+(a+1)/2−1]}(X_{n}))

× det (M_{[0,n−(b+1)/2−1]}(X_{n}) ⋮ M_{[n+(a+1)/2−(b+1)/2,n+(a+1)/2−1]}(X_{n})) ,

Pf ( \begin{matrix}N(X_{n+1}, X_{n+1}) & M_{R}(X_{n+1}) \\ −^{t}M_{R}(X_{n+1}) & 0 \end{matrix} )

= \frac{(−1)^{(b+1)(n−b)+(n−b)(n−b−1)/2}}{Δ(X_{n+1})}

× det (M_{[0,n−(b+1)/2]}(X_{n+1}) ⋮ M_{[n+(a−1)/2−(b+1)/2+1,n+(a−1)/2]}(X_{n+1}))

× det (M_{[0,n−(b+1)/2]}(X_{n+1}) ⋮ M_{[n+(a+1)/2−(b+1)/2+1,n+(a+1)/2]}(X_{n+1})) .

Proof. We use Theorem 7 and the following decomposition of the product of two Schur functions of rectangular shape [10, Theorem 2.4]: Let m ≤ n, then

s_{(s^{m})}(X_{N}) · s_{(t^{n})}(X_{N}) = ∑_{λ} s_{λ}(X_{N}),

where the sum is taken over all partitions λ with length ≤ m + n such that

λ_{i} + λ_{m+n−i+1} = s + t, i = 1, …, m,
λ_{m} ≥ max(s, t),
λ_{m+1} = ⋯ = λ_{n} = t.

Apply the minor summation formula in Theorem 7 to

G = M_{P} (X_{n}), H = M_{R}(X_{n}),

and the skew-symmetric matrix A = (a_{ij})_{i,j∈Γ} with nonzero entries

a_{i,a+b−i} = \{ \begin{matrix}1 & if 0 ≤ i ≤ (a + b)/2 − 1 \\ −1 & if (a + b)/2 + 1 ≤ i ≤ a + b. \end{matrix}

Then, by the same argument as in the proof of [10, Theorem 2.4], we obtain the first formula in item (1) of the Theorem. The other formulas are obtained by applying Theorem 7 to the above skew-symmetric matrix A and the matrices (G, H) = (M_{P} (X_{n+1}), M_{Q}(X_{n+1})), (M_{P} (X_{n}), M_{Q}(X_{n})), (M_{P} (X_{n+1}), M_{R}(X_{n+1})). □

Now we are in the position to complete the proof of Theorem 3. Proof of Theorem 3. Suppose that a and b are even. Combining (5.3), (5.4) and

(5.7), and using (5.2), we have

∑_{(λ,μ)∈R(a,b)} s_{λ}(X_{n+1})s_{μ}(X_{n})

= \frac{(−1)^{bn+n−b}}{Δ(X_{n})Δ(X_{n+1})} · (−1)^{bn+b(b−1)/2+n−b} · (−1)^{n−b} \frac{(−1)^{(n−b)^{2}+b/2}}{Δ(X_{n})Δ(X_{n+1})}

× det (M_{[0,n−b/2−1]}(X_{n}) ⋮ M_{[n+a/2−b/2,n+a/2−1]}(X_{n})) × det (M_{[0,n−b/2−1]}(X_{n}) ⋮ M_{[n+a/2−b/2+1,n+a/2]}(X_{n})) × det (M_{[0,n−b/2]}(X_{n+1}) ⋮ M_{[n+a/2−b/2+1,n+a/2]}(X_{n+1})) × det (M_{[0,n−b/2−1]}(X_{n+1}) ⋮ M_{[n+a/2−b/2,n+a/2]}(X_{n+1}))

= (−1)^{b^{2}/2}s(_{(_{\frac{a}{2}})}_{b/2})(X_{n})s(_{(\frac{a}{2} +1)}_{b/2})(X_{n})s(_{(_{\frac{a}{2}})}_{b/2})(X_{n+1})s(_{(_{\frac{a}{2}})}_{b/2+1})(X_{n+1}) = s(_{(_{\frac{a}{2}})}_{b/2})(X_{n})s(_{(\frac{a}{2} +1)}_{b/2})(X_{n})s(_{(_{\frac{a}{2}})}_{b/2})(X_{n+1})s(_{(_{\frac{a}{2}})}_{b/2+1})(X_{n+1}).

If a and b are odd, then it follows from (5.3), (5.4), (5.8), and (5.2) that

∑_{(λ,μ)∈R(a,b)} s_{λ}(X_{n+1})s_{μ}(X_{n})

= \frac{(−1)^{bn+n−b}}{Δ(X_{n})Δ(X_{n+1})} · (−1)^{bn+b(b−1)/2+n−b} · (−1)^{n−b} \frac{(−1)^{2b(n−b)+(n−b)^{2}+(b−1)/2}}{Δ(X_{n})Δ(X_{n+1})}

× det (M_{[0,n−(b−1)/2−1]}(X_{n}) ⋮ M_{[n+(a+1)/2−(b−1)/2,n+(a+1)/2−1]}(X_{n}))

× det (M_{[0,n−(b+1)/2−1]}(X_{n}) ⋮ M_{[n+(a+1)/2−(b+1)/2,n+(a+1)/2−1]}(X_{n}))

× det (M_{[0,n−(b+1)/2]}(X_{n+1}) ⋮ M_{[n+(a−1)/2−(b+1)/2+1,n+(a−1)/2]}(X_{n+1}))

× det (M_{[0,n−(b+1)/2]}(X_{n+1}) ⋮ M_{[n+(a+1)/2−(b+1)/2+1,n+(a+1)/2]}(X_{n+1}))

= (−1)^{(b^{2}−1)/2}s(_{(_{\frac{a+1}{2}})}_{(b−1)/2})(X_{n})s(_{(_{\frac{a+1}{2}})}_{(b+1)/2})(X_{n})

× s(_{(_{\frac{a−1}{2}})}_{(b+1)/2})(X_{n+1})s(_{(_{\frac{a+1}{2}})}_{(b+1)/2})(X_{n+1})

= s(_{(_{\frac{a+1}{2}})}_{(b−1)/2})(X_{n})s(_{(_{\frac{a+1}{2}})}_{(b+1)/2})(X_{n})s(_{(_{\frac{a−1}{2}})}_{(b+1)/2})(X_{n+1})s(_{(_{\frac{a+1}{2}})}_{(b+1)/2})(X_{n+1}).

□

References

1. M. Ciucu, Enumeration of perfect matchings in graphs with reflective symmetry, J. Combin. Theory Ser. A 77 (1997), 67-97.

2. M. Fulmek and C. Krattenthaler, Lattice path proofs for determinant formulas for symplectic and orthogonal characters, J. Combin. Theory Ser. A 77 (1997), 3–50.

3. I. M. Gessel and X. Viennot, Determinants, paths, and plane partitions, preprint, 1989.

4. M. Ishikawa and M. Wakayama, Minor summation formula for pfaffians, Linear and Multilinear Algebra 39 (1995), 285–305.

5. M. Ishikawa and M. Wakayama, Minor summation formula of pfaffians and Schur function identities, Proc. Japan Acad. Ser. A 71 (1995), 54–57.

6. M. Ishikawa, S. Okada and M. Wakayama, Applications of minor summation formulas I, Littlewood’s formulas, J. Algebra 183 (1996), 193–216.

7. G. Kuperberg, Symmetries of plane partitions and the permanent determinant method, J. Combin. Theory Ser. A 68 (1994), 115–151.

8. I. G. Macdonald, Symmetric Functions and Hall Polynomials, second edition, Oxford University Press, New York/London, 1995.

9. P. A. MacMahon, Combinatory Analysis, vol. 2, Cambridge University Press, 1916; reprinted by Chelsea, New York, 1960.

10. S. Okada, Applications of minor summation formulas to rectangular-shaped representations of classical groups, J. Algebra (to appear).

11. J. Propp, Twenty open problems on enumeration of matchings, preprint.

12. B. E. Sagan, The symmetric group, Wadsworth & Brooks/Cole, Pacific Grove, California, 1991.

13. J. R. Stembridge, Nonintersecting paths, pfaffians and plane partitions, Adv. in Math. 83 (1990),

96—131.

14. M. Ciucu, Enumeration of lozenge tilings of punctured hexagons, J. Combin. Theory Ser. A (1998).

Graduate School of Polymathematics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-01, Japan

Institut für Mathematik der Universität Wien, Strudlhofgasse 4, A-1090 Wien, Austria.