Generating functions for shifted plane partitions

C. Krattenthaler

Institut für Mathematik der Universität Wien, Strudlhofgasse 4, A-1090 Wien, Austria.

Abstract. With the help of a tableaux method, determinant formulas for trace generating functions for various classes of shifted plane partitions are derived. Proceeding from these determinants, generalizations of Gansner’s (J. Combin. Theory A 30 (1981), 71–89; Illinois J. Math. 25 (1985), 533–554) hook formulas for shifted plane partitions and alternative proofs of recent results of Proctor about alternating trace generating functions are given.

1. Introduction. Generating functions for certain families of plane partitions with a given shape and given row bounds can be written down in determinantal form. This has been shown by various methods ([Chu (1989), Gessel and Viennot (1989), Krattenthaler (1990a), Wachs (1985)]), the most prominent being the Gessel– Viennot [(1985, 1989)] method of nonintersecting lattice paths. In special cases these determinants can be evaluated to result into closed forms ([Gessel and Viennot (1989), Krattenthaler (1990a, 1990b)]). Among them are the famous hook-content formulas of Stanley [(1971)].

With the help of a tableaux method, which is related to the nonintersecting path method, and a simple determinant evaluation, the author ([Krattenthaler (1990a, 1990b)]) provided a uniform proof method to establish essentially all known results concerning generating functions for plane partitions of a given shape (without symmetries) and, moreover, some new results. In this paper we continue this work. We transfer the method of [Krattenthaler (1990a)] to the case of shifted plane partitions. These arise naturally when considering generating function problems for plane partitions with symmetries ([Andrews (1979), Mills et al. (1982, 1983, 1986), Okada (1989)]). Using our tableaux method we derive determinant formulas for generating functions for shifted plane partitions of a given shape with prescribed entries on the main diagonal (Theorems 1,2 below).

Again there are special cases in which the determinants may be evaluated. With the help of a summation formula for Schur functions we are able to generalize Gansner’s [(1981a, 1981b)] hook formulas for trace generating functions for two particular families of shifted plane partitions.

1991 Mathematics Subject Classification. Primary 05A15, 11P57; Secondary 05A10, 05A15, 11B65, 11P68, 15A15..

Key words and phrases. Generating functions, shifted plane partitions.

A particular specialization of trace generating functions leads to the alternating trace generating functions of Proctor [(1990)]. In the case of trapezoidal shapes, there are closed forms for the alternating trace generating function of shifted plane partitions with prescribed entries on the main diagonal. These formulas were first proved by Proctor [private communication] using group representation theory and bideterminants. With the help of a simple determinant evaluation (Lemma 7) we are able to provide alternative proofs which start with the determinantal formula of Theorem 1.

We should remark that recently Stembridge [(1990a)] discovered a powerful approach to the computation of generating functions for shifted plane partitions, which involves Pfaffians. For example, Gansner’s hook formula ([Gansner (1981a), Theorem 7.1]) for the generating function of shifted reverse plane partitions could be easily proved that way, too. (In [Stembridge (1990a)] only the case x_{i} = q is considered.) Other results about shifted plane partitions, which we were not able to prove using our methods, occur in [Proctor (1983, 1990), and Mills et al. (1986)].

The next section contains all relevant definitions. In section 3 we derive the determinant formulas for three types of generating functions of shifted plane partitions. The generalizations of Gansner’s results are given in section 4. Finally, in section 5 the computations concerning the alternating trace generating functions are discussed.

2. Definitions. We use the usual multidimensional notations. For r-tuples of integers λ = (λ_{1}, λ_{2}, …, λ_{r}) and μ = (μ_{1}, μ_{2}, …, μ_{r}), we write λ + μ for (λ_{1} + μ_{1}, λ_{2} + μ_{2}, …, λ_{r} + μ_{r}). If σ is an element of the symmetric group S_{r} of order r we define λ_{σ} to be the r-tuple (λ_{σ(1)}, λ_{σ(2)}, …, λ_{σ(r)}). An r-tuple λ of integers is called a partition if λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r} ≥ 0.

Let α, β be arbitrary integers and λ a partition with λ_{r} ≥ r. We call an array π of integers of the form

(2.1)  \begin{matrix}π_{11} & π_{12} & . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . & π_{1,λ_{1}} \\ π_{22} & . . . . . . . . . . . . . . . . . . . . . . . . . . & π_{2,λ_{2}} \\ ... & ... \\ π_{rr} & … & π_{r,λ_{r}} \end{matrix}

a shifted (α, β)–plane partition of shape λ if

(2.2)(a)  π_{ij} ≥ π_{i,j+1} + α  for 1 ≤ i ≤ r, i ≤ j < λ_{i}

and

(2.2)(b)  π_{ij} ≥ π_{i+1,j} + β  for 1 ≤ i ≤ r − 1, i ≤ j ≤ λ_{i+1} .

It should be noted that in our definition of the shifted shape there are λ_{i} − i + 1 integers in the i-th row, while usually there are λ_{i} integers in the i-th row.

An array π of the form (2.1) is called a shifted (α, β)–reverse plane partition of shape λ, if it satisfies

(2.3)(a)  π_{ij} ≤ π_{i,j+1} + α  for 1 ≤ i ≤ r, i ≤ j < λ_{i}

and

(2.3)(b)  π_{ij} ≤ π_{i+1,j} + β  for 1 ≤ i ≤ r − 1, i ≤ j ≤ λ_{i+1} .

To give an example, the array in Figure 1 below is a shifted (0, 2)–plane partition of shape (8,6,6,4).

11 9 7 7 6 6 3 1
6 5 5 4 3
3 2 1 1

−1

Figure 1

The entries of π are called parts of π. The sum of all parts of π, ∑ π_{ij}, is called the norm of π, for which we write n(π). The l-diagonal of an array π of the form (2.1) is the sequence of parts π_{ij} with j − i = l. The l-trace of π, denoted by t_{l}(π), is the sum of the parts of the l-diagonal of π.

We shall consider three types of generating functions for shifted (α, β)–plane partitions subject to certain restrictions. The part generating function for a certain class C of shifted (α, β)–plane partitions is ∑_{π} ∏ x_{π_{ij}} , where the sum is over all π belonging to C and the product is over all parts of π. The norm generating function for C is ∑_{π} x^{n(π)} , while the trace generating function for C is ∑_{π} ∏_{l∈ℤ} x_{l}^{t_{l}(π)} , where in both cases the sum is over all π being an element of that class C. (ℤ is the set of integers.) Of course, the norm generating function comes out of both the part and the trace generating function by setting x_{i} = x^{i}, respectively x_{i} = x.

For x being an indeterminate, we introduce the “x-notations” [β] = 1 − x^{β}, [n]! = [1][2] ⋯ [n], [0]! = 1,

[ \begin{matrix}n \\ k \end{matrix} ] = \frac{[n]!}{[k]! [n − k]!} ,

(a; x)_{∞} = ∏_{j=0}^{∞}(1 − ax^{j}) ,

and

(a; x)_{β} = (a; x)_{∞}/(ax^{β}; x)_{∞} .

Finally, for the integers m, n, m ≤ n, we set

x(m, n) = ∏_{j=m+1}^{n} x_{j} ,

with the empty product being equal to 1 by convention.

3. The determinant formulas. In this section we are going to prove the following theorems.

Theorem 1. Let α, β be arbitrary integers, λ be a partition with λ_{r} ≥ r, and let a, b be r-tuples of integers satisfying

(3.1)  a_{i} − α − β ≥ a_{i+1} and b_{i} + α(λ_{i} − λ_{i+1}) + (1 − β) ≥ b_{i+1}

for i = 1, 2, …, r − 1. Then the trace generating function for shifted (α, β)-plane partitions of shape λ in which the first part in row i is equal to a_{i} and the last part in row i is at least b_{i} is

(3.2) x_{0}^{∑_{i=1}^{r} a_{i}} ∏_{i=1}^{r} (x(0, λ_{i} − i)^{b_{i}} ∏_{l=0}^{λ_{i}−i−1} x(0, l)^{α})

× det_{1≤s,t≤r} (t(λ_{s} − s; a_{t} + (1 − α − β)(s − t) − α(λ_{s} − s) − b_{s})) ,

where t(p; m) is the coefficient of z^{m} in 1/ ∏_{l=0}^{p}(1 − x(0, l)z). The norm generating function for the same class of shifted (α, β)-plane partitions is

(3.3)  x^{N_{1}} det_{1≤s,t≤r} ([ \begin{matrix}(λ_{s} − s)(1 − α) + (1 − α − β)(s − t) + a_{t} − b_{s} \\ λ_{s} − s \end{matrix} ]) ,

where N_{1} = ∑_{i=1}^{r} (b_{i}(λ_{i} − i) + a_{i} + α\binom{λ_{i}−i}{2}). If β = 1 − α, the part generating function for this class is given by

(3.4)  ∏_{i=1}^{r} x_{a_{i}} det_{1≤s,t≤r} (h_{λ_{s}−s}^{(α)}(x; a_{t}, b_{s})) ,

where h_{n}^{(α)}(x; A, B) = ∑ x_{i_{1}} ⋯ x_{i_{n}}, the sum being over all i_{1}, …, i_{n} satisfying i_{1} ≤ A, i_{n} ≥ B, and i_{j} ≥ i_{j+1} + α for j = 1, 2, …, n − 1.

Theorem 2. Let α, β be arbitrary integers, λ be a partition with λ_{r} ≥ r, and let a, b be r-tuples of integers satisfying

(3.5)  a_{i} + α(λ_{i−1} − λ_{i}) + (1 − β) ≥ a_{i−1} and b_{i} − α − β ≥ b_{i−1}

for i = 1, 2, …, r − 1. Then the trace generating function for shifted (α, β)-reverse plane partitions of shape λ in which the first part in row i is equal to b_{i} and the last part in row i is at most a_{i} is

(3.6) ∏_{i=1}^{r} ∏_{l=0}^{λ_{i}−i−1} x(l, λ_{i} − i)^{α} det_{1≤s,t≤r} (x(−1, λ_{s} − s)^{b_{t}−(s−t)(1−α−β)}

· t′(λ_{s} − s; a_{s} + (1 − α − β)(s − t) − α(λ_{s} − s) − b_{t})) , where t′(p; m) is the coefficient of z^{m} in 1/ ∏_{l=0}^{p}(1 − x(l, p)z). The norm generating function for the same class of shifted (α, β)-plane partitions is (3.7)

x^{N_{2}} det_{1≤s,t≤r} (x^{(λ_{s}−s+1)(b_{t}+t(1−α−β))} [ \begin{matrix}(λ_{s} − s)(1 − α) + (1 − α − β)(s − t) + a_{s} − b_{t} \\ λ_{s} − s \end{matrix} ]) ,

where N_{2} = ∑_{i=1}^{r} (α\binom{λ_{i}−i+1}{2} − i(λ_{i} − i + 1)(1 − α − β)). If β = 1 − α, the part generating function for this class is given by

(3.8)  ∏_{i=1}^{r} x_{b_{i}} det_{1≤r,s≤t} (h_{λ_{s}−s}^{(α)}(x; a_{s}, b_{t})) .

In order to prove these theorems, we shall apply Lemma 2.1 of [Krattenthaler (1990a)], which we restate here without proof.

Lemma 3. Let {M_{σ} : σ ∈ S_{r}} be a collection of pairwise disjoint sets with a weight function w acting on ⋃_{σ∈S_{r}} M_{σ}. Let M_{id} be partitioned into two disjoint subsets, M_{id}^{+} and M_{id}^{−}, M_{id} = M_{id}^{+} ∪ M_{id}^{−}. Assume a weight-preserving involution φ acting on M_{id}^{−} ∪ ⋃_{σ≠id} M_{σ} satisfying the following property, holding for all σ ∈ S_{r}: For every π on which φ is applicable, π ∈ M_{σ}, there exists a transposition (i, j), 1 ≤ i, j ≤ r, for which φ(π) ∈ M_{σ(i,j)}. Then

(3.9)  F (M_{id}^{+}; w) = ∑_{σ∈S_{r}} sgn σ F (M_{σ}; w) ,

where F (M; w) denotes the generating function ∑_{π∈M} w(π). □

Proof of Theorems. In analogy with [Krattenthaler (1990a)] we first define a shifted α-tableaux of shape λ to be an array of the form (2.1) such that (2.2)(a) holds. Thus, for shifted α-tableaux there are only restrictions on the rows but none on the columns. All the notions of section 2, parts, norm, trace, etc., are extended to shifted α-tableaux in the obvious way.

For the set of all shifted α-tableaux of shape λ in which the first part in row i is equal to a_{i} and the last part in row i is at least b_{i} we write ST_{α}(λ; a, b). The subset of ST_{α}(λ; a, b) containing all shifted (α, β)-plane partitions of ST_{α}(λ; a, b) is denoted by ST_{α}^{+}(λ; a, b), its complement by ST_{α}^{−}(λ; a, b).

Given a permutation σ ∈ S_{r}, let the r-tuple e_{σ} be defined by (1 − σ(1), 2 − σ(2), …, r−σ(r)). Next we construct an involution φ_{α,β} acting on ⋃_{σ∈S_{r}} ST_{α}^{−}(λ; a_{σ}+ (1 − α − β)e_{σ}, b) which satisfies the properties required in Lemma 3. For a shifted α-tableaux τ of shape λ this involution is introduced in two steps:

(A) Consider all pairs (τ_{ij}, τ_{i+1,j}) of parts of τ with

(3.10)  τ_{ij} < τ_{i+1,j} + β .

Let J be the largest integer with τ_{iJ} < τ_{i+1,J} + β for some i. Let I be the smallest integer with τ_{IJ} < τ_{I+1,J} + β. (The idea is to search for the right-most pairs and, among them, for the highest pair satisfying (3.10) in the tableaux τ.)

(B) Let K be the smallest integer with τ_{IK} < τ_{I+1,K} + β. The image of τ under application of φ_{α,β} is the tableaux τ′, which is obtained by leaving all rows of τ unchanged, with the exception of the I’th and (I + 1)’th row. Here we set

τ′_{Ij} = \{ \begin{matrix}τ_{Ij} & j ≥ K \\ τ_{I+1,j+1} − (1 − α − β) & j < K \end{matrix}

and

τ′_{I+1,j} = \{ \begin{matrix}τ_{I+1,j} & j > K \\ τ_{I,j−1} + (1 − α − β) & j ≤ K \end{matrix} .

(For the geometrical interpretation of the action of φ_{α,β} confer [Krattenthaler (1990a), Def. 4.1].)

Let τ ∈ ST_{α}^{−}(λ; a_{σ} +(1−α−β)e_{σ}, b) and let I, K be the integers being determined in (A). It is clear from the definition of φ_{α,β} that φ_{α,β}(τ) again is a shifted (α, β)-plane partition of shape λ with diagonal entries being given by the vector a_{σ(I,I+1)} + (1 − α − β)e_{σ(I,I+1)}. Besides, the last part in row i of φ_{α,β}(τ) is again at least b_{i}: For i ≠ I +1 this is obvious, since the last parts in each row of τ remain unchanged under application of φ_{α,β}, with the exception of the (I +1)-th row. Here the last part might be changed, namely in case K = λ_{I+1}, where the last part of row (I + 1) of φ_{α,β}(τ) is given by τ_{I,λ_{I+1}−1} + (1 − α − β). Use of (2.2)(a) yields

τ_{I,λ_{I+1}−1} ≥ τ_{I,λ_{I}} + α(λ_{I} − λ_{I+1} + 1)
≥ b_{i} + α(λ_{I} − λ_{I+1} + 1) .

Hence, by the second part of (3.1), we obtain

τ_{I,λ_{I+1}−1} + (1 − α − β) ≥ b_{i+1} ,

just as being claimed.

Another fact is that, if σ ≠ id, we have

ST_{α}(λ; a_{σ} + (1 − α − β)e_{σ}, b) = ST_{α}^{−}(λ; a_{σ} + (1 − α − β)e_{σ}, b) .

Since σ ≠ id, there must exist an integer i, 1 ≤ i ≤ r − 1, such that σ(i) > σ(i + 1). Using the inequality

(3.11)  a_{R} < a_{S} + (R − S)(1 − α − β), for r ≥ R > S ≥ 1,

which is equivalent with (3.1), it can be shown that

a_{σ(i)} + (1 − α − β)(i − σ(i)) < a_{σ(i+1)} + (1 − α − β)(i + 1 − σ(i + 1)) + α + β .

This implies that for any τ ∈ ST_{α}(λ; a_{σ}+(1−α−β)e_{σ}, b) the inequality τ_{ii} < τ_{i+1,i+1}+ α + β is true, hence τ_{i,i+1} < τ_{i+1,i+1} + β, and thus τ ∈ ST_{α}^{−}(λ; a_{σ} + (1 − α − β)e_{σ}, b).

Summarizing we have the following properties of φ_{α,β}:

(1) φ_{α,β} is an involution on

ST_{α}^{−}(λ; a, b) ∪_{⋃_{σ∈S_{r}} ; σ≠id}ST_{α}(λ; a_{σ} + (1 − α − β)e_{σ}, b) .

If τ ∈ ST_{α}(λ; a_{σ} +(1−α−β)e_{σ}, b) and τ is an element of the domain of φ_{α,β}, then there exists a transposition (I, I + 1) with

φ_{α,β}(τ) ∈ ST_{α}(λ, a_{σ(I,I+1)} + (1 − α − β)e_{σ(I,I+1)}, b) .

(2) The l-trace and norm are invariant under application of φ_{α,β}. If α + β = 1 the set of parts remains unchanged under application of φ_{α,β}, too.

Since, because of (3.11), the map σ → a_{σ} + (1 − α − β)e_{σ} is injective, the sets ST_{α}(λ; a_{σ} +(1−α−β)e_{σ}, b) are pairwise disjoint. Hence, all assumptions of Lemma 3 are satisfied with M_{σ} = ST_{α}(λ; a_{σ} + (1 − α − β)e_{σ}, b), etc., and the trace weight w(τ) = ∏_{l∈ℤ} x_{l}^{t_{l}(τ)}. The generating functions F (ST_{α}(λ; a_{σ} + (1 − α − β)e_{σ}, b)) are equal to the product of the corresponding generating functions of the rows. Therefore, for this choice of M_{σ}, the sum on the right-hand side of (3.9) can be written in the form of a determinant, which finally, after taking some factors out of this determinant, results into the right-hand side of (3.2).

Formula (3.3) is obtained by setting x_{i} = x for all i in (3.2) and using the wellknown expansion for 1/(z; x)_{p+1} ([Andrews (1976), (3.3.7)]). Formula (3.4) is proved in analogy with (3.2) by observing that if α + β = 1, all assumptions of Lemma 3 are satisfied with M_{σ} = ST_{α}(λ; a_{σ}, b), etc., and w(τ) = ∏ x_{τ_{ij}}.

Theorem 2 can be proved by constructing a similar involution, which acts on shifted (α, β)-reverse tableaux, or by using the transformation π_{ij} → −π_{ij}, which maps shifted (α, β)-plane partitions onto shifted (α, β)-reverse plane partitions, and transforming the results of Theorem 1 into the “reverse” case. □

The special case α = 0, β = 1, b_{i} = 0 of (3.3) was used to obtain generating functions for symmetry classes of plane partitions ([Andrews (1979), Mills et al. (1982, 1983, 1986), Okada (1989)]). It is also interesting to note that a slight modification of φ_{01} serves to provide an alternative proof of Okada’s [(1990)] Theorem about partially strict shifted plane partitions.

4. Trace generating functions with no upper part bounds. The main tool which is used in this section to obtain closed forms for generating functions for shifted (α, β)-(reverse) plane partitions with no upper bound on the parts in each row, is the following summation theorem, which is proved by inserting the bideterminantal form of Schur functions ([Macdonald (1979), p. 24, (3.1)]) into the summation formula ([Mcdonald (1979), p. 45, Ex. 4]) for Schur functions.

Lemma 4. Let X_{1}, X_{2}, …, X_{r} be indeterminates. Then

∑_{a_{1}>a_{2}>···>a_{r}≥0} det_{1≤s,t≤r} (X_{s}^{a_{t}}) = \frac{∏_{1≤i<j≤r}(X_{i} − X_{j})}{∏_{i=1}^{r}(1 − X_{i}) ∏_{1≤i<j≤r}(1 − X_{i}X_{j})} . □

Since it is more straight-forward to deal with, we first consider the case of shifted (α, β)-reverse plane partitions.

Theorem 5. Let K, B be integers, K ≥ 1. The trace generating function for shifted (α, β)-reverse plane partitions π of shape λ in which the first part in row i is ≡ −i(1 − α − β) (mod K) and π_{11} ≥ (KB − (1 − α − β)) is

∏_{i=1}^{r} x(−1, λ_{i} − i)^{−i(1−α−β)+K(B+i−1)} ∏_{i=1}^{r} ∏_{l=0}^{λ_{i}−i−1} \frac{x(l, λ_{i} − i)^{α}}{(1 − x(l, λ_{i} − i))}

× \frac{∏_{1≤j<i≤r}(1 − x(λ_{i} − i, λ_{j} − j)^{K})}{∏_{i=1}^{r}(1 − x(−1, λ_{i} − i)^{K}) ∏_{1≤i<j≤r}(1 − x(−1, λ_{i} − i)^{K}x(−1, λ_{j} − j)^{K})}.

Proof. In (3.6) set b_{t} = KB_{t} − t(1 − α − β) and let a_{s} → ∞. Using the fact that, if lim_{i→∞} c_{i} = c then also lim_{z→1^{−}}(1 − z) ∑_{j} c_{j}z^{j} = c, we obtain, after taking some factors out of the determinant, for the trace generating function for shifted (α, β)-reverse plane partitions of shape λ in which the first part in row i is equal to KB_{i} − i(1 − α − β) the expression

\frac{∏_{i=1}^{r} ∏_{l=0}^{λ_{i}−i−1} x(l, λ_{i} − i)^{α} ∏_{i=1}^{r} x(−1, λ_{i} − i)^{−i(1−α−β)}}{∏_{i=1}^{r} ∏_{l=0}^{λ_{i}−i−1}(1 − x(l, λ_{i} − i))} det (x(−1, λ_{s} − s)^{KB_{t}}) .

When taking into account (3.5), one observes that, to obtain the desired generating function, these expressions have to be summed up over all B_{1}, B_{2}, …, B_{r} with B ≤ B_{1} < ⋯ < B_{r}. Application of Lemma 4 after some manipulation yields the claimed expression. □

Gansner’s [(1981a)] Theorem 7.1 is the special case α = β = 0, K = B = 1 of Theorem 5.

In the case of ordinary shifted plane partitions there is only a closed form result analogous to the previous one in case of a triangular shape (r^{r}), the shape λ with λ_{i} = r for all i.

Theorem 6. Let K, B be integers, K ≥ 1. The trace generating function for shifted (α, β)-plane partitions π of shape (r^{r}) in which the first part in row i is ≡ i(1 − α − β) (mod K) and π_{rr} ≥ (KB + r(1 − α − β)) is

∏_{i=0}^{r−1} x_{i}^{N_{3}(i)} ∏_{0≤i<j≤r−1} \frac{(1 − x(i, j)^{K})}{(1 − x(i, j))(1 − x(−1, i)^{K}x(−1, j)^{K})} ∏_{i=0}^{r−1} \frac{1}{(1 − x(−1, i)^{K})},

where N_{3}(i) = (r − i)(r + i + 1)/2 + (KB − rα)(r − i) + (α + K)\binom{r−i}{2} − β\binom{r−i+1}{2}.

Proof. Let B̂ = KB − rα. First we consider the generating function of shifted (α, β)-plane partitions of shape (r^{r}) where the first part in row i is equal to a_{i} and the last part in row i is at least B̂ + i(1 − β). If these choices are inserted into (3.2), we get (4.1)

x_{0}^{∑ a_{i}} ∏_{i=1}^{r} x(0, r−i)^{B̂+i(1−β)} ∏_{l=0}^{r−1} x(0, l)^{α(r−l−1)} det (t(r−s; a_{t} −t(1−α−β)−αr−B̂)) ,

where t(p; m) is the coefficient of z^{m} in the power series expansion given in Theorem 1. Using the partial fraction expansion of this power series, it turns out that the determinant in (4.1) by simple row manipulations may be simplified to

(4.2)  det (x(0, r − s)^{a_{t}−t(1−α−β)−αr−B̂} ∏_{l=0}^{r−s−1} \frac{(−1)^{r−s}x(l, r − s)}{(1 − x(l, r − s))}) .

Because of (2.2)(b), any shifted (α, β)-plane partition of shape (r^{r}) with π_{rr} ≥ KB + r(1−α −β) automatically possesses the property that the last part in row i is greater than B̂ +i(1−β). Therefore, by setting a_{i} = KA_{i} +i(1−α−β) in (4.1) and summing up over all A_{1}, …, A_{r} with A_{1} > ⋯ > A_{r} ≥ KB+r(1−α−β), the desired generating function will be obtained. Writing the determinant in (4.1) in the form (4.2), it is seen that again Lemma 4 may be applied. Some tedious manipulations finally result into the claimed expression. □

The case K = 2 of Theorem 6 is somewhat related to a result of Proctor [(1990) (“CYI” case of Theorem 1)] (cf. also [Stembridge (1990b)]), although there is no overlap. Gansner’s [(1981b) Theorem 4.2] is obtained from Theorem 6 by setting α = β = 0, K = 1, and B = −r.

5. Alternating trace generating functions. Recently, [Proctor (1990)] considered unusual weightings for shifted plane partitions π, namely w_{1}(π) = x^{∑_{i≥0}(−1)^{i−1}it_{i}(π)} and w_{2}(π) = x^{∑_{i≥0}(−1)^{i}(i+1)t_{i}(π)}, which he called “alternating trace” weightings. He also observed [private communication] that with these weightings there are closed expressions for trapezoidal shapes, the shapes λ, where λ_{i} = Λ−i for some Λ. His proofs base on the representation theory of the symplectic group. We give more elementary proofs below, which use the determinants in Theorems 1,2 and the following computational lemma. Special cases of these results earlier appeared in [Proctor (1988), Prop. 4.1, and Proctor (1990), Theorem 1].

Lemma 7. Let A_{1}, A_{2}, …, A_{r}, B_{2}, …, B_{r} be indeterminates. Then we have

(5.1) det ((C/A_{t} + B_{r}) ⋯ (C/A_{t} + B_{s+1})(A_{t} + B_{r}) ⋯ (A_{t} + B_{s+1}))

= ∏_{i=2}^{r} B_{i}^{i−1} ∏_{1≤i<j≤r}(A_{i} − A_{j})(1 − C/A_{i}A_{j}) .

Proof. The determinant on the left-hand side of (5.1) is a polynomial in A_{1}, …, A_{r} of degree 3\binom{r}{2} divided by ∏_{i} A_{i}^{r−1}. Since, for 1 ≤ t_{1} < t_{2} ≤ r, the determinant is zero for A_{t_{1}} = A_{t_{2}} or A_{t_{1}} = C/A_{t_{2}}, respectively, the left-hand side is equal to

p(A_{1}, …, A_{r}) ∏_{i<j}(A_{i} − A_{j})(A_{i}A_{j} − C)/ ∏_{i} A_{i}^{r−1} ,

where p(A_{1}, …, A_{r}) is some polynomial in the A_{i}’s. Comparing the degrees yields that p is a constant, which is computed by comparing coefficients of ∏_{i} A_{i}^{r−i} on both sides of (5.1). □

Theorem 8. Let λ be a trapezoidal shape, λ_{i} = Λ − i, with Λ ≥ 2r. Given an integer B and an r-tuple a satisfying (3.1), the generating function with respect to the alternating trace weight w_{1} for shifted (α, β)-plane partitions of shape λ in which the first part in row i is equal to a_{i} and the last part in row i is at least B − βi + αi is

x^{N_{4}} ∏_{i=1}^{r} \frac{[Λ − r + a_{i} − i(1 − α − β) − Λα − B]!}{[Λ − 2i]! [r + a_{i} − i(1 − α − β) − Λα − B]!}×

∏_{1≤i<j≤r}[a_{i} −a_{j} −(i−j)(1−α−β)][a_{i} +a_{j} −(i+j)(1−α−β)−Λ(2α−1)−2B +1] ,

where

N_{4} = ∑_{i=1}^{r} a_{i}(i − ⌊\frac{Λ}{2}⌋) + α (rΛ⌊\frac{Λ}{2}⌋ − \frac{Λ}{2}r(2r + 1) + \frac{r(4r^{2}+3r−1)}{6}) + χ(Λ odd)

· (α(Λ\frac{r^{2}}{2} − 4\binom{r+1}{3}) + β(\frac{r(4r^{2}+3r−1)}{6} − r(r + 1)⌊\frac{Λ}{2}⌋) + B(Λr − r(r + 1))) .

(χ is the usual truth function.)

Proof. The alternating trace weight w_{1} comes out of the general trace weighting ∏_{l∈ℤ} x_{l}^{t_{l}(π)} by setting x_{i} = x^{(−1)^{i−1}i}. Since for this choice of the variables x_{i}, we have

t(p; m) = x^{−m⌊ \frac{p}{2} ⌋} [ \begin{matrix}p + m \\ p \end{matrix} ] ,

which is again shown using [Andrews (1976), (3.3.7)], by (3.2) the desired generating function allows the form

(5.2) x^{N_{5}} det (x^{(−⌊ \frac{Λ}{2} ⌋+s)(a_{t}−t(1−α−β))} [ \begin{matrix}Λ(1 − α) − s + a_{t} − t(1 − α − β) − B \\ Λ − 2s \end{matrix} ]) ,

where we suppress the exact expression for N_{5} for brevity. Taking some factors out of the determinant, this expression may be written

x^{N_{6}} ∏_{i=1}^{r} \frac{[Λ − r + Â_{i}]!}{[Λ − 2i]! [r + Â_{i}]} det ((x^{−Λ−1−Â_{t}} − x^{−r}) ⋯ (x^{−Λ−1−Â_{t}} − x^{−s−1})

· (xÂ_{t} − x^{−r}) ⋯ (xÂ_{t} − x^{−s−1})) ,

where Â_{t} = a_{t} − t(1 − α − β) − Λα − B. The determinant can be evaluated using Lemma 7 with A_{i} = xÂ_{i}, B_{i} = −x^{−i}, and C = x^{−Λ−1}. Simplifications finally lead to the claimed expression. □

For the weighting w_{2} we only obtained a closed form expression for the first row length of the shifted shape being even. In case of odd first row length, it does not seem to be possible to evaluate the determinant.

Theorem 9. Let λ be a trapezoidal shape, λ_{i} = 2Λ − 1 − i, with Λ > r. Given an integer B and an r-tuple a satisfying (3.1), the generating function with respect to the alternating trace weight w_{2} for shifted (α, β)-plane partitions of shape λ in which the first part in row i is equal to a_{i} and the last part in row i is at least B − βi + αi is

x^{N_{7}} ∏_{i=1}^{r} \frac{(1 + x^{a_{i}−i(1−α−β)−α(2Λ−1)−B+Λ})}{(1 + x^{Λ−i})}

× \frac{[2Λ − 1 − r + a_{i} − i(1 − α − β) − (2Λ − 1)α − B]!}{[2Λ − 1 − 2i]! [r + a_{i} − i(1 − α − β) − (2Λ − 1)α − B]!}

× ∏_{1≤i<j≤r}[a_{i}−a_{j}−(i−j)(1−α−β)][a_{i}+a_{j}−(i+j)(1−α−β)−(2Λ−1)(2α−1)−2B+1] ,

where N_{7} = ∑_{i=1}^{r} a_{i}(i − Λ) + α (2Λ^{2}r − Λr(3r + 2) + \frac{r(2r^{2}+3r−2)}{3}) + Λr^{2} + r.

Proof. The alternating trace weight w_{2} comes out of the general trace weighting by setting x_{i} = x^{(−1)^{i}(i+1)}. For this choice of the variables x_{i}, we get, another time using [Andrews (1976), (3.3.7)],

t′(p; m) = x^{−m⌊ \frac{p+3}{2} ⌋} [ \begin{matrix}p + m + 1 \\ p + 1 \end{matrix} ] − x^{−1−(m−1)⌊ \frac{p+3}{2} ⌋} [ \begin{matrix}p + m \\ p + 1 \end{matrix} ]

= x^{−m⌊ \frac{p+3}{2} ⌋} [ \begin{matrix}p + m \\ p \end{matrix} ] \frac{[⌊\frac{p+1}{2}⌋] + x^{m+⌊ \frac{p+1}{2} ⌋}[⌊\frac{p+2}{2}⌋]}{[p + 1]} .

Using this identity, by (3.2) we obtain, after taking some factors out of the determinant, for the desired generating function

x^{N_{7}} ∏_{i=1}^{r} \frac{(1 + x^{a_{i}−i(1−α−β)−α(2Λ−1)−B+Λ})}{(1 + x^{Λ−i})} det (x^{(−Λ+1+s)(a_{t}−t(1−α−β))}

· [ \begin{matrix}(2Λ − 1)(1 − α) − s + a_{t} − t(1 − α − β) − B \\ 2Λ − 1 − 2s \end{matrix} ] ) .

The determinant in this expression is a special case of the determinant in (5.2) (replace Λ by 2Λ − 1 in (5.2)). □

References

Andrews, G. E., The Theory of Partitions, Encyclopedia of Mathematics and its Applications, Vol. 2, Addison–Wesley, Reading, 1976. Andrews, G. E., Plane partitions (III): The weak Macdonald conjecture, Inventiones Math. 53 (1979), 193–225. Chu, W. C., Dominance method for plane partitions. III. Flagged skew plane partitions, Acta Math. Sinica 5 (1989), 321–328. Gansner, E., The Hillman–Grassl correspondence and the enumeration of reverse plane partitions, J. Combin. Theory A 30 (1981a), 71–89.

Gansner, E., The enumeration of plane partitions via the Burge correspondence, Illinois J. Math. 25 (1981b), 533–554. Gessel, I., G. Viennot, Binomial determinants, paths, and hook length formulae, Adv. in Math. 58 (1985), 300–321. Gessel, I., G. Viennot, Determinants, paths, and plane partitions, preprint. Krattenthaler, C., Generating functions for plane partitions of a given shape, Manuscripta Math. 69 (1990a), 173–202. Krattenthaler, C., A determinant evaluation and some enumeration results for plane partitions, Number-Theoretic Analysis”, E. Hlawka, R. F. Tichy eds., Lect. Notes in Math. 1452, Springer-Verlag, Berlin, New York, 1990b, pp. 121–131. Macdonald, I. G., Symmetric Functions and Hall Polynomials, Oxford University Press, New York/London, 1979. Proctor, R. A., Shifted plane partitions of trapezoidal shape, Proc. Amer. Math. Soc. 89 (1983), 553–559. Proctor, R. A., Odd symplectic group, Inventiones Math. 92 (1988), 307–332. Proctor, R. A., New symmetric plane partition identities from invariant theory work of De Concini and Procesi, Europ. J. Combinatorics 11 (1990), 289–300. Mills, W. H., D. H. Robbins, H. Rumsey, Proof of the Macdonald conjecture, Inventiones Math. 66 (1982), 73–87. Mills, W. H., D. H. Robbins, H. Rumsey, Alternating sign matrices and descending plane partitions, J. Combin. Theory A 34 (1983), 340–359. Mills, W. H., D. H. Robbins, H. Rumsey, Self–complementary totally symmetric plane partitions, J. Combin. Theory A 42 (1986), 277–292. Okada, S., On the generating functions for certain classes of plane partitions, J. Combin. Theory A 51 (1989), 1–23. Okada, S., Partially strict shifted plane partitions, J. Combin. Theory A 53 (1990), 143–156. Stanley, R. P., Theory and applications of plane partitions: Part 2, Stud. Appl. Math 50 (1971), 259–279. Stembridge, J., Nonintersecting paths, Pfaffians, and plane partitions, Adv. in Math. 83 (1990a), 96–131. Stembridge, J., Hall-Littlewood functions, plane partitions, and the Rogers-Ramanujan identities, Trans. Amer. Math. Soc. 319 (1990b), 469–498. Wachs, M. L., Flagged Schur functions, Schubert polynomials, and symmetrizing operators, J. Combin. Theory A 40 (1985), 276–289.

Institut für Mathematik der Universität Wien, Strudlhofgasse 4, A-1090 Wien, Austria.