LATTICE PATH PROOFS FOR DETERMINANTAL FORMULAS FOR SYMPLECTIC AND ORTHOGONAL CHARACTERS

MARKUS FULMEK AND CHRISTIAN KRATTENTHALER^{†}

Abstract. We give bijective proofs for Jacobi–Trudi-type and Giambelli-type identities for symplectic and orthogonal characters. These proofs base on interpreting King and El-Sharkaway’s symplectic tableaux, Proctor’s odd and intermediate symplectic tableaux, Proctor’s and King and Welsh’s orthogonal tableaux, and Sundaram’s odd orthogonal tableaux in terms of certain families of nonintersecting lattice paths. This work is intended to be the counterpart of the Gessel–Viennot proof of the Jacobi–Trudi identities for Schur functions for the case of symplectic and orthogonal characters.

1. Introduction Schur functions, which are irreducible general linear characters, can be combinatori-

ally defined by means of semistandard Young tableaux (cf. section 3 for all background information). There are several determinant formulas for Schur functions. Those which are relevant for this paper are the Jacobi–Trudi identity and its dual form, the Nägelsbach–Kostka formula, and the Giambelli identity (see (3.3), (3.4), (3.5)). In their well-known (yet unpublished) paper [11] (cf. [34, sec. 7]) Gessel and Viennot give a beautiful bijective proof for the Jacobi–Trudi identities for Schur functions. It bases on interpreting semistandard tableaux as families of nonintersecting lattice paths. As was shown by Stembridge [34, sec. 9], also the Giambelli identity allows a bijective proof by using nonintersecting lattice paths. (The first bijective proof of the Giambelli identity is due to Eğecioğlu and Remmel [6].)

There are Jacobi–Trudi-type and Giambelli-type determinant formulas for irreducible symplectic and orthogonal characters (see (3.9), (3.10), (3.11) (3.26), (3.27), (3.28)), too. Since there are also tableau descriptions for symplectic and orthogonal characters, it is natural to ask for bijective proofs of the symplectic and orthogonal Jacobi–Trudi identities (being partly due to Weyl [37] and partly to Koike and Terada [14]) and Giambelli identities (being due to Abramsky, Jahn and King [1]). First attempts in this direction for the Jacobi–Trudi identities were made by Bressoud and Wei [2], and more successfully by Okada [24]. However, the determinant formulas that Okada proved bijectively are different from the Jacobi–Trudi identities (but are interesting in their own right). Additional algebraic steps were necessary to prove the Jacobi–Trudi identities itself.

In this paper we solve the problem completely for the symplectic case and almost completely for the orthogonal case.

† Supported in part by EC’s Human Capital and Mobility Program, grant CHRX-CT93-0400, the second author was also supported by the Austrian Science Foundation FWF, grant P10191-MAT

2 MARKUS FULMEK AND CHRISTIAN KRATTENTHALER^{†}

For the bijective proofs of the symplectic identities we utilize lattice path interpretations of the tableaux given by King and El-Sharkaway [12], which are translations of the Gelfand patterns of Zhelobenko [38]. Particularly nice is the bijective proof for the dual symplectic Jacobi–Trudi identity (see (3.10)), which combines the Gessel–Viennot method with a “modified” reflection principle. This idea originates from [16] and is also exploited in [17, 18, 19]. A bijective proof for the “ordinary” symplectic Jacobi– Trudi identity (see (3.9)) requires more work. We give a proof that uses the concept of dual lattice paths due to Gessel and Viennot [10] and relies on our bijection for the symplectic dual Jacobi–Trudi identity. In [8] we provided a different bijective proof which is inspired by Okada’s combinatorial-algebraic proof [24, Lemma 3.1C, Corollary 4.2(2)]. On the other hand, the bijective proof of the symplectic Giambelli identity is almost trivial. In addition, we provide bijective proofs for all of Proctor’s [26, 28, 29] Jacobi–Trudi identities for his odd symplectic and intermediate symplectic characters (see (3.14), (3.15), (3.19)). Besides we add a new Giambelli-type identity (3.16) for his intermediate symplectic characters.

There are several candidates for orthogonal tableaux. We show that Proctor’s [29] and King and Welsh’s [13] orthogonal tableaux are the “right” tableaux for proving the orthogonal Jacobi–Trudi identities. As in the symplectic case, a bijective proof of the orthogonal dual Jacobi–Trudi identity (see (3.27)) is easier found. It also employs the Gessel–Viennot method and some kind of reflection argument. Our bijective proof of the “ordinary” orthogonal Jacobi–Trudi identity (see (3.26)) uses once more Gessel and Viennot’s idea of dual lattice paths and relies on our bijection for the orthogonal dual Jacobi–Trudi identity. Unfortunately, we are not able to use the same tableaux for a bijective proof of the orthogonal Giambelli identity. As a substitute, by using Sundaram’s tableaux [36] we give at least a bijective proof for the odd orthogonal Giambelli identity. There are other determinant formulas which can be proved by means of Sundaram’s tableaux. They are only slight variations of the odd orthogonal Jacobi–Trudi identities. Since this work also led to explicit bijections between the various odd orthogonal tableaux mentioned here [7], the only formula which we could not prove bijectively is the even orthogonal Giambelli identity.

It is remarkable that, while in case of Schur functions the bijective proofs for the “ordinary” and the dual Jacobi–Trudi identities are more or less identical, in the symplectic and orthogonal case the situation is quite different. Here only the combinatorics of the dual Jacobi–Trudi identities is transparent while the combinatorics of the “ordinary” Jacobi–Trudi identities is more intricate.

Our paper is organized in the following way. In the next section we introduce some standard notation concerning partitions and lattice paths. All relevant information about tableaux and Jacobi–Trudi and Giambelli identities is listed in section 3. This section is intended to serve as a reference. So it may be skipped first and used if it is referred to in later sections. Subsequently, in section 4 we briefly review the Gessel– Viennot proof of the Jacobi–Trudi identity for Schur functions and Stembridge’s proof of the Giambelli identity for Schur functions. This section can be skipped by readers who are familiar with the Gessel–Viennot business. Our bijective proofs for the (even) symplectic identities are the contents of section 5. Bijective proofs for Proctor’s Jacobi– Trudi identities for odd and intermediate symplectic characters and the new Giambelli identity for intermediate symplectic characters are the topic of section 6. Section 7 discusses the combinatorics of Proctor’s and King and Welsh’s orthogonal tableaux and how they can be used to provide bijective proofs for the orthogonal Jacobi–Trudi identities. Finally, in section 8 we exhibit which determinant formulas result from Sundaram’s odd orthogonal tableaux, including the odd orthogonal Giambelli identity. As a curious by-product of our methods we define “even Sundaram tableaux” which turn out to enumerate the dimensions of irreducible representations of the even orthogonal groups.

2. Notation As usual, an r-tuple λ = (λ_{1}, λ_{2}, …, λ_{r}) with λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r} > 0 is called a

partition of length r. We denote the length of the partition λ by ℓ(λ). The Ferrers board of λ is an array of cells with ℓ(λ) left-justified rows and λ_{i} cells in row i. The conjugate of λ is the partition (λ′_{1}, …, λ′_{λ_{1}}) where λ′_{j} is the length of the j-th column in the Ferrers board of λ.

Let λ = (λ_{1}, …, λ_{r}) be a partition. Call s = |{i : λ_{i} ≥ i}| the rank of λ (s is the number of diagonal cells in the Ferrers board of λ). Define α = (α_{1}, …, α_{s}) and β = (β_{1}, …, β_{s}) by α_{i} := λ_{i} − i, β_{j} := λ′_{j} − j. Clearly, α, β provide another unique encoding of partitions λ. It is called the Frobenius notation of λ, denote it by (α|β). A partition of rank 1 is called a hook. Let B be the Ferrers board of a partition λ. We define the k-th principal hook of B to be the set of cells B_{i,j} such that i = k ≤ j or j = k ≤ i. The parameters α_{i} and β_{i} can then be interpreted as the number of cells in the i-th principal hook that are above or below the main diagonal, respectively.

By paths we usually mean lattice paths in the plane integer lattice ℤ^{2} consisting of unit horizontal and vertical steps in the positive direction, unless we explicitly allow other steps. Given points u and v, we denote the set of all lattice paths from u to v by P (u, v). If u = (u_{1}, …, u_{m}) and v = (v_{1}, …, v_{m}) are vectors of points, we denote the set of all m-tuples (P_{1}, …, P_{m}) of paths, where P_{i} runs from u_{i} to v_{i}, i = 1, …, m, by P (u, v). A set of paths is said to be nonintersecting if no two paths of this set have a point in common, otherwise it is called intersecting. If to each horizontal edge a in ℤ^{2} a weight w(a) is assigned, the weight w(P) of a path P is defined to be the product of the weights of all its horizontal steps. The weight w(P) of an m-tuple P = (P_{1}, …, P_{m}) of paths is defined to be the product ∏_{i=1}^{m} w(P_{i}) of the weights of all the paths in the m-tuple. Given any weight function w defined on a set A, by the generating function GF (A) we mean ∑_{x∈A} w(x).

If u = (u_{1}, u_{2}, …, u_{m}) is an m-tuple of points and σ ∈ S_{m} (S_{m} is the permutation group on {1, 2, …, m}) we write u_{σ} for the permuted m-tuple (u_{σ(1)}, u_{σ(2)}, …, u_{σ(m)}). Another multi-notation which will be frequently used is the following. If some reflection R is given and ε ∈ {1, −1}^{m}, by u^{(ε)} we mean the m-tuple (u_{1}^{(ε)}, u_{2}^{(ε)}, …, u_{m}^{(ε)}) where the i-th component u_{i}^{(ε)} is u_{i} if ε_{i} = 1 and R (u_{i}) if ε_{i} = −1.

3. Tableaux, Jacobi–Trudi identities, Giambelli identities Let λ be a partition. A filling of the cells of the Ferrers board of λ with elements

of the set {1, 2, …, n} which is nondecreasing in rows and strictly increasing in the columns is called an n-semistandard (Young) tableau of shape λ. Frequently we shall say tableau for short. For a tableau T the filling of the cell (i, j) is denoted by T_{ij}. The following example shows such a semistandard tableau of shape (4, 3, 2).

2 3 4 6 4 4 6 5 6^{Figure 1}

Let x = ⟨x_{1}, x_{2}, …, x_{n}⟩ be a sequence of n variables. The weight x^{T} of an nsemistandard tableau T is defined as

x^{T} := ∏ x_{T_{i,j}}, (3.1)

where the product is over all entries T_{ij} of T. The Schur function s_{n} (λ, x) associated to a partition λ is combinatorially defined by (see [23, I, (5.12)], [30, Def. 4.4.1], [33, Def. 5.1])

s_{n} (λ, x) = ∑_{T} x^{T} , (3.2)

where the sum is over all n-semistandard tableaux T of shape λ.

With these definitions, the “classical” Jacobi-Trudi identity reads as follows: Let λ be a partition of length r ≤ n, x a set of variables. The Schur function s_{n} (λ, x) can be expressed as the following r × r-determinant (see [23, I, (3.4)]):

s_{n} (λ, x) = |h_{λ_{j}−j+i} (x)|_{r×r} , (3.3)

where h_{m} (x) = ∑_{1≤i_{1}≤···≤i_{m}≤n} x_{i_{1}} ⋯ x_{i_{m}} denotes the m-th complete homogeneous symmetric function. For obvious reasons, we call (3.3) the “h-formula”.

The “dual” Jacobi-Trudi identity, the Nägelsbach–Kostka formula (see [23, I, (3.5)]), looks quite similar (here the conjugate partition λ′ is involved):

s_{n} (λ, x) = |e_{λ′_{j}−j+i} (x)|_{λ_{1}×λ_{1}} , (3.4)

where e_{m} (x) = ∑_{1≤i_{1}<···<i_{m}≤n} x_{i_{1}} ⋯ x_{i_{m}} denotes the m-th elementary symmetric function. Call this the “e-formula”.

With the Frobenius notation for partitions which was explained in section 2 the Giambelli identity reads (see [23, p. 30, Ex. 9])

s_{n} ((α|β) , x) = |s_{n} ((α_{i}|β_{j}) , x)|_{s×s} . (3.5)

Now we turn to symplectic Schur functions (symplectic characters). Let again λ = (λ_{1}, …, λ_{r}) be a partition of length r ≤ n. A tableau T of shape λ is called a 2nsymplectic tableau if its entries are elements of {1, 2, …, 2n} and if it obeys the additional constraint

T_{i,j} ≥ 2i − 1. (3.6)

These tableaux were introduced by King and El-Sharkaway [12]. Figure 2 shows a symplectic tableau of shape (4, 3, 2).

2 3 4 6 4 4 6 5 6^{Figure 2}

For the symplectic case let x = ⟨x_{1}, x_{1}^{−1}, x_{2}, x_{2}^{−1}, …, x_{n}, x_{n}^{−1}⟩. Here the weight of a 2n-symplectic tableau T is given by

x^{T} = ∏_{l=1}^{n} x_{l}^{|{T_{i,j}=2l−1}|−|{T_{i,j}=2l}|}. (3.7)

So an odd entry 2i − 1 of T contributes x_{i} to the weight of T, while an even entry 2i of T contributes x_{i}^{−1} to the weight of T. The symplectic Schur function associated to λ is combinatorially defined by (see [35, Theorem 2.3])

sp_{2n} (λ, x) = ∑_{T} x^{T} , (3.8)

where the sum is over all 2n-symplectic tableaux T of shape λ.

The Jacobi–Trudi-type identities for symplectic Schur functions read as follows. The analogue of the “h-formula” (3.3) is (see [9, Prop. 24.22])

sp_{2n} (λ, x) = |h_{λ_{j}−j+1} (x) ⋮ h_{λ_{j}−j+i} (x) + h_{λ_{j}−j−i+2} (x)|_{r×r} . (3.9)

The notation of the determinant means that the first expression gives the entries of the first row and the second the entries for the remaining rows, i ≥ 2.

The symplectic analogue of the “e-formula” (3.4) is (see [9, Cor. 24.24])

sp_{2n} (λ, x) = |e_{λ′_{j}−j+i} (x) − e_{λ′_{j}−j−i} (x)|_{λ_{1}×λ_{1}} . (3.10)

Finally, with the Frobenius notation, which was explained in section 2, the symplectic analogue of the Giambelli identity (3.5) reads (see [9, (24.47)])

sp_{2n} ((α|β) , x) = |sp_{2n} ((α_{i}|β_{j}) , x)|_{s×s} . (3.11)

Next we consider Proctor’s intermediate symplectic Schur functions [28, 26]. Let n, m be nonnegative integers, and let x = ⟨x_{1}, x_{1}^{−1}, x_{2}, x_{2}^{−1}, …, x_{n}, x_{n}^{−1}⟩ and z = ⟨z_{1}, z_{2}, …, z_{m}⟩. In [28] Proctor defined polynomials sp_{n,m} (λ, μ; x, z) which interpolate between Schur functions and (ordinary) symplectic characters. To be more precise, for n = 0 they are Schur functions, and for m = 0 they are (ordinary) symplectic characters. These intermediate symplectic Schur functions are indexed by circled partitions, which are denoted by a pair (λ, μ) of partitions. Here, λ is a partition of length at most n + m, in which some parts are circled. Which parts of λ are not circled is indicated by the partition μ of length at most n. If there are both uncircled and circled parts of λ of the same size, then the circled parts are listed after the uncircled parts of that size. Thus the notation (λ, μ) uniquely determines which parts in λ are circled. For example, the circled partition which is denoted by ((5, 4, 4, 3, 1), (5, 4, 1)) means (5, 4, 4^{∘}, 3^{∘}, 1), where ^{∘} denotes the circled parts. The intermediate symplectic Schur functions sp_{n,m} (λ, μ; x, z) are defined by means of (n, m)-symplectic tableaux of shape (λ, μ). An (n, m)-symplectic tableau of shape (λ, μ) is a tableau T in the usual sense with entries from {1, 2, …, 2n + m} which obeys the symplectic constraint (3.6) for the first n+1 rows, i.e., for i ≤ n+1, and satisfies another rather intricate constraint which involves the jeu de taquin. We shall not give the general definition since we are only interested in special cases where the second constraint is either superfluous or very simple.

The first case which we are interested in is the case that μ consists of the first n parts of λ, i.e., that all parts λ_{i} with i ≤ n are uncircled and all parts λ_{i} with i > n are circled. Proctor denotes this special subpartition μ = (λ_{1}, …, λ_{n}) of λ by λ^{trnc} [28, p. 678]. For the shape (λ, λ^{trnc}) the second constraint is superfluous so that an (n, m)-symplectic tableau of shape (λ, λ^{trnc}) is a tableau in the usual sense with entries from {1, 2, …, 2n + m} such that (3.6) holds for i ≤ n + 1. The left tableau in Figure 3 is a (3, 2)-symplectic tableau of shape ((5, 4, 4, 3, 1), (5, 4, 4)).

2 2 4 4 5 3 4 5 6 5 7 7 7 7 8 8 8 ^{2 2 4 4 9 3 4 5 6 6 7 7 7 7 9 9 9}^{Figure 3}

The weight of an (n, m)-symplectic tableau is defined by

(x, z)^{T} = ∏_{l=1}^{n} x_{l}^{|{T_{i,j}=2l−1}|−|{T_{i,j}=2l}|} ∏_{k=1}^{m} z_{k}^{|{T_{i,j}=2n+k}|}. (3.12)

The (n, m)-symplectic Schur function associated to (λ, λ^{trnc}) is combinatorially defined by [28, p. 665]

(3.13) sp_{n,m} (λ, λ^{trnc}; x, z) = ∑_{T}(x, z)^{T} ,

where the sum is over all (n, m)-symplectic tableaux of shape (λ, λ^{trnc}).

If ℓ(λ) = r ≤ n + 1 there are Jacobi–Trudi-type identities for sp_{n,m} (λ, λ^{trnc}; x, z). The analogue of the “h-formula” (3.3) is [26, p. 317]

sp_{n,m} (λ, λ^{trnc}; x, z) = |h_{λ_{j}−j+1} (x, z) ⋮ h_{λ_{j}−j+i} (x, z) + h_{λ_{j}−j−i+2} (x, z)|_{r×r} . (3.14)

Here, h_{m}(x, z) means the m-th complete homogeneous symmetric function in x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1}, z_{1}, …, z_{m}.

We state Proctor’s analogue [28, Prop. 8.1] of the “e-formula” (3.4) in the following equivalent form,

(3.15) sp_{n,m} (λ, λ^{trnc}; x, z) = |e_{λ′_{j}−j+i} (x, z) − e_{λ′_{j}−j−i} (x, z)|_{λ_{1}×λ_{1}} .

If ℓ(λ) ≤ n + 1, we also find a Giambelli identity for sp_{n,m} (λ, λ^{trnc}; x, z), which is new. Let the Frobenius notation of λ be (α|β). The intermediate symplectic analogue of the Giambelli identity (3.5) reads

sp_{n,m} ((α|β) , (α|β)^{trnc}; x, z) = |sp_{n,m} ((α_{i}|β_{j}) , (α_{i}| min{β_{j}, n − 1}); x, z)|_{s×s} . (3.16)

The second case which we are interested in is the case that m = 1 and μ consists only of one part of λ, λ_{d} say. I.e., all parts of λ except one, namely the first part of λ of size λ_{d}, are circled. In this case the above mentioned second constraint translates into the following: An (n, 1)-symplectic tableau of shape (λ, λ_{d}) is a tableau in the usual sense with entries from {1, 2, …, 2n + 1} such that (3.6) holds for i ≤ n + 1 and such that the last entries in columns 1, 2, …, λ_{d} are equal to 2n + 1. The right tableau in Figure 3 is a (4, 1)-symplectic tableau of shape ((5, 4, 4, 3, 1), 3).

The weight of an (n, 1)-symplectic tableau is defined by

(x, z)^{T} = z^{|{T_{i,j}=2n+1}|} ∏_{l=1}^{n} x_{l}^{|{T_{i,j}=2l−1}|−|{T_{i,j}=2l}|}. (3.17)

The (n, 1)-symplectic Schur function associated to (λ, λ_{d}) is combinatorially defined by [28, p. 665]

sp_{n,1} (λ, λ_{d}, x, z) = ∑_{T}(x, z)^{T} , (3.18)

where the sum is over all (n, 1)-symplectic tableaux of shape (λ, λ_{d}). The (n, 1)- symplectic Schur functions are (partially conjectural) characters for Proctor’s odd symplectic groups [26].

Let ℓ(λ) = r. Since m = 1 we must have r ≤ n + 1. There is also a Jacobi–Truditype identity for sp_{n,1} (λ, λ_{d}, x, z). It is an analogue of the “e-formula” (3.4). We state Proctor’s formula [28, Prop. 9.1] in the following equivalent form,

(3.19) sp_{n,1} (λ, λ_{d}, x, z)

= |ze_{λ′_{j}−j+i−1} (x) − ze_{λ′_{j}−j−i−1} (x) ⋮ e_{λ′_{j}−j+i} (x, z) − e_{λ′_{j}−j−i} (x, z)|_{λ_{1}×λ_{1}} .

The notation of the determinant has to be understood in the sense that the first expression gives the entries of the columns j for 1 ≤ j ≤ λ_{d} and the second the entries for the remaining columns j, j > λ_{d}.

The definition of orthogonal tableaux is more complicated than that of (ordinary) symplectic tableaux. Actually there are several different candidates [12, 13, 15, 25, 29, 27, 36]. We shall use Proctor’s [29, 27] and King and Welsh’s [13] orthogonal tableaux in the first place. These can be divided into two groups, let us call them coarse orthogonal tableaux and fine orthogonal tableaux. The number of coarse orthogonal tableaux of shape λ equals the dimension of the irreducible representation of the orthogonal group indexed by λ. However, coarse orthogonal tableaux are not very well suited for describing the irreducible characters. This task is better performed by the fine orthogonal tableaux, which even appear in 4 variations, to be more precise, 2 for the even case and 4 for the odd case.

Let us begin with the definition of coarse orthogonal tableaux. Let λ = (λ_{1}, …, λ_{r}) be a partition of length r. A tableau is called coarse orthogonal if it satisfies the c-th orthogonal condition for all c ∈ ℕ. A tableau T of shape λ is said to satisfy the c-th orthogonal condition if the number of entries ≤ c in T ’s first two columns does not exceed c, i.e.,

|{T_{i,j} ≤ c : 1 ≤ j ≤ 2; 1 ≤ i ≤ λ′_{j}}| ≤ c. (3.20)

The following tableau satisfies the 1-st, 3-rd and 5-th orthogonal condition, but not the 2-nd and 4-th.

1 2 2 4 4 2 4 5 4^{Figure 4}

To define fine orthogonal tableaux, we need to introduce two further conditions, the c-th protection condition and the c-th broken strip condition. A tableau is said to satisfy the c-th protection condition if whenever i + j = c and T_{i,1} = c − 1 and T_{j,2} = T_{j,3} = ⋯ = T_{j,h−1} = c − 1 and T_{j,h} = c, we have T_{j−1,h} = c − 1. Stated in a more formally manner, this is

i + j = c and T_{i,1} = c − 1
and T_{j,2} = ⋯ = T_{j,h−1} = c − 1 (for h ≥ 2)
and T_{j,h} = c
(3.21)  implies T_{j−1,h} = c − 1.

We say that the entry T_{j,h} = c is protected above by c − 1.

The following tableau satisfies the 6-th protection condition if and only if 5 is inserted instead of ∗. (Choose i = 4 and j = 2.)

1 3 3 ∗ 6 2 5 5 6 3 7 8 5^{Figure 5}

A tableau is said to satisfy the c-th broken strip condition if whenever i + j = c, T_{i,1} = c and T_{i−1,1} ≠ c − 1 then we have T_{j,2} ≠ c − 1.

Now, fine orthogonal tableaux are defined by mixing the orthogonal, protection, and broken strip conditions appropriately. We call a tableau N-orthogonal of the first kind, if its entries are elements of {1, 2, …, N}, and if it satisfies the 2c-th orthogonal condition for all c, the N-th orthogonal condition (for even N this is already included in the first requirement, it is a new requirement only for odd N), and the 2c-th protection condition for all c. The tableau in Figure 5 is a fine orthogonal tableau of the first kind if 5 is inserted instead of ∗. We call a tableau N-orthogonal of the second kind, if its entries are elements of {1, 2, …, N}, and if it satisfies the 2c-th orthogonal condition for all c, the N-th orthogonal condition, the 2c-th broken strip condition for all c, and if T_{c,1} = 2c − 1 then all entries 2c in the c-th row have to be protected above by 2c − 1. (The last requirement is the 2c-th protection condition for the special case i = j = c.) Next we define the weight for these tableaux. If N is even, N = 2n, let x = ⟨x_{1}, x_{1}^{−1}, … x_{n}, x_{n}^{−1}⟩. If N is odd, N = 2n + 1, let x = ⟨x_{1}, x_{1}^{−1}, … x_{n}, x_{n}^{−1}, 1⟩. As in the symplectic case, the weight of an N-orthogonal tableau T of the first or second kind is given by

x^{T} = ∏_{l=1}^{n} x_{l}^{|{T_{i,j}=2l−1}|−|{T_{i,j}=2l}|}. (3.22)

Note that in the odd case N = 2n + 1 the entries 2n + 1 do not contribute anything to the weight of the tableau.

Now let N be odd, N = 2n+1. We call a tableau (2n+1)-orthogonal of the third kind, if its entries are elements of {1, 2, …, 2n+1}, and if it satisfies the (2c+1)-st orthogonal condition for all c, and the (2c + 1)-st protection condition for all c. We call a tableau (2n + 1)-orthogonal of the fourth kind, if its entries are elements of {1, 2, …, 2n + 1}, and if it satisfies the (2c+1)-st orthogonal condition for all c, and the (2c+1)-st broken strip condition for all c.

With x = ⟨x_{1}, x_{1}^{−1}, … x_{n}, x_{n}^{−1}, 1⟩, the weight of an N-orthogonal tableau T of the third or fourth kind is given by

x^{T} = ∏_{l=1}^{n} x_{l}^{|{T_{i,j}=2l}|−|{T_{i,j}=2l+1}|}. (3.23)

Note that here entries 1 do not contribute anything to the weight, entries 2 contribute x_{1}, entries 3 contribute x_{1}^{−1}, etc.

All the orthogonal tableaux just described are due to Proctor except for the fine orthogonal tableaux of the second kind, which are due to King and Welsh.

Both the coarse orthogonal tableaux with entries being at most N and the various fine N-orthogonal tableaux satisfy the N-th orthogonal condition. This implies that the shape λ for such tableaux has at most N cells in the first two columns. We call a partition with at most N cells in the first two columns an N-orthogonal partition. In particular, the length of an N-orthogonal partition is at most N.

The orthogonal Schur function associated to an N-orthogonal partition λ is combinatorially defined by (see [29, Theorems 6.1–6.3] or [27, p. 35])

o_{N} (λ, x) = ∑_{T} x^{T} , (3.24)

where the sum is over all N-orthogonal tableaux T of the κ-th kind which are of shape λ, where κ is any (fixed) number out of {1, 2, 3, 4}.

As is well-known, when we replace each x_{i} by 1 in o_{N} (λ, x) we obtain the dimension of the corresponding irreducible representation. As mentioned above, this dimension is equinumerous with the coarse orthogonal tableaux of shape λ. Therefore, we may write

(3.25) o_{N} (λ, ⟨1, 1, …, 1⟩) = |{T : T coarse orthogonal of shape λ, T_{i,j} ≤ N}| .

The Jacobi–Trudi-type identities for orthogonal Schur functions read as follows. The analogue of the h-formula (3.3) is (see [9, Ex. 24.46])

o_{N} (λ, x) = |h_{λ_{j}−j+i} (x) − h_{λ_{j}−j−i} (x)|_{r×r} . (3.26)

The analogue of the “e-formula” (3.4) is (see [9, Cor. 24.35])

o_{N} (λ, x) = |e_{λ′_{j}−j+1} (x) ⋮ e_{λ′_{j}−j+i} (x) + e_{λ′_{j}−j−i+2} (x)|_{λ_{1}×λ_{1}} . (3.27)

As in (3.9), the notation of the determinant means that the first expression gives the entries of the first row and the second the entries for the remaining rows, i ≥ 2. Recall that for both (3.26) and (3.27) the sequence x is different for N even or odd.

Finally, with the Frobenius notation, which was explained in section 2, the orthogonal analogue of the Giambelli identity (3.5) reads (see [9, (24.47)])

o_{N} ((α|β) , x) = |o_{N} ((α_{i}|β_{j}) , x)|_{s×s} . (3.28)

For the case N = 2n+1, Sundaram [36] defined different orthogonal tableaux. These (2n + 1)-Sundaram tableaux are tableaux with entries from the alphabet 1 < 2 < ⋯ < 2n < ∞ in the usual sense, except that column-strictness does not extend to symbol ∞ (i.e., symbol ∞ may occur more than once in a column), which have at most one entry ∞ in each row, and which obey the symplectic constraint (3.6). The weight of such a tableau is defined by (3.22), entries ∞ do not contribute to the weight. Then, if λ is a partition with ℓ(λ) ≤ n + 1, the odd orthogonal Schur functions o_{2n+1} (λ, x) may be also written as [36, Def. 2.3])

o_{2n+1} (λ, x) = ∑_{T} x^{T} , (3.29)

where the sum is over all Sundaram tableaux T of shape λ.

We should remark that different symplectic and orthogonal tableaux, which we do not consider at all in this paper, have been introduced by DeConcini, Procesi, Lakshmibai, Musili, Seshadri [4, 5, 20, 21, 31]. These tableaux are particularly useful for formulating a Littlewood–Richardson rule for all classical groups [22]. The symplectic tableaux of these authors and the symplectic tableaux of King and El-Sharkaway, which are considered in this paper, have recently been related by Sheats [32].

4. Lattice path proofs for Schur functions In this section we review the Gessel–Viennot method [11] (see [30, ch. 4]) of proving

the Jacobi–Trudi identities (3.3) and (3.4) for Schur functions and Stembridge’s lattice path proof [34, sec. 9] for the Giambelli identity for Schur functions. (A different combinatorial proof for the Giambelli identity has been given by Eğecioğlu and Remmel [6]).

The idea of Gessel and Viennot is to convert tableaux into nonintersecting lattice paths. Subsequently, from the nonintersecting lattice paths the determinants (3.3) and (3.4) can be easily deduced.

For proving the h-formula (3.3), given an n-semistandard tableau T of shape λ = (λ_{1}, …, λ_{r}), λ being a partition of length r ≤ n, we map T to an r-tuple (P_{1}, …, P_{r}) of nonintersecting lattice paths by associating to row i of T the path P_{i} from u_{i} = (−i, 1) to v_{i} = (λ_{i} − i, n), i = 1, 2, …, r, such that the entries in the i-th row are the heights of horizontal steps in the path P_{i}. That this r-tuple of paths is indeed nonintersecting comes from the column-strictness of T. Figure 6 shows the triple of paths associated to a 6-semistandard tableau T of shape λ = (4, 3, 2).

[figure: -3; -2; -1; 1; 2; 3; 2; 3; 5; 6; 3; 4; 6; 4; 5] ^{2 3 5 6 3 4 6 4 5 T =}^{Figure 6}

Conversely, given a set of nonintersecting lattice paths (P_{1}, …, P_{r}) where P_{i} runs from u_{i} = (−i, 1) to v_{i} = (λ_{i} − i, n), i = 1, 2, …, r, we can reconstruct the tableau by labelling each horizontal step of the paths by its height and then read off the i-th row of the tableau from the labels of the i-th path. Following [30] we call this labelling of horizontal steps by their height the h-labelling (see Figure 7).

[figure: 1; 2; 3; 4; 5; 1; 2; 3; 4; 5; h-labelling:; 1; 1; 1; 1; 1; 2; 2; 2; 2; 2; 3; 3; 3; 3; 3; 4; 4; 4; 4; 4; 5; 5; 5; 5; 5; level 1; level 2; level 3; level 4; level 5; -1; 1; 2; 3; 4; 1; 2; 3; 4; 5; e-labelling:; 1; 2; 3; 4; 2; 3; 4; 5; 3; 4; 5; 6; 4; 5; 6; 7; 5; 6; 7; 8; 6; 7; 8; 9]

Figure 7

For proving the e-formula (3.4), we have to introduce yet another labelling. Now we label each horizontal step from (x − 1, y) to (x, y) by x + y. Call this labelling the e-labelling (see Figure 7). Given an n-semistandard tableau T of shape λ, we map T to a λ_{1}-tuple (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths by associating to column i of T a path P_{i} from u_{i} = (−i+1, i−1) to v_{i} = (λ′_{i} −i+1, n−λ′_{i} +i−1), i = 1, 2, …, λ_{1}, such that the entries of the i-th column can be read off from the labels of the horizontal steps of P_{i}. That the paths are nonintersecting in this case comes from the fact that entries are nondecreasing along rows of T. Figure 8 shows the quadruple of paths associated to a 6-semistandard tableau T of shape λ = (4, 3, 2).

[figure: -4; -3; -2; -1; 1; 2; 3; 4; 5; 2; 4; 5; 3; 4; 6; 4; 6; 6] ^{2 3 4 6 4 4 6 5 6 T =}^{Figure 8}

Let l be any of these two labellings. We define the weight w_{l}(a) of a horizontal step a to be w_{l}(a) := x_{l(a)}. This defines a weight for paths and m-tuples of paths (see section 2). Obviously, both of the above mappings from tableaux to nonintersecting lattice paths are weight-preserving.

The proof for both the h- and the e-formula can now be given in a uniform manner. In both cases we are interested in finding the generating function (using the h-labelling in the first case, respectively the e-labelling in the second case) for m-tuples (P_{1}, …, P_{m}) of nonintersecting lattice paths, where P_{i} runs from u_{i} to v_{i}, i = 1, 2, …, m, (m = r or m = λ_{1}, respectively, u_{i} and v_{i} are defined in each of the two cases as above.) We claim that these generating functions can be written as the determinant

|GF (P (u_{i}, v_{j}))|_{m×m} . (4.1)

To see this, with the notation of section 2 we expand the determinant in (4.1) into

|GF (P (u_{i}, v_{j}))|_{m×m} = ∑_{σ∈S_{m}} sgn (σ) ∏_{i=1}^{m} GF (P (u_{i}, v_{σ(i)}))

= ∑_{σ∈S_{m}, P∈P(u,v_{σ})} sgn (σ) w_{l}(P).

Thus the determinant in (4.1) is expressed as a generating function for m-tuples (P_{1}, …, P_{m}) of paths where P_{i} runs from u_{i} to v_{σ(i)}, i = 1, 2, …, m, for some permutation σ. We shall find a weight-preserving but sign-reversing involution on the set of all those m-tuples of paths that are intersecting. Given such an involution, after cancelling all such m-tuples of paths by that involution, only the nonintersecting m-tuples will survive, as was claimed. And, because of the location of the starting and end points of the paths, the associated permutation for such m-tuples must be the identity. As we saw above, these m-tuples of paths exactly correspond to n-semistandard tableaux of shape λ. It is easy to see that GF (P (u_{i}, v_{j})) evaluates to h_{λ_{j}−j+i} (x) in the case of the h-labelling, and to e_{λ′_{j}−j+i} (x) in the case of the e-labelling. This would prove the assertions.

Consider an m-tuple (P_{1}, …, P_{m}) ∈ P (u, v_{σ}) of paths that is intersecting. In the lexicographic order of the integer lattice look for the largest point of intersection of this m-tuple, p say. Let P_{i}, P_{j} be the paths having point p in common. Now interchange terminal portions (starting at p) of P_{i} and P_{j}, i.e., replace P_{i} by

P′_{i} := [subpath of P_{i} from u_{i} to p joined with subpath of P_{j} from p to v_{σ(j)}]

and P_{j} by

P′_{j} := [subpath of P_{j} from u_{j} to p joined with subpath of P_{i} from p to v_{σ(i)}] .

Clearly, this is an involution that preserves weight and reverses sign (since it corresponds to multiplying the initial permutation σ with the transposition (i, j)). We call this involution the Gessel–Viennot involution.

Now we turn to Stembridge’s [34, sec. 9] bijective proof of the Giambelli identity (3.5). We present Stembridge’s construction in a slightly modified fashion. Again, we consider paths in the integer lattice consisting of unit horizontal and vertical steps. While the direction of horizontal steps is always positive, the direction of a vertical step, however, is only positive if it is strictly to the right of the y-axis, and is negative if it does not lie strictly to the right of the y-axis. As we shall see, this change of direction corresponds to change of increasing/decreasing behaviour when reading the entries of a tableau’s principal hooks.

To the left of the horizontal axis we consider a “shifted” h-labelling by assigning label i + 1 to an edge at height i, while we consider the “usual” e-labelling to the right (recall Figure 7). As before, the weight of a horizontal edge with label l is defined to be x_{l}. Given an n-semistandard tableau of shape (α|β), we associate to its i-th principal hook a path P_{i} from (−α_{i}, n − 1) to (β_{i} + 1, n − β_{i} − 1) by interpreting the entries of the hook (read from “right to bottom”) as labels of the corresponding steps of P_{i}.

Figure 9 shows the pair of nonintersecting lattice paths associated to a 6-semistandard tableau T of shape λ = (4, 3, 2) (rank 2, Frobenius notation (α|β) = (3, 1|2, 1)).

[figure: -4; -3; -2; -1; 1; 2; 3; 4; 6; 4; 3; 2; 4; 5; 6; 4; 6] T = ^{2 3 4 6 4 4 6 5 6}^{Figure 9}

The construction might look complicated at first sight, but note that the right half of the picture is simply the same as in the case of associated “e-paths”, while the left half corresponds precisely to associated “h-paths”, shifted one unit down and reflected in some vertical line. Hence nonintersecting s-tuples (here, s is the rank of the partition λ = (α|β)) of such paths bijectively correspond to tableaux. This is clear by the above observation for both half-planes and by the fact that steps contained in the vertical axis are downwards directed. (The picture’s left and right half correspond to upper and lower part of the partition’s Ferrers board cut in two just above the diagonal. Nonintersecting paths in both halves imply nondecreasing rows and strictly increasing columns for both of these parts; downward directed vertical steps on the y-axis imply nondecreasing entries along the rows when glued together, from which column-strictness follows immediately.)

In particular, this means that for a hook (a|b) (a partition with rank 1) the corresponding Schur function s_{n} ((a|b), x) is just the generating function of the family of lattice paths starting in (−a, n − 1) and ending in (b + 1, n − b − 1). Hence application of the Gessel–Viennot method immediately leads to the Giambelli identity (3.5).

5. Symplectic identities: The tableaux of King and El-Sharkaway In this section we give bijective proofs for (3.9), (3.10), and (3.11). We start with

the proof of the “e-formula” (3.10).

Bijective proof of the symplectic dual Jacobi–Trudi identity (3.10). Let T be a 2n-symplectic tableau of shape λ, where λ is a partition of length r ≤ n. In the same way as in section 4 (see Figure 8), with T we associate a λ_{1}-tuple (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths where P_{i} runs from u_{i} = (−i + 1, i − 1) to v_{i} = (λ′_{i} − i + 1, 2n − λ′_{i} + i − 1), i = 1, 2, …, λ_{1}, by reading P_{i} off the i-th column. Obviously, T obeys the symplectic constraint if and only if the first of the associated paths (and hence all of the associated paths) does not cross the line y = x−1. Figure 10 gives an example for n = 3 and λ = (4, 3, 2).

[figure: -4; -3; -2; -1; 1; 2; 3; 4; 5; 2; 4; 5; 3; 4; 6; 4; 6; 6] ^{2 3 4 6 4 4 6 5 6 T =}^{Figure 10}

If we define the weight w(a) of a horizontal edge a with e-label (cf. Figure 7) 2i − 1 to be x_{i}, and the weight w(a) of a horizontal edge a with e-label 2i to be x_{i}^{−1} then the correspondence depicted in Figure 10 is weight-preserving with respect to the weight (3.7). Therefore the left side of (3.10) can be interpreted as generating function for all λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i+1, i−1) to (λ′_{i} − i + 1, 2n − λ′_{i} + i − 1), i = 1, 2, …, λ_{1}, and does not cross y = x − 1.

Our bijective proof combines the Gessel–Viennot involution with a modification of the reflection-principle (cf. e.g. [3, p. 22]). This idea originates from [16].

First we find a lattice path interpretation of the determinant at the right side of (3.10) by expanding it. With R denoting the reflection in the line y = x − 2, we have

|e_{λ′_{j}−j+i} (x) − e_{λ′_{j}−j−i} (x)|_{λ_{1}×λ_{1}} = |GF (P (u_{i}, v_{j})) − GF (P (R(u_{i}), v_{j}))|_{λ_{1}×λ_{1}} (5.1)

= ∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} sgn (σ) ∏_{i=1}^{λ_{1}} ε_{i} ∏_{i,ε_{i}=1} GF (P (u_{i}, v_{σ(i)})) · ∏_{i,ε_{i}=−1} GF (P (R(u_{i}), v_{σ(i)}))

= _{∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) sgn (ε) w(P),

where sgn (ε) stands for ∏_{i=1}^{λ_{1}} ε_{i}. The notation u^{(ε)} is explained in section 2. The weight w(P) is induced (as described in section 2) by the edge weight that was introduced just before.

By (5.1) we have written the determinant in (3.10) as a generating function for λ_{1}- tuples (P_{1}, …, P_{λ_{1}}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, λ_{1}. In order to establish (3.10), we shall give an involution that cancels all those λ_{1}-tuples of paths that are either intersecting or contain a path that crosses the line y = x − 1.

[figure: -1; 1; 2; 3; 4; 5; 6; 7; 8; p] Figure 11

First, we define the announced modification of the reflection-principle. The reflectionprinciple itself could not be used for our purposes since reflection of paths is not weightpreserving with respect to the weight of paths that we are considering. A path P crossing the line y = x − 1 must meet the line y = x − 2. Let p be the last meeting point. To P ’s initial portion up to p we may apply the following modified reflection in y = x − 2. (See Figure 11.) All points (x, y) ∈ P with x + y ≡ 0 (mod 2) (call them even points) are reflected in the usual way, i.e., (x, y) ↦ (y + 2, x − 2) (see the points (0, 0), (0, 2), (2, 2), (3, 3), (4, 4), (6, 4) in Figure 11), and so are all odd points (x + y ≡ 1 (mod 2)) whose adjacent steps are both vertical or both horizontal (see the points (0, 1), (1, 2), (5, 4) in Figure 11). The remaining case is a “kink” in an odd point q, i.e., a horizontal step of the path meets a vertical one in q. Here the whole kink is shifted until its even points have reached their new (reflected) positions. That is, if q = (x, y) is reached by a vertical step and left by a horizontal step then (x, y) is mapped to (y + 1, x − 1) (see the point (2, 3) in Figure 11), and if q = (x, y) is reached by a horizontal step and left by a vertical step then (x, y) is mapped to (y+3, x−3) (see the point (4, 3) in Figure 11). Figure 11 gives an example of this modified reflection.

Now we are able to describe the desired weight-preserving and sign-reversing involution. For a λ_{1}-tuple P containing a path meeting the forbidden line y = x − 2, choose i minimal such that P_{i} meets y = x − 2, and replace P_{i}’s portion up to the last meeting point with the line y = x − 2 by its modified reflection. Clearly, this mapping is weight-preserving and sign-reversing. It reverses sign since it changes the sign sgn (ε) of ε while leaving σ invariant. On λ_{1}-tuples of paths that do not contain any path crossing y = x − 1 but are intersecting we apply the Gessel-Viennot involution. Clearly, we thus obtain again a λ_{1}-tuple of paths that does not contain any path crossing y = x − 1 but is intersecting. It is therefore straight-forward to see that this mapping is a weightpreserving and sign-reversing involution. Thus only those λ_{1}-tuples remain that are nonintersecting and where none of their paths cross the line y = x − 1. Because the paths do not cross y = x − 1, the starting points must lie above y = x − 1, hence ε = (1, 1, …, 1). Again, since they are nonintersecting, because of the location of the starting and ending points the associated permutation must be the identity. But as was exhibited above, these λ_{1}-tuples correspond to symplectic tableaux.

Remark. In [24, Lemma 3.2C] basically the same lattice path interpretation is found but without observing that the restriction on the paths can be formulated by a noncrossing condition. Therefore, the reflection argument does not appear, and the formula (3.10) is not proved purely bijectively.

Bijective proof of the symplectic Jacobi–Trudi identity (3.9). Proving the h-formula (3.9) bijectively is more difficult. In [8] we gave an algorithmic proof inspired by Okada’s combinatorial-algebraic proof [24, Lemma 3.1C, Corollary 4.2(2)]. In the proof that we are going to describe here we use the concept of dual lattice paths, which is due to Gessel and Viennot [10, sec. 4]. The idea is to pass from “h-paths” to “e-paths” by “dualization”, and then rely on a variation of the bijection for the symplectic dual Jacobi–Trudi identity, which we have given just above.

Let T be a 2n-symplectic tableau of shape λ, where λ is a partition of length r ≤ n. In the same way as in section 4 (see Figure 6), by reading paths from the rows, we map T to an r-tuple (P_{1}, …, P_{r}) of nonintersecting lattice paths, where P_{i} runs from u_{i} = (−i, 1) to v_{i} = (λ_{i} − i, 2n), i = 1, 2, …, r. Obviously, the symplectic constraint translates into the condition that the first horizontal step of path P_{i} must be at least at height 2i − 1. We define the weight w(a) of a horizontal edge a with h-label (cf. Figure 7) 2i − 1 to be x_{i}, and the weight w(a) of a horizontal edge a with h-label 2i to be x_{i}^{−1} so that this correspondence is weight-preserving with respect to the weight (3.7).

Next we expand the determinant in (3.9) and find its lattice path interpretation. In the following R denotes the reflection in the line x = −1. Using again notation from section 2, we have

|h_{λ_{j}−j+1} (x) ⋮ h_{λ_{j}−j+i} (x) + h_{λ_{j}−j−i+2} (x)|_{r×r} (5.2)

= |GF (P (u_{1}, v_{j})) ⋮ GF (P (u_{i}, v_{j})) + GF (P (R(u_{i}), v_{j}))|_{r×r}

= ∑_{σ∈S_{λ_{1}}, ε∈{1}×{1,−1}^{r−1}} sgn (σ) ∏_{i,ε_{i}=1} GF (P (u_{i}, v_{σ(i)})) · ∏_{i,ε_{i}=−1} GF (P (R(u_{i}), v_{σ(i)}))

= _{∑_{σ∈S_{r}, ε∈{1}×{1,−1}^{r−1}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) w(P),

where the weight w(P) is induced (as explained in section 2) by the edge weight which was introduced just before. Thus the determinant in (3.9) is expressed as a generating function for r-tuples (P_{1}, …, P_{r}) of paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, r, and where P_{1} has to start in u_{1}. We shall give a weight-preserving and sign-reversing involution that among these r-tuples of paths cancels all those that are either intersecting, or contain a path P_{i} the first horizontal step of which is strictly below the line y = 2i − 1, or where the associated ε differs from (1, 1, …, 1). Suppose that this had been done. The remaining r-tuples are nonintersecting and the associated ε equals (1, 1, …, 1), so again the associated permutation must be the identity. Finally, the first horizontal step of P_{i} is at least at height 2i − 1, i = 1, 2, …, r. As we saw above, these r-tuples of paths exactly correspond to 2n-symplectic tableaux of shape λ. This would prove (3.9).

Let P = (P_{1}, …, P_{r}) be one of the r-tuples of paths in P (u^{(ε)}, v_{σ}) which have to be cancelled. Again note that P_{1} has to start in u_{1}.

If P is intersecting we apply the Gessel–Viennot involution. If P is nonintersecting, then in the first step we map P = (P_{1}, …, P_{r}) to a λ_{1}-tuple Q = (Q_{1}, …, Q_{λ_{1}}) of nonintersecting lattice paths, which we call the dual of P.

The correspondence between a set of paths and its dual is best described with a picture at hand. To start with, add a vertical step at the beginning of each of the paths of P so that, in abuse of notation, we obtain an r-tuple P of nonintersecting lattice paths where P_{i} starts at either u_{i} = (−i, 0) or R(u_{i}) with a vertical step and runs to v_{σ(i)}. The (bold) paths in Figure 12.a show a 6-tuple P of nonintersecting lattice paths with λ = (4, 4, 4, 4, 2, 1), σ = (2, 3, 1, 4, 5, 6), and ε = (1, 1, −1, 1, 1, 1), n is chosen to be 6.

The starting points of P occupy r lattice points on the line y = 0, the end points occupy r lattice points on the line y = 2n. Now, in the region −r ≤ x ≤ λ_{1} − 1 let w_{1}, w_{2}, …, w_{λ_{1}} (from left to right) be the lattice points on y = 0 not occupied by P, and \overline{v}_{1}, \overline{v}_{2}, …, \overline{v}_{λ_{1}} be the lattice points on y = 2n not occupied by P. Then we connect w_{i} to \overline{v}_{i} by a lattice path, i = 1, 2, …, λ_{1}, by starting at w_{i} and going up vertically unless we would meet a lattice path of P. In the latter case we proceed by an upward diagonal step in direction (−1, 1). The dotted paths in Figure 12.a show the result in our example. Besides, in Figure 12.a the starting points of the new paths are also labelled in the following way: The points (j − 1, 0), j = 1, 2, …, λ_{1}, which are not

[figure: a. The ,,h-paths“ P; b. Picture reflected in x = 0; c. Marking of paths interchanged; d. After deformation of the plane, the “e-paths” Q,; the dual of P; y = x − 2; ^{\overline{u}^{1}}; ^{\overline{v}^{2}}; ^{\overline{u}^{3}}; ^{\overline{v}^{3}}; R\overline{u}_{2}; ^{\overline{v}^{1}}; ^{\overline{u}^{4}}; ^{\overline{v}^{4}}; Ru_{3}; v_{1}; u_{1}; v_{2}; u_{2}; v_{3}; u_{4}; v_{4}; u_{5}; v_{5}; u_{6}; v_{6}; x = 1; ^{\overline{u}^{1}}; ^{\overline{v}^{2}}; ^{\overline{u}^{3}}; ^{\overline{v}^{3}}; R\overline{u}_{2}; ^{\overline{v}^{1}}; ^{\overline{u}^{4}}; ^{\overline{v}^{4}}; Ru_{3}; v_{1}; u_{1}; v_{2}; u_{2}; v_{3}; u_{4}; v_{4}; u_{5}; v_{5}; u_{6}; v_{6}; x = 1; ^{\overline{u}^{1}}; ^{\overline{v}^{2}}; ^{\overline{u}^{3}}; ^{\overline{v}^{3}}; R\overline{u}_{2}; ^{\overline{v}^{1}}; ^{\overline{u}^{4}}; ^{\overline{v}^{4}}; Ru_{3}; v_{1}; u_{1}; v_{2}; u_{2}; v_{3}; u_{4}; v_{4}; u_{5}; v_{5}; u_{6}; v_{6}; x = −1; ^{\overline{u}^{1}}; ^{\overline{v}^{2}}; w_{2}; ^{\overline{u}^{3}}; ^{\overline{v}^{3}}; w_{3}; R\overline{u}_{2}; ^{\overline{v}^{1}}; w_{1}; ^{\overline{u}^{4}}; ^{\overline{v}^{4}}; w_{4}; Ru_{3}; v_{1}; u_{1}; v_{2}; u_{2}; v_{3}; u_{4}; v_{4}; u_{5}; v_{5}; u_{6}; v_{6}]

Figure 12

occupied by P are labelled by \overline{u}_{j}, and the points (−j − 1, 0), j = 1, 2, …, r − 1, which are not occupied by P are labelled by R(\overline{u}_{j}).

In the next step we reflect the whole picture in the line x = 0, see Figure 12.b. Then we exchange the roles of P, the old paths, and the new paths. In Figure 12.c this is indicated by marking the new paths with bold lines and the old paths with dotted lines. In the final step we deform the plane by the linear transformation (x, y) → (x, y − x), see Figure 12.d.

The λ_{1}-tuple Q of paths (bold in Figure 12.d) that we finally obtained by this procedure is defined to be the dual of P. The bold paths in Figure 12.d should remind us of the objects that were involved in the bijective proof of the symplectic dual Jacobi– Trudi identity. Indeed, let \overline{u}_{i} = (−i + 1, i − 1) and \overline{v}_{i} = (λ′_{i} − i + 1, 2n − λ′_{i} + i − 1), and let \overline{R} denote the reflection in the line y = x − 2. Then the λ_{1}-tuple of paths Q = (Q_{1}, …, Q_{λ_{1}}) is nonintersecting, and, for some permutation \overline{σ} ∈ S_{λ_{1}} and some sign vector \overline{ε} ∈ \{1, −1\}^{λ_{1}}, the path Q_{i} runs from \overline{R}^{(1−\overline{ε}_{i})/2}(\overline{u}_{i}) (i.e., either \overline{u}_{i} or \overline{R}(\overline{u}_{i}) depending on whether \overline{ε}_{i} = 1 or −1) to \overline{v}_{\overline{σ}(i)}. In the example of Figure 12 we have λ′ = (6, 5, 4, 4), \overline{σ} = (2, 1, 3, 4) and \overline{ε} = (1, −1, 1, 1). Moreover, there holds

(5.3) sgn (σ) = sgn (\overline{σ}) · sgn (\overline{ε}) .

This is easily checked in our example, but before we verify this fact in general, we finish the description of the involution.

The basic idea is to apply the mapping involving the “modified reflection” used in the proof of the symplectic dual Jacobi–Trudi identity (i.e., choose i minimal such that Q_{i} meets y = x − 2, and replace Q_{i}’s portion up to the last meeting point with the line y = x − 2 by its modified reflection; cf. Fig. 11) and after that map the resulting λ_{1}- tuple of paths to its preimage under dualization. However, the application of modified reflection may introduce new points of intersections of paths, which we cannot permit, since dualization does not work with intersecting paths.

There is an easy way to come around this problem. Let A denote the mapping involving the “modified reflection”, and let B denote the Gessel–Viennot involution. Now, starting with Q, apply A, then B, then A again, etc., as long as possible. Formally this is B^{[0 or 1]} ∘ (A ∘ B)^{m} ∘ A, for some m. The algorithm terminates because the set of λ_{1}-tuples under consideration is finite and there cannot be any cycle. For, both A and B are actually involutions on the set of λ_{1}-tuples of paths containing at least one path that meets the line y = x − 2, B being not defined for nonintersecting λ_{1}-tuples. Now think of this set of λ_{1}-tuples as the vertices of a graph where two vertices O_{1} and O_{2} are connected by an edge if O_{1} and O_{2} are mapped to each other by either A or B. This graph has maximal degree 2, hence consists of paths and cycles. The “starting objects” for our algorithm, which are nonintersecting λ_{1}-tuples, all have degree 1 in this graph (only operation A is applicable), therefore we cannot run into a cycle. Since A can always be applied, the algorithm has to stop with an application of A. But that means that we finally obtained some nonintersecting λ_{1}-tuple, Q′ say.

Next note that, if σ′ is the permutation corresponding to Q′ and if ε′ is the sign vector corresponding to Q′, then we have

(5.4) sgn (\overline{σ}) · sgn (\overline{ε}) = − sgn (σ′) · sgn (ε′) .

This is because we started from Q by an application of A and arrived at Q′ by an application of A, therefore, to go from Q to Q′, we applied an odd number of operations A and B, A changing the sign of the corresponding sign vector, B changing the sign of the corresponding permutation.

In the last step we determine the preimage of Q′ under dualization, P̂ say. In fact, it is easy to check that each step of the dualization process can be reversed so that P̂ is well-defined.

Obviously, the map from P to P̂ is weight-preserving. To check that it is also signreversing, let σ̂ be the permutation corresponding to P̂. Because of (5.3), which also applies to σ̂, σ′, ε′, and (5.4), we have

sgn (σ) = sgn (\overline{σ}) · sgn (\overline{ε}) = − sgn (σ′) · sgn (ε′) = − sgn \binom{}{σ̂} .

Thus the mapping from P to P̂ is sign-reversing.

Finally, we have to prove (5.3). Recall that P and Q are related by dualization as exhibited in Figure 12. Let s and \overline{s} be the number of reflected paths of P and Q, respectively. Since P is nonintersecting, the permutation σ corresponding to P must satisfy the inequalities

σ^{−1}(1) > σ^{−1}(2) > ⋯ > σ^{−1}(s); σ^{−1}(s + 1) < σ^{−1}(s + 2) < ⋯ < σ^{−1}(r).

Likewise, the permutation \overline{σ} corresponding to Q must satisfy

\overline{σ}^{−1}(1) > \overline{σ}^{−1}(2) > ⋯ > \overline{σ}^{−1}(s̄); \overline{σ}^{−1}(s̄ + 1) < \overline{σ}^{−1}(s̄ + 2) < ⋯ < \overline{σ}^{−1}(λ_{1}).

Therefore the number of inversions of σ^{−1} is

inv(σ^{−1}) = ∑_{k=1}^{s}(σ^{−1}(k) − 1), (5.5)

while the number of inversions of \overline{σ}^{−1} is

inv(\overline{σ}^{−1}) = ∑_{k=1}^{\overline{s}}(\overline{σ}^{−1}(k) − 1). (5.6)

How are s and \overline{s}, σ^{−1}(k) and \overline{σ}^{−1}(k) related? By construction of dualization, the reflected starting points of P are exactly the reflections of the reflected starting points of Q, see Figure 12. In fact, if a path of P starts at some point then there cannot be a path of P starting at its reflection, hence it must be occupied by a starting point of a path of Q, and vice versa. Therefore the number of reflected starting points in P and Q is the same, which simply says that s = \overline{s}. Besides, because of the particular numbering of the starting points of P and Q (which is caused by the fact that P’s first starting point, u_{1}, has to be on the reflection line, and, hence, Q’s starting points avoid the reflection line) we have

σ^{−1}(k) − 1 = \overline{σ}^{−1}(k) for k = 1, 2, …, s. (5.7)

A combination of (5.5), (5.6), (5.7), and s = \overline{s}, gives

sgn (σ) = (−1)^{inv(σ^{−1})}
= (−1)^{inv(\overline{σ}^{−1})+s}
= sgn (\overline{σ}) · sgn (\overline{ε}) ,

which confirms (5.3). This finishes the proof.

Bijective proof of the symplectic Giambelli identity (3.11). We use the same encoding of symplectic tableaux as was used in the proof of the “ordinary” Giambelli identity (3.5) for tableaux (recall Figure 9). That is, we map a symplectic tableau of shape (α, β), where (α, β) is the Frobenius notation of a partition of rank s, to an s-tuple (P_{1}, …, P_{s}) of lattice paths where P_{i} runs from (−α_{i}, 2n − 1) to (β_{i} + 1, 2n − β_{i} − 1), i = 1, 2, …, s. Figure 13 shows an example (with n = 3) of this correspondence between nonintersecting lattice paths and symplectic tableaux.

[figure: -4; -3; -2; -1; 1; 2; 3; 4; 6; 4; 3; 2; 4; 5; 6; 4; 6] T = ^{2 3 4 6 4 4 6 5 6}^{Figure 13}

As is clear from the picture, the symplectic constraint translates into the condition that the first path must not cross the line y = x − 1. If, as before, we define the weight of a horizontal edge with label 2i − 1 to be x_{i}, and the weight of a horizontal edge with label 2i to be x_{i}^{−1}, then this correspondence is weight-preserving with respect to the weight (3.7).

Obviously, the set of all s-tuples of lattice paths subject to this condition is invariant under Gessel–Viennot involution. Therefore the same arguments as in Stembridge’s proof of the “ordinary” Giambelli identity, which was reviewed in section 4, establish (3.11).

6. Symplectic identities: The odd and intermediate symplectic tableaux of Proctor In this short section we give bijective proofs for (3.14), (3.15), (3.16), and (3.19).

We are going to rely on the bijections of the previous section, so there is almost no additional work to be done.

Bijective proof of the (n, m)-symplectic dual Jacobi–Trudi identity (3.15). In the same way as in the proof of the (ordinary) symplectic dual Jacobi–Trudi identity (3.10) given in the previous section (see Figure 10), we map (n, m)-symplectic tableaux of shape (λ, λ^{trnc}) to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i + 1, i − 1) to (λ′_{i} − i + 1, 2n + m − λ′_{i} + i − 1), i = 1, 2, …, λ_{1}. In general, the symplectic constraint would translate into the condition that the path P_{1} (and hence all other paths) must not cross the line segment {(x, y) : y = x − 1 and x ≤ n + 1}. However, since for (3.15) we have to assume ℓ(λ) ≤ n + 1, the number of horizontal steps of each path cannot exceed n + 1, and therefore all the paths do not cross the complete line y = x−1. This shows that for ℓ(λ) ≤ n+1 the (n, m)-symplectic tableaux of shape (λ, λ^{trnc}) correspond to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from u_{i} = (−i+1, i−1) to v_{i} = (λ′_{i}−i+1, 2n+m−λ′_{i}+i−1), i = 1, 2, …, λ_{1}, and does not cross y = x−1. If we define the weight w(a) of a horizontal edge a with e-label (cf. Figure 7) 2i − 1, i ≤ n, to be x_{i}, the weight w(a) of a horizontal edge a with e-label 2i, i ≤ n, to be x_{i}^{−1}, and the weight of a horizontal edge with e-label i, i > 2n, to be z_{i−2n}, then the above correspondence is weight-preserving with respect to the weight (3.12).

Let R be the reflection in the line y = x − 2. Again, the determinant in (3.15) may be written as a generating function for λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, λ_{1},

|e_{λ′_{j}−j+i} (x, z) − e_{λ′_{j}−j−i} (x, z)|_{λ_{1}×λ_{1}} = _{∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) sgn (ε) w(P).

The involution that cancels λ_{1}-tuples P where a path of P crosses y = x−1 is defined in the same way as in the proof of (3.10) in the previous section. This is a weightpreserving mapping since the right-most possible crossing point of a path in P with y = x − 1 is (n, n − 1), and hence the steps that are involved in the modified reflection, depicted in Figure 11, all have e-label at most 2n. Of course, for the intersecting λ_{1}-tuples we apply the Gessel–Viennot involution.

Bijective proof of the (n, m)-symplectic Jacobi–Trudi identity (3.14). In the same way as in the proof of the (ordinary) symplectic Jacobi–Trudi identity (3.9) given in the previous section (see Figure 6), we map (n, m)-symplectic tableaux of shape (λ, λ^{trnc}) to r-tuples (P_{1}, …, P_{r}) of nonintersecting lattice paths, where P_{i} runs from (−i, 1) to (λ_{i} − i, 2n + m), i = 1, 2, …, r. In general, the symplectic constraint would translate into the condition that the first horizontal step of path P_{i}, i ≤ n + 1, must be at least at height 2i − 1. However, since also for (3.14) we have to assume r = ℓ(λ) ≤ n + 1, this condition holds for all paths P_{i}. Thus for ℓ(λ) ≤ n + 1 the (n, m)-symplectic tableaux of shape (λ, λ^{trnc}) correspond to r-tuples (P_{1}, …, P_{r}) of nonintersecting lattice paths, where P_{i} runs from u_{i} = (−i, 1) to v_{i} = (λ_{i} − i, 2n + m), i = 1, 2, …, r, and the first horizontal step of P_{i} is at least at height 2i − 1. If we define the weight of a horizontal edge with h-label (cf. Figure 7) 2i − 1, i ≤ n, to be x_{i}, the weight of a horizontal edge with h-label 2i, i ≤ n, to be x_{i}^{−1}, and the weight of a horizontal edge with h-label i, i > 2n, to be z_{i−2n}, then the above correspondence is weight-preserving with respect to the weight (3.12).

For the proof of (3.14) we let R be the reflection in the line x = −1, as in the proof of (3.9) in the previous section. Then the determinant in (3.14) may be written as a generating function for r-tuples (P_{1}, …, P_{r}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, r, namely

|h_{λ_{j}−j+1} (x, z) ⋮ h_{λ_{j}−j+i} (x, z) + h_{λ_{j}−j−i+2} (x, z)|_{r×r}

= _{∑_{σ∈S_{r}, ε∈{1}×{1,−1}^{r−1}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) w(P).

For the rest of the proof we proceed as in the proof of (3.9) given in the previous section.

Bijective proof of the (n, m)-symplectic Giambelli identity (3.16). In the same way as in section 5, (n, m)-symplectic tableaux of shape ((α, β), (α|β)^{trnc}), where (α, β) is the Frobenius notation of a partition of rank s, correspond to s-tuples (P_{1}, …, P_{s}) of lattice paths where P_{i} runs from (−α_{i}, 2n + m − 1) to (β_{i} + 1, 2n + m − β_{i} − 1), i = 1, 2, …, s, and does not cross the line y = x − 1 (see Figure 13). The weight of a horizontal edge with label 2i − 1, i ≤ n, is defined to be x_{i}, the weight of a horizontal edge with label 2i, i ≤ n, is defined to be x_{i}^{−1}, and the weight of a horizontal edge with label i, i > 2n, is defined to be z_{i−2n}. Then this correspondence is weight-preserving with respect to the weight (3.12).

The rest of the proof is identical with the proof of the (ordinary) symplectic Giambelli identity (3.11) given in section 5.

Bijective proof of the odd symplectic dual Jacobi–Trudi identity (3.19). Again, as in the proof of the (ordinary) symplectic dual Jacobi–Trudi identity (3.10) given in the previous section (see Figure 10), we map (n, 1)-symplectic tableaux of shape (λ, λ_{d}) to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i + 1, i − 1) to (λ′_{i} − i + 1, 2n − λ′_{i} + i), i = 1, 2, …, λ_{1}. Also here, because of ℓ(λ) ≤ n + 1, the symplectic constraint translates into the condition that all the paths do not cross the line y = x − 1. Besides, there is the constraint that the last entries in columns 1, 2, …, λ_{d} of (n, 1)-symplectic tableaux are equal to 2n + 1. It is easy to see that altogether this means that (n, 1)-symplectic tableaux of shape (λ, λ_{d}) correspond to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from u_{i} = (−i + 1, i − 1) to v_{i} = (λ′_{i} − i + 1, 2n − λ′_{i} + i), i = 1, 2, …, λ_{1}, and does not cross y = x − 1, and in addition the paths P_{1}, …, P_{λ_{d}} end with a horizontal step (which has e-label 2n + 1). If we define the weight of a horizontal edge with e-label (cf. Figure 7) 2i − 1, i ≤ n, to be x_{i}, the weight of a horizontal edge with e-label 2i, i ≤ n, to be x_{i}^{−1}, and the weight of a horizontal edge with e-label 2n + 1 to be z, then the above correspondence is weight-preserving with respect to the weight (3.17).

Again it is easy to see that the determinant in (3.19) may be written as a generating function for certain λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, λ_{1},

|ze_{λ′_{j}−j+i−1} (x) − ze_{λ′_{j}−j−i−1} (x) ⋮ e_{λ′_{j}−j+i} (x, z) − e_{λ′_{j}−j−i} (x, z)|_{λ_{1}×λ_{1}}

= _{∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} ; P∈P^{(1)}(u^{(ε)},v_{σ})} sgn (σ) sgn (ε) GF (P^{(1)} (u^{(ε)}, v_{σ})) .

Here P^{(1)} denotes the additional constraint that paths P_{σ^{−1}(1)}, …, P_{σ^{−1}(λ_{d})} end with a horizontal step. The rest of the proof is identical with the proof for (3.15). One only has to note that the modified reflection as well as the Gessel–Viennot involution preserve the additional constraint regarding the last steps of P_{σ^{−1}(1)}, …, P_{σ^{−1}(λ_{d})}.

7. Orthogonal identities: Proctor’s tableaux N-orthogonal tableaux must satisfy rather complicated constraints, compared to the

simple symplectic tableaux. So we start with the easier case of coarse orthogonal tableaux and give a proof of a “weak form” of the orthogonal e-formula, where all the variables involved are set to 1. Recall that (3.27), when evaluated at x_{1} = ⋯ = x_{n} = 1, just reads

(7.1) o_{N} (λ, ⟨1, 1, …, 1⟩)

= |e_{λ′_{j}−j+1} (1, 1, …, 1) ⋮ e_{λ′_{j}−j+i} (1, 1, …, 1) + e_{λ′_{j}−j−i+2} (1, 1, …, 1)|_{λ_{1}×λ_{1}} .

By (3.25), the left side of the equation equals the number of coarse orthogonal tableau.

Bijective proof of the dimension formula (7.1). First we describe the lattice path interpretation of coarse orthogonal tableaux. A coarse orthogonal tableau T with entries ≤ N and of shape λ = (λ_{1}, …, λ_{r}), where λ is an N-orthogonal partition, is mapped to a λ_{1}-tuple (P_{1}, …, P_{λ_{1}}) of lattice paths, where P_{i} runs from (−i + 1, i − 1) to (λ′_{i} − i + 1, N − λ′_{i} + i − 1), i = 1, 2, …, λ_{1}, by using the correspondence of section 4 that based on the e-labelling, i.e., by reading the path P_{i} from the i-th column of T. Figure 14 depicts an example of this correspondence where λ = (4, 3, 2, 1) and the entries of the tableau are bounded by N = 9.

[figure: -3; -2; -1; 1; 2; 3; 4; 5; 1; 2; 4; 6; P_{1}; 4; 6; 7; P_{2}; 4; 6; P_{3}; 5; P_{4}] T = ^{1 4 4 5 2 6 6 4 7 6}^{Figure 14}

It is obvious that this λ_{1}-tuple must be nonintersecting because T is a tableau. Let R denote the reflection in the line y = x. This notation holds for the remainder of this section. We claim that the coarse orthogonal condition (3.20) translates into the condition that the second path P_{2} and the reflection R (P_{1}) of the first path in the line y = x must not intersect. This is seen as follows.

Let c be fixed. Denote the number of elements in the i-th column of T that are smaller or equal c by μ_{i}, i = 1, 2. The coarse orthogonal condition then says

μ_{1} + μ_{2} ≤ c.

If (P_{1}, P_{2}, …, P_{λ_{1}}) corresponds to T then the meeting point of P_{1} with the antidiagonal line x + y = c is p_{1} = (μ_{1}, c − μ_{1}) and the meeting point of P_{2} with x + y = c is p_{2} = (μ_{2} − 1, c − μ_{2} + 1). The reflection R (p_{1}) of p_{1} in the line y = x is the point (c − μ_{1}, μ_{1}). Obviously we have that μ_{1} + μ_{2} ≤ c if and only if R (p_{1}) lies strictly south-east of p_{2}. In other words: T obeys the c-th orthogonal condition for all c if and only if for all c the reflected meeting point of P_{1} and x + y = c lies strictly south-east of the meeting point of P_{2} and x + y = c. This is equivalent to our claim.

If we can show that the number of all λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i + 1, i − 1) to (λ′_{i} − i + 1, N − λ′_{i} + i − 1), i = 1, 2, …, λ_{1}, and where the reflection of P_{1} does not intersect P_{2} equals the determinant in (7.1), then (7.1) is established. In fact, we are going to prove a more general path counting result from which (7.1) immediately follows.

Theorem 1. Let R denote the reflection in the line y = x. Let u_{1}, …, u_{m} be pairwise distinct points where u_{i+1} lies to the north-west of u_{i}, i = 1, 2, …, m−1, and in addition u_{2} lies to the north-west of R (u_{1}). Let v_{1}, …, v_{m} be another collection of pairwise distinct points with the same property. Besides, u_{i} is assumed to lie to the south-west of v_{i}, i = 1, 2, …, m. Then the number of nonintersecting m-tuples (P_{1}, …, P_{m}) of lattice paths, where P_{1} runs from either u_{1} or R (u_{1}) to v_{1}, and for i ≥ 2 the paths P_{i} run from u_{i} to v_{i}, and where in addition there is no intersection of P_{2} with R (P_{1}), is given by

| |P (u_{i}, v_{j})| + |P (R (u_{i}) , v_{j})| |_{m×m}. (7.2)

(As usual, |A| denotes the number of elements of the set A.) If u_{1} lies on the line y = x then R (u_{1}), the reflection of u_{1} in y = x, is considered to be different from u_{1}. Thus in this case every m-tuple is counted twice.

Remark. For the λ_{1}-tuples that result from coarse orthogonal tableaux the point u_{1} is (0, 0) which lies on y = x. Hence in this case the last remark applies and therefore the actual number that we are interested in is the determinant (7.2) divided by two. This two can be cancelled in the first row which consists of entries of the form 2 |P (u_{1}, v_{j})|. Thus (7.1) is established once the theorem is proved.

Proof: With R being the reflection in the line y = x, we expand the determinant,

| |P (u_{i}, v_{j})| + |P (R (u_{i}) , v_{j})| |_{m×m}

= ∑_{σ∈S_{m}, ε∈{1,−1}^{m}} sgn (σ) ∏_{i,ε_{i}=1} |P (u_{i}, v_{σ(i)})| ∏_{i,ε_{i}=−1} |P (R (u_{i}) , v_{σ(i)})|

= _{∑_{σ∈S_{m}, ε∈{1,−1}^{m}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) .

(The meaning of u^{(ε)} is that of section 2.) We have expressed the determinant as a weighted sum over m-tuples P = (P_{1}, …, P_{m}) of lattice paths, where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, m. We shall give a sign-reversing involution that cancels all m-tuples (P_{1}, …, P_{m}) that are either intersecting or contain two paths P_{i}, P_{j}, i ≠ j, such that P_{i} intersects R (P_{j}). Suppose that this had been done. The remaining m-tuples are nonintersecting and for all pairs P_{i}, P_{j} the path P_{i} and the reflected path R (P_{j}) do not intersect. In particular, the paths P_{i}, i ≥ 2, neither meet P_{1} nor R (P_{1}). A moment’s thought shows that, because of the order of the end points, this implies that the end point of P_{1} must be v_{1}. Since for l ≥ 2 the end points v_{l} lie to the north-west of v_{1} and R (v_{1}), the complete paths P_{i}, i ≥ 2, must lie to the north-west of P_{1} and R (P_{1}), not intersecting both of them. Since also P_{2}, …, P_{m} are nonintersecting, because of the order of the starting and end points, P_{i} must run from u_{i} to v_{i} for i ≥ 2. In other words, the remaining m-tuples are those for which σ = id and ε = (1, …, 1) or (−1, 1, …, 1), which are nonintersecting, and where R (P_{1}) does not meet P_{i} for i ≥ 2. This would prove the assertion.

[figure: u_{i}; u_{j}; P_{j}; R (u_{j}); P_{i}; R (P_{j}); p; ⟷; u_{i}; R (u_{i}); R (P′_{i}); R (u_{j}); P′_{j}; P′_{i}; p]

Figure 15

Now we define the involution. Given an m-tuple of paths, look for the highest level x + y = c that contains a point p where two paths P_{i} and P_{j}, i ≠ j, of the m-tuple intersect or where some P_{i} and R (P_{j}), i ≠ j, intersect. If there is a point of (ordinary) intersection between two paths on this level, then choose (i, j), i ≠ j, to be minimal in lexicographic order such that P_{i} and P_{j} intersect on this level and apply the Gessel-Viennot involution. If not, then choose (i, j), i ≠ j, to be minimal in lexicographic order such that P_{i} and R (P_{j}) intersect on this level. Now, as in the Gessel–Viennot involution, interchange terminal portions of R (P_{j}) and P_{i} beginning from p, obtaining P′_{i} and P_{j}′. Then reflect back P′_{i}, thus obtaining R (P′_{i}) and P′_{j}. Figure 15 illustrates this operation. Thus, from the original m-tuple of paths we obtain a new m-tuple by replacing P_{i} by R (P′_{i}) and P_{j} by P′_{j}. What actually happens is that, beginning from the meeting point p, respectively its reflection (both points are circled in Figure 15), terminal portions of paths are interchanged, while, up to p, respectively its reflection, initial portions are reflected. For later use, let us call this operation reflection-Gessel-Viennot mapping.

The above described mapping is sign-reversing since the associated permutations differ by the transposition (i, j). Finally, the mapping is an involution since the Gessel– Viennot involution is and since the reflection-Gessel-Viennot mapping does not change anything on the level x + y = c or on higher levels and so does not introduce points of (ordinary) intersection between paths there. Hence, renewed application of the mapping will give back the original m-tuple.

Bijective proof of the orthogonal dual Jacobi–Trudi identity (3.27). Things are a bit more complicated in the (non-coarse) orthogonal case. First, because now we have to take care of weights, we have to use the “modified” reflection that we already used in the symplectic case. Second, only (approximately) half of the orthogonal conditions hold, while the odd cases are replaced by protection or/and broken strip conditions. Actually, all 4 different fine orthogonal tableaux that were defined in section 3 could be used for a bijective proof of the orthogonal dual Jacobi–Trudi identity. Since the proofs are very similar, we give all the details only for the orthogonal tableaux of the first kind and sketches of proofs for the tableaux of second, third and fourth kind.

By fine orthogonal tableaux of the first kind. First we translate the N-orthogonal tableaux of the first kind into the lattice path language. Given an N-orthogonal tableau T of the first kind which is of shape λ = (λ_{1}, …, λ_{r}), where λ is an N-orthogonal partition, in the same way as for coarse orthogonal tableaux we map it to a λ_{1}-tuple (P_{1}, …, P_{λ_{1}}) of lattice paths, where P_{i} runs from (−i+1, i−1) to (λ′_{i}−i+1, N−λ′_{i}+i−1), i = 1, 2, …, λ_{1}, by reading the path P_{i} from the i-th column of T. Figure 16 depicts an example of this correspondence where λ = (4, 4, 3, 1) and N = 8.

T = ^{1 3 3 ∗ 2 5 5 6 3 7 8 5} ∗ = 5 or 3 [figure: -3; -2; -1; 1; 2; 3; 4; 1; 2; 3; 5; P_{1}; 3; 5; 7; P_{2}; 3; 5; 8; P_{3}; 5; 6; P_{4}; 3; 6; R̃ (P_{1})]

Figure 16

The last entry in the first row of this tableau must be 5 in order that the tableau is orthogonal. (Otherwise the 6-th protection condition (3.21) would be violated.) Right now the reader should ignore all dotted lines in Figure 16 and assume that ∗ = 5.

It is obvious that the λ_{1}-tuple of paths that results from an N-orthogonal tableau T of the first kind must be nonintersecting because T is a tableau. Let R̃ denote the following modified reflection in the line y = x. If N is even, R̃ is the modified reflection that was explained in section 5 (see Figure 11) but in the line y = x instead of y = x−2. If N is odd, then on the region below the line x + y = N − 1 R̃ acts as the modified reflection in y = x, while it acts as the usual reflection R between x + y = N − 1 and x + y = N. This definition holds for the remainder of this section.

It was shown at the beginning of this section that the 2c-th orthogonal condition (3.20) translates into the condition that the intersection point p_{1} of the line x + y = 2c with the reflection R (P_{1}) of the first path in the line y = x lies strictly to the south-east of the intersection point p_{2} of P_{2} and the line x + y = 2c. Since in even points, i.e., in points (x, y) with x + y ≡ 0 (mod 2), the modified reflection agrees with the ordinary reflection, the preceding statement is also true if we replace ‘reflection’ by ‘modified reflection’. (The same is true for points on level x + y = N if N is odd.)

It remains to see how the 2c-th protection condition translates into the lattice path language. Here one is advised to come back to Figure 16 with dotted lines included. If ∗ does not equal 5 but something less, 3 say, then in the strip between the lines x + y = 4 and x + y = 6 we have the situation of Figure 17.

[figure: R̃ (P_{1})]^{Figure 17}

That is, in the strip between these two lines the (modified) reflected first path exhibits an east-north turn, there are (possibly) some more paths, also exhibiting an east-north turn, differing from its predecessor by a shift in direction (−1, 1), with the exception of the last which exhibits a north-east turn whose even points also differ from the even points of the preceding east-north turn by a shift in direction (−1, 1). (Note that violations of the 2c-th protection condition might very well look different from the situation described above, but then would already form a violation of the 2c-th orthogonal condition.) So we have shown that N-orthogonal tableaux of the first kind and of shape λ correspond to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i+1, i−1) to (λ′_{i} −i+1, N −λ′_{i} +i−1), i = 1, 2, …, λ_{1}, where for all c the meeting of R̃ (P_{1}) and x + y = 2c lies strictly to the south-east of the meeting of P_{2} with x + y = 2c, and where for all c in the region between the lines x + y = 2c − 2 and x + y = 2c we never have the situation that is exemplified in Figure 17.

The weight that we consider here is basically the same as in the symplectic “e-case”, namely, the weight of a horizontal edge with e-label (cf. Figure 7) 2i is defined to be x_{i}^{−1}, and the weight of a horizontal edge with e-label 2i − 1 is defined to be x_{i} except that the weight of a horizontal edge with e-label N is defined to be 1 if N is odd. Obviously, for N being even or odd, this makes the correspondence depicted in Figure 16 weight-preserving with respect to the weight (3.22).

Let u_{i} = (−i + 1, i − 1) and v_{i} = (λ′_{i} − i + 1, N − λ′_{i} + i − 1), i = 1, …, λ_{1} and let R be the reflection in y = x. We expand the determinant in (3.27),

|e_{λ′_{j}−j+1} (x) ⋮ e_{λ′_{j}−j+i} (x) + e_{λ′_{j}−j−i+2} (x)|_{λ_{1}×λ_{1}} (7.3)

= |GF (P (u_{1}, v_{j})) ⋮ GF (P (u_{i}, v_{j})) + GF (P (R (u_{i}) , v_{j}))|_{λ_{1}×λ_{1}}

= _{∑_{σ∈S_{λ_{1}}} ; ε∈{1}×{1,−1}^{λ_{1}−1}} sgn (σ) ∏_{i,ε_{i}=1} GF (P (u_{i}, v_{σ(i)})) · ∏_{i,ε_{i}=−1} GF (P (R (u_{i}) , v_{σ(i)}))

= _{∑_{σ∈S_{λ_{1}}, ε∈{1}×{1,−1}^{λ_{1}−1}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) w(P),

which is thereby written as a generating function for λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, λ_{1}. Again, the notation u^{(ε)} is from section 2. Note that here, in contrast to Theorem 1, we do not consider reflections of u_{1}.

We have to construct a weight-preserving and sign-reversing involution such that after the cancellation only those λ_{1}-tuples P = (P_{1}, …, P_{λ_{1}}) survive which correspond to N-orthogonal tableaux of the first kind. In order to make this more precise, we introduce a few terms. We say that P exhibits a reflected intersection in point p if there are two paths P_{i} and P_{j}, i ≠ j, such that R̃ (P_{i}) and P_{j} intersect in p. We say that P exhibits an odd reflected crossing in point p if there are two paths P_{i} and P_{j} such that R̃ (P_{i}) and P_{j} cross in an odd point p (i.e., in a point p = (x, y) with x + y ≡ 1 (mod 2)), in the way that is exemplified in the left picture in Figure 18.

odd reflected crossings:

[figure: P_{j}; R̃(P_{i}); p; y=x; P_{j}; R̃(P_{i}); P_{i}; R̃(P_{j})]

Figure 18

For later reference we remark that if there is an odd reflected crossing of R̃(P_{i}) and P_{j} above y = x then there is an odd reflected crossing of R̃(P_{j}) and P_{i} below y = x, and vice versa, see the right picture in Figure 18. We call an odd reflected crossing special if the situation is as exemplified in Figure 19. That is, the odd reflected crossing is located above y = x, and in the strip between lines x + y = 2c − 2 and x + y = 2c containing this odd reflected crossing there are (possibly) some more paths, exhibiting an east-north turn, each differing from its predecessor by a shift in direction (−1, 1), the turning point of the first differing from the odd reflected crossing also by a shift in direction (−1, 1), and then follows a point (marked by a circle in Figure 19) that is not occupied by any path and differs from the turning point of the last east-north turn again by a shift in direction (−1, 1). It will be important later to observe that an odd reflected crossing in the strip between lines x + y = 2c − 2 and x + y = 2c can only be non-special if there are (ordinary) intersections or even reflected intersections in this strip. This is straight-forward for odd reflected crossings above y = x. If we encounter an odd reflected crossing below y = x, we just have to remember that this is equivalent to an odd reflected crossing above y = x, so we are in the same case again.

special odd reflected crossing:

[figure: y=x; P_{i_{h}}; P_{i_{2}}; ^{R̃ (P_{i_{1}})}; p]

Figure 19

We say that P exhibits an odd reflected violation in point p if there are paths P_{i_{1}}, P_{i_{2}}, …, P_{i_{h}}, h ≥ 2, such that in some region between lines x + y = 2c − 2 and x + y = 2c and above y = x (this means that the even points of the reflected path R̃ (P_{i_{1}}) have to be above or on y = x) the paths R̃ (P_{i_{1}}) , P_{i_{2}}, …, P_{i_{h}} exhibit the situation that is exemplified in Figure 20 (compare Figure 17). The “reference point” p, which is for later use, is marked by a bold dot.

odd reflected violation:

[figure: y=x; P_{i_{h}}; P_{i_{2}}; ^{R̃ (P_{i_{1}})}; p]

Figure 20

We shall give a weight-preserving and sign-reversing involution that cancels all λ_{1}- tuples (P_{1}, …, P_{λ_{1}}) that are either intersecting, or exhibit a reflected intersection in an even point (i.e., in a point (x, y) with x + y ≡ 0 (mod 2)), or a special odd reflected crossing, or an odd reflected violation. (We shall call any of these situations sometimes simply a “violation” for short.) Suppose that this had been done. The remaining λ_{1}- tuples are nonintersecting. The modified reflection R̃ (P_{1}) does not touch the path P_{2} in even points. In particular, R̃ (P_{1}) and P_{2} cannot cross in even points. Moreover, R̃ (P_{1}) and P_{2} cannot cross in an odd point. For, either this is already an odd reflected crossing (cf. Figure 18), or it is a crossing as exemplified in Figure 21. In the latter case there has to be an odd reflected crossing or an even reflected intersection of R̃ (P_{1}) and P_{2} “later”, because, by N-orthogonality of λ, the end point of P_{2}, which is u_{2}, lies in the north-west of the end point of R̃ (P_{1}), which is R (u_{1}). But, an odd reflected crossing is either special or, if non-special, implies an (ordinary) intersection or an even reflected intersection, as we observed before. Hence, R̃ (P_{1}) and P_{2} cannot cross at all and do not even touch in even points and on the line x + y = N. Besides, R̃ (P_{1}) , P_{2}, …, P_{h} in any region between lines x + y = 2c − 2 and x + y = 2c cannot exhibit the situation that is exemplified in Figure 20.

not an odd reflected crossing:

[figure: R̃(P_{1}); P_{2}]

Figure 21

Summarizing, by essentially the same arguments as in the proof of Theorem 1, what this tells is that only λ_{1}-tuples remain for which σ = id and ε = (1, …, 1), which are nonintersecting, where R̃ (P_{1}) is always strictly to the south-east of P_{2} on even levels (i.e., on lines of the form x + y = 2c), and where for R̃ (P_{1}) , P_{2}, …, P_{h} in any region between x+y = 2c−2 and x+y = 2c never the situation that is exemplified in Figure 17. occurs. These are exactly those λ_{1}-tuples which correspond to N-orthogonal tableaux of the first kind. This would prove (3.27).

Now we define the involution. Given a λ_{1}-tuple (P_{1}, P_{2}, …, P_{λ_{1}}) that has to be cancelled, we look for the highest level x + y = c containing an (ordinary) intersection between two paths or an even reflected intersection or a special odd reflected crossing or an odd reflected violation in some point, respectively reference point, p. If there is a point of (ordinary) intersection between two paths on this level, then choose (i, j), i ≠ j, to be minimal in lexicographic order such that P_{i} and P_{j} intersect on this level and apply the Gessel-Viennot involution. If not, then we basically apply the reflection-Gessel-Viennot mapping from the proof of Theorem 1 except that ordinary reflection is replaced by modified reflection (because we have to preserve the weight). However, the details are a little bit delicate.

To be precise, let (i_{1}, i_{2}) be minimal in lexicographic order such that P_{i_{1}} and P_{i_{2}} are the paths that are “involved” in such an even reflected intersection, special odd reflected crossing, or odd reflected violation on that level. (“Involved” has an obvious meaning for even reflected intersections. For special odd reflected crossings and odd reflected violations P_{i_{1}} and P_{i_{2}} are as in the definition of special odd reflected crossings, see Figure 19, respectively odd reflected violations, see Figure 20.) If we are considering an even reflected intersection then we carry out the reflection-Gessel-Viennot mapping straight-forwardly, with ordinary reflection replaced by modified reflection. If we are considering a special odd reflected crossing or an odd reflected violation we apply a further modification.

Let us start with an odd reflected violation with reference point p. See the left-most picture in Figure 22. First flip the east-north turn of R̃ (P_{i_{1}}) into a north-east turn, and the north-east turn of P_{i_{h}} into a east-north turn (see the middle picture of Figure 22), thus obtaining the paths P′_{i_{1}} and P′_{i_{h}}. We have introduced a meeting point between P′_{i_{1}} and P_{i_{2}} (which could also be P′_{i_{h}} if h = 2). Now interchange terminal portions of P′_{i_{1}} and P_{i_{2}}, thus obtaining P′′_{i_{1}} and P′_{i_{2}} (see the right-most picture in Figure 22). Finally apply the modified reflection to P′_{i_{2}}, thus obtaining R̃(P′_{i_{2}}). From the original λ_{1}-tuple by this procedure we obtain another λ_{1}-tuple where P_{i_{1}} is replaced by P′′_{i_{1}}, P_{i_{2}} by R̃(P′_{i_{2}}), and P_{i_{h}} by P′_{i_{h}}.

[figure: special odd reflected crossing:; y=x; P′_{i_{2}}; P′′_{i_{1}}; P′_{i_{h}}; p; q; ↔; intermediate:; y=x; P′_{i_{1}}; P_{i_{2}}; P′_{i_{h}}; p; q; ↔; odd reflected violation:; y=x; ^{R̃ (P_{i_{1}})}; P_{i_{2}}; P_{i_{h}}; p]

Figure 22

It should be noted that we end up with a λ_{1}-tuple that contains a special odd reflected crossing in p (compare Figure 18). The point q is circled to indicate that it is not occupied by any path.

On the other hand, if we start with a special odd reflected crossing in p, we simply go in the reverse direction, i.e., we go from right to left in Figure 22. All we have to do is to determine the point q, which is simply the “first” point on the same level as p that is not occupied by any path.

For later use, let us call the mapping defined for even reflected intersections, special odd reflected crossings, and odd reflected violations modified-reflection-Gessel-Viennot mapping. It is obvious that this mapping is weight-preserving and sign-reversing, the latter because in any case the associated permutations differ by the transposition (i_{1}, i_{2}). It is also an involution since we do not introduce new “violations” of paths on or above the level that we are considering, as is not difficult to see.

Thus the construction of our weight-preserving and sign-reversing involution is complete, which finishes the proof.

By fine orthogonal tableaux of the second kind. Note that the orthogonal tableaux of the first kind differ from the ones of the second kind only in that “most” of the protection conditions are replaced by broken strip conditions. So it is easy to set up a weight-preserving bijection between these two types of orthogonal tableaux. Given an orthogonal tableau of the first kind, we look for all violations of some broken strip condition. It is not hard to see that in lattice path language they look precisely as depicted in Fig. 23. Note that the even points on the reflected kink must lie strictly above the line y = x to be in accordance with the definition of the broken strip condition.

[figure] Figure 23

The situation of Fig. 23 appears in the intermediate picture in Fig. 22 (if we forget about y = x). So we can transform it into an odd reflected violation strictly above the line y = x by the operation that leads from the intermediate picture to the left picture in Fig. 22, and vice versa. (The case where these even points on the reflected kink actually would lie on line y = x precisely corresponds to the case of the single protection condition that is not replaced by a broken strip condition.) This gives a weight-preserving bijection between the set of all orthogonal tableaux of the first kind and orthogonal tableaux of the second kind. This finishes the proof.

The same weight-preserving bijection (formulated in the “language of tableaux”, without employing a lattice path interpretation) is given in [13, App. A, p. 276f]. Note that the above operation mapping odd reflected violations to broken strip violations is not feasible in situations where the even points of the reflected kinks lie on line y = x, since then it would introduce an intersection of the second path with the first path itself (which coincides with its own reflection in the relevant strip in this case).

By fine orthogonal tableaux of the third kind. The definition of orthogonal tableaux of the third kind is very similar to the definition of orthogonal tableaux of the first kind. Therefore we can more or less copy the proof by orthogonal tableaux of the first kind. At each occurrence, 2c has to be replaced by 2c + 1, and the meaning of “modified reflection” has to be adjusted. Namely, now all odd points (i.e., points (x, y) with x + y ≡ 1 (mod 2)) are reflected in the usual way and so are all even points (i.e., points (x, y) with x + y ≡ 0 (mod 2)) whose adjacent steps are both vertical or both horizontal, while a “kink” in an even point is shifted accordingly (compare with Figure 11). Besides, steps between the lines x + y = 0 and x + y = 1 are reflected in the usual way.

By fine orthogonal tableaux of the fourth kind. The definition of orthogonal tableaux of the fourth kind is very similar to the definition of orthogonal tableaux of the second kind. Therefore we can map them by a weight-preserving bijection to orthogonal tableaux of the third kind in much the same way as we mapped orthogonal tableaux of the second kind to orthogonal tableaux of the first kind.

Naturally, the question arises why we do not need any protection conditions here (as was the case for tableaux of the second kind). The reason for this is that there are no odd points on the line y = x; and odd levels play the role that even levels played for tableaux of the second kind. Hence all odd reflected violations can be converted into broken strip violations.

Bijective proof of the orthogonal Jacobi–Trudi identity (3.26). Let T be an N-orthogonal tableau (of first, second, third, or fourth kind, respectively) of shape λ, where λ is an N-orthogonal partition, i.e., λ′_{1} + λ′_{2} ≤ N. As in section 4 (see Figure 6), by reading paths from the rows and using the h-labelling (see Figure 7) we map T to an r-tuple (P_{1}, …, P_{r}) of nonintersecting lattice paths, where P_{i} runs from u_{i} = (−i, 1) to v_{i} = (λ_{i} − i, N), i = 1, 2, …, r. The orthogonal, protection, and broken strip conditions translate into rather complicated constraints on the heights of horizontal steps of these paths. We do not need to make this precise since, as in the proof of the symplectic Jacobi–Trudi identity, we take a detour and pass from “h-paths” to “e-paths” by applying the concept of dual lattice paths due to Gessel and Viennot [10, sec. 4]. This allows us to rely on the bijections for the orthogonal dual Jacobi–Trudi identity, which were presented just before.

To complete the lattice path interpretation of the left-hand side of (3.26), we address the weights of orthogonal tableaux and lattice paths. Since the weight definition (3.22) for fine orthogonal tableaux of the first and second kind differs from the weight definition (3.23) for fine orthogonal tableaux of the third and fourth kind, we have to split the definition of edge weights. If we started with an orthogonal tableau of the first or second kind, we define the weight of a horizontal edge with h-label 2i to be x_{i}^{−1}, and the weight of horizontal edge with h-label 2i − 1 to be x_{i}, except that the weight of a horizontal edge with e-label N is defined to be 1 if N is odd. This makes the above correspondence weight-preserving with respect to (3.22). If we started with an orthogonal tableau of the third or fourth kind, we define the weight of a horizontal edge with h-label 1 to be 1, the weight of a horizontal edge with h-label 2i to be x_{i}, and the weight of horizontal edge with h-label 2i + 1 to be x_{i}^{−1}. This makes the above correspondence weight-preserving with respect to (3.23).

Now we expand the determinant in (3.26) and find its lattice path interpretation. In the following computation R is the reflection in the line x = 0.

|h_{λ_{j}−j+i} (x) − h_{λ_{j}−j−i} (x)|_{λ_{1}×λ_{1}} = |GF (P (u_{i}, v_{j})) − GF (P (R(u_{i}), v_{j}))|_{λ_{1}×λ_{1}}

= ∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} sgn (σ) sgn (ε) ∏_{i,ε_{i}=1} GF (P (u_{i}, v_{σ(i)})) · ∏_{i,ε_{i}=−1} GF (P (R(u_{i}), v_{σ(i)}))

= _{∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} ; P∈P(u^{(ε)},v_{σ})} sgn (σ) sgn (ε) w(P),

where again sgn (ε) stands for ∏_{i=1}^{λ_{1}} ε_{i} and the notation u^{(ε)} is from section 2. The weight w(P) is induced (as explained in section 2) by the edge weight that was introduced just before. Thus the determinant in (3.26) is written as a generating function for r-tuples P = (P_{1}, …, P_{r}) of lattice paths where P_{i} starts at either u_{i} or R(u_{i}) and runs to v_{σ(i)}, i = 1, 2, …, r, for some permutation σ. We have to give a weight-preserving and signreversing involution that cancels all those r-tuples of paths that either contain a path in a reflected starting point, or are intersecting, or (if σ = id, ε = (1, 1, …, 1)) where the tableau corresponding to P violates one of the relevant orthogonal, protection, or broken strip conditions.

Let P = (P_{1}, …, P_{r}) be one of these r-tuples of paths in P (u^{(ε)}, v_{σ}) which have to be cancelled. If P is intersecting we apply the Gessel–Viennot involution. If P is nonintersecting, then in the first step we determine the dual of P, compare with Figure 12. It can be seen that here it is a λ_{1}-tuple Q = (Q_{1}, …, Q_{λ_{1}}) of paths, where Q_{i} runs from either \overline{u}_{i} = (−i + 1, i − 1) or \overline{R}(\overline{u}_{i}) to \overline{v}_{σ(i)}, \overline{v}_{j} being given by \overline{v}_{j} = (λ′_{j} − j + 1, N − λ′_{j} + j − 1), for some permutation σ ∈ S_{λ_{1}}. In this case \overline{R} denotes the reflection in the line y = x. In particular, it should be observed that the first path, Q_{1}, has to start in \overline{u}_{1} = (0, 0). (This is an implication of the fact that no path of P starts in (0, 0).) So the duals are exactly the nonintersecting objects that are involved in the bijective proof of the orthogonal dual Jacobi–Trudi identity.

Now we want to apply one of the bijections (depending on which type of tableaux we want to base our proof) that proved the orthogonal dual Jacobi–Trudi identity. However, the same problem as in the proof of the symplectic Jacobi–Trudi identity appears as this may introduce new points of intersections of paths. Again, this cannot be permitted, since dualization does not work with intersecting paths. To come around this problem we use the same trick as in the proof of the symplectic Jacobi–Trudi identity. Let A denote the appropriate mapping involving the “modified-reflection-Gessel-Viennot mapping” (cf. Fig. 22) used in the proof of the orthogonal dual Jacobi– Trudi identity, and let B denote the Gessel–Viennot involution. Again, starting with Q, apply A, then B, then A again, etc., as long as possible. By the same arguments as in the proof of the symplectic Jacobi–Trudi identity it is seen that the algorithm terminates. Since A (the mapping involving the “modified-reflection-Gessel-Viennot mapping” in y = x) can always be applied to these objects, the algorithm has to stop with an application of A. So we finally obtained some nonintersecting λ_{1}-tuple, Q′ say. In the last step we determine the preimage of Q′ under the dualization procedure, P′ say. That the map from P to P′ is weight-preserving is obvious. That it is also signreversing is checked in much the same way as it is done in the proof of the symplectic Jacobi–Trudi identity.

8. Orthogonal identities: Sundaram’s tableaux Sundaram’s tableaux are not too far from symplectic tableaux. Therefore it is possible

to modify the bijections of section 5 in order to be applicable for Sundaram’s tableaux. Let λ be a partition with ℓ(λ) = r ≤ n + 1. Throughout this section x denotes the sequence ⟨x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1}, 1⟩. The identities which result from lattice path interpretations of Sundaram’s tableaux are: A variant (see [9, Prop. 24.44]) of the odd orthogonal h-formula ((3.26) for N = 2n + 1)

o_{2n+1} (λ, x) = |h′_{λ_{j}−j+1} (x) ⋮ h′_{λ_{j}−j+i} (x) + h′_{λ_{j}−j−i+2} (x)|_{r×r} , (8.1)

where h′_{m} (x) = h_{m} (x) − h_{m−2} (x), a variant of the odd orthogonal e-formula ((3.27) for N = 2n + 1)

(8.2) o_{2n+1} (λ, x)

= |∑_{k≥0} e_{λ′_{j}−j+i−k} (x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1}) − ∑_{k≥0} e_{λ′_{j}−j−i−k} (x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1})|_{λ_{1}×λ_{1}} ,

and the odd orthogonal Giambelli identity ((3.28) for N = 2n + 1)

o_{2n+1} ((α|β) , x) = |o_{2n+1} ((α_{i}|β_{j}) , x)|_{s×s} . (8.3)

The equivalence of (3.26) for N = 2n + 1 and (8.1), and that of (3.27) for N = 2n + 1 and (8.2) is easily verified by simple row operations.

Bijective proof for the odd orthogonal dual Jacobi–Trudi-type identity (8.2). Here we shall consider lattice paths consisting of arbitrary vertical steps, of horizontal steps lying strictly below the line x + y = 2n + 1, and of downward diagonal steps in direction (1, −1) on the line x + y = 2n + 1. We label horizontal edges by the usual e-labelling (see Figure 7) and diagonal edges by ∞. The weight of horizontal edges is defined as in the symplectic case, i.e., the weight of a horizontal edge with label 2i − 1 is x_{i}, and the weight of a horizontal edge with label 2i is x_{i}^{−1}. Diagonal edges do not contribute to the weight.

Let T be a (2n + 1)-Sundaram tableau of shape λ, where λ is a partition of length r ≤ n+1. By reading paths from the columns, T corresponds to λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of nonintersecting lattice paths, where P_{i} runs from (−i+1, i−1) to (λ′_{i}−i+1, 2n−λ′_{i}+i), i = 1, 2, …, λ_{1}, and does not cross the line y = x−1, and where P_{i} consists of arbitrary vertical steps, of horizontal steps strictly below x + y = 2n + 1, and possibly downward diagonal steps on the line x + y = 2n + 1. As in section 5 the non-crossing condition results from the symplectic constraint (3.6). Note that with the above weight this mapping is weight-preserving with respect to (3.22). Basically the same correspondence already appears in [24, Lemma 3.2B]. An example with n = 3 and λ = (4, 4, 1, 1) is given in Figure 24.

[figure: -3; -2; -1; 1; 2; 3; 4; 1; 4; ∞; ∞; 3; 4; 4; 6; ∞; ∞] T = ^{1 3 4 ∞ 4 4 6 ∞ ∞ ∞}^{Figure 24}

With R being the reflection in the line y = x − 2, we write the determinant in (8.2) as a generating function for λ_{1}-tuples (P_{1}, …, P_{λ_{1}}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, λ_{1}, namely

|∑_{k≥0} e_{λ′_{j}−j+i−k} (x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1}) − ∑_{k≥0} e_{λ′_{j}−j−i−k} (x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1})|_{λ_{1}×λ_{1}}

= _{∑_{σ∈S_{λ_{1}}, ε∈{1,−1}^{λ_{1}}} ; P∈P^{(2)}(u^{(ε)},v_{σ})} sgn (σ) sgn (ε) w(P).

Here P^{(2)} indicates that now we are considering paths with horizontal steps only strictly below x + y = 2n + 1 and downward diagonal steps on x + y = 2n + 1. u^{(ε)} has again the meaning of section 2.

The involution that cancels λ_{1}-tuples P where a path of P crosses y = x−1 is defined in the same way as in the proof of (3.10) in section 5. Again, this is a weight-preserving mapping since the right-most possible crossing point of a path in P with y = x − 1 is (n, n − 1), and hence the steps that are involved in the modified reflection, depicted in Figure 11, all have e-label at most 2n. Of course, for the intersecting λ_{1}-tuples we apply the Gessel–Viennot involution.

Bijective proof for the odd orthogonal Jacobi–Trudi-type identity (8.1). Now we consider lattice paths consisting of arbitrary vertical steps, of horizontal steps up to height at most 2n, and of possibly one upward diagonal step in direction (1, 1) from some point (x, 2n) to (x + 1, 2n + 1). We label horizontal edges by the usual h-labelling (see Figure 7) and diagonal edges by ∞. The weight of horizontal edges is defined as before, i.e., the weight of a horizontal edge with label 2i − 1 is x_{i}, and the weight of a horizontal edge with label 2i is x_{i}^{−1}. As before, diagonal edges do not contribute to the weight.

This time we apply “row-wise” reading to Sundaram’s tableaux. Thus a (2n + 1)- Sundaram tableau of shape λ, where λ is a partition of length r ≤ n, corresponds to r-tuples (P_{1}, …, P_{r}) of nonintersecting lattice paths, where P_{i} runs from (−i, 1) to (λ_{i} − i, 2n + 1), i = 1, 2, …, r, and where P_{i} consists of arbitrary vertical steps, of horizontal steps at height between 2i − 1 and 2n, and possibly one upward diagonal step from height 2n to height 2n + 1. As in section 5 the lower bound for the heights of paths results from the symplectic constraint (3.6). Note that with the above described labelling this mapping is weight-preserving with respect to (3.22). An example with n = 3 and λ = (4, 4, 1, 1) is given in Figure 25.

[figure: -5; -4; -3; -2; -1; 1; 2; 3; ∞; ∞; 4; 4; 6; ∞; 1; 3; 4; ∞] T = ^{1 3 4 ∞ 4 4 6 ∞ ∞ ∞}^{Figure 25}

Now let R be the reflection in the line x = −1. A moment’s thought shows that the determinant in (8.1) may be written as a generating function for (P_{1}, …, P_{r}) of lattice paths where P_{i} runs from either u_{i} or R(u_{i}) to v_{σ(i)}, i = 1, 2, …, r, namely

|h′_{λ_{j}−j+1} (x) ⋮ h′_{λ_{j}−j+i} (x) + h′_{λ_{j}−j−i+2} (x)|_{r×r} = _{∑_{σ∈S_{λ_{1}}, ε∈{1}×{1,−1}^{r−1}} ; P∈P^{(3)}(u^{(ε)},v_{σ})} sgn (σ) w(P).

The notation P^{(3)} indicates that here we are considering paths with arbitrary vertical steps, with horizontal steps at height at most 2n, and possibly one upward diagonal steps from height 2n to height 2n + 1.

The definition of the involution that cancels those r-tuples (P_{1}, …, P_{r}) where either the associated ε is different from (1, 1, …, 1) or where there is some P_{i} whose first horizontal step is strictly below the line y = 2i − 1 is basically identical with the one in the proof of (3.9) in section 5. We just have to say what to do with the diagonal steps, representing ∞, during the dualization procedure. Namely, to the dualization rules in the proof of (3.9) (“ … connect w_{i} to \overline{v}_{i} by a lattice path, i = 1, 2, …, λ_{1}, by starting at w_{i} and going up vertically unless we would meet a lattice path of P. In the latter case proceed by an upward diagonal step in direction (−1, 1) … ”) we have to add that when we meet the end point of a diagonal step we proceed by a left-handed horizontal step.

Bijective proof for the odd orthogonal Giambelli identity (8.3). The proof of the odd orthogonal Giambelli identity by means of Sundaram’s tableaux is similar to the one given for the symplectic Giambelli identity (3.11) (see Figure 13). The lattice path interpretation of Sundaram’s tableaux is the same as in the symplectic case, besides the entries ∞ are interpreted as downward diagonal steps. Thus, if (α|β) is a partition (in Frobenius notation) of rank s, (2n + 1)-Sundaram tableaux of shape (α|β) correspond to s-tuples (P_{1}, …, P_{s}) of nonintersecting lattice paths where P_{i} runs from (−α_{i}, 2n) to (β_{i} + 1, 2n − β_{i}), i = 1, 2, …, s, and do not cross the line y = x − 1 in the way as is depicted in Figure 26 below. The labelling is defined as in the proofs of the ordinary and symplectic Giambelli identities; in addition, diagonal steps are labelled by ∞. Again, the weight of a horizontal edge with label 2i − 1 is defined to be x_{i}, and the weight of a horizontal edge with label 2i is defined to be x_{i}^{−1}. Again, diagonal edges do not contribute to the weight. Figure 26 gives an example for n = 4 and (α|β) = (3, 2|3, 0). As is clear from the picture and the form of the determinant in (8.3), application of the Gessel–Viennot method immediately implies (8.3).

[figure: -3; -2; -1; 1; 2; 3; 4; ∞; 6; 3; 2; 3; ∞; ∞; ∞; 7; 4] T = ^{2 3 6 ∞ 4 4 7 ∞ ∞ ∞}^{Figure 26}

“Even” Sundaram tableaux. Let us define 2n-Sundaram tableaux to be tableaux with entries from the alphabet 1 < 2 < ⋯ < 2n < ∞ in the usual sense, except that column-strictness does not extend to symbol ∞ (i.e., symbol ∞ may occur more than once in a column), which have at most one entry ∞ in each row, such that the entries in row i are at least 2i. Note that the only difference to the definition of (2n + 1)- Sundaram tableaux is the requirement that the entries in row i have to be at least 2i (instead of only 2i − 1). In particular, the entry 1 actually cannot appear in a 2n-Sundaram tableau. It turns out that 2n-Sundaram tableaux of a given shape enumerate the dimension of the corresponding irreducible representation of the even orthogonal group. However, we were not able to find appropriate weights for these tableaux such that they would generate the full characters.

Theorem 2. Let λ = (λ_{1}, …, λ_{r}) be partition of length r ≤ n. Then there holds

(8.5) o_{2n} (λ, ⟨1, 1, …, 1⟩) = |{T : T 2n-Sundaram of shape λ}| .

Proof: If we use the same arguments as in the proof of the odd orthogonal dual Jacobi–Trudi-type identity (8.2), with the modified reflection in y = x − 2 replaced by the “ordinary” reflection in the line y = x − 1, we obtain

|{T : T 2n-Sundaram of shape λ}|

= |∑_{k≥0} e_{λ′_{j}−j+i−k} (1, 1, …, 1) − ∑_{k≥0} e_{λ′_{j}−j−i−k+1} (1, 1, …, 1)|_{λ_{1}×λ_{1}} ,

where (1, 1, …, 1) stands for a sequence of exactly 2n 1’s. Simple row manipulations turn this determinant into

|e_{λ′_{j}−j+1} (⟨1, 1, …, 1⟩) ⋮ e_{λ′_{j}−j+i} (⟨1, 1, …, 1⟩) + e_{λ′_{j}−j−i+2} (⟨1, 1, …, 1⟩)|_{λ_{1}×λ_{1}} ,

which by (3.27) is identified as o_{2n} (λ, ⟨1, 1, …, 1⟩).

References

1. J. Abramsky, H. A. Jahn and R. C. King, Frobenius symbols and the groups S_{n}, GL(n), O(n), and Sp(n), Can. J. Math. 25 (1973), 941–959.

2. D. M. Bressoud and S.-Y. Wei, Determinental formulae for complete symmetric functions, J. Combin. Theory Ser. A 60 (1992), 277–286.

3. L. Comtet, Advanced Combinatorics, D. Reidel, Dordrecht, Holland, 1974.

4. C. DeConcini, Symplectic standard tableaux, Adv. in Math. 34 (1979), 1–27.

5. C. DeConcini and C. Procesi, A characteristic-free approach to invariant theory, Adv. in Math. 21 (1976), 330–354.

6. Ö. N. Eğecioğlu and J. B. Remmel, A combinatorial proof of the Giambelli identity for Schur functions, Adv. in Math. 70 (1988), 59–86.

7. M. Fulmek and C. Krattenthaler, A bijection between Proctor’s and Sundaram’s odd orthogonal tableaux, Discrete Math. (to appear).

8. M. Fulmek and C. Krattenthaler, Lattice path proofs for determinant formulas for symplectic and orthogonal characters (extended abstract), Proc. of the 5th Conference on Formal Power Series and Algebraic Combinatorics 1994, DIMACS, Rutgers Univ., New Brunswick, New Jersey, 191–200.

9. W. Fulton and J. Harris, Representation Theory, Springer–Verlag, New York, 1991.

10. I. M. Gessel and X. Viennot, Binomial determinants, paths, and hook length formulae, Adv. in Math. 58 (1985), 300–321.

11. I. M. Gessel and X. Viennot, Determinants, paths, and plane partitions, preprint, 1988.

12. R. C. King and N. G. I. El-Sharkaway, Standard Young tableaux and weight multiplicities of the classical Lie groups, J. Physics A 16 (1983), 3153–3178.

13. R. C. King and T. A. Welsh, Construction of orthogonal group modules using tableaux, Linear and Multilinear Algebra 33 (1993), 251–283.

14. K. Koike and I. Terada, Young–diagrammatic methods for the representation theory of the classical groups of type B_{n}, C_{n}, D_{n}, J. Algebra 107 (1987), 466–511.

15. K. Koike and I. Terada, Young diagrammatic methods for the restriction of representations of complex classical Lie groups to reductive subgroups of maximal rank, Adv. in Math 79 (1990), 104–135.

16. C. Krattenthaler, The major counting of nonintersecting lattice paths and generating functions for tableaux, Mem. Amer. Math. Soc., vol. 115, no. 552, Providence, R. I., 1995.

17. C. Krattenthaler, Nonintersecting lattice paths and oscillating tableaux, J. Statist. Plann. Inference (to appear).

18. C. Krattenthaler, Non-crossing two-rowed arrays, in preparation.

19. C. Krattenthaler and B. E. Sagan, More on nonintersecting lattice paths and identities for classical group characters, in preparation.

20. V. Lakshmibai, Geometry of G/P — VIII: The group SO(2n + 1) and the involution σ, J. Algebra 108 (1987), 435–471.

21. V. Lakshmibai, C. Musili and C. S. Seshadri, Geometry of G/P, Bull. Amer. Math. Soc. 1 (1979), 432–435.

22. P. Littelmann, A generalization of the Littlewood–Richardson rule, J. Algebra 130 (1990), 328–368.

23. I.G. Macdonald, Symmetric Functions and Hall Polynomials, Oxford University Press, New York/London, 1979.

24. S. Okada, Lattice path method and characters of classical groups, preprint.

25. , A Robinson–Schensted-type algorithm for 𝔰𝔬(2n, ℂ), J. Algebra 143 (1991), no. 2, 334–372.

26. R. A. Proctor, Odd symplectic groups, Invent. Math. 92 (1988), 307–332.

27. , A Schensted algorithm which models tensor representations of the orthogonal group, Can. J. Math. 42 (1990), 28–49.

28. , A generalized Berele–Schensted algorithm and conjectured Young tableaux for intermediate symplectic groups, Trans. Amer. Math. Soc. 324 (1991), 655–692.

29. , Young tableaux. Gelfand patterns and branching rules for classical Lie groups, J. Algebra 164 (1994), 299–360.

30. B. E. Sagan, The symmetric group, Wadsworth & Brooks/Cole, 1991.

31. C. S. Seshadri, Geometry of G/P — I (Standard monomial theory for a minuscule P), in: P. Ramanujan: A Tribute, Springer–Verlag, 1978.

32. J. T. Sheats, A symplectic jeu de taquin bijection between the tableaux of King and of DeConcini, to appear.

33. R. P. Stanley, Theory and applications of plane partitions: Part 1,2, Stud. Appl. Math 50 (1971), 167–188, 259–279.

34. J. R. Stembridge, Nonintersecting paths, pfaffians and plane partitions, Adv. Math. 83 (1990), 96–131.

35. S. Sundaram, Tableaux in the representation theory of the classical Lie groups, Invariant Theory and Tableaux (Dennis Stanton, ed.), Springer Verlag, 1990, pp. 191–225.

36. , Orthogonal tableaux and an insertion algorithm for SO(2n+1), J. Combin. Theory Ser. A 53 (1990), 239–256.

37. H. Weyl, Classical groups, Princeton University Press, Princeton, New Jersey, 1939; second edition, 1946.

38. D. P. Zhelobenko, The classical groups: Spectral analysis of their finite-dimensional representations, Russian Math. Surveys 17, 1–94.

Dept. of Economics, University of Vienna, Hohenstaufengasse 9, A-1010 Vienna, Austria

Institut für Mathematik, Universität Wien, Strudlhofgasse 4, A-1090 Vienna, Austria