THE DETERMINANT OF AN ELLIPTIC SYLVESTERESQUE MATRIX

GAURAV BHATNAGAR* AND CHRISTIAN KRATTENTHALER**

Abstract. We evaluate the determinant of a matrix whose entries are elliptic hypergeometric terms and whose form is reminiscent of Sylvester matrices. A hypergeometric determinant evaluation of a matrix of this type has appeared in the context of approximation theory, in the work of Feng, Krattenthaler and Xu. Our determinant evaluation is an elliptic extension of their evaluation, which has two additional parameters (in addition to the base q and nome p found in elliptic hypergeometric terms). We also extend the evaluation to a formula transforming an elliptic determinant into a multiple of another elliptic determinant. This transformation has two further parameters. The proofs of the determinant evaluation and the transformation formula require an elliptic determinant lemma due to Warnaar, and the application of two C_{n} elliptic formulas that extend Frenkel and Turaev’s _{10}V_{9} summation formula and _{12}V_{11} transformation formula, results due to Warnaar, Rosengren, Rains, and Coskun and Gustafson.

1. Introduction

The determinant of a Sylvester matrix is used to determine whether two polynomials have a common root. Recently, in the context of approximation theory, a determinant of a hypergeometric matrix was evaluated by Feng, Xu and the second author [2]. The matrix they considered resembles a Sylvester matrix. The objective of this paper is to give an elliptic extension of their determinant evaluation.

We briefly discuss Sylvester matrices. Consider the Sylvester matrix that corresponds to the polynomials

x^{2} + 2x + 1 = (x + 1)^{2}

and

x^{3} + 3x^{2} + 3x + 1 = (x + 1)^{3}

given by

(\begin{matrix}1 & 2 & 1 & 0 & 0 \\ 0 & 1 & 2 & 1 & 0 \\ 0 & 0 & 1 & 2 & 1 \\ 1 & 3 & 3 & 1 & 0 \\ 0 & 1 & 3 & 3 & 1 \end{matrix}) .

Its determinant is 0, which indicates the obvious fact that the two polynomials (x + 1)^{2} and (x + 1)^{3} have a common root.

2010 Mathematics Subject Classification. Primary 33D67; Secondary 15A15. Key words and phrases. Determinant, C_{n} elliptic hypergeometric series, Sylvester matrix. *Research supported by a grant of the Austrian Science Fund (FWF): START grant Y463. **Research partially supported by the Austrian Science Fund (FWF), grant F50-N15, in the frame-

work of the Special Research Program “Algorithmic and Enumerative Combinatorics”.

More generally, consider the Sylvester matrix defined as follows. Let

b_{ij}(s, r) := \binom{r}{j − i}s^{j−i}

and consider the (r_{1} + r_{2}) × (r_{1} + r_{2}) matrix B = (b′_{ij}), where

b′_{ij} = \begin{cases}b_{ij}(s_{1}, r_{1}), & for 0 ≤ i ≤ r_{2} − 1, \\ b_{i−r_{2},j}(s_{2}, r_{2}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

Then its determinant is given by

det B = (−1)^{r_{1}r_{2}}(s_{1} − s_{2})^{r_{1}r_{2}}.  (1.1)

For example, for r_{1} = 2 and r_{2} = 3, we have

det (\begin{matrix}1 & 2s_{1} & s_{1}^{2} & 0 & 0 \\ 0 & 1 & 2s_{1} & s_{1}^{2} & 0 \\ 0 & 0 & 1 & 2s_{1} & s_{1}^{2} \\ 1 & 3s_{2} & 3s_{2}^{2} & s_{2}^{3} & 0 \\ 0 & 1 & 3s_{2} & 3s_{2}^{2} & s_{2}^{3} \end{matrix}) = (s_{1} − s_{2})^{6}.

This explains why the determinant is 0 when s_{1} and s_{2} are both 1. The matrix of this last example is the Sylvester matrix corresponding to the polynomials

x^{2} + 2s_{1}x + s_{1}^{2} = (x + s_{1})^{2}

and

x^{3} + 3s_{2}x^{2} + 3s_{2}^{2}x + s_{2}^{3} = (x + s_{2})^{3}.

The formula for det B follows from a well-known result concerning the determinant of Sylvester polynomials, given in, for example, Loos [7, Theorem 1, p. 177]. Consider the polynomials

C ∏_{i=1}^{r_{1}}(x − γ_{i}) and K ∏_{j=1}^{r_{2}}(x − β_{j}),

where C, K are constants. Then the determinant of the corresponding Sylvester matrix is given by

C^{r_{2}}K^{r_{1}} ∏_{i=1}^{r_{1}} ∏_{j=1}^{r_{2}}(γ_{i} − β_{j}).

However, this classic idea fails in the evaluation of the determinant of the following “Sylvesteresque” matrix, which appears in the context of approximation theory in work of the second author with Feng and Xu [2].

To state their formula, we require the rising factorial (or Pochhammer symbol) (a)_{n}, which is defined by (a)_{0} := 1 and

(a)_{k} := ∏_{j=0}^{k−1}(a + j), for k = 1, 2, ….

Let

m_{ij}(s_{1}, s_{2}, r) := \binom{r}{j − i}\begin{matrix}(s_{1} + i) & j−i \\ (s_{1} + s_{2} + i + j − 1)_{j−i} & (s_{1} + s_{2} + r + 2i)_{j−i} \end{matrix} .

Consider the (r_{1} + r_{2}) × (r_{1} + r_{2}) matrix M = (m′_{ij}), where

m′_{ij} = \begin{cases}m_{ij}(s_{1}, s_{2}, r_{1}), & for 0 ≤ i ≤ r_{2} − 1, \\ (−1)^{j−i−r_{2}}m_{i−r_{2},j}(s_{2}, s_{1}, r_{2}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

This matrix is not quite a Sylvester matrix, but still has a nice determinant evaluation (see [2, Theorem 4.1])

det M = (−1)^{r_{1}r_{2}} ∏_{j=1}^{r_{1}} \frac{1}{(s_{1} + s_{2} + r_{1} + r_{2} + j − 2)_{r_{2}}} .  (1.2)

It is not obvious, but the formula for det M extends the one for det B. To obtain (1.1) from (1.2), multiply row i of det M by

\begin{cases}\frac{t^{i}}{(s_{1} + s_{2})^{2i}} , & for 0 ≤ i ≤ r_{2} − 1, \\ \frac{t^{i−r_{2}}}{(s_{1} + s_{2})^{2(i−r_{2})}} , & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

column j by

\frac{(s_{1} + s_{2})^{2j}}{t^{j}} , for 0 ≤ j ≤ r_{1} + r_{2} − 1,

and multiply by the corresponding products on the right hand side of (1.2). Next, replace s_{1} by s_{1}t, s_{2} by s_{2}t and take the limit as t → ∞. Equation (1.1) follows after replacing s_{2} by −s_{2} on both sides.

In this paper, we extend the formula for det M to one with elliptic hypergeometric terms. Our first result (stated in Section 2) is the evaluation of an elliptic determinant, which has two additional parameters even when specialized to the hypergeometric case.

Our proof (in Section 3) is an elliptic extension of the one given in [2], and requires a determinant lemma due to Warnaar [12, Lemma 5.3] and a C_{n} summation theorem. Warnaar’s determinant lemma is an elliptic extension of a very useful determinant lemma due to the second author [5, Lemma 5]. The C_{n} summation theorem was conjectured by Warnaar [12, Corollary 6.2, x = q], proved by Rosengren [9] (and in more generality by Rains [8, Theorem 4.9] and, independently by Coskun and Gustafson [1]). A combinatorial proof was given by Schlosser [11].

The summation formula is a C_{n} extension of Frenkel and Turaev’s [3]_{10}V_{9} summation formula (see [4, (11.4.1)]). There is a more general transformation formula which is a C_{n} extension of Frenkel and Turaev’s _{12}V_{11} transformation formula, given in [4, (11.5.1)]. Again, this was conjectured by Warnaar [12, Conjecture 6.1, x = q], and proved—in more generality, and independently—by Rains [8, Theorem 4.9] and by Coskun and Gustafson [1].

Naturally, we consider what happens if we use the C_{n} _{12}V_{11} transformation formula. This leads (in Section 4) to a surprisingly elegant transformation formula between two elliptic determinants. In Section 5, we show how to recover our determinant evaluation from this transformation formula by using elementary determinant operations.

Acknowledgements. We thank Michael Schlosser for helpful discussions. We also thank the referees for many useful suggestions.

2. An elliptic determinant evaluation

In this section we state our first theorem, an elliptic determinant evaluation. To define the entries of the matrix under consideration, we need the notation for elliptic shifted factorials. For these notations, and background results, Gasper and Rahman [4, Ch. 11] is the standard reference. We recommend Rosengren [10] for a friendly introduction to elliptic hypergeometric series.

We follow Gasper and Rahman [4, (11.6.2)] and define products as follows:

∏_{j=k}^{n} a_{j} := \begin{cases}a_{k}a_{k+1} ⋯ a_{n}, & if n ≥ k, \\ 1, & if n = k − 1, \\ (a_{n+1}a_{n+2} ⋯ a_{k−1})^{−1}, & if n ≤ k − 2. \end{cases}  (2.1)

The q-shifted factorials, for k any integer, are defined as

(a; q)_{k} := ∏_{j=0}^{k−1}(1 − aq^{j}),

and for |q| < 1,

(a; q)_{∞} := ∏_{j=0}^{∞}(1 − aq^{j}).

The modified Jacobi theta function is defined as

θ(a; p) := (a; p)_{∞} (p/a; p)_{∞},

where a ≠ 0 and |p| < 1. The q, p-shifted factorials (or theta shifted factorials), for k an integer, are defined as

(a; q, p)_{k} := ∏_{j=0}^{k−1} θ(aq^{j}; p) .

The parameters p and q are called the nome and base, respectively. When p = 0, the modified theta function θ(a; p) reduces to (1 − a); and thus (a; q, p)_{k} reduces to (a; q)_{k}.

Further, we use the short-hand notations

θ(a_{1}, a_{2}, …, a_{r}; p) := θ(a_{1}; p) θ(a_{2}; p) ⋯ θ(a_{r}; p) ,
(a_{1}, a_{2}, …, a_{r}; q, p)_{k} := (a_{1}; q, p)_{k}(a_{2}; q, p)_{k} ⋯ (a_{r}; q, p)_{k},
(a_{1}, a_{2}, …, a_{r}; q)_{k} := (a_{1}; q)_{k} (a_{2}; q)_{k} ⋯ (a_{r}; q)_{k}.

Next, we define the elliptic matrix under consideration. Let s_{1}, s_{2}, t_{1} and t_{2} be arbitrary complex numbers, r_{1}, r_{2}, i, and j be non-negative integers, and let

f_{ij} = f_{ij}(s_{1}, s_{2}, t_{1}, t_{2}, r_{1}, r_{2}) := q^{\binom{j−i}{2}+r_{2}(j−i)} \begin{matrix}(q; q, p) & r_{1} \\ (q; q, p) & r_{1}−j+i \end{matrix}

× \begin{matrix}(s_{1}q^{i}, t_{1}q^{i}, t_{2}q^{i}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}+i}/t_{1}t_{2}; q, p) & j−i \\ (q, s_{1}s_{2}q^{i+j−1}, s_{1}s_{2}q^{r_{1}+2i}; q, p) & j−i \end{matrix} . (2.2)

We consider the (r_{1} + r_{2}) × (r_{1} + r_{2}) matrix F = (f′_{ij}), where

f′_{ij} = \begin{cases}f_{ij}(s_{1}, s_{2}, t_{1}, t_{2}, r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ f_{i−r_{2},j}(s_{2}, s_{1}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

The form of F is similar to that of Sylvester matrices. Observe that, in view of (2.1), we have

\frac{1}{(q)_{−r}} = 0, for r = 1, 2, 3, ….

Thus, for the first r_{2} rows, m_{ij} = 0 if j < i, and if j > r_{1} + i. Further, the first non-zero entry is 1 (when i = j). Similar remarks apply for the next r_{1} rows of F.

Theorem 1. Let F be the matrix defined above. Then

det F = (−1)^{r_{1}r_{2}} (\frac{t_{1}t_{2}}{s_{2}} )^{r_{1}r_{2}} q^{\frac{1}{2} r_{1}r_{2}(r_{1}+4r_{2}−3)}

× ∏_{j=1}^{r_{1}} \begin{matrix}(s_{2}q^{−r_{2}+j}/t_{1}, s_{2}q^{−r_{2}+j}/t_{2}, s_{1}s_{2}q^{−r_{2}+j}/t_{1}t_{2}; q, p) & r_{2} \\ (s_{1}s_{2}q^{r_{1}+r_{2}+j−2}; q, p) & r_{2} \end{matrix} . (2.3)

Remark. Theorem 1 can also be stated as

det F = s_{1}^{r_{1}r_{2}}q^{r_{1}r_{2}(r_{1}+r_{2}−1)+r_{1}\binom{r_{2}}{2}}

× ∏_{j=1}^{r_{1}} \begin{matrix}(s_{2}q^{−r_{2}+j}/t_{1}, s_{2}q^{−r_{2}+j}/t_{2}, t_{1}t_{2}q^{−r_{1}+j}/s_{1}s_{2}; q, p) & r_{2} \\ (s_{1}s_{2}q^{r_{1}+r_{2}+j−2}; q, p) & r_{2} \end{matrix} . (2.4)

Next we take a special case to obtain a q-analogue of (1.2). Let

u_{ij} = u_{ij}(s_{1}, s_{2}, r_{1}, r_{2}) := q^{\binom{j−i}{2}} \begin{matrix}(q; q)_{r_{1}} & (s_{1}q^{i}; q)_{j−i} \\ (q; q)_{r_{1}−j+i} & (q, s_{1}s_{2}q^{i+j−1}, s_{1}s_{2}q^{r_{1}+2i}; q)_{j−i} \end{matrix} .  (2.5)

We consider the (r_{1} + r_{2}) × (r_{1} + r_{2}) matrix U = (u′_{ij}), where

u′_{ij} = \begin{cases}q^{\binom{j}{2}−\binom{i}{2}} u_{ij}(s_{1}, s_{2}, r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ (−1/s_{2})^{j−i+r_{2}} u_{i−r_{2},j}(s_{2}, s_{1}, r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

Corollary 2. Let U be the matrix defined above. Then

det U = \frac{(−1)^{r_{1}r_{2}}}{s_{2}^{r_{1}r_{2}}} q^{r_{1}\binom{r_{2}}{2}} ∏_{j=1}^{r_{1}} \frac{1}{(s_{1}s_{2}q^{r_{1}+r_{2}+j−2}; q)_{r_{2}}} .  (2.6)

Remarks. (1) This determinant evaluation is a q-analogue of (1.2). To see this, we replace s_{1} and s_{2} by q^{s_{1}} and q^{s_{2}} (respectively), multiply both sides of (2.6) by (1−q)^{r_{1}r_{2}} and take the limits as q → 1.

To take entry-wise limits in the determinant, we need to multiply each entry by an appropriate power of (1 − q). Thus we multiply each entry of column j by (1 − q)^{j}, and divide the first r_{2} rows by (1 − q)^{i}, and the last r_{1} rows by (1 − q)^{i−r_{2}}. We compensate by multiplying the resulting determinant by

∏_{j=0}^{r_{1}+r_{2}−1} \frac{1}{(1 − q)^{j}} ∏_{i=0}^{r_{2}−1} (1 − q)^{i} ∏_{i=r_{2}}^{r_{1}+r_{2}−1} (1 − q)^{i−r_{2}} = \frac{1}{(1 − q)^{r_{1}r_{2}}}

This explains why we need to multiply by (1 − q)^{r_{1}r_{2}} on the left hand side. The limit on the right hand side too requires this additional power of (1 − q).

(2) Corollary 2 being a q-analogue of (1.2), it may be the starting point for finding a q-analogue for the best approximation result in [2].

Proof. We take p = 0 in (2.3), and after dividing both sides by (t_{1}t_{2})^{r_{1}r_{2}}, take the limits as t_{1} → ∞ and t_{2} → ∞. From the right hand side, we obtain

\frac{(−1)^{r_{1}r_{2}}}{s_{2}^{r_{1}r_{2}}} q^{\frac{1}{2} r_{1}r_{2}(r_{1}+4r_{2}−3)} ∏_{j=1}^{r_{1}} \frac{1}{(s_{1}s_{2}q^{r_{1}+r_{2}+j−2}; q)_{r_{2}}} .

Before taking the limits on the left hand side, we use the elementary modifications

(t_{1}q^{i}, t_{2}q^{i}; q)_{j−i} = (q^{1−j}/t_{1}, q^{1−j}/t_{2}; q)_{j−i}(t_{1}t_{2})^{j−i}q^{2\binom{j−i}{2}+2i(j−i)}

to reverse some of the products in the first r_{2} rows of det F, and

(t_{1}t_{2}q^{−r_{1}+i}/s_{2}; q)_{j−i+r_{2}} = (q^{1+r_{1}−r_{2}−j}s_{2}/t_{1}t_{2}; q)_{j−i+r_{2}}

× (−1)^{j−i+r_{2}} (\frac{t_{1}t_{2}}{s_{2}} )^{j−i+r_{2}} q^{\binom{j−i+r_{2}}{2}+(−r_{1}+i)(j−i+r_{2})}

in the last r_{1} rows of the determinant. From the jth column of the determinant, we take out (t_{1}t_{2})^{j}. From the first i rows, we can take out (t_{1}t_{2})^{−i}, and from the last r_{1} rows, we take out (t_{1}t_{t})^{−(i−r_{2})}. This results in the product (t_{1}t_{2})^{r_{1}r_{2}} outside the determinant that cancels the product we divided earlier.

Now we take the limits as t_{1}, t_{2} → ∞ term-wise in the determinant, to obtain a determinant whose entries are given by

\begin{cases}q^{2\binom{j}{2}−2\binom{i}{2}+r_{2}(j−i)} u_{ij}(s_{1}, s_{2}, r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ (−1/s_{2})^{j−i+r_{2}} q^{\binom{j−i+r_{2}}{2}+i(j−i+r_{2})} u_{i−r_{2},j}(s_{2}, s_{1}, r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

Again, we take out some powers of q from the determinant, and cancel them from the other side to obtain Corollary 2. □

A slightly different limiting case yields another determinant evaluation. Again, let u_{ij} be defined by (2.5). We consider the matrix V = (v′_{ij}), where

v′_{ij} = \begin{cases}u_{ij}(s_{1}, s_{2}, r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ (s_{1}/s_{2})^{(j−i+r_{2})/2} u_{i−r_{2},j}(s_{2}, s_{1}, r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

Corollary 3. Let V be the matrix defined above. Then

det V = (\frac{s_{1}}{s_{2}} )^{\frac{1}{2} r_{1}r_{2}} q^{r_{1}\binom{r_{2}}{2}} ∏_{j=1}^{r_{1}} \begin{matrix}((s_{2}/s_{1})^{1/2}q^{−r_{2}+j}; q) & r_{2} \\ ((s_{1}s_{2})^{1/2}q^{j−1}, s_{1}s_{2}q^{r_{1}+r_{2}+j−2}; q) & r_{2} \end{matrix} .  (2.7)

Remark. When we take the hypergeometric case of Corollary 3, by replacing s_{1} by q^{s_{1}}, s_{2} by q^{s_{2}}, and taking the limit as q → 1, we obtain the determinant of a matrix very similar to the matrix M considered in (1.2). The difference is that there are no alternating signs in the last r_{1} rows of the matrix.

Proof. We take p = 0 in (2.3), divide both sides by t_{2}^{r_{1}r_{1}} and take limits as t_{2} → ∞. Next we replace t_{1} by (s_{1}s_{2})^{1/2}.

In the resulting determinant on the left hand side we multiply the ith row by

\begin{cases}((s_{1}s_{2})^{1/2}; q)_{i}, & for 0 ≤ i ≤ r_{2} − 1, \\ ((s_{1}s_{2})^{1/2}; q)_{i−r_{2}}, & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

and, to compensate, divide the determinant by

∏_{i=0}^{r_{2}−1} ((s_{1}s_{2})^{1/2}; q)_{i} ∏_{i=r_{2}}^{r_{1}+r_{2}−1} ((s_{1}s_{2})^{1/2}; q)_{i−r_{2}}.

In the resulting determinant, column j has the common factor

((s_{1}s_{2})^{1/2}; q)_{j}

that we can take out from each column. In addition, we take out negative signs and powers of q to obtain

(−1)^{r_{1}r_{2}}q^{\frac{1}{2} r_{1}r_{2}(r_{1}+3r_{2}−2)} \begin{matrix}∏_{j=0}^{r_{1}+r_{2}−1} & ((s_{1}s_{2})^{1/2}; q)_{j} \\ ∏_{i=0}^{r_{2}−1} & ((s_{1}s_{2})^{1/2}; q)_{i} & ∏_{i=0}^{r_{1}−1} & ((s_{1}s_{2})^{1/2}; q)_{i} \end{matrix} det V.

Now comparing with what we obtain on the right hand side after taking the limit t_{2} → ∞ and t_{1} = (s_{1}s_{2})^{1/2}, we obtain Corollary 3. □

3. Proof of Theorem 1

In this section we give a proof of Theorem 1. First we record the required results in a form suitable for our use. From now on, we suppress q and p and denote the theta shifted factorials by (a)_{k}.

We begin with a determinant lemma due to Warnaar [12, Lemma 5.3].

Proposition 4. Let A_{0}, …, A_{n−1}, c, and x_{0}, …, x_{n−1}, be arbitrary complex numbers. If, for i = 0, 1, …, n − 1, P_{i} is analytic in 0 < |x| < ∞, and satisfies the following two conditions:

(1) P_{i}(px) = (c/x^{2}p)^{i}P_{i}(x) (quasi-periodicity), and

(2) P_{i}(c/x) = P_{i}(x) (symmetry), then

det_{0≤i,j≤n−1} (P_{i}(x_{j}) ∏_{k=i+1}^{n−1} θ(A_{k}x_{j}) θ(cA_{k}/x_{j}))

= ∏_{0≤i<j≤n−1}(cA_{j}/x_{j})θ(x_{j}/x_{i}) θ(x_{i}x_{j}/c) ∏_{i=0}^{n−1} P_{i}(1/A_{i}).

Warnaar’s determinant lemma is used to evaluate the following determinant.

Lemma 5. For all positive integers r_{2}, we have

det_{0≤i,j≤r_{2}−1} (q^{−ik_{j}}(q^{k_{j}−i+1}, aq^{k_{j}})_{i} (q^{r_{1}−k_{j}+i+1}, aq^{r_{1}+k_{j}+i+1})_{r_{2}−i−1})

= q^{− ∑_{i=0}^{r_{2}−1} ik_{i}} ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, aq^{k_{i}+k_{j}}) ∏_{i=0}^{r_{2}−1} (q^{r_{1}+1}, aq^{r_{1}+i})_{i}.

Proof. We apply Proposition 4 with

P_{i}(x) = (c^{\frac{1}{2}} /x)^{i} (xq^{1−i}, x/c)_{i}.

The quasi-periodicity and symmetry properties of P_{i}(x) are easy to verify using the elementary identities [4, (11.2.55)]

(a)_{n} = (pa)_{n}(−a)^{n}q^{\binom{n}{2}}

and [4, (11.2.48)]

(a)_{n} = (q^{1−n}/a)_{n}(−a)^{n}q^{\binom{n}{2}}.

The special case n = r_{2}, x_{j} = q^{k_{j}}, A_{k} = q^{−r_{1}−k}, c = 1/a of Proposition 4 implies that

det_{0≤i,j≤r_{2}−1}(q^{2\binom{i+1}{2}−2\binom{r_{2}}{2}−(2r_{1}−1)(r_{2}−i−1)} (1/a)^{i/2} (1/aq)^{r_{2}−i−1}

· q^{−ik_{j}}(q^{k_{j}−i+1}, aq^{k_{j}})_{i} (q^{r_{1}−k_{j}+i+1}, aq^{r_{1}+i+k_{j}+1})_{r_{2}−i−1})

= ∏_{0≤i<j≤r_{2}−1}(q^{−r_{1}−j−k_{j}}/a) θ(q^{k_{j}−k_{i}}, aq^{k_{i}+k_{j}})

× ∏_{i=0}^{r_{2}−1}(1/a)^{i/2} (q^{−r_{1}−i})^{i} (q^{r_{1}+1}, aq^{r_{1}+i})_{i}.

To complete the proof of the lemma, we take out the common factors from each row of the determinant, cancel common terms, and simplify. □

Next, we require a C_{n} extension of Frenkel and Turaev’s _{10}V_{8} summation formula. As explained earlier, this is due to Warnaar and Rosengren, but we prefer to use a formulation due to Schlosser. In Schlosser [11, Theorem 3.1], we take r = r_{2}, use the summation indices k_{0}, k_{1}, …, k_{r_{2}−1}, make the substitutions a ↦ s_{1}s_{2}/q, b ↦ s_{1}, c ↦ t_{1}, d ↦ t_{2} and take m = r_{1} + r_{2} − 1, to obtain

∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1} ∏_{j=0}^{r_{2}−1} \begin{matrix}(s_{1}s_{2}/q, s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}, q^{−(r_{1}+r_{2}−1)}) & k_{j} \\ (q, s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}, s_{1}s_{2}q^{r_{1}+r_{2}−1}) & k_{j} \end{matrix}

· ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, s_{1}s_{2}q^{k_{i}+k_{j}−1})^{2} ∏_{j=0}^{r_{2}−1} \frac{θ(s_{1}s_{2}q^{2k_{j}−1})}{θ(s_{1}s_{2}/q)}q^{∑_{i=0}^{r_{2}−1}(2r_{2}−2i−1)k_{i}}

= q^{−4\binom{r_{2}}{3}} (\frac{s_{2}}{t_{1}t_{2}q^{2}} )^{\binom{r_{2}}{2}} ∏_{j=1}^{r_{2}} (q, s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2})_{j−1} (q, s_{1}s_{2})_{r_{1}+r_{2}−1}

× ∏_{j=1}^{r_{2}} \begin{matrix}(s_{2}q^{1−j}/t_{1}, s_{2}q^{1−j}/t_{2}, s_{1}s_{2}q^{1−j}/t_{1}t_{2}) & r_{1} \\ (q, s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, s_{2}q^{1−2r_{2}+j}/t_{1}t_{2}) & r_{1}+r_{2}−j \end{matrix} .  (3.1)

These are all the ingredients required for the proof of Theorem 1.

Proof of Theorem 1. Let F_{b_{1},a_{2},...,b_{r}}^{a_{1},a_{2},...,a_{r}} denote the submatrix of F consisting of rows a_{1}, …, a_{r} and columns b_{1}, …, b_{r}. By taking the Laplace expansion of det F with respect to the first r_{2} rows, we find that

det F = ∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1}(−1)^{\binom{r_{2}}{2}+ ∑_{i=0}^{r_{2}−1} k_{i}} det F_{k_{0},k_{1},...,k_{r_{2}−1}}^{0,1,...,r_{2}−1} det F_{l_{0},l_{1},...,l_{r_{1}−1}}^{r_{2},r_{2}+1,...,r_{1}+r_{2}−1},

(3.2)

where {l_{0}, l_{1}, …, l_{r_{1}−1}} is the complement of the set {k_{0}, k_{1}, …, k_{r_{2}−1}} in the set {0, 1, 2, …, r_{1} + r_{2} − 1}. Let D_{1} and D_{2} denote the two determinants in the sum. Then

D_{1} = det F_{k_{0},k_{1},...,k_{r_{2}−1}}^{0,1,...,r_{2}−1} = det (f_{i,k_{j}}(s_{1}, s_{2}, t_{1}, t_{2}, r_{1}, r_{2}))

and

D_{2} = det F_{l_{0},l_{1},...,l_{r_{1}−1}}^{r_{2},r_{2}+1,...,r_{1}+r_{2}−1} = det (f_{i,l_{j}}(s_{2}, s_{1}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, r_{2}, r_{1})) ,

where the f_{ij}’s are as in (2.2). After evaluating these determinants using Lemma 5, the resulting sum can be evaluated using the elliptic C_{n} summation theorem given in (3.1). The result follows after performing a large amount of simplification. Here are some more details.

We first evaluate D_{1} = det (f_{i,k_{j}}), where we suppress the dependence on other parameters for the time being. We first take out many factors out of the rows (indexed by i) and columns (indexed by j), with the goal that the entries of the resulting determinant is a polynomial. We have

D_{1} = det (f_{i,k_{j}}) = q^{∑_{j=0}^{r_{2}−1} \binom{k_{j}}{2}+r_{2} ∑_{j=0}^{r_{2}−1} k_{j}− \frac{1}{3} (2r_{2}−1)\binom{r_{2}}{2}} ∏_{i=0}^{r_{2}−1} \begin{matrix}(q)_{r_{1}} & (s_{1}s_{2}q^{r_{1}})_{2i} \\ (s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}) & i \end{matrix}

× ∏_{j=0}^{r_{2}−1} \begin{matrix}(s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}) & k_{j} \\ (q, s_{1}s_{2}q^{k_{j}−1})_{k_{j}} & (q)_{r_{1}+r_{2}−k_{j}−1} & (s_{1}s_{2}q^{r_{1}})_{r_{2}+k_{j}−1} \end{matrix}

× det_{0≤i,j≤r_{2}−1} (q^{−ik_{j}}(q^{k_{j}−i+1}, s_{1}s_{2}q^{k_{j}−1})_{i} (q^{r_{1}−k_{j}+i+1}, s_{1}s_{2}q^{r_{1}+k_{j}+i})_{r_{2}−i−1})

= q^{∑_{j=0}^{r_{2}−1} \binom{k_{j}}{2}+r_{2} ∑_{j=0}^{r_{2}−1} k_{j}− \frac{1}{3} (2r_{2}−1)\binom{r_{2}}{2}− ∑_{i=0}^{r_{2}−1} ik_{i}}

× ∏_{i=0}^{r_{2}−1} \begin{matrix}(q)_{r_{1}} & (q^{r_{1}+1}, s_{1}s_{2}q^{r_{1}+i−1})_{i} & (s_{1}s_{2}q^{r_{1}})_{2i} \\ (s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}) & i \end{matrix}

× ∏_{j=0}^{r_{2}−1} \begin{matrix}(s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}) & k_{j} \\ (q, s_{1}s_{2}q^{k_{j}−1})_{k_{j}} & (q)_{r_{1}+r_{2}−k_{j}−1} & (s_{1}s_{2}q^{r_{1}})_{r_{2}+k_{j}−1} \end{matrix}

× ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, s_{1}s_{2}q^{k_{i}+k_{j}−1}) ,

where the determinant evaluation in the last line is from Lemma 5, with a ↦ s_{1}s_{2}/q. The second determinant

D_{2} = det (f_{i,l_{j}}(s_{2}, s_{1}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, r_{2}, r_{1}))

is obtained from the above by replacing k_{j} by l_{j}, and simultaneously replacing

(s_{1}, s_{2}, t_{1}, t_{2}, r_{1}, r_{2}) by (s_{2}, s_{1}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, r_{2}, r_{1}).

Note that the indices of summation are k_{j} and the expression we obtain for D_{2} is in terms of the l_{j}’s. We use elementary algebraic manipulations to express the products in terms of the k_{j}’s. For example, we use the “inclusion-exclusion” formula

∏_{0≤i<j≤r_{1}−1}θ(q^{l_{j}−l_{i}}) = \frac{∏_{0≤i<j≤r_{1}+r_{2}−1}θ(q^{j−i}) ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}})}{∏_{j=0}^{r_{2}−1} ∏_{i=0}^{k_{j}−1} θ(q^{k_{j}−i}) ∏_{i=0}^{r_{2}−1} ∏_{j=k_{i}+1}^{r_{1}+r_{2}−1} θ(q^{j−k_{i}})}

= \frac{∏_{i=0}^{r_{1}+r_{2}−1} (q)_{i} ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}})}{∏_{j=0}^{r_{2}−1} (q)_{k_{j}} ∏_{i=0}^{r_{2}−1} (q)_{r_{1}+r_{2}−k_{i}−1}};

and, using the same idea,

q^{− ∑_{i=0}^{r_{1}−1} il_{i}} = ∏_{0≤i<j≤r_{1}−1}q^{−l_{j}}

= q^{− ∑_{i=0}^{r_{2}−1} ik_{i}+ ∑_{i=0}^{r_{2}−1} \binom{k_{i}}{2}−(r_{1}−1)\binom{r_{1}+r_{2}}{2}+ ∑_{i=1}^{r_{1}+r_{2}−1} \binom{i}{2}}.

After some further algebraic simplification, we obtain

D_{2} = q^{\frac{1}{3} (2r_{1}+2r_{2}−1)\binom{r_{1}+r_{2}}{2}−r_{1} ∑_{j=0}^{r_{2}−1} k_{j}− \frac{1}{3} (2r_{1}−1)\binom{r_{1}}{2}− ∑_{i=0}^{r_{2}−1} ik_{i}}

× ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, s_{1}s_{2}q^{k_{i}+k_{j}−1})

× ∏_{j=0}^{r_{1}+r_{2}−1} \begin{matrix}(s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}) & j \\ (s_{1}s_{2}q^{r_{2}})_{r_{1}+j−1} & (q)_{r_{1}+r_{2}−j−1} \end{matrix}

× ∏_{j=0}^{r_{1}−1} \begin{matrix}(q)_{r_{2}} & (q^{r_{2}+1}, s_{1}s_{2}q^{r_{2}+j−1})_{j} & (s_{1}s_{2}q^{r_{2}})_{2j} \\ (s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}) & j \end{matrix}

× ∏_{j=0}^{r_{2}−1} \begin{matrix}(s_{1}s_{2}q^{r_{2}}) & r_{1}+k_{j}−1 \\ (s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2})_{k_{j}} & (s_{1}s_{2}q^{2k_{j}})_{r_{1}+r_{2}−k_{j}−1} \end{matrix} .

Next we substitute the above expressions for D_{1} and D_{2} in (3.2) to obtain, after some algebraic manipulation, the multiple sum

det F = (−1)^{\binom{r_{2}}{2}}q^{r_{1}r_{2}(r_{1}+r_{2}−1)}

× ∏_{j=0}^{r_{1}+r_{2}−1} \begin{matrix}(s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}) & j \\ (s_{1}s_{2}q^{r_{2}})_{r_{1}+j−1} & (q)_{r_{1}+r_{2}−j−1} \end{matrix}

× ∏_{j=0}^{r_{1}−1} \begin{matrix}(q)_{r_{2}} & (q^{r_{2}+1}, s_{1}s_{2}q^{r_{2}+j−1})_{j} & (s_{1}s_{2}q^{r_{2}})_{2j} \\ (s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}) & j \end{matrix}

× ∏_{j=0}^{r_{2}−1} \begin{matrix}(q)_{r_{1}} & (q^{r_{1}+1}, s_{1}s_{2}q^{r_{1}+j−1})_{j} & (s_{1}s_{2}q^{r_{1}})_{2j} & (s_{1}s_{2})_{r_{1}} \\ (s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2})_{j} & (s_{1}s_{2})_{r_{2}} & (s_{1}s_{2})_{r_{1}+r_{2}−1} & (q)_{r_{1}+r_{2}−1} \end{matrix}

× ∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1} ( ∏_{j=0}^{r_{2}−1} \begin{matrix}(s_{1}s_{2}/q, s_{1}, t_{1}, t_{2}, s_{1}s_{2}^{2}q^{r_{1}−r_{2}}/t_{1}t_{2}, q^{−(r_{1}+r_{2}−1)}) & k_{j} \\ (q, s_{2}, s_{1}s_{2}/t_{1}, s_{1}s_{2}/t_{2}, t_{1}t_{2}q^{r_{2}−r_{1}}/s_{2}, s_{1}s_{2}q^{r_{1}+r_{2}−1}) & k_{j} \end{matrix}

· ∏_{0≤i<j≤r_{2}−1} (θ(q^{k_{j}−k_{i}}, s_{1}s_{2}q^{k_{i}+k_{j}−1}))^{2} ∏_{j=0}^{r_{2}−1} \frac{θ(s_{1}s_{2}q^{2k_{j}−1})}{θ(s_{1}s_{2}/q)}q^{∑_{i=0}^{r_{2}−1}(2r_{2}−2i−1)k_{i}}).

This sum can be evaluated using (3.1). After replacing the multiple sum by the corresponding products, we again require a large number of algebraic simplifications. For example, we use

∏_{i=1}^{r_{2}} (s_{2}q^{1−i}/t_{1})_{r_{1}} = ∏_{j=1}^{r_{1}} (s_{2}q^{−r_{2}+j}/t_{1})_{r_{2}},

∏_{j=0}^{r_{1}−1} (s_{1}s_{2}q^{r_{2}+j−1})_{j} = ∏_{i=0}^{r_{1}−1} (s_{1}s_{2}q^{r_{2}+2i})_{r_{1}−i−1},

∏_{i=0}^{r_{2}−1} \begin{matrix}(s_{1}s_{2})_{r_{1}} & (s_{1}s_{2}q^{r_{1}})_{2i} & (s_{1}s_{2}q^{r_{1}+i−1})_{i} \\ (s_{1}s_{2}) & r_{1}+r_{2}+i−1 \end{matrix} = 1,

and other elementary identities. The result then condenses to the right hand side of (2.3). □

4. A transformation formula for elliptic determinants

In this section, we derive a transformation formula between two determinants that extends Theorem 1, by adding two further parameters. It is apparent from our proof in Section 3 that the matrices we consider are closely linked with very-well-poised elliptic hypergeometric series (see [4] for the terminology, if required). We make this connection transparent by re-labelling the parameters.

The Sylvesteresque matrices which we study now are defined as follows. Let a, b, c, d, e, f be arbitrary complex numbers, r_{1}, r_{2}, i, and j be non-negative integers, and let

g_{ij}(a; b, c, d, e, f; r_{1}, r_{2}) := q^{\binom{j−i}{2}+r_{2}(j−i)} \begin{matrix}(q) & r_{1} \\ (q) & r_{1}−j+i \end{matrix}

× \begin{matrix}(bq^{i}, cq^{i}, dq^{i}, eq^{i}, fq^{i}, a^{3}q^{r_{1}−r_{2}+i+3}/bcdef) & j−i \\ (q, aq^{i+j}, aq^{r_{1}+2i+1}) & j−i \end{matrix} . (4.1)

We consider the (r_{1} + r_{2}) × (r_{1} + r_{2}) matrices G = (g′_{ij}) and H = (h′_{ij}), where

g′_{ij} = \begin{cases}g_{ij}(a; b, c, d, e, f; r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ g_{i−r_{2},j}(a; aq/b, aq/c, aq/d, aq/e, aq/f; r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

and

h′_{ij} = \begin{cases}g_{ij}(λ; λb/a, λc/a, λd/a, e, f; r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ g_{i−r_{2},j}(λ; aq/b, aq/c, aq/d, λq/e, λq/f; r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. \end{cases}

Theorem 6. With G and H as defined above, and λ = a^{2}q^{2−r_{2}}/bcd, we have

det G = (\frac{a}{λ})^{r_{1}r_{2}} ∏_{j=1}^{r_{1}} \begin{matrix}(λq^{r_{1}+r_{2}+j−1}) & r_{2} \\ (aq^{r_{1}+r_{2}+j−1}) & r_{2} \end{matrix} det H.

Remark. We reiterate that we have suppressed the q, p in our notation.

Sketch of proof. The first few steps in the proof are similar to those of Theorem 1. We begin with the Laplace expansion of det G with respect to the first r_{2} rows to obtain an expression of the form

det G = ∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1}(−1)^{\binom{r_{2}}{2}+ ∑_{i=0}^{r_{2}−1} k_{i}}D_{1} · D_{2},

where D_{1} and D_{2} are the determinants

D_{1} = det (g_{i,k_{j}}(a; b, c, d, e, f; r_{1}, r_{2}))

and

D_{2} = det (g_{i,l_{j}}(a; aq/b, aq/c, aq/d, aq/e, aq/f; r_{2}, r_{1})) .

Here the g_{ij}’s are as in (4.1) and the l_{j}’s have the same meaning as in the proof of Theorem 1.

Again, both determinants D_{1} and D_{2} can be evaluated using Lemma 5, after taking out common factors from rows and columns of each determinant; and again, the expression for D_{2} is in terms of the l_{j}’s and we have to write it in terms of the indices of summation k_{j}. After some algebraic manipulations, just as done earlier, we arrive at a C_{n} multiple sum. This multiple sum can be transformed into a multiple of another multiple sum using the C_{n} transformation formula (4.2) below.

Next, we perform these steps in reverse. Using Lemma 5, and some algebraic simplification, we write the summand of the resulting sum in the form

(⋆ ⋆ ⋆) ∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1}(−1)^{\binom{r_{2}}{2}+ ∑_{i=0}^{r_{2}−1} k_{i}}D_{3} · D_{4},

where (⋆ ⋆ ⋆) are some explicit products, and D_{3} and D_{4} are the determinants

D_{3} = det (g_{i,k_{j}}(λ; λb/a, λc/a, λd/a, e, f; r_{1}, r_{2}))

and

D_{4} = det (g_{i,l_{j}}(λ; aq/b, aq/c, aq/d, λq/e, λq/f; r_{2}, r_{1})) .

This is the Laplace expansion of det H with respect to the first r_{2} rows, multiplied by some products.

These products simplify considerably and we obtain the right hand side of Theorem 6. □

The C_{n} transformation formula we require for our proof is a formula due to Warnaar, Rains, and Coskun and Gustafson, as explained earlier. We use a formulation presented in the second author’s paper with Schlosser [6, Theorem 2], where we use the summation indices k_{0}, k_{1}, …, k_{r_{2}−1}, take r = r_{2} and m = r_{1} + r_{2} − 1, in order to write it in the form

∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1} (q^{∑_{i=0}^{r_{2}−1}(2r_{2}−2i−1)k_{i}} ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, aq^{k_{i}+k_{j}})^{2}

· ∏_{j=0}^{r_{2}−1} \frac{θ(aq^{2k_{j}}) (a, b, c, d, e, f, λaq^{r_{1}+1}/ef, q^{−(r_{1}+r_{2}−1)})_{k_{j}}}{θ(a) (q, aq/b, aq/c, aq/d, aq/e, aq/f, efq^{−r_{1}}/λ, aq^{r_{1}+r_{2}})_{k_{j}}} )

= ∏_{j=1}^{r_{2}} \begin{matrix}(b, c, d, ef/a)_{j−1} & (aq)_{r_{1}+r_{2}−1} & (aq/ef)_{r_{1}} & (λq/e, λq/f)_{r_{1}+r_{2}−j} \\ (λb/a, λc/a, λd/a, ef/λ)_{j−1} & (λq)_{r_{1}+r_{2}−1} & (λq/ef)_{r_{1}} & (aq/e, aq/f)_{r_{1}+r_{2}−j} \end{matrix}

× ∑_{0≤k_{0}<···<k_{r_{2}−1}≤r_{1}+r_{2}−1} (q^{∑_{i=0}^{r_{2}−1}(2r_{2}−2i−1)k_{i}} ∏_{0≤i<j≤r_{2}−1}θ(q^{k_{j}−k_{i}}, λq^{k_{i}+k_{j}})^{2}

· ∏_{j=0}^{r_{2}−1} \frac{θ(λq^{2k_{j}}) (λ, λb/a, λc/a, λd/a, e, f, λaq^{r_{1}+1}/ef, q^{−(r_{1}+r_{2}−1)})_{k_{j}}}{θ(λ) (q, aq/b, aq/c, aq/d, λq/e, λq/f, efq^{−r_{1}}/a, λq^{r_{1}+r_{2}})_{k_{j}}} ),

(4.2)

where λ = a^{2}q^{2−r_{2}}/bcd.

5. How Theorem 6 extends Theorem 1

It is apparent that the transformation formula in Theorem 6 contains two additional parameters than the determinant evaluation in Theorem 1, and is thus formally an extension of Theorem 1. By examining the proofs of the two theorems, this fact is confirmed. In this section, we show how to obtain the determinant evaluation in (2.4) directly from the transformation formula in Theorem 6, by using elementary determinant operations.

We take d = aq/c in Theorem 6. Let G′ = (g′′_{ij}) be the resulting matrix. So

g′′_{ij} = \begin{cases}g_{ij}(a; b, c, aq/c, e, f; r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ g_{i−r_{2},j}(a; aq/b, aq/c, c, aq/e, aq/f; r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

where the g_{ij}’s are as defined in (4.1). Further, let F′ = (f′′_{ij}) be the matrix whose entries are given by

f′′_{ij} = \begin{cases}f_{ij}(b, aq/b, e, f, r_{1}, r_{2}), & for 0 ≤ i ≤ r_{2} − 1, \\ f_{i−r_{2},j}(aq/b, b, aq/e, aq/f, r_{2}, r_{1}), & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

where the f_{ij}’s are as in (2.2).

We first observe that

det G′ = ∏_{i=0}^{r_{1}−1} \begin{matrix}(c, aq/c) & r_{2}+i \\ (c, aq/c) & i \end{matrix} · det F′.  (5.1)

To see this, we multiply the ith row of det G′ by

\begin{cases}(c, aq/c)_{i}, & for 0 ≤ i ≤ r_{2} − 1, \\ (c, aq/c)_{i−r_{2}}, & for r_{2} ≤ i ≤ r_{1} + r_{2} − 1, \end{cases}

and, to compensate, divide det G′ by

∏_{i=0}^{r_{2}−1} (c, aq/c)_{i} ∏_{i=r_{2}}^{r_{1}+r_{2}−1} (c, aq/c)_{i−r_{2}}.

In this manner we get a determinant equivalent to det G′. But in the resulting determinant, column j has the common factor

(c, aq/c)_{j}

that we can take out from each column, to obtain

\begin{matrix}∏_{j=0}^{r_{1}+r_{2}−1} & (c, aq/c)_{j} \\ ∏_{i=0}^{r_{2}−1} & (c, aq/c)_{i} & ∏_{i=0}^{r_{1}−1} & (c, aq/c)_{i} \end{matrix} det F′.

After some cancellation, we obtain (5.1).

Next, we consider det H when d = aq/c. Again, let H′ = (h′′_{ij}), where h′′ is obtained from h′ by replacing d by aq/c. Note that under this substitution, λ = aq^{1−r_{2}}/b and

λb/a = q^{1−r_{2}}.

Thus, in the first r_{2} rows of H′ (i.e., for 0 ≤ i ≤ r_{2} − 1), h′′_{ij} contains the factor (q^{1−r_{2}+i})_{j−i}. But

(q^{1−r_{2}+i})_{j−i} = (q^{−(r_{2}−i−1)})_{j−i} = 0, for j − i > r_{2} − i − 1,

so

h′′_{ij} = 0, for j > r_{2} − 1.

This shows that in the first r_{2} rows of H′, the entries in all the columns after the first r_{2} columns are 0.

Similarly, for the next r_{1} rows we find that due to the presence of the factor

\frac{1}{(q)_{(r_{2}−j)+(i−r_{2})}} = \frac{1}{(q)_{i−j}}

in h′_{ij}, we have

h′′_{ij} = 0, for j > i,

for r_{2} ≤ i ≤ r_{1} + r_{2} − 1. Thus the matrix H′ is of the form

( \begin{matrix}1 & ∗ & ∗ & ∗ & 0 & . . . . . . & 0 \\ 0 & 1 & ∗ & ∗ & 0 & . . . . . . & 0 \\ ⋮ & ⋮ & 1 & ∗ & 0 & . . . . . . & 0 \\ 0 & 0 & … & 1 & 0 & . . . . . . & 0 \\ ∗ & ∗ & ∗ & ∗ & ∗ & 0 & … & 0 \\ 0 & ∗ & ∗ & ∗ & ∗ & ∗ & 0 & 0 \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ∗ & 0 \\ 0 & 0 & … & ∗ & . . . . . . . . . & ∗ \end{matrix} ) .

Here the top-left block is an r_{2}×r_{2} upper-triangular matrix, and the bottom right block is an r_{1} × r_{1} lower-triangular matrix. The determinant of H′ is given by the product of the diagonal entries. For the first r_{2} rows, the diagonal entry is 1. To compute the diagonal entries when r_{2} ≤ i ≤ r_{1} + r_{2} − 1, note that

h′′_{ij} = q^{\binom{j−i+r_{2}}{2}+r_{1}(j−i+r_{2})} \begin{matrix}(q) & r_{2} \\ (q) & i−j \end{matrix}

× \begin{matrix}(aq^{i−r_{2}+1}/b, cq^{i−r_{2}}, aq^{i−r_{2}+1}/c, λq^{i−r_{2}+1}/e, λq^{i−r_{2}+1}/f, λbefq^{−r_{1}+i−1}/a^{2}) & j−i+r_{2} \\ (q, λq^{i+j−r_{2}}, λq^{2i−r_{2}+1}) & j−i+r_{2} \end{matrix} .

An expression for det H′ is given by the product of the diagonal entries

det H′ = ∏_{i=r_{2}}^{r_{1}+r_{2}−1} h′′_{ii}

and this gives, after some simplification,

det H′ = q^{r_{1}^{2}r_{2}+r_{1}\binom{r_{2}}{2}} ∏_{i=0}^{r_{1}−1} \begin{matrix}(c, aq/c) & r_{2}+i \\ (c, aq/c)_{i} & (λq^{r_{1}+r_{2}+i})_{r_{2}} \end{matrix}

× ∏_{i=1}^{r_{1}} (aq^{−r_{2}+i+1}/be, aq^{−r_{2}+i+1}/bf, efq^{−r_{1}+i−1}/a)_{r_{2}},

where λ = aq^{1−r_{2}}/b. From here, we see that, when d = aq/c, the right hand side of Theorem 6 reduces to

(\frac{a}{λ})^{r_{1}r_{2}} ∏_{j=1}^{r_{1}} \begin{matrix}(λq^{r_{1}+r_{2}+j−1}) & r_{2} \\ (aq^{r_{1}+r_{2}+j−1}) & r_{2} \end{matrix} det H′ = b^{r_{1}r_{2}}q^{r_{1}r_{2}(r_{1}+r_{2}−1)+r_{1}\binom{r_{2}}{2}}

× ∏_{i=0}^{r_{1}−1} \begin{matrix}(c, aq/c) & r_{2}+i \\ (c, aq/c) & i \end{matrix}

× ∏_{i=1}^{r_{1}} \begin{matrix}(aq^{−r_{2}+i+1}/be, aq^{−r_{2}+i+1}/bf, efq^{−r_{1}+i−1}/a) & r_{2} \\ (aq^{r_{1}+r_{2}+i−1}) & r_{2} \end{matrix} . (5.2)

Now comparing (5.1) and (5.2), and simultaneously replacing

(a, b, e, f) by (s_{1}s_{2}/q, s_{1}, t_{1}, t_{2}),

we obtain Theorem 1 in the form (2.4).

References

[1] H. Coskun and R. A. Gustafson. Well-poised Macdonald functions W_{λ} and Jackson coefficients ω_{λ} on BC_{n}. In Jack, Hall–Littlewood and Macdonald polynomials, vol. 417 of Contemp. Math., pp. 127–155. Amer. Math. Soc., Providence, RI, 2006.

[2] H. Feng, C. Krattenthaler, and Y. Xu. Best polynomial approximation on the triangle. preprint,

16 pp.; arχiv:1802.0599.

[3] I. B. Frenkel and V. G. Turaev. Elliptic solutions of the Yang–Baxter equation and modular hypergeometric functions. In The Arnold–Gelfand mathematical seminars, pp. 171–204. Birkhäuser Boston, Boston, MA, 1997.

[4] G. Gasper and M. Rahman. Basic Hypergeometric Series, vol. 96 of Encyclopedia of Mathematics and its Applications. Cambridge University Press, Cambridge, second edition, 2004. With a foreword by Richard Askey.

[5] C. Krattenthaler. Advanced determinant calculus. Sém. Lothar. Combin., 42:Art. B42q, 67 pp., 1999. The Andrews Festschrift (Maratea, 1998).

[6] C. Krattenthaler and M. J. Schlosser. The major index generating function of standard Young tableaux of shapes of the form “staircase minus rectangle”. In Ramanujan 125, vol. 627 of Contemp. Math., pp. 111–122. Amer. Math. Soc., Providence, RI, 2014.

[7] R. Loos. Computing in algebraic extensions. In B. Buchberger, G. E. Collins, R. Loos, and R. Albrecht, editors, Computer algebra, pp. 173–187. Springer, Vienna, second edition, 1983. Symbolic and algebraic computation.

[8] E. M. Rains. BC_{n}-symmetric Abelian functions. Duke Math. J., 135(1):99–180, 2006.

[9] H. Rosengren. A proof of a multivariable elliptic summation formula conjectured by Warnaar. In q-Series with applications to combinatorics, number theory, and physics (Urbana, IL, 2000), vol. 291 of Contemp. Math., pp. 193–202. Amer. Math. Soc., Providence, RI, 2001.

[10] H. Rosengren. Elliptic hypergeometric functions. In Lectures at OPSF-S6, College Park, Maryland, July 2016, 64 pp. 2016.

[11] M. Schlosser. Elliptic enumeration of nonintersecting lattice paths. J. Combin. Theory Ser. A, 114(3):505–521, 2007.

[12] S. O. Warnaar. Summation and transformation formulas for elliptic hypergeometric series. Constr. Approx., 18(4):479–502, 2002.

Fakultät für Mathematik, Universität Wien, Oskar-Morgenstern-Platz 1, A-1090 Vienna, Austria.

E-mail address: bhatnagarg@gmail.com

Fakultät für Mathematik, Universität Wien, Oskar-Morgenstern-Platz 1, A-1090 Vienna, Austria.

URL: http://www.mat.univie.ac.at/~kratt