Date: Wed, 6 Mar 1996 11:30:21 -0800 (PST)
From: Mark Crispin <MRC@Panda.COM>
Sender: Mark Crispin <mrc@Ikkoku-Kan.Panda.COM>
Subject: Re: re PIPELINING 
To: Keith Moore <moore@cs.utk.edu>
Cc: drums@cs.utk.edu
In-Reply-To: <199603060903.EAA03504@wilma.cs.utk.edu>
Message-Id: <MailManager.826140621.7721.mrc@Ikkoku-Kan.Panda.COM>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII

On Wed, 06 Mar 1996 04:03:26 -0500, Keith Moore wrote:
> My experience contradicts this.  We run a mailing list with
> 5000 recipients.  It used to send each message in a separate envelope
> and took an average of eight hours to attempt delivery to each
> recipient.  I rewrote it to put all the recipients in the same
> envelope and it took twenty-four hours or more -- the time was
> highly variable due to the variance in the number of unreachable
> hosts for which the connection had to time out before the MTA
> would go to the next recipient.  I then rewrote the code again
> to parallize delivery, but to put up to twenty different domains
> in an envelope.  Now it takes an average of two hours to attempt
> delivery to each recipient.

Note that this behavior comes about because of sendmail's queue implementation
strategy; it delays the easy cases because of delays that it encounters in the
hard cases.

What Bernstein claims is a protocol issue is really a queue management issue;
Bernstein's algorithm is intended to improve real-time latency with sendmail's
type of queueing strategy.  What Bernstein does not understand is that that
algorithm can, with different queue management strategies, make matters
considerably worse.

Nor does it scale; the benefits observed by Bernstein are real but in small
scale, limited cases.  The issue here is scheduling/fairness.  Bernstein's
algorithm is, in effect, a way for a particular message to monopolize the
parallelism provided in simple sendmail-style queueing strategies.  This is
fine if there are no other contenders for that parallelism.

A related example has to do with TCP retransmit times on lossy networks with
high latency.  It benefits a single TCP sending application to set a low
ceiling on the retransmit timer, and in other ways to retransmit aggressively.
This behavior guarantees that a necessary retransmission happens as quicly as
possible, and real-time performance is unquestionably improved.  However,
because of the high latency, you end up with a lot of spurious retransmission
which ultimately degrades the overall infrastructure performance.

I recommend that we make the observation that using a single-threaded FIFO
queue in a mailer leads to performance problems, and move on.