Changeset 78 for pa_proposals/trunk

Show
Ignore:
Timestamp:
01/06/04 14:16:16 (5 years ago)
Author:
rossbencina
Message:

revised proposal to require paFramesPerBufferUnspecified with
paNeverDropInput, and also to loosen the definition of whrere
discontinuities indicated by underflow/overflow flags were located

in the buffer if paFramesPerBufferUnspecified was not used.

Files:
1 modified

Legend:

Unmodified
Added
Removed

  • pa_proposals/trunk/001-UnderflowOverflowHandling.html

    r55 r78  
    1818<P><A href="index.html">Enhancement Proposals Index</A>, 
    1919<A href="http://www.portaudio.com/">PortAudio Home Page</A></P> 
    20 <P>Updated: October 26, 2002 </P> 
     20<P>Updated: January 7, 2004 </P> 
    2121 
    2222<H4>Status</H4> 
    … …  
    2626</P> 
    2727 
     28<H4>Dependencies</H4> 
     29 
     30<P> 
     31This proposal depends on <A HREF="004-CallbackVariableFramesPerBuffer.html">004 - Allow Callbacks to Accept Variable Number of Frames per Buffer</A> because the paNeverDropInput flag proposed here requires callbacks to vary the number of frames per buffer. 
     32</P> 
     33 
    2834<H4>Background</H4> 
    2935 
    3036<P> 
    31 There are conditions where a full-duplex stream needs to generate output but doesn't have any input available, or where it has too much input so some input needs to be discarded (not passed to the output.) There is also the case where output is needed, but the callback has (transiently) consumed so much CPU time that output has to be generated without the callback being called. 
    32 </P> 
    33  
    34 <P>Currently (V17) the Stream Callback Function handles these underflow/overflow conditions by passing NULL buffer pointers to the callback. This can happen if the output is pre-rolled and there is not yet any input data. It can also happen if the input underflows.  
    35 </P> 
    36  
    37 <P> 
    38 A number of concerns have been raised about the current system: For PortAudio to discard input just because it is not needed for output is considered unacceptable for some recording applications. Passing of NULL buffer pointers has been deemed to be too error prone and requires too much housekeeping for simple programs. 
    39 </P> 
    40  
    41 <P> 
    42 See <A HREF="http://techweb.rfa.org/pipermail/portaudio/2001-October/000222.html">http://techweb.rfa.org/pipermail/portaudio/2001-October/000222.html</A> and subsequent messages in various threads. 
    43 </P> 
    44  
    45 <P> 
    46 This proposal seeks to make implementing the Stream Callback as simple as possible for ease-of-use while providing full access to overflow/underflow information, and all input and output sample data when clients require it. 
    47 </P> 
     37Once a PortAudio stream is running, every input and/or output sample should be passed between the hardware and the Stream Callback. Unfortunately in the real-world contexts in which PortAudio operates it is not always possible to achieve this goal. Sometimes buffers will underflow (no data is available when needed) or overflow (more data is available than can be processed). These underflows and overflows are often the result of the failure of some subsystem to keep up with real-time deadlines. Sometimes this is caused by client code, for example if the Stream Callback consumes too much CPU time before returning. At other times it is the result of system level interactions between the scheduling of interrupt handlers, processes, timers or threads. PortAudio introduces another potential source of underflow and overflow: when combining separate input and output devices into a full duplex device it must account for any skew or drift between these devices. It does this by inserting silence in the case of an underflow, or discarding samples in case of an overflow. Note that this skew correction is a boundary case which should not arise with true full-duplex hardware, however it must be dealt with as a possibility when fusing independent input and output devices into a full duplex stream. 
     38</P> 
     39 
     40<P> 
     41From the previous description it may be observed that there are cases where PortAudio is at-best only able to detect that an underflow or overflow has occured, and there are other cases where PortAudio is in full control of the underflow or overflow. In the former case samples may be dropped by a separate subsystem (the driver for example) and hence are impossible to recover. In the latter case excess (overflowed) input samples may be received by PortAudio but not passed to the Stream Callback, and an input underflow may cause PortAudio to send additional zero (silent) samples to the callback -- equivalent operations may occur with respect to the callback's output. 
     42</P> 
     43 
     44<p> 
     45Currently (V18) handles underflow conditions by passing NULL buffer pointers to the callback. This can happen if the output is pre-rolled and there is not yet any input data. It can also happen if the input underflows in a full-duplex stream. However aside from observing that this sometimes happens, the functionality was never designed, well documented or uniformly implemented.  
     46</p> 
     47<p> 
     48Below are listed the three main shortcomings in the way V18 handles underflow and overflow conditions: 
     49</p> 
     50 
     51<ol> 
     52<li> 
     53Clients should be able to depend on PortAudio to implement a policy to manage underrun and overrun conditions without requiring the client to write any underrun or overrun related code. Passing NULL buffer pointers when an underrun occurs requires even simple clients to check for a valid buffer pointer. This is an error prone approach.  
     54</li> 
     55<li> 
     56In most cases there is no way for clients who are concerned with underflows and overflows to determine when they have occurred. The only condition which is currently signalled are underflows where PortAudio needs to call the callback and has no data (in which case it may pass a NULL buffer pointer). 
     57</li> 
     58<li> 
     59In the case of PortAudio managed underrun and overrun (such as full-duplex skew correction) there is no consistent, clearly stated policy on how PortAudio is to discard overflowed data or zero-pad when underflow occurs. An "ouput driven" full-duplex model has been proposed where the quality of the output stream is maintained and input is discarded or padded to keep input in sync with output. Similarly an "input driven" model is possible. Although the output-driven model is favoured for many applications, it is unacceptable for PortAudio to discard and/or silently zero-pad input data to assure the quality of output when used in applications primarily concerned with audio capture.  
     60</li> 
     61</ol> 
     62 
     63<P> 
     64This proposal seeks to address the above issues by providing full access to detected underflow and overflow information, and by giving the client access to all available input and output sample data when required. At the same time this proposal seeks to make implementing the Stream Callback as simple as possible to ensure ease-of-use. 
     65</P> 
     66 
     67<P> 
     68See <A HREF="http://techweb.rfa.org/pipermail/portaudio/2001-October/000222.html">http://techweb.rfa.org/pipermail/portaudio/2001-October/000222.html</A> and subsequent messages in various threads. This proposal was reopened in order to fix a problem described here: 
     69<A HREF="http://techweb.rfa.org/pipermail/portaudio/2003-December/002926.html">http://techweb.rfa.org/pipermail/portaudio/2003-December/002926.html</A> 
     70and discussed extensively in that thread including: 
     71<A HREF="http://techweb.rfa.org/pipermail/portaudio/2004-January/002946.html">http://techweb.rfa.org/pipermail/portaudio/2004-January/002946.html</A> which provides an  explanation of the issues surrounding the proposed paNeverDropInput flag.</P> 
    4872 
    4973<H4>Proposal</H4> 
    5074 
    5175<P> 
    52 For streams providing input, the inputBuffer parameter will always point to a valid memory location containing framesPerBuffer frames of sample data in the requested format. The inputBuffer parameter will be NULL for output only streams. Similarly, the outputBuffer parameter will be NULL for input only streams, otherwise it will point to a valid memory location containing framesPerBuffer frames of sample data. 
    53 </P> 
    54  
    55 <P> 
    56 A new parameter and corresponding type will be added to the Stream Callback Function that gives the status of the data as bit flags.  
     76For streams providing input, the inputBuffer Stream Callback Function parameter will always point to a valid memory location containing framesPerBuffer frames of sample data in the requested format. The inputBuffer parameter will be NULL for output only streams. Similarly, the outputBuffer parameter will be NULL for input only streams, otherwise it will always point to a valid memory location with sufficient space for framesPerBuffer frames of sample data. 
     77</P> 
     78 
     79<P> 
     80A new parameter and corresponding type will be added to the Stream Callback Function that indicates the status of the callback data as a set of bit flags:  
    5781</P> 
    5882 
    … …  
    7195 
    7296<PRE> 
    73 #define paInputUnderflow&nbsp;&nbsp; ((PaStreamCallbackFlags)0x01) /* Input data is all zeros because no real data is available. */  
    74 #define paInputOverflow    ((PaStreamCallbackFlags)0x02) /* Input data was discarded by PortAudio */ 
    75 #define paOutputUnderflow  ((PaStreamCallbackFlags)0x04) /* Output data was inserted by PortAudio because the callback is using too much CPU */ 
    76 #define paOutputOverflow   ((PaStreamCallbackFlags)0x08) /* Output data will be discarded because no room is available. */</PRE> 
     97#define paInputUnderflow&nbsp;&nbsp; ((PaStreamCallbackFlags)0x01) 
     98#define paInputOverflow    ((PaStreamCallbackFlags)0x02) 
     99#define paOutputUnderflow  ((PaStreamCallbackFlags)0x04) 
     100#define paOutputOverflow   ((PaStreamCallbackFlags)0x08)  
     101</PRE> 
    77102 
    78103<P>New rules will govern when the Stream Callback is called:</P> 
    79104 
    80 <UL> 
    81 <LI>For input-only streams, the callback will be called for every available input buffer. If the callback takes too long to complete and input samples have to be discarded the paInputOverflow flag will be set the next time the callback is called. An input-only stream will never be called with the paInputUnderflow flag set. </LI> 
    82 <LI>For output-only streams, the callback will be called whenever an output buffer needs to be filled, except when doing so would cause the stream to fall further behind real-time due to the CPU load being too high. In such cases PortAudio will insert silence or repeat the previous buffer to the output and the paOutputUnderflow bit will be set next time the callback is called. An output-only stream will never be called with the paOutputOverflow flag set. </LI> 
    83 <LI>By default, a full duplex stream will behave according to the same rules as an output-only stream. If input is not available, the paInputUnderflow bit will be set and the input buffers will contain zeros. In the case of an input buffer overflow, PortAudio will discard input - in such cases the input samples will <B>not</B> be passed to a callback, and paInputOverflow will be set next time the callback is called to generate more output. A full-duplex stream in default mode will never be called with the paOutputOverflow flag set. </LI> 
    84 <LI>A new mode flag to Pa_OpenStream(), <I>paNeverDropInput</I> specifies that a full duplex stream will not discard overflowed input samples without calling the callback. When an input buffer overflow occurs, the callback will be passed an input buffer containing valid data and a valid output buffer pointer, the paOutputOverflow flag will be set, when the callback completes the output buffer will be discarded.</LI></UL> 
    85  
    86 <P> 
    87 Note that the default full-duplex mode is intended to cover most common uses of PortAudio, where the client wants a simple audio streaming interface, and is happy to let PortAudio handle buffer underflow/overflow conditions when they occur. 
    88 </P> 
     105 
     106<p> 
     107<table width=80% cellpadding=2 cellspacing=1> 
     108<tr> 
     109<td bgcolor=#000099 colspan=2 align=center> 
     110<font color=#ffffff size=+2>Input-only streams</font> 
     111</td> 
     112</tr> 
     113<tr> 
     114<td bgcolor=#eeeeee width=50%><b>Input underflow:</b></td> 
     115<td bgcolor=#eeeeee width=50%><b>Input overflow:</b></td> 
     116</tr> 
     117<tr> 
     118<td>Never happens</td> 
     119<td>If callback is slow, input data is discarded.</td> 
     120</tr> 
     121</table> 
     122</p> 
     123 
     124<P> 
     125For input-only streams, the callback will be called once for every 
     126available input buffer, except that in cases where deadlines are missed an 
     127input overflow may occur. This overflow may be indicated to PortAudio by the 
     128host API, or may result from a PortAudio implementation choosing to discard 
     129samples. In either case one or more input samples will be discarded and not 
     130passed to the callback. Whenever an input overflow is detected by PortAudio 
     131the paInputOverflow flag will be set in the callback containing the first 
     132input sample following the overflow. Note that unless the stream was opened 
     133with the paFramesPerBufferUnspecified flag, the first input sample following 
     134the overflow may not be the first sample in the callback buffer. An 
     135input-only stream will never be called with the paInputUnderflow flag set,  
     136nor will it be called with the paOutputUnderflow or paOutputOverflow flags set. 
     137</P> 
     138<BR> 
     139 
     140<p> 
     141<table width=80% cellpadding=2 cellspacing=1> 
     142<tr> 
     143<td bgcolor=#000099 colspan=2 align=center> 
     144<font color=#ffffff size=+2>Output-only streams</font> 
     145</td> 
     146</tr> 
     147<tr> 
     148<td bgcolor=#eeeeee width=50%><b>Output underflow:</b></td> 
     149<td bgcolor=#eeeeee width=50%><b>Output overflow:</b></td> 
     150</tr> 
     151<tr> 
     152<td>If callback is slow, output to DAC is padded</td> 
     153<td>Never happens</td> 
     154</tr> 
     155</table> 
     156</p> 
     157 
     158<P> 
     159For output-only streams, the callback will be called whenever an output 
     160buffer needs to be filled, except when doing so would cause the stream to 
     161underflow its available buffer resources (for example due to the CPU load 
     162being too high).  In such cases PortAudio may employ an efficient technique 
     163such as inserting silence or repeating previous buffers to recover from the 
     164underflow. An underflow may also arise between the host API and PortAudio in 
     165which case PortAudio may be able to detect the underflow. In either case one 
     166or more spurious output samples (or gaps) will have been sent to the DACs. 
     167Whenever an output underflow is detected by PortAudio, the paOutputUnderflow 
     168flag will be set in the callback containing the first output sample 
     169following the underflow. Note that unless the stream was opened with the 
     170paFramesPerBufferUnspecified flag, the first output sample following the 
     171underflow may not be the first sample in the callback buffer. An output-only 
     172stream will never be called with the paOutputOverflow flag set, nor will it 
     173be called with the paInputUnderflow or paInputOverflow flags set. 
     174</P> 
     175<BR> 
     176 
     177<p> 
     178<table width=80% cellpadding=2 cellspacing=1> 
     179<tr> 
     180<td bgcolor=#000099 colspan=2 align=center> 
     181<font color=#ffffff size=+2>Full-duplex</font><br> 
     182<font color=#ffffff>without paNeverDropInput (default mode)</font> 
     183</td> 
     184</tr> 
     185 
     186<tr> 
     187<td bgcolor=#eeeeee width=50%><b>Input underflow:</b></td> 
     188<td bgcolor=#eeeeee width=50%><b>Input overflow:</b></td> 
     189</tr> 
     190<tr> 
     191<td>If output would underflow if the callback wasn't  
     192  called, zeroed input is passed to callback.</td> 
     193<td>If callback is slow or output queue is  
     194  already full, input data is discarded.</td> 
     195</tr> 
     196 
     197<tr> 
     198<td bgcolor=#eeeeee width=50%><b>Output underflow:</b></td> 
     199<td bgcolor=#eeeeee width=50%><b>Output overflow:</b></td> 
     200</tr> 
     201<tr> 
     202<td>If callback is slow, output to DAC is padded.</td> 
     203<td>Never happens</td> 
     204</tr> 
     205</table> 
     206</p> 
     207 
     208<P> 
     209By default, a full duplex stream will have output behaviour the same as an 
     210output-only stream, and will have input behaviour similar to an input-only 
     211stream with the following exceptions and additions: If sufficient input is 
     212not available, the paInputUnderflow bit will be set and one or more ranges 
     213of samples in the input buffer will contain silence. If the stream was 
     214opened with the paFramesPerBufferUnspecified flag, the paInputUnderflow flag 
     215indicates that the whole input buffer contains silence, otherwise it 
     216indicates that some portions of the input buffer contain silence. In the 
     217case of an input overflow, PortAudio will discard input as for a half-duplex 
     218input stream and the paInputOverflow flag will be set in the callback 
     219containing the first input sample following the overflow. Note that unless 
     220the stream was opened with the paFramesPerBufferUnspecified flag, the first 
     221input sample following the overflow may not be the first sample in the 
     222callback buffer. A full-duplex stream in default mode will never be called 
     223with the paOutputOverflow flag set. 
     224</P> 
     225<BR> 
     226 
     227<p> 
     228<table width=80% cellpadding=2 cellspacing=1> 
     229<tr> 
     230<td bgcolor=#000099 colspan=2 align=center> 
     231<font color=#ffffff size=+2>Full-duplex</font><br> 
     232<font color=#ffffff>with paNeverDropInput</font> 
     233</td> 
     234</tr> 
     235 
     236<tr> 
     237<td bgcolor=#eeeeee width=50%><b>Input underflow:</b></td> 
     238<td bgcolor=#eeeeee width=50%><b>Input overflow:</b></td> 
     239</tr> 
     240<tr> 
     241<td>If output would underflow if the callback wasn't  
     242  called, zeroed input is passed to callback.</td> 
     243<td>If callback is slow, input data is discarded. 
     244    (If output queue is already full, callback is called 
     245         with input only so that no input data is lost.)</td> 
     246</tr> 
     247 
     248<tr> 
     249<td bgcolor=#eeeeee width=50%><b>Output underflow:</b></td> 
     250<td bgcolor=#eeeeee width=50%><b>Output overflow:</b></td> 
     251</tr> 
     252<tr> 
     253<td>If callback is slow, output to DAC is padded.</td> 
     254<td>If input would otherwise overflow if the  
     255    callback wasn't called, output is discarded.</td> 
     256</tr> 
     257</table> 
     258</p> 
     259 
     260<P> 
     261A new flag for the Pa_OpenStream() streamFlags parameter is introduced: 
     262paNeverDropInput. This flag may be used to request that a full 
     263duplex stream will make all possible efforts to pass "potentially 
     264overflowing" input samples to the callback. This flag may only be used for  
     265full duplex streams, and only in 
     266combination with paFramesPerBufferUnspecified, otherwise Pa_OpenStream() 
     267will return an error (paInvalidFlag). The semantics for a full duplex 
     268stream with the paNeverDropInput flag differ from the default full-duplex 
     269stream mode in the following way: When a potential input buffer overflow 
     270condition occurs, such that PortAudio has access to overflowed input data, 
     271the Stream Callback will be passed this input data in the input buffer, a valid 
     272output buffer pointer will be provided, and the paOutputOverflow flag 
     273will be set. When the callback completes, any data written to the output 
     274buffer will be discarded. Input buffer overflows where PortAudio does not 
     275have access to the overflowed data are indicated to the callback using the 
     276paInputOverflow flag as for input-only and default full-duplex streams. 
     277</P> 
     278<BR> 
     279 
     280 
     281<P> 
     282As noted earlier, underflow and overflow conditions give rise to discontinuities  
     283where samples have or will be inserted into or removed from the sample stream. 
     284These discontinuities are indicated to the client by the four underflow and overflow flags defined earlier. 
     285When a stream is opened with the paFramesPerBufferUnspecified flag such 
     286discontinuities are guarenteed to fall on the boundaries between the last sample 
     287of one callback and the first sample of the next. No such guarantee is  
     288made if paFramesPerBufferUnspecified is not used (ie a fixed, client specified 
     289callback buffer size is requested). In this case the flags indicate that an  
     290underflow or overflow condition has occurred (and hence a discontinuity is present) 
     291adjacent to <i>one or more</i> samples within the current callback buffers. 
     292This imprecision in identifying the location of underflows and overflows  
     293is a side effect of potential adaption between fixed callback and host  
     294API buffer sizes.  
     295</P> 
     296 
     297<P> 
     298It is important to note two important cases when paFramesPerBufferUnspecified  
     299is used with a full-duplex stream: 
     300(1) When the input underflows, the Stream Callback will be called with the paInputUnderflow flag set and 
     301the entire input buffer will contain silence (zeros);  
     302(2) in paNeverDropInput mode, an excess of input data will force an output overflow. The excess input will be passed to the callback, and the paOutputOverflow flag will be set. The entire input buffer will contain only the excess samples and the entire output buffer will be discarded when the callback returns. 
     303</P> 
     304 
     305<H4>Discussion</H4> 
     306 
     307<P> The original definition of this proposal ommitted consideration of any underflow and overflow conditions which did not directly arise from the PortAudio implementation. It said nothing of detected low-level underflow and overflow conditions, or of the difference between an overflow where the data is discarded by PortAudio, and an overflow where the overflowed data is discarded by an inaccessible subsystem and is hence unrecoverable. </P> 
     308 
     309<P> 
     310paNeverDropInput was only considered after it became clear that it was necessary to specify sematics for handling input/output skew correction and other conditions which might call for PortAudio to discard samples. paNeverDropInput resulted from some clients being more interested in preserving input coherence, while others were concerned primarily with output coherence. The choice to favour output in the default case arose primarily because of the bias of the designers. It can also be justified on the grounds that (1) full duplex audio processing clients are more common than full duplex clients favouring high-quality capture over output quality, (2) that calling a full-duplex callback only to throw away the output may result in a CPU load spike, and (3) to a full duplex effects processing application there is not a lot of difference between a corrupted input or output if input/output skew must be compensated somehow. 
     311</P> 
     312 
     313<P> 
     314The paFramesPerBufferUnspecified flag was defined elsewhere for other reasons. 
     315The requirement of paFramesPerBufferUnspecified for paNeverDropInput was only decided at a very late stage once implementation problems arose with combining this proposal with the buffer adapter (pa_process.c) used in the V19 implementations. The buffer adapter meant that it was not possible to place discontinuities on buffer boundaries in some instances, so the position of discontinuities was relaxed, except when using paFramesPerBufferUnspecified. Dominic Mazzoni noted that  
     316"when the 
     317application is just trying to capture audio to disk (a primary goal of 
     318Audacity and also probably some other apps), the buffer size doesn't 
     319matter (as long as it's less than some maximum) but avoiding dropped 
     320samples does." 
     321</P> 
     322 
     323<P> 
     324Note that some of the language used to describe the paNeverDropInput case is biased towards a model that assumes the output device is the master. For example "excess input data" could equally be phrased "lack of need for output data". 
     325</P> 
     326 
     327<P> 
     328At one stage it was suggested that excess input data could be supplied by providing a "partially full" input or output buffer, however this would have required separate frame count parameters for the input and output buffers -- a situation which is not satisfactory because one of the main benefits of full duplex PortAudio is it manages the issues of keeping input and output buffers in sync for the client. 
     329</P> 
     330 
     331<P> 
     332Note that both full-duplex modes fulfill the guarantee that both the input and output buffers are always valid. This has the benefit of offering minimal surprise to clients -- clients who ignore the paInputUnderflow flag will not crash if 
     333they try to read the buffer, they will just get a bit of silence in 
     334their stream.  Simple clients may use either mode without implementing special handling for underflow or overflow conditions. Each mode has its own benefits, the default favours output quality, paNeverDropInput offers improved capture quality and supports equivalent output quality if the client is careful to not generate output samples when the paOutputOverflow flag is set. 
     335</P> 
     336 
    89337 
    90338<H4>Impact Analysis</H4> 
    … …  
    96344</BODY> 
    97345</HTML> 
     346