audio

Comparison of two playback mechanisms: HTML5 audio element and web audio.

HTML 5 audio

pros:

• buffers automatically and does a reasonable job
• can easily set start position
• built-in playback rate control

cons:

• some glitching when setting currentTime (used to point the player at the start of a clip)
• have to use timeout/setinterval to monitor for end of file

If we allow for +/- 11ms when setting the currentTime, then glitching seems to go away.

If we encounter issues with loading files, we can create an asset manager to pre-load audio assets. In this case, it would be helpful to list all the audio assets related to SMIL when the book loads; of course, this can't be done without analyzing all the SMIL files, which could get expensive. One compromise might be to have the MO component, which deals with one file at a time, pre-load audio assets of a single file.

The remaining issue is that if the tab moves to the background, audio playback is affected because the interval is checked only every second. What ends up happening is that a clip overplays and is reset; this is especially noticeable with short clips. In this case, when the tab comes back into the foreground, the system should perform an integrity check that will reset the smil tree position to the correct point. While this is a bit of an expensive search operation, its use is minimized in this case to when the user is switching tabs.

Web audio

pros:

• no glitching
• can specify clip duration up front
• can accept audio filters, for example to scale the playback rate without affecting the pitch

cons:

• have to use timeout/setinterval to monitor the status of an audio clip
• have to manage buffer manually, and it can be slow to load

future pros:

• need for timeout/setinterval monitoring will go away in the future (see this bug)

If we were to use web audio, we would have to progressively buffer windows of data (for example, 1MB at a time). We might not need the full power of web audio.