audio: Support loading external MP3s

[Herschel]

• Add AudioBackend::register_mp3 to register MP3 data
• Add SoundLoader variant
• avm1: Implement Sound.load, Sound.getBytesLoaded, Sound.getBytesTotal

TODO:

• Currently minimp3-rs provides no way to seek thru MP3 frames without decoding, so we end up decoding the entire MP3 on desktop to discover its length.
  • This might be a good time to switch over to symphonia for our default MP3 decoder, as it allows for seeking thru frames without decoding audio. (You still have to step thru all of the MP3 frames, but this is the nature of MP3).
• Figure out what to do on web. web: Decode and mix audio via Rust #4273 switched to decoding audio via Rust, so the web target should operate the same as desktop.

[Toad06]

  * We could use `HtmlAudioElement` to stream and decode MP3s.

FWIW I think this approach could be problematic on Safari (at least on iOS and iPadOS) because unlike all other browsers, each new <audio> node (or new Audio() call) requires a user gesture before playing the sound.

[Herschel]

  * We could use `HtmlAudioElement` to stream and decode MP3s.

FWIW I think this approach could be problematic on Safari (at least on iOS and iPadOS) because unlike all other browsers, each new <audio> node (or new Audio() call) requires a user gesture before playing the sound.

Do you know if this includes playing the audio with a MediaElementAudioSourceNode attached to an audio context?

[Toad06]

Using MediaElementAudioSourceNode doesn't seem to change anything unfortunately. :(

In case this could be useful, I'm attaching the tests I've made:

• index1.html: Plays a sound after a user gesture, then tries to create new Audio instances playing the same sound. Doesn't work in Safari.
• index2.html: Plays a sound after a user gesture, then re-use the Audio instance to play another sound, changing its src. Works in Safari.
• index3.html: Similar to the first test, but uses MediaElementAudioSourceNode. Also doesn't work in Safari.

[torokati44]

I'd just like to bring up again one of the problems Ruffle faces on web even with embedded MP3s: the decodeAudioData function has no way of telling the browser what kind of content is supposed to be in there, so it has to find it out somehow by trying to find different kinds of headers, or resorting to heuristics.
This already causes some content to be mute in Firefox, but have sound in Chrome.
So, my point is, that at least from this perspective, if at all possible, having a method of decoding that provides a way to also supply a format or content type might be better.

[Herschel]

<audio>/MediaElementAudioSourceNode is nice in that respect because it allows specifying a MIME type, but it still puts us at the mercy of browsers (such as the Safari autoplay behavior).

[sombraguerrero]

As requested, examples of Onda's Animecuentos -> Toto activities to further examine the sound.onLoad issues.
@Herschel
toto.zip

[n0samu]

With this PR, the external MP3 sound effects in the Flexitwins app work perfectly, but the background music (techno.mp3) that's supposed to play on the game instructions screen doesn't play. I found that this is because the offset and loop parameters of Sound.start are used when playing this MP3, whereas Sound.start is called without parameters when playing the other external MP3s.

[Herschel]

This is unblocked now that #4273 is merged and could use some testing!

@n0samu I tried the linked Flexitwins app and seems like it's good now, I hear the music on the instructions screen. Could you verify?

[Toad06]

This looks good to me, just a few observations/questions:

• When loadSound is called, some slowdown can be observed. You can check this on https://www.argentine-music.com/ for instance (happens on the first frame and also - though less noticeable - when clicking any Play button in "Catalogue musical" -> any snapshot).

• MP3s on https://www.argentine-music.com/ don't play as they rely on the isStreaming argument (set to true) to autoplay the file.

• Could this check in fn load_sound: if activation.swf_version() >= 6 be moved to declare_properties (like avm1: Implement property version checking #6856 did)?

[n0samu]

I tried the linked Flexitwins app and seems like it's good now, I hear the music on the instructions screen. Could you verify?

Yes, with the latest updates the extension is working perfectly now! It doesn't seem to work in the desktop app yet though.

[n0samu]

Did some more testing:

• In Eeekoworld, the external MP3s play fine! But when mousing over an object that plays an MP3, it doesn't first stop any MP3s that are already playing, like it does in Flash Player. Sounds start layering over each other if you mouse over more than one thing in a short timespan.
• The intro of Binky's Facts and Opinions works! But for some reason, the lip-sync between the character's mouth and the audio is wrong and the character's mouth stops moving long before the audio ends. (The rest of the game won't work reliably on Wayback because of missing assets, but you can find the full game in Flashpoint if you want to test further.)
• The MP3 voicelines in Morningstar work perfectly now! (The intro uses an external FLV, so it needs to be skipped by clicking the button in the bottom-right corner.)

[Herschel]

Thank you! Those cases should be fixed, although possibly the layering sounds has some edge cases still (controlled by the isStreaming flag of Sound.loadSound -- if this is true, the Sound object can only seem to play a single instance of the sound.)

[Toad06]

Tested some good old MP3 players and they all worked great. 👍

https://www.argentine-music.com/ still has issues and seems to be one of these edge cases though. The main SWF loads a music when the file is ready to play, then the user can load another music from an inner SWF by clicking on a button. In any case, the loadSound method is called with the isStreaming argument to true. For some reason, this can cause different issues, depending on the user's interactions. I wrote down my observations as steps to reproduce below:

Steps to reproduce the issues on argentine-music.com

1- When the file is ready to play, a "main music" is loaded and then played. Click the note in the bottom left corner of the screen: this will stop the music. Click another time to play it again. Everything works perfectly here.

2- Click "Catalogue musical" on the left. Click on some pictures (randomly). Notice that the main music keeps playing. Again, this matches Flash.

3- Click the picture with a clown ("Drôle de jeu"). On the right, click the Play button on the same line as "Générique". Notice that the main music fades out (which is correct) but the new music also fades out, so you can just hear half of a second of it. Wait about 18 seconds, so that the music ends (the Play button changes back to its original aspect) and the main music fades in (which is correct).

4- Click on some pictures again. Notice that the main music unexpectedly stops.

5- Click the picture with a clown. Click the note in the bottom left corner twice (to play the main music again) and click the same Play button as before. While the sound plays, click the note twice again. Wait about 18 seconds and notice that the main music won't fade in this time. Click the Stop button.

6- Click the note twice and the same Play button. Click one of the category at the bottom ("Divertissement", "Fiction", "Documentaire" or "Divers"). Notice that both of the sounds fade in.

[Herschel]

Thanks for testing! It seems like the behavior with loadSound/isStreaming is a little bit different -- usually setVolume affects the volume of all sounds played within the same clip (the clip you pass to new Sound()). But in the case of dynamically loaded sounds, it only affects that specific instance. AVM1 sound is strange!

So in argentine-music, there are separate new Sound() instances. Even though they are separate objects, calling setVolume on any of them would normally affect the global sound volume for every sound, and this is what Ruffle does currently. But in FP, after loadSound is called, setVolume only affects that specific loaded sound.

I'm thinking we'll need something like enum SoundType { Embedded; External; ExternalStreaming; } in SoundObject to differentiate between the behavior. I might defer to this to a later PR.