Mixing in feature space

[desh2608]

When mixing audios at the feature level, each feature type defines a mix() function. For the TorchaudioFbank feature, this is defined here and is given as:

    def mix(
        features_a: np.ndarray, features_b: np.ndarray, energy_scaling_factor_b: float
    ) -> np.ndarray:
        return np.log(
            np.maximum(
                # protection against log(0); max with EPSILON is adequate since these are energies (always >= 0)
                EPSILON,
                np.exp(features_a) + energy_scaling_factor_b * np.exp(features_b),
            )
        )

Is there some analysis about how accurate this mixing is, compared with mixing the raw audio and then performing extraction? STFT is linear and the log-exp addition takes care of the log part, but wouldn't the Mel-scale filters make some difference? Does anyone know of any papers about this?

[pzelasko]

I don’t recall seeing any analysis of this. I remember checking it in some ASR experiment and the difference was very small, about 1% relative to the WER. Saurabh tried it on speech enhancement and the difference was more pronounced when evaluated on EER in speaker ID task, maybe 5% relative. More detailed analysis would be welcome.

[pzelasko]

In ASR I don’t remember which was the “winner”, but in enhancement + speaker ID the audio domain mixing was better.

[pzelasko]

See here for spectrograms analysis (beginnings of Lhotse) #19 (comment)

[desh2608]

Thanks for the pointer! The discussions on that thread are very enlightening.