How to extract audio features for each video frame using pyAudioAnalysis

Question

I'm trying something like extracting audio features for each video frame. I know there are 30 video frames and 16000 audio frames per second in the video file. I'm using pyAudioAnalysis python lib to achieve the goal but no success. Here is my code.

from __future__ import print_function
from pyAudioAnalysis import audioBasicIO
from pyAudioAnalysis import ShortTermFeatures,MidTermFeatures
import matplotlib.pyplot as plt
import os,shlex, subprocess
import pandas as pd
import numpy as np

# -acodec pcm_s16le  -vn -ar 16000
command_line = "ffmpeg -i test.mp4 -ac 1 -ar 16000  -vn test_mono.wav"
#command_line = "ffmpeg -i test.mp4 -ab 160k -ac 2 -ar 44100 -vn test_stereo.wav"
args = shlex.split(command_line)
print(args)
processResult = subprocess.call(args) # Success!
print(processResult)
[SamplingRate, signals] = audioBasicIO.read_audio_file("test_mono.wav")
print(SamplingRate)

Output

['ffmpeg', '-i', 'test.mp4', '-ac', '1', '-ar', '16000', '-vn', 'test_mono.wav'] 1

16000

#mid_feature_extraction(signal, sampling_rate, mid_window, mid_step,short_window, short_step):
MidFeatures,ShortFeatures,MidFeatureLabels=MidTermFeatures.mid_feature_extraction(signals, SamplingRate, 0.043*SamplingRate, 
                                                                                  0.043*SamplingRate,0.00016*SamplingRate,
                                                                                  0.00016*SamplingRate)
print('Mid Features Extr Success')
MidFeatures_Dataframe = pd.DataFrame(data=MidFeatures.transpose(), columns=MidFeatureLabels)
print(type(MidFeatures))
#print(MidFeatures)
print(type(ShortFeatures))
#print(ShortFeatures)
print(type(MidFeatureLabels))
#print(MidFeatureLabels)
MidFeatures_Dataframe.to_csv('Audio_MidFeatures.csv')
print('Mid Features File Success')

Output

Mid Features Extr Success

<class 'numpy.ndarray'>

<class 'numpy.ndarray'>

<class 'list'>

Mid Features File Success

As per my calculations I should get 338 rows of audio features, and after a long time of struggle I'm getting 326 with the above parameters but still don't know how. If anyone can help me how, window and steps are working here. I know the basic concepts of window and step as work in CNN but not getting in this context.

Answer 1

I don't know all your calculations, but looking at the doc in the code here, short_window and short_step should be in samples (probably as well as mid_window and mid_step).

However in your code:

#mid_feature_extraction(
    signal, sampling_rate, 
    mid_window, mid_step,
    short_window, short_step
)
midFeat,shortFeat,midFeatLabels=MidTermFeatures.mid_feature_extraction(
    signals, SamplingRate, 
    0.043*SamplingRate, 0.043*SamplingRate,
    0.00016*SamplingRate, 0.00016*SamplingRate
)

it seems that short_step=short_window=0.00016*16000=2.56 which is not in samples. So it will be cast to integer and will be equal to 2 instead of 2.56.

Hope it helps.