# How to extract audio features for each video frame using pyAudioAnalysis

I'm trying something like extracting audio features for each video frame. I know there are 30 video frames and 16000 audio frames per second in the video file. I'm using pyAudioAnalysis python lib to achieve the goal but no success. Here is my code.

from __future__ import print_function
from pyAudioAnalysis import audioBasicIO
from pyAudioAnalysis import ShortTermFeatures,MidTermFeatures
import matplotlib.pyplot as plt
import os,shlex, subprocess
import pandas as pd
import numpy as np

# -acodec pcm_s16le  -vn -ar 16000
command_line = "ffmpeg -i test.mp4 -ac 1 -ar 16000  -vn test_mono.wav"
#command_line = "ffmpeg -i test.mp4 -ab 160k -ac 2 -ar 44100 -vn test_stereo.wav"
args = shlex.split(command_line)
print(args)
processResult = subprocess.call(args) # Success!
print(processResult)
[SamplingRate, signals] = audioBasicIO.read_audio_file("test_mono.wav")
print(SamplingRate)


Output

['ffmpeg', '-i', 'test.mp4', '-ac', '1', '-ar', '16000', '-vn', 'test_mono.wav'] 1

16000

#mid_feature_extraction(signal, sampling_rate, mid_window, mid_step,short_window, short_step):
MidFeatures,ShortFeatures,MidFeatureLabels=MidTermFeatures.mid_feature_extraction(signals, SamplingRate, 0.043*SamplingRate,
0.043*SamplingRate,0.00016*SamplingRate,
0.00016*SamplingRate)
print('Mid Features Extr Success')
MidFeatures_Dataframe = pd.DataFrame(data=MidFeatures.transpose(), columns=MidFeatureLabels)
print(type(MidFeatures))
#print(MidFeatures)
print(type(ShortFeatures))
#print(ShortFeatures)
print(type(MidFeatureLabels))
#print(MidFeatureLabels)
MidFeatures_Dataframe.to_csv('Audio_MidFeatures.csv')
print('Mid Features File Success')



Output

Mid Features Extr Success

<class 'numpy.ndarray'>

<class 'numpy.ndarray'>

<class 'list'>

Mid Features File Success

As per my calculations I should get 338 rows of audio features, and after a long time of struggle I'm getting 326 with the above parameters but still don't know how. If anyone can help me how, window and steps are working here. I know the basic concepts of window and step as work in CNN but not getting in this context.

• Could you also put output? You have print commands, but knowing what the printed output was might improve my understanding here. – EngrStudent Aug 24 '20 at 17:56
• Sure, wait a minute – DevLoverUmar Aug 24 '20 at 17:57
• @EngrStudent Please take a look now – DevLoverUmar Aug 24 '20 at 18:06
• I think "process results" needs a newline in front of it. I would also like to see the shape of the numpy arrays instead of the type. How long is sample? what is its dim? – EngrStudent Aug 24 '20 at 18:09
• Could you detail your calculations please ? – etiennedm Aug 25 '20 at 12:19

## 1 Answer

I don't know all your calculations, but looking at the doc in the code here, short_window and short_step should be in samples (probably as well as mid_window and mid_step).

However in your code:

#mid_feature_extraction(
signal, sampling_rate,
mid_window, mid_step,
short_window, short_step
)
midFeat,shortFeat,midFeatLabels=MidTermFeatures.mid_feature_extraction(
signals, SamplingRate,
0.043*SamplingRate, 0.043*SamplingRate,
0.00016*SamplingRate, 0.00016*SamplingRate
)


it seems that short_step=short_window=0.00016*16000=2.56 which is not in samples. So it will be cast to integer and will be equal to 2 instead of 2.56.

Hope it helps.

• Thanks for your answer but I have already calculated that to be 2.56 – DevLoverUmar Aug 25 '20 at 13:18
• Ok. Just to be sure, are you aware that it will be cast to int, so short_step and short_window will be equal to 2 ? – etiennedm Aug 25 '20 at 13:35
• Oh no, I'm not. Good point. Thanks! – DevLoverUmar Aug 26 '20 at 18:45
• I have updated the answer to be more clear. Hope it helps, otherwise, you could post your calculations, it will be easier to understand your expected result. – etiennedm Aug 27 '20 at 7:21