0
$\begingroup$

I'm trying something like extracting audio features for each video frame. I know there are 30 video frames and 16000 audio frames per second in the video file. I'm using pyAudioAnalysis python lib to achieve the goal but no success. Here is my code.

from __future__ import print_function
from pyAudioAnalysis import audioBasicIO
from pyAudioAnalysis import ShortTermFeatures,MidTermFeatures
import matplotlib.pyplot as plt
import os,shlex, subprocess
import pandas as pd
import numpy as np

# -acodec pcm_s16le  -vn -ar 16000
command_line = "ffmpeg -i test.mp4 -ac 1 -ar 16000  -vn test_mono.wav"
#command_line = "ffmpeg -i test.mp4 -ab 160k -ac 2 -ar 44100 -vn test_stereo.wav"
args = shlex.split(command_line)
print(args)
processResult = subprocess.call(args) # Success!
print(processResult)
[SamplingRate, signals] = audioBasicIO.read_audio_file("test_mono.wav")
print(SamplingRate)

Output

['ffmpeg', '-i', 'test.mp4', '-ac', '1', '-ar', '16000', '-vn', 'test_mono.wav'] 1

16000

#mid_feature_extraction(signal, sampling_rate, mid_window, mid_step,short_window, short_step):
MidFeatures,ShortFeatures,MidFeatureLabels=MidTermFeatures.mid_feature_extraction(signals, SamplingRate, 0.043*SamplingRate, 
                                                                                  0.043*SamplingRate,0.00016*SamplingRate,
                                                                                  0.00016*SamplingRate)
print('Mid Features Extr Success')
MidFeatures_Dataframe = pd.DataFrame(data=MidFeatures.transpose(), columns=MidFeatureLabels)
print(type(MidFeatures))
#print(MidFeatures)
print(type(ShortFeatures))
#print(ShortFeatures)
print(type(MidFeatureLabels))
#print(MidFeatureLabels)
MidFeatures_Dataframe.to_csv('Audio_MidFeatures.csv')
print('Mid Features File Success')

Output

Mid Features Extr Success

<class 'numpy.ndarray'>

<class 'numpy.ndarray'>

<class 'list'>

Mid Features File Success

As per my calculations I should get 338 rows of audio features, and after a long time of struggle I'm getting 326 with the above parameters but still don't know how. If anyone can help me how, window and steps are working here. I know the basic concepts of window and step as work in CNN but not getting in this context.

$\endgroup$
5
  • 1
    $\begingroup$ Could you also put output? You have print commands, but knowing what the printed output was might improve my understanding here. $\endgroup$ Aug 24, 2020 at 17:56
  • $\begingroup$ Sure, wait a minute $\endgroup$ Aug 24, 2020 at 17:57
  • 1
    $\begingroup$ @EngrStudent Please take a look now $\endgroup$ Aug 24, 2020 at 18:06
  • 1
    $\begingroup$ I think "process results" needs a newline in front of it. I would also like to see the shape of the numpy arrays instead of the type. How long is sample? what is its dim? $\endgroup$ Aug 24, 2020 at 18:09
  • $\begingroup$ Could you detail your calculations please ? $\endgroup$
    – etiennedm
    Aug 25, 2020 at 12:19

1 Answer 1

1
$\begingroup$

I don't know all your calculations, but looking at the doc in the code here, short_window and short_step should be in samples (probably as well as mid_window and mid_step).

However in your code:

#mid_feature_extraction(
    signal, sampling_rate, 
    mid_window, mid_step,
    short_window, short_step
)
midFeat,shortFeat,midFeatLabels=MidTermFeatures.mid_feature_extraction(
    signals, SamplingRate, 
    0.043*SamplingRate, 0.043*SamplingRate,
    0.00016*SamplingRate, 0.00016*SamplingRate
)

it seems that short_step=short_window=0.00016*16000=2.56 which is not in samples. So it will be cast to integer and will be equal to 2 instead of 2.56.

Hope it helps.

$\endgroup$
4
  • $\begingroup$ Thanks for your answer but I have already calculated that to be 2.56 $\endgroup$ Aug 25, 2020 at 13:18
  • $\begingroup$ Ok. Just to be sure, are you aware that it will be cast to int, so short_step and short_window will be equal to 2 ? $\endgroup$
    – etiennedm
    Aug 25, 2020 at 13:35
  • $\begingroup$ Oh no, I'm not. Good point. Thanks! $\endgroup$ Aug 26, 2020 at 18:45
  • 1
    $\begingroup$ I have updated the answer to be more clear. Hope it helps, otherwise, you could post your calculations, it will be easier to understand your expected result. $\endgroup$
    – etiennedm
    Aug 27, 2020 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.