0
$\begingroup$

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used scipy.find_peaks and it works great, but I don't quite understand how to adjust this method arguments in order to capture only outstanding spikes - now it captures even slightest of them.

Here I'd like to replace spikes 1,2 and maybe 3 with median value from some local area around those spikes. enter image description here

Here is whole code with sample data:

import matplotlib.pyplot as plt
from scipy import signal
import numpy as np

data = np.array([1499.92837649,1498.74695136,1501.65412106,1852.75606625,1529.10227068,1527.2338009,1511.42044108,1508.18740648,1505.78652903,1515.93546365,1511.83021892,1504.59584562,1508.78471494,1500.78264115,1405.06811723,1397.95364197,1441.55203344,1482.71423213,1478.62308578,1497.64108386,1501.68473585,1506.23520923,1496.6881723,1498.70582285,1496.92464156,1491.36344958,1497.02274755,1493.25840383,1467.99193355,1488.16412732,1564.90609222,4324.09218032,1504.01615122,1516.59231739,1502.65908262,1491.34087737,1495.48497145,1482.18263694,1486.75207092,1488.28305886,1503.25046412,1488.96239452,1481.77735161,1486.36756273,1487.62106583,1401.90394972,1312.80471197,1245.19721745,1243.21820291,1241.83373451,1252.64588912,1323.309521,1373.16110944,1382.73759737,1406.10184314,1386.36355816,1363.1847865,1372.95915991,1316.68933536,1198.69830381,1377.63603792,1352.5725527,1164.1890967,1415.80938398,1404.66987602,1385.51846367,1379.27067776,1378.43735237,1382.84139049,1384.81287032,1384.59375778,1385.73733668,1386.34247221,1384.36957678,1380.06376123,1264.0448045,1234.94478379,1229.29513939,1211.72444676,1233.43773096,1225.94634645,1230.723957,1237.08116637,1240.80832322,1240.97114186,1239.03900484,1284.61014877,1376.50536986,1335.85100319,1306.7951141,1310.61421202,1313.23339974,1589.72468022,1308.64232315,1349.56164181,1436.42676762,1437.19472574,1437.04182744,1444.09334231,1451.14090997,1459.07449958,1441.82901671,1433.05944969,1440.50813024,1433.57889913,1429.69352137,1388.76264747,1415.18110448,1296.92282912,1312.63891787,1324.12978022,1323.27683689,1323.63259773,1317.64058711,1321.30544196,1322.08050495,1333.83595527,1319.98486284,1322.5660459,1326.1957005,1340.72903052,1348.75908412,1329.65293999,1320.77210104,1327.04623036,1316.11686802,1337.68561317,1322.63096511,1316.94664622,1325.19155977,1321.63982623,1331.69975677,1401.04362381,1451.22796634,1447.52458809,1447.92768292,1435.01876725,1403.47455162,1365.8405731,1364.47540946,1382.85646065,1396.10868696,1400.26717039,1374.84874018,1377.38365093,1458.15925098,1517.75673687,1527.29117179,1520.68639153,1520.06411655,1517.46319186,1518.01639946,1521.99181722,1539.93004727,1428.81560541,1507.31029851,1509.14927289,1500.07776239,1502.50659647,1498.35194354,1501.50056196,1504.92608285,1493.98376338,1501.16215364,1498.95507777,1441.07291997,1260.26816612,1178.65026678,1200.91257562,1313.28164252,1349.3797309,1354.86841693,1350.31761625,1351.56776315,1357.5037897,1349.92620054,1363.31005578,1355.73050569,1355.63120482,1359.48963968,1350.41379651,1390.8234566,1336.75550519,1487.59234266,1491.72431846,1489.40352151,1486.81409867,1502.45161666,1485.33489538,1498.11824785,1490.02329115,1490.50769554,1492.0315521,1513.04979179,1490.14435257,1498.66988545,1396.369713,1257.39311729,1258.08524592,1189.43829894])

peaks, props = signal.find_peaks(data, width=(None, 1), wlen=2, prominence=0, rel_height=0.3)

plt.bar(range(0, len(data)), data, label='data')
plt.plot(peaks, data[peaks], "x", color='r')
plt.get_current_fig_manager().window.state('zoomed')
plt.show()
showPlot(np.array(data))

I guess I need somehow specify prominence, but I don't know how to figure out the required value. I have some other data sets and fixed value might not be appropriate for them, thus this must be evaluated from data. Would appreciate any help or piece of advice.

$\endgroup$
0
$\begingroup$

I suggest you play with the height parameter. According to your figure the peaks are easy to detect. You could use the most frequent value as offset for the height parameter, but I think you should play with those values. After you find the desired peaks, you can simply do a left and right search so that you calculate the average and replace it instead of the peak.

$\endgroup$
0
$\begingroup$

One way to potentially do this is to choose peak widths such that those under a certain value are no longer detected as peaks and instead replaced with Median like Niels has suggested above.

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jan 14 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.