0
$\begingroup$

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used scipy.find_peaks and it works great, but I don't quite understand how to adjust this method arguments in order to capture only outstanding spikes - now it captures even slightest of them.

Here I'd like to replace spikes 1,2 and maybe 3 with median value from some local area around those spikes. enter image description here

Here is whole code with sample data:

import matplotlib.pyplot as plt
from scipy import signal
import numpy as np

data = np.array([1499.92837649,1498.74695136,1501.65412106,1852.75606625,1529.10227068,1527.2338009,1511.42044108,1508.18740648,1505.78652903,1515.93546365,1511.83021892,1504.59584562,1508.78471494,1500.78264115,1405.06811723,1397.95364197,1441.55203344,1482.71423213,1478.62308578,1497.64108386,1501.68473585,1506.23520923,1496.6881723,1498.70582285,1496.92464156,1491.36344958,1497.02274755,1493.25840383,1467.99193355,1488.16412732,1564.90609222,4324.09218032,1504.01615122,1516.59231739,1502.65908262,1491.34087737,1495.48497145,1482.18263694,1486.75207092,1488.28305886,1503.25046412,1488.96239452,1481.77735161,1486.36756273,1487.62106583,1401.90394972,1312.80471197,1245.19721745,1243.21820291,1241.83373451,1252.64588912,1323.309521,1373.16110944,1382.73759737,1406.10184314,1386.36355816,1363.1847865,1372.95915991,1316.68933536,1198.69830381,1377.63603792,1352.5725527,1164.1890967,1415.80938398,1404.66987602,1385.51846367,1379.27067776,1378.43735237,1382.84139049,1384.81287032,1384.59375778,1385.73733668,1386.34247221,1384.36957678,1380.06376123,1264.0448045,1234.94478379,1229.29513939,1211.72444676,1233.43773096,1225.94634645,1230.723957,1237.08116637,1240.80832322,1240.97114186,1239.03900484,1284.61014877,1376.50536986,1335.85100319,1306.7951141,1310.61421202,1313.23339974,1589.72468022,1308.64232315,1349.56164181,1436.42676762,1437.19472574,1437.04182744,1444.09334231,1451.14090997,1459.07449958,1441.82901671,1433.05944969,1440.50813024,1433.57889913,1429.69352137,1388.76264747,1415.18110448,1296.92282912,1312.63891787,1324.12978022,1323.27683689,1323.63259773,1317.64058711,1321.30544196,1322.08050495,1333.83595527,1319.98486284,1322.5660459,1326.1957005,1340.72903052,1348.75908412,1329.65293999,1320.77210104,1327.04623036,1316.11686802,1337.68561317,1322.63096511,1316.94664622,1325.19155977,1321.63982623,1331.69975677,1401.04362381,1451.22796634,1447.52458809,1447.92768292,1435.01876725,1403.47455162,1365.8405731,1364.47540946,1382.85646065,1396.10868696,1400.26717039,1374.84874018,1377.38365093,1458.15925098,1517.75673687,1527.29117179,1520.68639153,1520.06411655,1517.46319186,1518.01639946,1521.99181722,1539.93004727,1428.81560541,1507.31029851,1509.14927289,1500.07776239,1502.50659647,1498.35194354,1501.50056196,1504.92608285,1493.98376338,1501.16215364,1498.95507777,1441.07291997,1260.26816612,1178.65026678,1200.91257562,1313.28164252,1349.3797309,1354.86841693,1350.31761625,1351.56776315,1357.5037897,1349.92620054,1363.31005578,1355.73050569,1355.63120482,1359.48963968,1350.41379651,1390.8234566,1336.75550519,1487.59234266,1491.72431846,1489.40352151,1486.81409867,1502.45161666,1485.33489538,1498.11824785,1490.02329115,1490.50769554,1492.0315521,1513.04979179,1490.14435257,1498.66988545,1396.369713,1257.39311729,1258.08524592,1189.43829894])

peaks, props = signal.find_peaks(data, width=(None, 1), wlen=2, prominence=0, rel_height=0.3)

plt.bar(range(0, len(data)), data, label='data')
plt.plot(peaks, data[peaks], "x", color='r')
plt.get_current_fig_manager().window.state('zoomed')
plt.show()
showPlot(np.array(data))

I guess I need somehow specify prominence, but I don't know how to figure out the required value. I have some other data sets and fixed value might not be appropriate for them, thus this must be evaluated from data. Would appreciate any help or piece of advice.

$\endgroup$
0
$\begingroup$

I suggest you play with the height parameter. According to your figure the peaks are easy to detect. You could use the most frequent value as offset for the height parameter, but I think you should play with those values. After you find the desired peaks, you can simply do a left and right search so that you calculate the average and replace it instead of the peak.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.