I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used scipy.find_peaks and it works great, but I don't quite understand how to adjust this method arguments in order to capture only outstanding spikes - now it captures even slightest of them.
Here I'd like to replace spikes 1,2 and maybe 3 with median value from some local area around those spikes.
Here is whole code with sample data:
import matplotlib.pyplot as plt
from scipy import signal
import numpy as np
data = np.array([1499.92837649,1498.74695136,1501.65412106,1852.75606625,1529.10227068,1527.2338009,1511.42044108,1508.18740648,1505.78652903,1515.93546365,1511.83021892,1504.59584562,1508.78471494,1500.78264115,1405.06811723,1397.95364197,1441.55203344,1482.71423213,1478.62308578,1497.64108386,1501.68473585,1506.23520923,1496.6881723,1498.70582285,1496.92464156,1491.36344958,1497.02274755,1493.25840383,1467.99193355,1488.16412732,1564.90609222,4324.09218032,1504.01615122,1516.59231739,1502.65908262,1491.34087737,1495.48497145,1482.18263694,1486.75207092,1488.28305886,1503.25046412,1488.96239452,1481.77735161,1486.36756273,1487.62106583,1401.90394972,1312.80471197,1245.19721745,1243.21820291,1241.83373451,1252.64588912,1323.309521,1373.16110944,1382.73759737,1406.10184314,1386.36355816,1363.1847865,1372.95915991,1316.68933536,1198.69830381,1377.63603792,1352.5725527,1164.1890967,1415.80938398,1404.66987602,1385.51846367,1379.27067776,1378.43735237,1382.84139049,1384.81287032,1384.59375778,1385.73733668,1386.34247221,1384.36957678,1380.06376123,1264.0448045,1234.94478379,1229.29513939,1211.72444676,1233.43773096,1225.94634645,1230.723957,1237.08116637,1240.80832322,1240.97114186,1239.03900484,1284.61014877,1376.50536986,1335.85100319,1306.7951141,1310.61421202,1313.23339974,1589.72468022,1308.64232315,1349.56164181,1436.42676762,1437.19472574,1437.04182744,1444.09334231,1451.14090997,1459.07449958,1441.82901671,1433.05944969,1440.50813024,1433.57889913,1429.69352137,1388.76264747,1415.18110448,1296.92282912,1312.63891787,1324.12978022,1323.27683689,1323.63259773,1317.64058711,1321.30544196,1322.08050495,1333.83595527,1319.98486284,1322.5660459,1326.1957005,1340.72903052,1348.75908412,1329.65293999,1320.77210104,1327.04623036,1316.11686802,1337.68561317,1322.63096511,1316.94664622,1325.19155977,1321.63982623,1331.69975677,1401.04362381,1451.22796634,1447.52458809,1447.92768292,1435.01876725,1403.47455162,1365.8405731,1364.47540946,1382.85646065,1396.10868696,1400.26717039,1374.84874018,1377.38365093,1458.15925098,1517.75673687,1527.29117179,1520.68639153,1520.06411655,1517.46319186,1518.01639946,1521.99181722,1539.93004727,1428.81560541,1507.31029851,1509.14927289,1500.07776239,1502.50659647,1498.35194354,1501.50056196,1504.92608285,1493.98376338,1501.16215364,1498.95507777,1441.07291997,1260.26816612,1178.65026678,1200.91257562,1313.28164252,1349.3797309,1354.86841693,1350.31761625,1351.56776315,1357.5037897,1349.92620054,1363.31005578,1355.73050569,1355.63120482,1359.48963968,1350.41379651,1390.8234566,1336.75550519,1487.59234266,1491.72431846,1489.40352151,1486.81409867,1502.45161666,1485.33489538,1498.11824785,1490.02329115,1490.50769554,1492.0315521,1513.04979179,1490.14435257,1498.66988545,1396.369713,1257.39311729,1258.08524592,1189.43829894])
peaks, props = signal.find_peaks(data, width=(None, 1), wlen=2, prominence=0, rel_height=0.3)
plt.bar(range(0, len(data)), data, label='data')
plt.plot(peaks, data[peaks], "x", color='r')
plt.get_current_fig_manager().window.state('zoomed')
plt.show()
showPlot(np.array(data))
I guess I need somehow specify prominence, but I don't know how to figure out the required value. I have some other data sets and fixed value might not be appropriate for them, thus this must be evaluated from data.
Would appreciate any help or piece of advice.
