You can use Robust Squared Mahalanobis Distance to detect outliers in Multivariate. Then run your model one time using all data values and compute the Mean square error. Run it for the second time without these outliers and compute again MSE. See the difference. If you have so many outliers, you can use first principle component (PC) to reduce the ...
I suggest reading only the index from the CSV file and do your modification and copy it back instead of reading the entire CSV file.
You can do that with:
df = pd.read_csv("sample.csv", names=column_names)
Instead of df.index.map(), use panda's .str accessor so that the slicing is vectorized. That will speed up processing on each chunk.
First of all, reading and then writing seems not to be very efficient since every line will be fully overwritten, which is not necessary, right?
Unless the entries in your index are of uniform length (so the final character ...
You can read a file line by line, process each line and write to a new file line by line, this is probably not the most efficient way, but will certainly solve the RAM issue.
with open("my_file.csv") as f_in, open("new_file.csv", "w") as f_out:
for line in f_in:
new_line = line # do you processing here