1
$\begingroup$

I have a large data set in .txt form, and was trying to split it into three for testing, validation, and training by transforming the data into a .npy form then load it and use it on my model.

Is such a thing even possible? And if so, what should I do to make it happen?

Please excuse if my question is not logical or unreasonable, I am just trying to learn stuff.

the data should look something like this, keep in mind it is a huge amount of data enter image description here

the first 2 lines as requested:

0.60927 -0.35816 -0.10597 0.095495 0 0.063161 -0.064612 0 0 -0.053995 -0.0051178 0 0 0 0 0 0.96824 0.32552 0.44348 0.011149 0 0.13371 0.0014431 0 0 0.074547 -0.0050623 0 0 0 0 0 0.65083 0.090205 0.05407 0.046624 0 0.14346 -0.086913 0 0 -0.01106 0.0072273 0 0 0 0 0 0.057936 0.5131 0.18093 -0.089536 0 0.023838 0.0032454 0 0 0.10316 -0.00034063 0 0 0 0 0

$\endgroup$
5
  • 1
    $\begingroup$ Is that tabulated data? If so can you show 2-3 lines of it for the discussion? $\endgroup$ Jan 23, 2022 at 1:29
  • $\begingroup$ No it is not tabulated, I attached a photo of how it looks like you may want to take a look. $\endgroup$
    – besa
    Jan 23, 2022 at 22:59
  • $\begingroup$ I am wondering whether the space between two numbers is by ' ' or by \t. For me to verify, can you just copy the complete first and second lines of the file and paste it here? It's okay even if it is too long. I expect the first line would be the header and the second would be data. $\endgroup$ Jan 24, 2022 at 1:19
  • $\begingroup$ I copied them is that how you want to see them? or did I done it wrong? $\endgroup$
    – besa
    Jan 24, 2022 at 10:16
  • $\begingroup$ Thank you. Your text data helped because now I know that there is only one space character between two numbers, unlike what I see in the photo. Text data is generally needed in this place :) $\endgroup$ Jan 24, 2022 at 10:25

2 Answers 2

2
$\begingroup$

You can read the txt like this. Note that I added header=None because I assume your file does NOT contain a header line, but please remove it if it does have it.

import pandas as pd
data = pd.read_csv('file_path.txt', sep=' ', header=None)

Then convert it to the numpy array format

data_npy = data.values

(Optional) Shuffle it.

import numpy as np
np.random.seed(1)
np.random.shuffle(data_npy)

Split it into three sets, in a ratio of 6:2:2

num_rows = data_npy.shape[0]
data_npy1 = data_npy[:int(num_rows*.6)]
data_npy2 = data_npy[int(num_rows*.6): int(num_rows*.8)]
data_npy3 = data_npy[int(num_rows*.8):]

And lastly save them.

np.save('file_path1.npy', data_npy1)
np.save('file_path2.npy', data_npy2)
np.save('file_path3.npy', data_npy3)

UPDATE

Check number of entities per line

num_lines = 10

with open('file_path.txt', 'r') as f:
    for i in range(num_lines):
        num_entities = f.readline().split(' ')
        print(f'Line {i} has {num_entities} entities')
$\endgroup$
10
  • $\begingroup$ when I try to read the data it gives ParserError: Error tokenizing data. C error: Expected 235 fields in line 2, saw 237 $\endgroup$
    – besa
    Jan 24, 2022 at 14:44
  • $\begingroup$ Seems that the program saw only 235 fields in line 1 so it expects the same number in all subsequent lines. Now I think the most important thing is to find out what the correct number of field is, before deciding what to do next. If only line 1 or line 2 has the wrong number of field, the quickest way is to make manual adjustment. if not, then we may need more dedicated code for that. $\endgroup$ Jan 24, 2022 at 15:05
  • $\begingroup$ I've been trying to solve this but it feels like nothing works, how am I supposed to know what is the correct number of fields? $\endgroup$
    – besa
    Jan 24, 2022 at 16:08
  • $\begingroup$ Is the first line of your data file a header line, or not a header line? $\endgroup$ Jan 25, 2022 at 0:36
  • $\begingroup$ No it's not a header line. $\endgroup$
    – besa
    Jan 25, 2022 at 8:53
1
$\begingroup$

import numpy as np

file_path = 'data.txt'
data = np.loadtxt(file_path)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.