# I have a 20.2 GB dataset which is entirely consisting of 1 file. Need help on how to open the file [closed]

Whenever I am dealing with datasets that have weird extensions that I have not encountered before, I typically open them with notepad to see how the data looks like before using pandas to analyze them. But this dataset (link), that has an extension of .tr (stands for trace files I suppose), is not supported by notepad due to its huge size (that's what the error message says).

Which software should I use to open the trace file? Also is there a safe way to segment the file into smaller parts, as I typically upload my data to kaggle and work on their kernel.

I am not sure if this is the right place to post, if not please point me towards the correct place before removing the post.

• Can you provide more details about the data format? What is the .tr file? You may want to try R, since it support to manage most data format. Aug 1 '19 at 19:58
• Some network simulators like NS3 generate a trace file where all the events are logged: for example in the dataset that I am working on, it is supposed to contain the vehicle ID, x-coordinate and y-coordinate in simple text format. Aug 1 '19 at 20:31
• A better place for this kind of question would be superuser.com. There are lots of gnu tools you could use to look (less), open (vi) or cut (split) this kind of file, see gnu.org/software/coreutils/coreutils.html Aug 1 '19 at 20:55

This file should be a plain text file with n rows and m columns separated by a tabulator. Here is another much smaller file which they offer for download and claim that it has the same data format.

Since the file is too big to fit in memory, you can use the data format HDF5. This will allow you to load slices of your dataset without having to load the whole file into memory. However, first, you need to convert the file to a .h5 file. This can be done with h5py.

I put the following code together right now. It reads the text-file line by line and writes each line to the .h5-file. It works fine on the smaller file which they offer for download. I haven't tested it on a larger file, however. If you try it out, let me know how it goes.

import h5py
import numpy as np

n_rows_dataset = 1000

file = h5py.File('/path/to/your/output.h5', 'w')
dataset = file.create_dataset(
'my_dataset',
(n_rows_dataset, N_COLS),
chunks=(5, N_COLS),
maxshape=(None, N_COLS))

current_row = 0
# this does not load the whole file but creates an iterator instead
for line in open("/path/to/the/large/file"):
if current_row == n_rows_dataset-1:
n_rows_dataset = n_rows_dataset + 1000
dataset.resize((n_rows_dataset, N_COLS))
dset[current_row] = np.fromstring(line, sep=" ", dtype=np.float64)
current_row += 1


Note: You can find out the number of colums of your dataset by printing the first lines of your file:

i = 0
for line in open("/path/to/the/large/file"):
print(line)
i += 1
if i >= 5:
break



If you just want to peek inside a very large file as text with a basic GUI + search, I'd recommend glogg. It is designed for that very purpose.

When it comes to splitting a large file into many smaller ones, you can refer to these solutions for Windows or Unix commands and use the method that suits you best.