0
$\begingroup$

I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. For the process I should split the dataset into 10 smaller datasets. And the other task is using Weka4WS (weka for web services) for the process (For clustering part).

So my questions:
1. How can I split the dataset using python code?
2. What is Weka4WS and how should I use it, I mean the implementation (because it's not something like apps like the Weka itself)?

splitting dataset size of datasets

$\endgroup$
  • $\begingroup$ How does the downloaded dataset look like? Do you get CSV files or something else? $\endgroup$ – BrunoGL Aug 6 '18 at 17:38
0
$\begingroup$

Splitting your dataset

First you can load your data set to memory:

# Load the data to memory
with open('file.txt', 'r', encoding='utf-8') as file:
    header = file.readline()
    data = []
    for line in file.readlines():
        data.append(line)

Next, you use KFold to split your data into $10$ splits and save to $10$ different files.

# Split the data and save the data
from sklearn.model_selection import KFold
kf = KFold(n_splits=10)

split = 1
for _, split_index in kf.split(data): # Iterate over the splits

    with open('file_{}.txt'.format(split), 'w+', encoding='utf-8') as file: # Open a file with name file_{split number}
        file.write(header) # write the header of the file
        for index in split_index: # Write each point
            file.write(data[index])
    split += 1 # Increase split number

Weka4WS

The Weka4WS is an extension to Weka itself. Yes, it is like an application like Weka, with a Graphical User Interface. From the docs:

Thus, Weka4WS also extends the Weka GUI to enable the invocation of the data mining algorithms that are exposed as Web Services on remote Grid nodes.

However, it is now as easy to run as the normal Weka. You need some additional steps to use it.

The official installation is here: http://scalab.dimes.unical.it/weka4ws/howtos/installation/

Here is a tutorial to run the application on Windows: http://scalab.dimes.unical.it/weka4ws/howtos/run-the-client-on-windows/

This is how the app looks like, this is the KnowledgeFlow screen:

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ The dataset is a txt file and its size is 360MB. Here's the link of dataset: kdd.ics.uci.edu/databases/census1990/USCensus1990.data.txt $\endgroup$ – Nebula Aug 6 '18 at 18:06
  • $\begingroup$ I used wc -l in terminal and it was 2458286 line of samples in dataset file. $\endgroup$ – Nebula Aug 6 '18 at 18:08
  • $\begingroup$ Alright, I edited my answer so you can load your data set, split it and save to 10 different files. $\endgroup$ – BrunoGL Aug 6 '18 at 18:17
  • $\begingroup$ Let me know if that helps you $\endgroup$ – BrunoGL Aug 6 '18 at 18:29
  • $\begingroup$ Sure, I'll tell the result, now I'm trying the code and I've got an Error I'm trying to solve it. Thanks:) By the way can I determine size of the files? $\endgroup$ – Nebula Aug 6 '18 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.