0
$\begingroup$

I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. For the process I should split the dataset into 10 smaller datasets. And the other task is using Weka4WS (weka for web services) for the process (For clustering part).

So my questions:
1. How can I split the dataset using python code?
2. What is Weka4WS and how should I use it, I mean the implementation (because it's not something like apps like the Weka itself)?

splitting dataset size of datasets

$\endgroup$
1
  • $\begingroup$ How does the downloaded dataset look like? Do you get CSV files or something else? $\endgroup$ Aug 6, 2018 at 17:38

1 Answer 1

0
$\begingroup$

Splitting your dataset

First you can load your data set to memory:

# Load the data to memory
with open('file.txt', 'r', encoding='utf-8') as file:
    header = file.readline()
    data = []
    for line in file.readlines():
        data.append(line)

Next, you use KFold to split your data into $10$ splits and save to $10$ different files.

# Split the data and save the data
from sklearn.model_selection import KFold
kf = KFold(n_splits=10)

split = 1
for _, split_index in kf.split(data): # Iterate over the splits

    with open('file_{}.txt'.format(split), 'w+', encoding='utf-8') as file: # Open a file with name file_{split number}
        file.write(header) # write the header of the file
        for index in split_index: # Write each point
            file.write(data[index])
    split += 1 # Increase split number

Weka4WS

The Weka4WS is an extension to Weka itself. Yes, it is like an application like Weka, with a Graphical User Interface. From the docs:

Thus, Weka4WS also extends the Weka GUI to enable the invocation of the data mining algorithms that are exposed as Web Services on remote Grid nodes.

However, it is now as easy to run as the normal Weka. You need some additional steps to use it.

The official installation is here: http://scalab.dimes.unical.it/weka4ws/howtos/installation/

Here is a tutorial to run the application on Windows: http://scalab.dimes.unical.it/weka4ws/howtos/run-the-client-on-windows/

This is how the app looks like, this is the KnowledgeFlow screen:

enter image description here

$\endgroup$
9
  • $\begingroup$ The dataset is a txt file and its size is 360MB. Here's the link of dataset: kdd.ics.uci.edu/databases/census1990/USCensus1990.data.txt $\endgroup$
    – Nebula
    Aug 6, 2018 at 18:06
  • $\begingroup$ I used wc -l in terminal and it was 2458286 line of samples in dataset file. $\endgroup$
    – Nebula
    Aug 6, 2018 at 18:08
  • $\begingroup$ Alright, I edited my answer so you can load your data set, split it and save to 10 different files. $\endgroup$ Aug 6, 2018 at 18:17
  • $\begingroup$ Let me know if that helps you $\endgroup$ Aug 6, 2018 at 18:29
  • $\begingroup$ Sure, I'll tell the result, now I'm trying the code and I've got an Error I'm trying to solve it. Thanks:) By the way can I determine size of the files? $\endgroup$
    – Nebula
    Aug 6, 2018 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.