Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

Evaluating likelihood of egg breaking when falling in random container on concrete [closed]

I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete. I am looking at the different factors that play a role ...
0
votes
0answers
23 views

Help required for medical imaging research | Deep Learning Project

I am a cs student currently working on Brain tumour segmentation using cascading of two U-Net research project. I have researched over the internet about the cascading of CNN but I found nothing about ...
0
votes
0answers
7 views

How far I can apply ARIMA forecast algorithm to my model?

I created a ARIMA model with 1224 month wise data (102 years - From Jan 1900 to Dec 2002), totalling to 1224 data items. How far I can forecast. Typically i got some pattern in forecast only for 36 ...
1
vote
1answer
31 views

Sequence to carry out data analysis?

I have a dataset with 4700 records and it's a classification problem. Proportion of classes is 33 and 67% few questions 1) does this proportion qualify dataset as imbalanced ? 2) should I do ...
0
votes
2answers
23 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
1
vote
1answer
18 views

On which step should use SMOTE technique for over sampling?

I want to use SMOTE technique for over sampling but I don't know on which step on pre-processing I should use it. My preprocessing steps are: Missing values Removing Outliers Smoothing Data Should ...
0
votes
0answers
25 views

What is the effect of changing the seed value on the classification results?

I used weka program to make a classification, first I used a dataset using the explorer and the seed value was equal to one, then I used the experimenter for the same dataset and the seed value was ...
0
votes
1answer
18 views

what does random seed value mean in weka?

I use weka to make a dataset classification but there is an option in classifier evaluation ( random seed for XVAL/% split). What does this option mean? ,and what is the seed value ? ,and What is the ...
1
vote
1answer
27 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
1answer
19 views

Question mark on Correlation Matrix with RapidMiner

I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'. Someone can help me? This is a sample of data
0
votes
1answer
12 views

What is the output polytree after aplying the Ramex algorithm to this graph?

I've been trying to understand the way this algorithm works, but I can't get a consistent result. It has two phases: the first one coverts a table of events into a graph, and the second where the ...
1
vote
1answer
32 views

Maybe wrong values for precision and recall

I'm trying to do some data mining with RapidMiner studio. I've applied the K-nearest neighbor algorithm with different values of K. As I expected, accuracy increase and after K=5, it decrease. But I ...
2
votes
1answer
48 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
2answers
25 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
1
vote
3answers
28 views

How to extract insights from the given data?

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in ...
0
votes
0answers
18 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
0
votes
0answers
20 views

Clustering data using KMeans centroids of base period for pattern analysis

I have a data frame consisting of 12 months of Customer Transaction Level Data. The data is unsupervised. The data is divided into 6 sets of 2 months period each. Taking first period as the base, I am ...
0
votes
1answer
42 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
0answers
9 views

models and algorithms to measure the difference and trend of a matrix sequence

I have a matrix sequence, i.e, a sequence consists of about 5000 matrix, each matrix are of the same dimension, e.g., 25*20. Are there any models/metrics to characterize the trend of these matrix. For ...
0
votes
0answers
16 views

how to analyse open ended responses

I have a dataset in which 5 fields are open-ended responses. I need to find interesting insights from this data. My approach is: create a word2vec and then form different clusters and make these ...
0
votes
0answers
8 views

Sample the transaction set with close lift or/and confidence estimate with original set[association rules]

In frequency set mining, for large sets, we can use sample to reduce the calculation while not miss too much for the support using formula: ...
0
votes
0answers
12 views

How to get a sample/summarised data from a table for data analysis?

I am currently working on EHR data analysis. I have a measurement table which has information lab test information of the patients. However the record count is huge. These are table count in my EHR ...
0
votes
0answers
16 views

Plot line at X axis based on Values of Y axis - maplotlib/seaborn

I have made a dist plot and now I want to plot a line on the x-axis based on the value of the y-axis. ie. I want to plot a line at various maximas/peaks and also shade a portion of the x-axis between ...
0
votes
0answers
12 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
70 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
22 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
1answer
24 views

Computing Jaccard Similarity between two documents

Data Mining: Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) = D1 = the quick brown fox jumps over the lazy dog D2 = jeff typed the quick brown dog jumps over the lazy fox ...
0
votes
0answers
13 views

Besides RFM segmentation, how to assess user consistency

In many RFM (Recency-Frequency-Monetary) tutorials, total times of purchase is used as "frequency". However, I wonder if this could be used to reflect user consistency (or some better words to ...
2
votes
0answers
37 views

Why decision tree algorithm performs better in small dataset in compare to other classification algorithm?

I read in some paper that decision tree algorithms have better result in small dataset like(400-600 records) when compared to other classification algorithms like SVM, Naive Bayes and KNN. I want to ...
0
votes
0answers
15 views

Selecting only the fittest chromosome for reproduction in genetic algorithms

Let's assume I track all my chromosomes with the fittness level in a database. The initial population will contain all available allels at least once. So, every solution exists at least in one ...
4
votes
1answer
46 views

Mutation on demand in genetic algorithms

genetic algorithm usually use a "mutation rate" to control the rate of chromosome mutation. Most of the researchers at researchgate recommend to keep this rate low in order to converge quickly, to be ...
1
vote
1answer
26 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
0
votes
1answer
20 views

appending file text into list with file name in python

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
2
votes
3answers
118 views

References for longitudinal data analysis

So my goal is to study longitudinal data (data in time series) by applying some data mining techniques. Ultimately I want to be able to "predict" outcomes. For example, a study of patients along the ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
1
vote
0answers
42 views

Dependency Graph

I want to create a dependency graph of some sensors in the network ( based on their reported value). Please note that a change in the values of sensors is related to each other. For example, if the ...
4
votes
0answers
27 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
0
votes
0answers
43 views

How to calculate Fuzzy C-Means problem by hand

I figured that this doubt next can interest another students like me and help others also that are trying to understand mathematically the fuzzy c-means mathematical mechanism already that some books ...
1
vote
1answer
31 views

T-DBSCAN - Implementing STOP logic

I am attempting to implement the T-DBSCAN algorithm described in T-DBSCAN: A Spatiotemporal Density Clustering for GPS Trajectory Segmentation. I have been able to implement most of the logic between ...
0
votes
0answers
17 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
2
votes
1answer
40 views

measuring flip-flop behaviour across several topics

I'm trying to analyze a behavior called "sentiment flipping" of users in a dataset, but I'm not able to step on. Let's suppose that I have two groups of users, say them good and bad users. My ...
2
votes
3answers
63 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
1
vote
1answer
53 views

Does AWS Sagemaker continue running when I go offline?

I'm pulling data from web services in an AWS Sagemaker Notebook. I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run. ...
1
vote
1answer
22 views

some algorithms to identify of a pattern in text data

I have few sentences like below in my project(around 25000), ...
0
votes
1answer
80 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
22 views

What are some key difference in functionnality between ProM and Apromore?

I would like to know what are some use cases where one would use ProM and where one would use Apromore. I have traces and events from logs that I would like to use process mining on (mapping a ...
0
votes
2answers
24 views

How to train a model on a data where there are multiple data inside a data point?

I'm trying to do prediction on capacity column, however each data point consist of more data. Each data point represent a cycle data. Each cycle has a capacity. Each cycle runs for some time duration,...
3
votes
3answers
9k views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
0
votes
3answers
40 views

Problem in implementing CNN

I am using the fashion MNIST dataset to try to work this out: I am using the data from the links: Training : http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ...