Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
2
votes
1answer
38 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
2answers
23 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
1
vote
3answers
27 views

How to extract insights from the given data?

Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in ...
0
votes
0answers
17 views

Is there a technique to calculate a monthly median from a known weekly median?

I currently have a dataset that conveys the financial details for each constituency in the UK. I wish to create monthly reports for income however the only associated value is "WeeklyMedian" - is ...
0
votes
0answers
20 views

Clustering data using KMeans centroids of base period for pattern analysis

I have a data frame consisting of 12 months of Customer Transaction Level Data. The data is unsupervised. The data is divided into 6 sets of 2 months period each. Taking first period as the base, I am ...
0
votes
1answer
37 views

Making Sense of this Error Message

I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code: ...
0
votes
0answers
6 views

models and algorithms to measure the difference and trend of a matrix sequence

I have a matrix sequence, i.e, a sequence consists of about 5000 matrix, each matrix are of the same dimension, e.g., 25*20. Are there any models/metrics to characterize the trend of these matrix. For ...
0
votes
0answers
16 views

how to analyse open ended responses

I have a dataset in which 5 fields are open-ended responses. I need to find interesting insights from this data. My approach is: create a word2vec and then form different clusters and make these ...
0
votes
0answers
8 views

Sample the transaction set with close lift or/and confidence estimate with original set[association rules]

In frequency set mining, for large sets, we can use sample to reduce the calculation while not miss too much for the support using formula: ...
0
votes
0answers
11 views

How to get a sample/summarised data from a table for data analysis?

I am currently working on EHR data analysis. I have a measurement table which has information lab test information of the patients. However the record count is huge. These are table count in my EHR ...
0
votes
0answers
15 views

Plot line at X axis based on Values of Y axis - maplotlib/seaborn

I have made a dist plot and now I want to plot a line on the x-axis based on the value of the y-axis. ie. I want to plot a line at various maximas/peaks and also shade a portion of the x-axis between ...
0
votes
0answers
12 views

Do I include a spatial features inside my training data for a SOM

I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
1answer
40 views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
0
votes
0answers
20 views

GPS Data Preprocessing Recommandations?

GPS Data Preprocessing I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
1answer
12 views

Should I have “normal” sampled data in my dataset?

I am busy working on a project to find the reasons why kids in normal households are doing badly in school. I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
1answer
20 views

Computing Jaccard Similarity between two documents

Data Mining: Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) = D1 = the quick brown fox jumps over the lazy dog D2 = jeff typed the quick brown dog jumps over the lazy fox ...
0
votes
0answers
12 views

Besides RFM segmentation, how to assess user consistency

In many RFM (Recency-Frequency-Monetary) tutorials, total times of purchase is used as "frequency". However, I wonder if this could be used to reflect user consistency (or some better words to ...
2
votes
0answers
36 views

Why decision tree algorithm performs better in small dataset in compare to other classification algorithm?

I read in some paper that decision tree algorithms have better result in small dataset like(400-600 records) when compared to other classification algorithms like SVM, Naive Bayes and KNN. I want to ...
0
votes
0answers
12 views

Selecting only the fittest chromosome for reproduction in genetic algorithms

Let's assume I track all my chromosomes with the fittness level in a database. The initial population will contain all available allels at least once. So, every solution exists at least in one ...
4
votes
1answer
46 views

Mutation on demand in genetic algorithms

genetic algorithm usually use a "mutation rate" to control the rate of chromosome mutation. Most of the researchers at researchgate recommend to keep this rate low in order to converge quickly, to be ...
1
vote
1answer
21 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
0
votes
1answer
18 views

appending file text into list with file name in python

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
2
votes
3answers
107 views

References for longitudinal data analysis

So my goal is to study longitudinal data (data in time series) by applying some data mining techniques. Ultimately I want to be able to "predict" outcomes. For example, a study of patients along the ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
1
vote
0answers
41 views

Dependency Graph

I want to create a dependency graph of some sensors in the network ( based on their reported value). Please note that a change in the values of sensors is related to each other. For example, if the ...
4
votes
0answers
26 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
0
votes
0answers
36 views

How to calculate Fuzzy C-Means problem by hand

I figured that this doubt next can interest another students like me and help others also that are trying to understand mathematically the fuzzy c-means mathematical mechanism already that some books ...
1
vote
1answer
27 views

T-DBSCAN - Implementing STOP logic

I am attempting to implement the T-DBSCAN algorithm described in T-DBSCAN: A Spatiotemporal Density Clustering for GPS Trajectory Segmentation. I have been able to implement most of the logic between ...
0
votes
0answers
17 views

Sampling user IDs from a large data set for retention analysis

Let's say I have a data set with user identities and their respective orders, of the form: ...
2
votes
1answer
40 views

measuring flip-flop behaviour across several topics

I'm trying to analyze a behavior called "sentiment flipping" of users in a dataset, but I'm not able to step on. Let's suppose that I have two groups of users, say them good and bad users. My ...
2
votes
3answers
63 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
1
vote
1answer
45 views

Does AWS Sagemaker continue running when I go offline?

I'm pulling data from web services in an AWS Sagemaker Notebook. I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run. ...
1
vote
1answer
22 views

some algorithms to identify of a pattern in text data

I have few sentences like below in my project(around 25000), ...
0
votes
1answer
65 views

Dealing with new outliers after capping

I'm trying to cap outliers in a column of my pandas DataFrame. Here's the boxplot for a column of my original data. So, using code from this stackoverflow answer, I tried capping outliers. Here's ...
0
votes
1answer
16 views

What are some key difference in functionnality between ProM and Apromore?

I would like to know what are some use cases where one would use ProM and where one would use Apromore. I have traces and events from logs that I would like to use process mining on (mapping a ...
0
votes
2answers
24 views

How to train a model on a data where there are multiple data inside a data point?

I'm trying to do prediction on capacity column, however each data point consist of more data. Each data point represent a cycle data. Each cycle has a capacity. Each cycle runs for some time duration,...
2
votes
3answers
5k views

Pandas change value of a column based another column condition

I have values in column1, I have columns in column2. What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90. Here is what i did so ...
0
votes
3answers
38 views

Problem in implementing CNN

I am using the fashion MNIST dataset to try to work this out: I am using the data from the links: Training : http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz ...
0
votes
0answers
29 views

Why FP-Growth is slower than Apriori?

My project is about Data mining (in MATLAB) and want to use Apriori and FP-GRowth to extract rules (Associate Rule Mining). I read in somewhere on the internet, FP-Growth should be faster than Apriori ...
0
votes
0answers
21 views

Getting Error while predicting during deploying simple ML model with Flask

I have deployed a simple titanic (doing for testing purpose) model with Python Flask. I am using below server.py (Flask app start and predict API code) and request.py (JSON request) code. Getting ...
1
vote
0answers
32 views

How to detect Class label in Apriori algorithm in MATLAB?

Sorry if my my question is simple. I just would like to know how to detect class labels in data mining algorithms such as Apriori. As I know, in the normal use of the Apriori algorithm our patterns ...
1
vote
0answers
7 views

Could perceptron use the standard logistic function as its activation function?

the perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to ...
0
votes
0answers
21 views

What could be the source of a systematic error in prediction?

While training a LSTM model on time series data I am getting a constant over prediction as below. The features are properly scaled and there is no evident over-fitting as checked from learning curves....
2
votes
0answers
21 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...
0
votes
0answers
9 views

What statistical test would I use to test that both data sources are matching

Suppose I am only working with two datasets, Data Source 1 with columns Customer ID and ...
0
votes
0answers
18 views

How to aggregate large data

I have a quite large data (frame) with 65 physical quantities and each with different time stamps. Some are gathered in intervals of several hours and some in milliseconds. Hence, the data frame ...
0
votes
0answers
8 views

Analysis on the 5 segment of the data to differentiate from each other using R

I have 1000 customers operational data which is already divided into segments. Now I have to do analysis on different segments to find out how they differ from each other. Why they fall into their ...
2
votes
3answers
86 views

Visualize/analyze data before or after imputing missing values?

My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible. To me, it would make ...
1
vote
0answers
23 views

How to perform Learning to Rank for a small dataset

I am very interested in applying Learning to rank to my problem doamin. When I read through the literature of Learning to rank I ...
0
votes
0answers
12 views

Extract data from mainly unstructured sets and derive risk metrics out of those

I have the following question (this was a real life example): Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...