Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
0
votes
1answer
237 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
2
votes
1answer
44 views

Association Rules

I'm using association rules for a project and noticed that there is a dearth of papers in the last 10-15 years on the topic although it seemed really popular 15-25 years ago. Is there any reason why?
2
votes
3answers
1k views

Combining Two CSV's in Jupyter Notebook

I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
3
votes
1answer
290 views

When to One-Hot encode categorical data when following Crisp-DM

I have a dataset that contains 15 categorical features (2 and 3 level factors which are non-ordinal) and 3 continuous numeric features. Seeing as most machine learning algorithms require numerical ...
-1
votes
1answer
49 views

How can I predict the true label for data with incomplete features based on the model learned by data with complete features? [closed]

for example, the model was learned by training data with complete features (f1,f2,f3,f4,f5,f6) but, I wonder the model can test data with incomplete features (f1,f2,f3) to attach the true label to ...
1
vote
1answer
113 views

Selecting threshold for F1 Score

When selecting a probability threshold to maximize the F1 score prior to deploying a model (based on the precision-recall curve), should the threshold be selected based on the training or holdout ...
1
vote
2answers
354 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
1
vote
1answer
54 views

PCA & Clustering Confusion

I have a question related to K-Means clustering and PCA. In my project, I have two target classes - 0 and 1- and I am trying to group the records that were predicted as 0 into 5 clusters. I am using ...
2
votes
1answer
43 views

How to visualize categorical data with numeric columns

I have data like as shown below I would like to represent the above tabular data in a visual form. However, the below graph may not work because my real data as 50K unique drug names. So, is there ...
0
votes
1answer
40 views

What to choose for null hypothesis and alternate hypothesis?

In this article which explains types of t-tests for “Independent Two-Sample t-test” example: We can confirm that the t-statistic is again less than the t-critical value so we fail to reject the null ...
1
vote
1answer
88 views

Association Rules with Python (coded dataset)

I have this dataset which I really need to use association rules techniques on. The dataset has like 90 variables, many of which are ordinal. Thing is, the data is already coded using numbers instead ...
1
vote
1answer
84 views

Best Approach to Forecasting Numerical Value Based on time series and categorical data?

Consider a dataset of thousands of car repairs that have been performed. In simplest of terms, the columns to consider are the time of year when it was broken (seasonal changes in demand for car ...
1
vote
2answers
60 views

How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?

I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best ...
0
votes
1answer
72 views

Is this attribute numeric or categorical (ordinal)? Help!

So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very ...
2
votes
1answer
222 views

How to do data augmentation for Machine Learning on statistical data?

I am training machine learning classification model. I have data in csv format with lets say 5 features(or columns) and 100 such observations(or rows). I want to add more similar data to improve my ...
4
votes
2answers
239 views

a data point - too little to spend resources during data preparation?

I am learning analytics online and have some quick questions. Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent? Let's say for ex: we have is ...
1
vote
0answers
14 views

Space partitioning to estimate product weight

Looking for some vocabulary to help me refine my research so I can tackle this problem. Here's an overview of the problem statement I'm working on. At my company we manufacture various products, ...
1
vote
2answers
303 views

Need help understanding data leakage

I am a newbie to this stuff so I am sorry if my question is stupid~ I need help understanding what data leakage between X_train and X_test is and when exactly it happens. I am currently working on a ...
0
votes
1answer
56 views

kMean clustering for recommendation

I have a file with 50000 rows from a library platform. Each individual row saves a user, and shows the order in which the user, has selected. The books could be from various categories (e.g. roman, ...
1
vote
1answer
2k views

How to get two boxplots on same axis?

I have two dataframes (both of different lengths, etc) but they have exact same columns and I would like to explore the distribution of one of these columns by plotting a boxplot. However, how do I ...
1
vote
2answers
261 views

Will one hot encoding / unbalanced columns cause bias to Clustering Analysis?

I'm wondering if having too many columns about one certain feature is gonna cause bias to the clustering analysis. For example, if my dataset has columns = ['incoming calls', 'outgoing calls', '...
1
vote
0answers
103 views

Genetic algorithm - Feature selection packages in Python

Can you share some packages in Python which are implemented that I can use for selecting features based on a genetic algorithm? I did refer to this AUTO-ML post and found out that it is useful but ...
3
votes
1answer
62 views

Replace human judgements with mathematical approach/theories

I would like to give a context of what I did. 1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below. 2) Now my task is to find matching ...
2
votes
1answer
26 views

Topology of Data

Do every data resembles some form of space like manifold? Meaning, do geometry of data (especially the big data) is a manifold? Is every data embedded to some sphere? If that is so, can we also say ...
1
vote
1answer
56 views

Automatically Detect Valid/Interesting pivot/drill downs of a dataset

Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to ...
0
votes
1answer
279 views

Error using corpus widget in Orange v 3.25.0 text mining and no import document option?

I am experimenting with Orange data mining tool. When I use the 'Corpus' widget from the text mining section it gives me the error: [![Corpus widget error][1]][1] I ...
1
vote
0answers
77 views

Is it necessary to transform data to normal distribution when removing outliers for xgboost?

sorry if this is statistics 101 but i cannot find a similar question. I am wanting to use xgboost to classify my data in two classifications. my data is numerical (financial statement data) and i can ...
0
votes
0answers
17 views

Find the most representative samples from a class

I have a dataset where each sample has 187 dimensions. The normal class has 70.000 samples while the other 4 have at most 5.000. Ideally, I would like to keep at most 15000 samples from the normal ...
0
votes
1answer
27 views

Find the time between two events by customer id [closed]

I need to find a customer has bought P1, and after how many days customer will buy P2. I am unable to find the days between order of P1 and the next order of P2 by the same customer. I have data as ...
0
votes
3answers
54 views

What's the best way to do classification basing on two given datasets (annual data and daily data)?

I want to do binary-classification basing on two given dataset, one is annual statistical data of a company and has the label I should be able to predict like this: ...
0
votes
1answer
20 views

Data mining: Clique based clustering to make comparison in social network analysis

I am a very beginner in data mining. I want to work on Clique based clustering method. I want to make a comparison between various datasets for social network analysis or community detection of social ...
0
votes
1answer
96 views

How to identify topic transition in consecutive sentences using Python?

I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without ...
1
vote
1answer
524 views

How do I perform K-Means clustering of the Olivetti Dataset

This Question pertains to Matrix Factorization and the full question is given below. Provide for k-means clustering of the Olivetti dataset the following visualizations: A scatter plot of the r = 2-...
1
vote
1answer
217 views

How to fix spelling mistakes in data?

I have an input data file which contains list of drug names. I have more than 1000 unique drug names. However, the drug names has spelling mistakes and space character issues. For ex: we have ...
1
vote
3answers
442 views

Decision tree too small

I have a data set of 2300 entries, with 5 variables one of them the dependent variable which is binary. I fitted a decision try using the rpart function in R over ...
1
vote
0answers
76 views

How to write a code on Link prediction on dynamic network in python?

I have read plenty of papers on link prediction on dynamic networks, but I am getting confusion on how to do it. I know link prediction on dynamic networks is different from static network.In dynamic ...
1
vote
0answers
15 views

Is there any problem using Euclidean Distance for a data set that includes a lot of values between 0 and 1?

I have a data set which includes 3 types of financial data (P/E, ROA, EPS growth) for each stock in a list of stocks. My goal is to use K-Medoids to cluster these stocks into groups based on this data....
0
votes
1answer
493 views

Increasing minNumObj increasing accuracy in decision tree

I have been using a J48 classifier in weka and have noticed that increasing minNumObj -- The minimum number of instances per leaf leads to a small accuracy increase. ...
1
vote
1answer
37 views

Can the dependency between variables be deduced from data? And if so, how?

I have a data set $X$ that consists of $m$ vectors $\vec{x}$ of $n$ real-valued components. Each vector component lies within a corresponding predefined interval of valid values, which is the same for ...
1
vote
1answer
210 views

What is the difference between biclustering and clustering?

After reading the wiki page for biclustering (https://en.m.wikipedia.org/wiki/Biclustering), I am really confused on what is the difference between biclustering and clustering? Any explanation/...
1
vote
1answer
32 views

What does test score provide in terms of real application?

Initially, I wanted to create a classification tool for some program codes in a specific environment based on the application patterns they show. I had 5 different patterns I wanted to look for and ...
4
votes
2answers
170 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
0
votes
2answers
95 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
0
votes
1answer
323 views

Python and data analytics how to get the unique number of orders per seller using groupby?

I am totally new to data analytics and python. I took a good course on Machine learning. But I didn't really worked with real life data. I managed to get data from a store who wants to create a ...
5
votes
3answers
2k views

Discrimination vs Calibration - Machine Learning Models

I came across a new term called Calibration while reading about prediction models. Can you please help me understand how different it is from ...
2
votes
2answers
49 views

Different approaches to label data

I have a dataset of patient records but they don't have labels. I would like to label them and would like to know what are the different approaches available that I can consider to label them. For ...
0
votes
3answers
116 views

Why to exclude features used for label generation during modeling?

I have a dataset like below without labels But with the help of experts opinion, we generate labels based on the below 3 rules (all 3 rules has to be met to label it as 1) So now the dataset looks ...
0
votes
1answer
60 views

Probabilistic gold standard vs Deterministic gold standard

I understand that we say something as a gold standard when it involves human intervention/judgement/review. But can someone help me understand what's the difference between probabilistic gold ...
1
vote
1answer
217 views

How to trust the labels generated using ML models?

I have a dataset of patient records. But I do not know whether he is +ve for a cancer or not. So, I do not have the labels in my dataset. Now I can run a machine learning models like clustering to ...
0
votes
1answer
140 views

Can we add other features along with Text in sentiment analysis

Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given ...

1 2 3
4
5
23