Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
-1
votes
1answer
17 views

What is the purpose of 'col' argument in 'pairs' graph function?

Hello I'm new to data science with R. I'm investigating the dataset Auto of ISLR, used for predicting if mpg is low or high using all other attributes. I'm using the following code to create a ...
0
votes
0answers
36 views

Unformatted data entries

I have been working recently on an independent project using a database for Cybersecurity Attack classification. I imported the database using Pandas (Python) and before starting the processing step, ...
-1
votes
0answers
17 views

Convert dataset to table format [closed]

I am trying to understand the given dataset. I want it to simplified in table format. Basically, there is no primary key here. SF and SR are values. Y1, L1, Y2, L2, Y3, L3 are columns. Can someone ...
0
votes
0answers
8 views

Storing event data in JSON vs. rows in another table

I want to better understand the constrains and considerations I need to keep in mind when processing site analytics. For example, let's say that I have a table that contains page view data. There is a ...
0
votes
0answers
11 views

How to select a model [closed]

I have a 3 years data set from an IT service desk with dates, calling parties, handler, region, place, who was it assigned to, who fixed the issue ... I assume you get the point Now I do not know what ...
0
votes
2answers
35 views

Data Cleaning, refining etc

I have data as below: Carpanter Carepnter Carpentor Labourer Labor Labour Housewife House Wife housewife. I want to clean data and rectify the spelling mistakes ...
0
votes
1answer
22 views

Python and data analytics how to get the unique number of orders per seller using groupby?

I am totally new to data analytics and python. I took a good course on Machine learning. But I didn't really worked with real life data. I managed to get data from a store who wants to create a ...
4
votes
2answers
64 views

Discrimination vs Calibration - Machine Learning Models

I came across a new term called Calibration while reading about prediction models. Can you please help me understand how different it is from ...
1
vote
2answers
32 views

Different approaches to label data

I have a dataset of patient records but they don't have labels. I would like to label them and would like to know what are the different approaches available that I can consider to label them. For ...
0
votes
3answers
31 views

Why to exclude features used for label generation during modeling?

I have a dataset like below without labels But with the help of experts opinion, we generate labels based on the below 3 rules (all 3 rules has to be met to label it as 1) So now the dataset looks ...
1
vote
1answer
34 views

Probabilistic gold standard vs Deterministic gold standard

I understand that we say something as a gold standard when it involves human intervention/judgement/review. But can someone help me understand what's the difference between probabilistic gold ...
1
vote
1answer
179 views

How to trust the labels generated using ML models?

I have a dataset of patient records. But I do not know whether he is +ve for a cancer or not. So, I do not have the labels in my dataset. Now I can run a machine learning models like clustering to ...
0
votes
1answer
26 views

Can we add other features along with Text in sentiment analysis

Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given ...
2
votes
1answer
10 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
-1
votes
0answers
20 views

Data source for global corona infection chains [migrated]

Even though there is plenty of public data, I cannot find a source for the global infection chains of Corona. In other words, I am interested in the data that shows from where each infection has been ...
1
vote
1answer
28 views

Insights betwwen two columns/variables in Dataframe

I have data in two columns one is range of old credit score (Input score range) and new credit score (cvsc100). How do i find insights from both of them ? where the old is range of values and other ...
0
votes
1answer
17 views

Which algorithms should I use for identifying similar characteristics between data points (the intersections)?

I am working with a dataset that has been coded and categorized, so that each datapoint has a set of coded characteristics. An example data point would be something like the following: Example Data ...
4
votes
1answer
25 views

Does k fold cross validation become less useful when number of observations is very large?

As seen in the accepted answer for variance of k-fold cross validation , the simulation shows that k-fold CV has the same test error rate for different values of k when n=200. Does this mean that k-...
3
votes
0answers
33 views

What are the data preprocessing steps required before running K-Modes?

I have a clustering task at hand. The data that I have contains only categorical variables. So, k-modes seemed like the best option. But I am not sure what are the data pre processing steps required ...
1
vote
0answers
42 views

Building a recommender system [closed]

I want to build a recommender system for shops, where I recommend items. I've learned about these systems like with content-based, collaborative filtering and so on. But now I want to make one on a ...
3
votes
2answers
81 views

Correlation vs Coefficient of multiple linear regression

I have 10000 samples. There are 4 independent variables and 1 dependent variable. The independent variables are all centered with 0 mean. I found the correlation between each of these variables ...
-1
votes
1answer
25 views

How to use entire file as unique sample for classification in scikit-learn?

My dataset is split in different files, since i'm using EEG data collected for BCI (Brain-Computer Interface) classification. Here is what i have: Each .txt file ...
1
vote
1answer
24 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
3
votes
1answer
123 views

Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-...
2
votes
1answer
28 views

How to find pattern in sequential data?

I have multiple sequential data that lead up to an event. All I want to do if find the equivalent sequential data or a pattern that leads to the event. The data is also multivariate so I want to ...
2
votes
2answers
51 views

How to predict an outcome within a specific time window?

I have a dataset which has around 10K records. My objective is to predict whether the customer will churn or not. Binary classification problem with each class representing around 55:45 proportion ...
1
vote
0answers
18 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
2answers
108 views

How to train my model efficiently?

I am new to ML and have been reading online about training bottlenecks when there are frequent updates to data. Let's say I have a built a model based on a dataset of 10M records. Now, in another 2 ...
0
votes
0answers
14 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
1
vote
0answers
22 views

What are some unsupervised clustering methods that are fairly easy to explain and scale well?

I have been tasked to identify some clusters in some data and then match these with real world counterparts to find the accuracy. To give a contrived example, say I have some data about profiles for ...
0
votes
1answer
38 views

How to select one record from multiple record for a subject during analysis?

I have a dataset where I am working on a binary classification. I have two classes of subjects. One is Outpatients and Other is Inpatients. (66:33 is the class proportion) My objective is to identify ...
3
votes
1answer
22 views

Can numerical discrete finite data be always treated also as categorical?

In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or ...
1
vote
1answer
25 views

Require Guidance - Identifying risk factors

I have a dataset of patients who have been inpatients (admitted to hospital) and not admitted (but visited as outpatients). Class proportion is 66:34. I have collected a list of features for all ...
0
votes
0answers
12 views

Using a GUI and ML to parse semi structured data from pdfs?

I have an issue where I have several different types of (PDF) documents. These documents are what I would call semi structured, in that there are often lines and borders around data, but the data ...
1
vote
2answers
37 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
0
votes
0answers
11 views

Which one is used to calculate the number of data needed for Machine Learning Models?

I am wondering which one is used to calculate the needed number of data for Machine Learning models between Hoeffding's Inequality and Power Analysis. I don't know which one people use when we need to ...
1
vote
0answers
16 views

Cyber crime - data set

I'm doing science project on my university to make an app, which uses AI to detect cyber crimes. I'm looking for sites to make my data set. Do you know any with ads like tobacco, alcohol, ...
1
vote
1answer
21 views

How to approach a new data set with no dependent variable

Basically I am taking a small course in data science. And one of our first assignments are that we have gotten a pretty huge data set (hundred of thousands of samples, and roughly 150 independent ...
2
votes
0answers
16 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
3
votes
0answers
140 views

How to do feature selection for clustering and implement it in python?

I am trying to implement k-means clustering on 60-70 features and I came across a post for feature selection technique on quora by Julian Ramos, but I fail to understand few steps mentioned. I am ...
5
votes
1answer
50 views

How do I interpret my result of clustering?

I am working on a clustering problem. I have 11 features. My complete data frame has 70-80% zeros. The data had outliers that I capped at 0.5 and 0.95 percentile. However, I tried k-means (python) on ...
3
votes
1answer
60 views

What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm? I understand that EDA helps us generate good and helpful insights ...
2
votes
2answers
97 views

How to deal with with rows with zero in every feature while clustering?

I am working on a clustering problem which has 13000 observations and 15 features. Around 3000 observations in the dataset has zero in every features ( i.e all values zero in 3000 rows). I am trying ...
0
votes
1answer
24 views

What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying

I am really in need of some explanation, I am working on an NLP cyber-bullying detection tool which I will deploy to the web using Django framework, however, am stuck on some idea, can someone explain ...
2
votes
1answer
54 views

Does K - Means clustering on data reduced using PCA and the original data make any difference?

I am working on clustering and I have 90 features with 13500 data points and after removing the correlated variables which had pearson correlation more than 90% my feature space reduced to 70. Also, ...
2
votes
1answer
56 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
0
votes
0answers
36 views

How to approach new project in Medical Image analysis?

I am working on a project in medical image analysis - Breast cancer detection. And since I am the one who has proposed this project I was wondering, what would be the (data science) steps its ...
0
votes
0answers
28 views

Fastest frequent itemset library in R or python

I am trying to map a problem with frequent itemset mining. I have only 60 transactions in total but most of them are big. Like the smallest transaction have 30 items and the largest transaction has ...
2
votes
1answer
49 views

Find Net Reclassification Improvement/Index metric using Python

I am working on a binary classification problem with ~5k records and class proportion of 33:67. I have 60 features/variables in my dataset and finally I have come to about 10 variables based on ...
2
votes
1answer
33 views

How to evaluate performance of a new feature in a model?

I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records. When I tried multiple models based on 6,7 and 8 features, I see the ...

1
2 3 4 5
20