Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

How to apply entropy discretization to a dataset

I have a simple dataset that I'd like to apply entropy discretization to. The program needs to discretize an attribute based on the following criteria When either the condition “a” or condition “b” is ...
1
vote
1answer
30 views

How to approach this: Percentage change in one KPI leading to change in other KPIs?

I want to know how can I approach or model this problem. I have 7 KPIs (3 of them dependent on each other) and one main KPI (total 8 KPIs). I want to understand effect of these 7 KPIs on the main KPIs....
2
votes
0answers
20 views

Is there a way to scrape tweets in realtime from a list of specified users?

I am trying to build a scraper that will run continuously and save the tweets from a list of users instantaneously or within seconds of the user tweeting it. It could save the tweet details to a ...
0
votes
1answer
22 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
4
votes
1answer
43 views

difference between novelty, concept drift and anomaly

Concept drift is when the relation between the input data and the target variable changes over time. like changes in the conditional distribution. is novelty an outlier? what should I think of? what ...
0
votes
0answers
15 views

KNN search of high-dimensional embedding vectors

I have a large database of n-dimensional numerical vectors. Each vector is the embedding of a vertex into n-dimensional vector space. Vertices belong to a graph, and an algorithm such as ...
1
vote
3answers
48 views

Does One-Hot encoding increase the dimensionality and sparsity of dataset?

There are two ways to convert object datatype into numeric datatype, first is One-Hot encoding and second is simply map the numerical tags to different values. For example for column Age containing ...
0
votes
0answers
12 views

Does this make data leakage in time series? # need help for understanding time series data

Does this make data leakage in time series? I already read this, data leakage when scaling time series Data leakage is when information from outside the training dataset is used to create the model. ...
0
votes
1answer
12 views

Can you explain how these precision, recall and purity are calculated in this context?

I am studying to get my first job as data scientist and I stumbled upon this. I have been banging my head on the wall for hrs to understand this concept. But couldn't get concrete answer. Just 1 ...
0
votes
0answers
35 views

Is there any difference between data mining architecture and Knowledge discovery from data architecture?

I am looking at architectures of these two and they seem too similar to me. this is KDD architecture-: This is data mining architecture. I see no difference between them. in data mining architecture ...
0
votes
0answers
9 views

Differential Gene Expression Analysis problem

I have my own gene expression data that I am trying to perform differential gene expression analysis on. When I load it I can see all the columns. When I run Differential Gene Expression Analysis I ...
1
vote
1answer
23 views

How to determine the correlation using $X^2$ values

I am trying to understand how to calculate the correlation between two variables using p values. I am familiar with using the classic $X^2$ formula from the The Morgan Kaufmann series in data mining. ...
0
votes
1answer
14 views

MaxDiff Histogram usage

I would like to understand how the MaxDiff algorithm works. The only example that I could find online was a set of slides by Chris Crifton. Given An attribute C 12, 5, 5, 5, 12, 1, 1, 20, 20, 2, 2, 12,...
1
vote
0answers
8 views

Identifying the best group that are most productive

can you please give some idea to solve this problem, Below is my data, I want to identify the best inventory based on the CPM(target value) .. For example: I want to be able to say - Business section, ...
1
vote
2answers
48 views

Calculating the dissimilarity between term frequency vectors

Given that a document is an object represented by what is called a term frequency vector. How can we calculate the dissimilarity between term frequency vectors?
0
votes
1answer
15 views

Calculating the dissimilarity between two numeric attributes

I was wondering if someone could explain how to calculate the dissimilarity between two numeric attributes. Given that a numeric attribute is a a measurable quantity, represented in integer or real ...
1
vote
2answers
49 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
1
vote
1answer
81 views

Calculating the supremum distance

In this example I would like to figure out how to calculate the supremum distance. Given the example shown below I would like to know what f and h represent. Any help would be greatly appreciated.
1
vote
1answer
10 views

Calculating Dissimilarity between attributes

I would like to figure out how to calculate the dissimilarity between attributes Jack and Jim. Given the attributes table shown below Given the Relational table. and the example calculations I would ...
0
votes
1answer
27 views

Confusions about fp tree growth numerical-:

If fcam=3,Would conditional fp tree still be c=4 or what would it be? My guess is that it should be f=3,c=4,a=3,m=3. Am I right? Or what else would it be? Please guide. Also, in this below figure(same ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
2
votes
1answer
99 views

How to cluster skills in job domain?

I have a problem related to clustering, where i need to cluster skill set from job domain. Let's say, in a resume a candidate can mention they familiarity with amazon s3 bucket. But each people can ...
2
votes
0answers
17 views

Who is supposed to label my sentiment analysis? Linguistics or psychologist?

I'm starting off my undergraduate research on text classification even though I'm still considered new to this topic. I've collected more than 20K data from Twitter. I've been trying to label the data ...
0
votes
2answers
16 views

where can we find the healthcare dataset [closed]

I'm researching Medical Data set on some diseases like tuberculosis,cirrhosis,chronic kidney disease with the columns of age, body mass index, cholesterol, glucose, blood pressure, and the like that. ...
0
votes
0answers
7 views

Categories for binning weather / meteorological data

For EDA purposes I would like to bin continuous weather variables (temperature, rainfall, etc.) into well defined intervals such as the ones you hear in your daily weather report such as: For ...
1
vote
0answers
12 views

Best Technologies opening Large Sets of Sensor Time-Series Data to Analytics

My team is exploring options to create a robust "analytics" capability that is well-suited for our large quantities of sensor test data. I'd appreciate any suggestions for technologies that ...
1
vote
1answer
35 views

How can I weight each point in one-class SVM?

I want to give weights to some data points Specifically, these are points related to anomalies (I'm implementing one-class SVM for anomaly detection) Exactly, I want to consider some data points that ...
0
votes
0answers
97 views

Why am I getting a different answer in Principal Component Analysis dimensional reduction?

Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8). Compute the principal component using PCA Algorithm. Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
21 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
27 views

What do we mean by permissible transformations in types of attributes-:nominal,ordinal,interval,ratio?

I am studying data mining and I stumbled upon types of attributes. They are Nominal Ordinal Interval Ratio Data mining book by Tan,Steinbech,Kumar says Permissible transformations for-: nominal-:...
0
votes
0answers
12 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
3
votes
1answer
104 views

Finding if an outcome is predictable

Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
0
votes
0answers
3 views

Why is an external resample + internal resample good in simulated annealing algorithm?

I have being taking notes of the following link about the simulated annealing method: https://bookdown.org/max/FES/simulated-annealing.html However I have been struggling to understand why this author ...
1
vote
0answers
24 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on a conference. I have a question There is a dataset below. this is temporal-spatial dataset. ...
2
votes
1answer
25 views

Removing Outliers within R Works on One Attribute But Not on Second

(Note: this is part of a homework assignment. I am not asking for a solution to the assigned problem, I am just confused by some behavior in R.) I have a dataset loaded into an R script as a dataframe....
0
votes
1answer
42 views

How is there an inverse relation between precision and recall?

What I know? Firstly, Precision= $\frac{TP}{TP+FP}$ Recall=$\frac{TP}{TP+FN}$ What book says? A model that declares every record has high recall but low precision. I understand that if predicted ...
0
votes
0answers
22 views

How do I verify and test a machine learning model against reality during time?

As a software engineers we familiar with a concept of testing (unit, integration, e2e) Tests give us a level of confidence about the code and changes in our code. Looks like for ML the "code"...
2
votes
1answer
105 views

Really confused with characteristics of Naive Bayes classifiers?

Naive Bayes classifiers have the following characteristics-: They are robust to isolated noise points because such points are averaged out when estimating contiditional probabilities from data. Naive ...
0
votes
0answers
11 views

Normalize chemical terms

I'm trying to detect text similarity among paragraphs of chemistry related literature. I am facing the problem of the multiplicity of ways to write down a specific compound. Per example: The compound ...
1
vote
3answers
35 views

How to detect outliers to lable my unsupervised data?

I have unsupervised sensor data and i want to lable each row of data as anomaly or normal. Here K-mean clustering will work?
0
votes
0answers
40 views

How to use genetic algorithm to generate association rules from a transaction dataset using Python?

Please help! I really need to use GA to obtain association rules. My dataset looks like this: ...
1
vote
1answer
52 views

How to structure unstructured data

I am analysing tweets and have collected them in an unstructured format. What is the best way to structure this data so I can begin the data mining processes? Somebody suggested using python packages ...
1
vote
1answer
30 views

Evaluation metric for time-series anomaly detection

I have a question for AI or data experts. I'm writing a paper My dataset are time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one ...
1
vote
0answers
11 views

Evaluation of reliability of suppliers by on time delivery

I am trying to create a weight that can help me properly evaluate the reliability of a company’s on time delivery ratios. I’m using historic delivery data. For example, comparing company A that ...
1
vote
0answers
41 views

What is the purpose of defining such measure as metric or non-metric?

proximity measures can be metrics or non-metrics. the following criteria defines a metric dissimilarity measurement: here is for a metric similarity measurement I would like to know the consequences ...
1
vote
0answers
22 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
0
votes
1answer
24 views

How to classify ordered labels(ordinal data)?

I have some data similar to movie ratings and the labels are ordered, like 1 to 10. since the target label is not a nominal but ordinal variable, what types of models should I be using for classifying ...
0
votes
1answer
34 views

Data info: Orange data mining

I am trying to use the widgets from Orange Data Mining in the practical world. When do you use Data Info in Orange Data Mining set?
2
votes
0answers
29 views

How to compare two one hot encoded data frames based on column names?

I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
0
votes
0answers
15 views

Is there a simple way or tool for creating authors network from arXiv raw data?

Actually, i used arXiv to extract paper titles and abstact information for each author. Now I want to create author networ(co-author) from this raw data shown below ...

1
2 3 4 5
22