Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

How to read multiple JSON files to a dataframe in R? can I get the code with the comments on it to understand it better?

I have multiple json files in a folder called "app users". These json files has NAN values as well in it.
0
votes
0answers
12 views

Nan number being returned from simple python calculation [closed]

I am working on implementing a Naive Bayes Classification algorithm. I have a method def prob_continous_value which is supposed to return the probability density ...
0
votes
0answers
9 views

How to extract rules for decision tree from ID3 classification

I am working on a program to implement the ID3 algorithm. The program takes in user input for setting a threshold and creating a decision tree. ...
1
vote
0answers
14 views

Is python & tableau enough to become data analyst? [closed]

Currently, I working as SEO Analyst. I want to become a data analyst and so I have started learning python with data science from udemy course. Can Anyone help me with guidance to learn data analytics?...
0
votes
1answer
28 views

How to convert duration column from 1 hr. 17 min to 77 min in pandas? [closed]

I am trying to convert all the hr., min, and sec into just mins. For example, 1 hr. 17 min to 77 min, and 34 sec to 0.56 min, not 0.34 min. So I have used this code: ...
1
vote
0answers
23 views

Association rules - Find 100% confidence rules [closed]

Suppose there are 100 items, numbered 1 to 100, and also 100 baskets, also numbered 1 to 100. Item i is in basket b if and only ...
1
vote
1answer
21 views

How to calculate the information conveyed in a message for a given dataset

Given the data sets. Test Set ...
0
votes
0answers
29 views

Is it possible to derive anything useful from this piece of data?

Let's say you have online Profile A. Profile A is present on 3 websites: X, Y, Z. ...
0
votes
0answers
17 views

Automatic text summarization in data mining [closed]

Can help me understand the relationship between automatic text summarization and data mining projects? If you can provide an example I will thankful.
1
vote
0answers
34 views

Apriori algorithm with tags

In apriori algorithm, we can create association rules with respect to the frequencies of the corresponding data set. My question is, what if we have tags data in addition to the transaction data. For ...
-1
votes
0answers
31 views

How to extract rules from training set to run against test set for ID3 algorithm

I am working on a data mining project. The goal of this project is to implement an ID3 classifier and Naive Bayes Classifer. I have to sets of data for ID3 the following test sets are provided A2 ...
0
votes
0answers
27 views

Identify the missing value in a table

How to compute the missing values x in the below table. which formula can be used to obtain the values of x. The values in the last row and last column are the sums of the rows and columns. What is ...
2
votes
0answers
22 views

Unstructured data not template based to structured data

I am working on a project where my goal is to extract data from a large diversity of pdf that does not follow a template (i.e unstructured data not template based). My ORC part works well and now I am ...
1
vote
0answers
8 views

What are the metrics to evaluate Data Quality?

Data quality refers to the overall utility of a dataset(s) as a function of its ability to be easily processed and analyzed for other uses, usually by a database, data warehouse, or data analytics ...
0
votes
0answers
5 views

Analyzing Microscopic Residue

My goal is to analyze a set microscopic residue. I'd like to apply entropy discretization to attribute A1, segmentation by natural partitioning to A2, then correlate A1 and A3. The first attribute of ...
0
votes
1answer
15 views

How to apply entropy discretization to a dataset

I have a simple dataset that I'd like to apply entropy discretization to. The program needs to discretize an attribute based on the following criteria When either the condition “a” or condition “b” is ...
1
vote
1answer
34 views

How to approach this: Percentage change in one KPI leading to change in other KPIs?

I want to know how can I approach or model this problem. I have 7 KPIs (3 of them dependent on each other) and one main KPI (total 8 KPIs). I want to understand effect of these 7 KPIs on the main KPIs....
2
votes
0answers
25 views

Is there a way to scrape tweets in realtime from a list of specified users? [closed]

I am trying to build a scraper that will run continuously and save the tweets from a list of users instantaneously or within seconds of the user tweeting it. It could save the tweet details to a ...
0
votes
1answer
28 views

Interpreting evaluation metrics with threshold/cutoff

I was doing churn prediction for a company. I've got the following results by applying 3 classifier. Model Accuracy AUC Logistic Regression 0.671 0.736 Decision Tree (pruned) 0.681 0.665 Decision ...
4
votes
1answer
56 views

difference between novelty, concept drift and anomaly

Concept drift is when the relation between the input data and the target variable changes over time. like changes in the conditional distribution. is novelty an outlier? what should I think of? what ...
0
votes
0answers
29 views

KNN search of high-dimensional embedding vectors

I have a large database of n-dimensional numerical vectors. Each vector is the embedding of a vertex into n-dimensional vector space. Vertices belong to a graph, and an algorithm such as node2vec or ...
1
vote
3answers
61 views

Does One-Hot encoding increase the dimensionality and sparsity of dataset?

There are two ways to convert object datatype into numeric datatype, first is One-Hot encoding and second is simply map the numerical tags to different values. For example for column Age containing ...
0
votes
0answers
17 views

Does this make data leakage in time series? # need help for understanding time series data

Does this make data leakage in time series? I already read this, data leakage when scaling time series Data leakage is when information from outside the training dataset is used to create the model. ...
0
votes
1answer
14 views

Can you explain how these precision, recall and purity are calculated in this context?

I am studying to get my first job as data scientist and I stumbled upon this. I have been banging my head on the wall for hrs to understand this concept. But couldn't get concrete answer. Just 1 ...
0
votes
0answers
39 views

Is there any difference between data mining architecture and Knowledge discovery from data architecture?

I am looking at architectures of these two and they seem too similar to me. this is KDD architecture-: This is data mining architecture. I see no difference between them. in data mining architecture ...
0
votes
0answers
9 views

Differential Gene Expression Analysis problem

I have my own gene expression data that I am trying to perform differential gene expression analysis on. When I load it I can see all the columns. When I run Differential Gene Expression Analysis I ...
1
vote
1answer
23 views

How to determine the correlation using $X^2$ values

I am trying to understand how to calculate the correlation between two variables using p values. I am familiar with using the classic $X^2$ formula from the The Morgan Kaufmann series in data mining. ...
0
votes
1answer
27 views

MaxDiff Histogram usage

I would like to understand how the MaxDiff algorithm works. The only example that I could find online was a set of slides by Chris Crifton. Given An attribute C 12, 5, 5, 5, 12, 1, 1, 20, 20, 2, 2, 12,...
1
vote
0answers
9 views

Identifying the best group that are most productive

can you please give some idea to solve this problem, Below is my data, I want to identify the best inventory based on the CPM(target value) .. For example: I want to be able to say - Business section, ...
1
vote
2answers
71 views

Calculating the dissimilarity between term frequency vectors

Given that a document is an object represented by what is called a term frequency vector. How can we calculate the dissimilarity between term frequency vectors?
0
votes
1answer
16 views

Calculating the dissimilarity between two numeric attributes

I was wondering if someone could explain how to calculate the dissimilarity between two numeric attributes. Given that a numeric attribute is a a measurable quantity, represented in integer or real ...
1
vote
2answers
49 views

Selecting Training Set from Model CV Error

What I would like to do is recursively: Train the model on all data Remove the sample(s) with highest error Repeat until the remaining samples have an acceptable error The hypothesis is: "To ...
1
vote
1answer
409 views

Calculating the supremum distance

In this example I would like to figure out how to calculate the supremum distance. Given the example shown below I would like to know what f and h represent. Any help would be greatly appreciated.
1
vote
1answer
11 views

Calculating Dissimilarity between attributes

I would like to figure out how to calculate the dissimilarity between attributes Jack and Jim. Given the attributes table shown below Given the Relational table. and the example calculations I would ...
0
votes
1answer
32 views

Confusions about fp tree growth numerical-:

If fcam=3,Would conditional fp tree still be c=4 or what would it be? My guess is that it should be f=3,c=4,a=3,m=3. Am I right? Or what else would it be? Please guide. Also, in this below figure(same ...
1
vote
0answers
8 views

Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the ...
2
votes
1answer
99 views

How to cluster skills in job domain?

I have a problem related to clustering, where i need to cluster skill set from job domain. Let's say, in a resume a candidate can mention they familiarity with amazon s3 bucket. But each people can ...
2
votes
0answers
17 views

Who is supposed to label my sentiment analysis? Linguistics or psychologist?

I'm starting off my undergraduate research on text classification even though I'm still considered new to this topic. I've collected more than 20K data from Twitter. I've been trying to label the data ...
0
votes
2answers
16 views

where can we find the healthcare dataset [closed]

I'm researching Medical Data set on some diseases like tuberculosis,cirrhosis,chronic kidney disease with the columns of age, body mass index, cholesterol, glucose, blood pressure, and the like that. ...
0
votes
0answers
7 views

Categories for binning weather / meteorological data

For EDA purposes I would like to bin continuous weather variables (temperature, rainfall, etc.) into well defined intervals such as the ones you hear in your daily weather report such as: For ...
1
vote
0answers
13 views

Best Technologies opening Large Sets of Sensor Time-Series Data to Analytics

My team is exploring options to create a robust "analytics" capability that is well-suited for our large quantities of sensor test data. I'd appreciate any suggestions for technologies that ...
1
vote
1answer
37 views

How can I weight each point in one-class SVM?

I want to give weights to some data points Specifically, these are points related to anomalies (I'm implementing one-class SVM for anomaly detection) Exactly, I want to consider some data points that ...
0
votes
0answers
210 views

Why am I getting a different answer in Principal Component Analysis dimensional reduction?

Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8). Compute the principal component using PCA Algorithm. Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
22 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
29 views

What do we mean by permissible transformations in types of attributes-:nominal,ordinal,interval,ratio?

I am studying data mining and I stumbled upon types of attributes. They are Nominal Ordinal Interval Ratio Data mining book by Tan,Steinbech,Kumar says Permissible transformations for-: nominal-:...
0
votes
0answers
13 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
3
votes
1answer
104 views

Finding if an outcome is predictable

Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
0
votes
0answers
4 views

Why is an external resample + internal resample good in simulated annealing algorithm?

I have being taking notes of the following link about the simulated annealing method: https://bookdown.org/max/FES/simulated-annealing.html However I have been struggling to understand why this author ...
1
vote
0answers
26 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on a conference. I have a question There is a dataset below. this is temporal-spatial dataset. ...
2
votes
1answer
26 views

Removing Outliers within R Works on One Attribute But Not on Second

(Note: this is part of a homework assignment. I am not asking for a solution to the assigned problem, I am just confused by some behavior in R.) I have a dataset loaded into an R script as a dataframe....

1
2 3 4 5
23