Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.

0
votes
1answer
9 views

what is step by step if I want to search prediction from 4 datasets?

the fourth data is (train_data, test_data, previous_data, and information_history_data). The goals is searching user's rate on the loan to bank. I confused first step to doing because there many ...
0
votes
1answer
13 views

How can I train a many-to-one RNN with an array of 2D matrices?

I have eye tracking data for every word of a novel. Features for every word is given separately. I want to take groups of 100 words to make a sample and then use each of these samples as a single ...
0
votes
1answer
10 views

Parsing and storing a large amount of HTML data

I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch ...
0
votes
1answer
31 views

Binary Encoding of Ordinal Categories

I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame. I want to encode these ordinal categories in columns with this ...
1
vote
3answers
50 views

Test data significantly different from training data

What are the risks if the test data is significantly different from the training data? Is the most significant problem related to both?
0
votes
0answers
6 views

How to mine spatial telemetry data

I have a dataset of telemetry data that consists of -- Unique ID geohash DTG (yyyy-mm-ddTHH:MM:SS.000UTC) Lat Long and I'm trying to see if there are associations between ID's within the geohash ...
0
votes
0answers
7 views

Connecting R to OMBD [closed]

Can anyone guide me for the two questions 1) How to connect R with ombd API? 2) How to deal with English text having hindi words in it? Thanks in advance.
1
vote
1answer
23 views

TDIDT Decision Trees algorithm

What is the Difference between TDIDT, ID3, CART, and C4.5? My main concern is about TDIDT, Is it first ever algorithm that came with Decision trees? Is it predecessor or successor of ID3, CART, ...
2
votes
1answer
31 views

Calculate a ranking function from classification features

I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range). I obtained how important each feature was in ...
1
vote
1answer
18 views

when and where dummy.data.frame would be used?

I am learning this post, in which, the author gives a piece of code ...
0
votes
0answers
11 views

How to solve a tie in one-r?

when using the one r algorithm with this data a triple tie happens, age group = 10/12, fresh fruit = 10/12 and id = 10/12 why does weka choose age group? Age group: Senior -> Anchovy ...
0
votes
1answer
42 views

Cohort analysis over 3 months

I am trying to look at the customer retention & churn by using cohorts for an e-commerce usecase. From a business perspective, a client is defined as churned if it hasn't performed any ...
0
votes
0answers
11 views

What is the accepted level of persona coverage for Recommender Systems?

Almost all of e-commerce companies use recommender systems which involve a set of personas. (explained in this post). "Your personas will never cover 100% of your users" In practice, what is the ...
1
vote
0answers
18 views

similarity measures for categorical data in data mining

I am working on my assignment in which i have to mention 5 similarity measures for categorical and continuous data in data mining.As a beginner i tried my best and found SQUARE DISTANCE,EUCLIDEAN AND ...
0
votes
0answers
11 views

is it possible to run a Monte Carlo simulation with a single sample? I guess not

I am learning data science through a series of HarvardX courses. to elaborate Sample Correlation, the professor assumes A less fortunate geneticist can only afford to take a random sample of 25 ...
0
votes
0answers
5 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
0answers
10 views

Determine learned topics in text

I have a large number of texts (each about 1000 words). Every text contains a various number of topics, usually expressed as a sentence. I've extracted and categorize 22 topics and found 5000 of them ...
0
votes
1answer
25 views

What is the convergence criteria of a semi-supervised learning algorithm?

I would like to know when to stop doing semi supervision? For example, if I learn a classifier from a small dataset and then use it to label a pool of unlabelled dataset. I then use the newly labelled ...
5
votes
1answer
47 views

Text extraction from large pool of documents of different formats

I have a collection of 6 million documents stored on a hard drive (around 500GB of data storage). Those documents contain text, tables, images and come in different formats: pdf, jpg, png, rar, vsd, ...
0
votes
0answers
26 views

Root cause analysis on text data

I am working on a project to automate the process of root cause analysis for hardware devices. The input here is a text containing the problem description (as in an e-mail) and I have access to ...
0
votes
0answers
13 views

Topic modeling / classification for a list

I'm trying to categorize a list of keywords to have a better overview of the different topics I'm targeting (this is for search engine optimization). I know there are plenty of algorithms to read a ...
0
votes
1answer
10 views

NAB benchmark (Numenta Anomaly Benchmark)

I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where ...
0
votes
1answer
28 views

Random Forest Techniques/Models

Can anyone tell about different Techniques/algorithms of Random forest? I know, Random Forest is itself an algorithm/model, but I'm looking for another version of it as we have in decision trees. List ...
0
votes
0answers
8 views

Automated Spatial data mining tools in python

How can spatial data raster be processed in order to apply random forest prediction algorithm on it using python programming language?
0
votes
1answer
18 views

comparing data from first quarter in two years

How can I compare data from two quarters? What software can I use to do it parallelly side by side so I can compare them. What kind of factors can I take into consideration, I'm posting a sample data ...
2
votes
0answers
31 views

Scraping financial web data

I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's ...
1
vote
0answers
36 views

Is it a good idea to train Neural Network for classification on dataset where each document has a different class i.e. no class is repeated again?

My goal is to build a recommendation model for which I want to use Neural Network (LSTM). The user will give some input keywords and the model should return the suggestions (classes) based on ...
0
votes
0answers
24 views

Isolation forest parameters

I am implementing Isolation Forest from scratch, I am in dilemma of how to set contamination value, and what if not used im the code. Is accuracy differ? What other parameters required to run ...
0
votes
0answers
15 views

Are there any tools out there to use for exploratory analysis on semi structured data sources (non-relational)?

I.e. server log files and other machine generated data. As far as I know most visualization tools (Power BI, Tableau, Excel) generally expect tabular data. Are there similar tools for exploratory ...
1
vote
1answer
29 views

Decision trees With branching Factor?

How to construct a Binary Tree from a decision tree with branching factor greater than 2 (b>2)?
1
vote
1answer
10 views

Text Dataset Help: Need to figure out how to get the publication date of a text

I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the ...
1
vote
3answers
68 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
0answers
46 views

How to build a symptom checker and medical diagnose chat bot

I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work; Application - A , User - U ...
0
votes
1answer
21 views

Comparison between two items

I have table like: Sales 2010 2011 2012 2013 State X Y X Y X Y X Y 1 2 3 4 5 6 7 ...
0
votes
0answers
11 views

How can I cluster data by using multiple views of the data with a common column using OpenRefine?

I need to cluster my data, but there are multiple ways to view the data such that I can't really place it all into a single table. I have been working with Open Refine to cluster this data but I keep ...
0
votes
1answer
16 views

One attribute includes another attribute

I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, ...
0
votes
0answers
13 views

Data set descriptions for frequent item-set mining data set

Recently I started working on Frequent Itemset Mining and came across this link (http://fimi.uantwerpen.be/data/) hosting popular data sets that many research articles used to evaluate their itemset ...
1
vote
1answer
35 views

Why does a confusion matrix in Orange not show results?

I ran a neural network on a data set and when I checked the confusion matrix I got nothing but a message that says: Confusion matrix cannot show regression results.
1
vote
0answers
14 views

robots.txt communication in webmining

I hope this is the right subforum as I did not really find a suiting one. I'm mining data from a website that does specifically exclude some crawlers from it's site in the robots.txt like this: ...
0
votes
1answer
27 views

How to use Machine Learning to discover important biomarkers in an unbalanced small data set

I have a project which I am just starting out, I am only just learning machine learning and statistics so I am somewhat unsure as to what approaches will be good to start off with, and I am sorry if ...
1
vote
0answers
16 views

Is the Apriori algorithm suitable for database tuples?

Problem description: I need to use an association rule algorithm that lets me use database tuples and I think Apriori is a good option, but I am not sure. Starting point: I know the Apriori ...
3
votes
2answers
77 views

Avoiding the zero problem

I have a dataset and I'm trying to predict the label for my sample but I couldn't map it since that case never showed here is my sample (I'm using naïve Bayesian method) X=(Age=middle, has_job=false, ...
0
votes
2answers
38 views

Loops in R programming,

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
1
vote
0answers
43 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my LightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
0
votes
0answers
25 views

Filtering unique row values in SQL, Advanced

My data looks like this: Number(String), Number2(String), TransactionType(String), Cost(Integer) For number 1, Cost 10 and -10 cancel out so the remaining cost is 100 For number 2, Cost 50 and -...
0
votes
0answers
15 views

Data Mining in Practice: Analysing Real Estate

I need to analyse a data set of around 30,000 properties listed or sold over 2 years The main aim is to: Minimize risk and boost profit investments in a short-term market and a long-term strategy. ...
1
vote
1answer
41 views

What kind of data visualization should I use?

I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
1
vote
0answers
27 views

What are some good fields to research in data science? [closed]

I am recently working on my thesis. I had started with recommender systems, but figured out that there is nothing more to optimize in this field since researchers have found around 8.5 RMSE for ...
1
vote
1answer
15 views

does index of my data which is of type “Date time index” plays a part in reggression?

I'm new to data science and I'm working on a regression problem. My question is the index of my data which is of type "Date time index" plays a part in regression? I mean is it Okay if i drop the ...