Questions tagged [data-mining]
An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.
1,186
questions
0
votes
0
answers
10
views
Looking for datasets on automobile parts information for machine learning
I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
- 1
0
votes
0
answers
12
views
Seeking datasets for training a Language Model on U.S. mortgage loan processes
I'm in the process of training a Language Model (LLM) and require datasets that encompass various aspects of the U.S. mortgage loan process. The model's aim is to understand and simulate decision-...
- 1
0
votes
0
answers
16
views
Pytrends package in python is not working correctly
I would like to download data for the keyword "Investment" for all countries from google trends. I wrote the following code:
...
- 1
0
votes
0
answers
13
views
Get accurate alias list of people without information on aliases
I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
- 13
0
votes
0
answers
10
views
Help with creating a panel dataset
I have some medical information of individuals spanning ten years which includes first name, last name, dob, and other variables (two/three that could help match observations i.e. state living, ...
0
votes
0
answers
14
views
How to choose the right typt of ANN architecture for a regression model
So, im working on a project where i am leveraging ai to get accurate price predictions in terms of houses and real estate properties. I would like to use an artificial neural network so now i have to ...
- 1
0
votes
1
answer
22
views
Clustering words with similar meanings
What methods are there to cluster words/word phrases with similar meanings together from a list of words/word phrases?
- 1
0
votes
0
answers
10
views
What is the best sampling strategy for correlation analysis?
I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context.
Thanks
- 21
0
votes
0
answers
34
views
Does clustering belong to the domain of data mining or to the domain of machine learning?
Question 1. Does clustering belong to the domain of data mining or to the domain of machine learning? Or to both domains?
Question 2. Depending on the answer to Question 1, could you please suggest a ...
- 101
0
votes
0
answers
27
views
Unable to create associations using Apriori association rule algorithm for my dataset where the support is 0.01, confidence=0.8 and maxlen=10
Here's the code that I'm using:
library(arules)
library(arulesViz)
library(tidyverse)
library(readxl)
library(knitr)
library(ggplot2)
library(lubridate)
library(plyr)
library(dplyr)
tr <- read....
0
votes
0
answers
34
views
Where to find the statement "time series discord is the best overall technique among all techniques" in Chandola et al.(2009)?
Senin et al. (2015) "Time series anomaly discovery with grammar-based compression" claim exactly the following (copied and pasted):
in a recent extensive empirical study by Chandola et al., ...
- 101
0
votes
0
answers
34
views
Using Bayesian statistics in time series forecasting
I would like to forecast demand count time series of taxi fleets at different locations on the map at different points in time. I.e. multivariate demand Time series forecasting.
Given hierarchinal ...
- 101
0
votes
2
answers
67
views
Been stuck on a DS problem. Just need to know whether this problem statement is solvable or not
I've been stuck on the following problem for weeks now. To be clear I'm not asking the community to provide a full solution. Just a few ideas or at least confirmation on whether this problem statement ...
- 173
0
votes
0
answers
55
views
Implicit feedback based recommendation - User - User recommendation
My dataset involves customer_id, item_id and Count of purchases. My datset size is small...we have 15 users and 317 items..
Currently, am trying to build a recommendation system based on user based ...
- 2,555
0
votes
0
answers
8
views
I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy
My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
- 1
0
votes
2
answers
60
views
How do I separate periodic data from time series data?
I am currently working on a classification task of gym exercises based on accelerometer data. I am trying modularize window extraction so I can train my model based on metrics within a window (which ...
0
votes
0
answers
24
views
Please give me suggestions to identify scammers from communication data
I have some communication behavior data of carrier subscribers, containing information on calls made, duration and base stations. What do I need to do with this kind of data or what kind of machine ...
- 9
-1
votes
2
answers
139
views
How to generate synthetic dataset with dependent and independent features?
I want to generate a synthetic dataset, in python, composed of 10 features. 5 are dependent among each others and 5 are independent.
Generate dependent features.
...
- 57
1
vote
1
answer
164
views
How to predict what someone will order?
Suppose Prof. X goes to a road side tea-coffee shop everyday at 5pm just after his office. After reaching there he tosses a coin, and places his order tea or coffee. The shop owner Y has been ...
- 173
0
votes
0
answers
29
views
SMOTE-Tomek links with imbalanced dataset
I have a question: if I want to use SMOTE-Tomek links for my imbalanced dataset, should I now apply the method before or after the split?
- 1
0
votes
2
answers
39
views
Data Analysis Process
How do you feed data from a data warehouse to Python for ad-hoc analysis?
My day-to-day work is to answer ad-hoc questions, and 95% of the data I need is in our data warehouse. I often query data from ...
1
vote
2
answers
163
views
what qualifies as a data leakage?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
- 2,555
4
votes
1
answer
147
views
Are imbalanced data problems solvable? [closed]
I am working as a data scientist for the past 2 years where I have worked on problems related to binary classification, revenue prediction etc.
In the past two years, I have had 2 problems that ...
- 2,555
0
votes
0
answers
21
views
Can TSNE and other visualisation methods separate multivariate normal blobs?
Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix.
Let's assume, that the densities of those distributions do not ...
0
votes
0
answers
20
views
How to assign a group to new data using k means?
can we assign a group or cluster to a new data(test data) using unsupervised learning techniques? For expamle k means clustering.
0
votes
1
answer
43
views
Hadoop, Spark and Cloud
It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
- 1
0
votes
0
answers
25
views
Anomaly detection on subset of categorical data
Imagine that I'm a health department, and I have a data stream that describes whether someone gets sick after eating food from a restaurant. Each data point in the data stream arrives 24 hours after ...
- 101
0
votes
1
answer
42
views
Study data science at university [closed]
Is it a waste to study data science in university, despite the advancement of artificial intelligence and automation of analysis?
regards
0
votes
0
answers
96
views
Orange Data Mining creating a target variable on a table at run time
In Orange Data Mining I have created a table at run time and I need to declare one of the columns as a target. I have seen all widgets, but with no solution. I think to use a Python Script but I don't ...
0
votes
0
answers
13
views
Gaining insights into commonly co-bought categories of products using association rules (frequent itemset mining)
Let's say I have a small database of supermarket transactions.
Each records consists of a tx with several bought products, each having a unique identifier (eg, 1,2,3,4,5 etc.). Running A-priori on ...
2
votes
1
answer
1k
views
Can I use LLM to explain codebase?
I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
- 158
0
votes
0
answers
20
views
GridSearchCV using best paramters
i have got a question, i want to use GridSearch on a MLP and my question is after finding out best parameters do i have to train my model with the data again with the best parameters ?
- 13
0
votes
0
answers
31
views
GitHub Archived Repositories
I'm trying to build a model that observes patterns of source control usage, from how many files are changed per commit, how many contributors there are, even semantic analysis on the commit messages.
...
- 101
1
vote
1
answer
45
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 125
1
vote
0
answers
47
views
Mining association rules between time series
I have a pandas dataframe that represents a time series. My time series is segmented over the phase type that the robot is performing (i.e. I have a column with the phase type per timestamp and the ...
- 45
0
votes
0
answers
22
views
Can support be accounted multiple time for same sequence in sequential pattern mining?
I want to find top-k frequent sequential patterns from a list of sequences. Order of occurrence matters here (subsequence (1,2) is not same as (2,1)). Suppose I have 2 sequences:
S1=[1,2,3,1,5,2]
S2=[...
- 1
2
votes
3
answers
455
views
How to forecast a timeseries with geolocation data?
I have created a dataset with my geolocations from the last three months. The data set contains longitude, latitude, and timestamp, with a frequency of every 5 minutes. Based on this data, I want to ...
- 93
0
votes
0
answers
30
views
Finding a relation between three variables
I am new to data mining and have learnt about association rules mining, classification analysis, cluster analysis and outlier analysis. So, to find relationship between three variables, regression ...
- 1
0
votes
1
answer
48
views
Classification of a noisy data
What method can be used to classify data in the following example?
There is a table (hundreds of strings and hundreds of columns). Several columns in this table uniquely allow you to classify each row:...
- 1
0
votes
0
answers
8
views
What models/techniques can I use to generalize industry specific datasets?
I have a few dictionaries pertaining to different industries (ie. tech, manufacturing, education, etc.). These dictionaries map phrases and keywords to a sentiment score. I'd like to create a ...
0
votes
0
answers
34
views
Refining AI problem statement - suggestions
I am looking for some guidance. My company is a electronic goods manufacturing company. We work with multiple distributors (around 7 distributors) across specific regions to sell our products. But ...
- 2,555
0
votes
0
answers
79
views
0
votes
0
answers
21
views
Creating song vectors from playlists
Given a large number of playlists, I would like to create song vectors using this data. Furthermore, I would like to measure the performance of the model, so that it can be optimized for some ...
- 111
1
vote
1
answer
40
views
How to increase retention?
As you might already know there is a concept of retention. Let's say I have created a game and today hundred people have downloaded my game. Let's say tomorrow 47 out of yesterday's hundred people are ...
- 121
0
votes
0
answers
7
views
Can Data Mining or traditional EDW also do Process Mining?
Can Data Mining also do Process Mining? Can frameworks and tools used for Data Mining or EDW be used for Process Mining?
Summarize the problem
I'm investigating Process Mining with a view to create ...
- 101
0
votes
0
answers
374
views
AttributeError: 'NoneType' object has no attribute 'longitude'
I'm trying to get the location longitude of a address using geopy.
normally this works completely fine.
...
0
votes
0
answers
21
views
Any ideas on deciding a hotel competitive group?
Suppose for each hotel, I need to find a group of hotels which are competitive to this hotel. Competitive hotels mean they probably share the same group of customers, and the customers will make a ...
1
vote
1
answer
75
views
revenue forecast using regression - what is the input for future?
I have a dataset with quarter wise revenue for past 3 years from Jan 2020 to Dec 2022. I have 4642 customers.
Each customer has 1 row of data which includes features based on his purchase frequency, ...
- 2,555
1
vote
0
answers
22
views
How to do unsupervised clustering on sentences to find intents
I am working on chatbot for students.
So, I have chatlogs on conversation between student and tutor, which is on mathematical problems (no labels).
First thing I want to find is intents, such as ...
- 23
0
votes
0
answers
8
views
User Actions in Browser mining
I'm looking for a way to collect user actions: opening new tabs, switching between tabs, entering text, following links in browser by user. it should log user actions as well as context: what was the ...
