Questions tagged [data-mining]
An activity that seeks patterns in large, complex data sets. It usually emphasizes algorithmic techniques, but may also involve any set of related skills, applications, or methodologies with that goal.
1,177 questions
0
votes
0
answers
16
views
getting ideas to start my learning
i have plan to start my career on data analytics and i need a guildline how to start and where to start ,if you are ready to give some hints through that I'll get some clarity and i'll start my ...
0
votes
1
answer
25
views
How to create a conversation dataset from a website without API?
I am currently doing my thesis on Natural Language Processing and it involves studying how people text online in a community so that it can be used to simulate conversational agents that can mimic ...
0
votes
0
answers
16
views
Performance Issues with Large Dataset for Frequent Itemsets Mining
I’m working on a data mining project, specifically focusing on mining frequent itemsets. My dataset contains 175 unique items and 1400 transactions, but I’m encountering significant performance issues ...
0
votes
0
answers
8
views
Creating object profiles based on their attributes
I'm working on a recommendation system to suggest alternative cities based on how similar city A is to the recommendations. To do this, I gathered information about each city's different points of ...
- 1
0
votes
0
answers
8
views
Creating an Index that measures sum AND count/frequency
I work at an R1 university and am trying to synthesize a dataset that has information about grants that faculty here have won. I want a quick-and-dirty index that balances two things: how much a given ...
2
votes
1
answer
154
views
What is appropriate Individual KPI for AI projects?
I work in the sales department of electronics component manufacturing company and we do data science projects using traditional algorithm like Random forests (success likelihood of design project), ...
- 2,655
0
votes
0
answers
20
views
Accessing PubMed Central full-texts via FTP?
I wish to process full-texts of many articles using europepmc::epmc_ftxt (so that I can later use tidypmc::pmc_text and tidypmc::separate_text). I find that the R coding (a) below is much too slow in ...
- 11
0
votes
0
answers
122
views
Change point detection in a timeseries
I have a few questions about using the PELT algorithm for change point detection in the ruptures library.
Optimal Penalty Value for PELT Algorithm
I'm using the Pruned Exact Linear Time (PELT) ...
- 1
1
vote
0
answers
101
views
How to implement hard negative mining for PyTorch RetinaNet_v2 object detection model?
I am currently using the PyTorch object detection model retinanet_resnet50_fpn_v2. I want to implement hard negative mining, such that I take the top X images with ...
- 11
0
votes
0
answers
14
views
i am facing problem in combining the forecasts generated by multiple stacked GRU models
first i have used CEEMDAN, to decompose multi-variate data into different IMFs, after that for each IMF stacked GRU model is trained, which gives results for each IMFs,
for the final predictions i ...
0
votes
0
answers
26
views
The origin of NaN in the dataset
I train on a dataset with Kaggle from Netflix, where information about the series is provided. I am interested in a question about NaN in the column with the name Director. I want to try to understand ...
0
votes
0
answers
10
views
Feedback on this plan to make a data pipeline
I am picturing something like the following:
List internet sources you want to acquire data from.
Study the source code structure for each site. Try to build an “extractor” for each individual site ...
- 131
0
votes
0
answers
50
views
How to visualize a decision tree classifier?
I'm using ID3 algorithm to build a classifier and was wondering if there is any way to visualize the decision tree that the algorithm builds.
This is my code for a decision tree in Python:
...
- 11
1
vote
1
answer
30
views
How to match the two datasets on Electoral bonds given by SBI?
Just stumbled across the Electoral bond dataset https://www.kaggle.com/datasets/shaundanielll/electoral-bond-data-state-bank-of-india
There are two csvs- One contains Encashment data(Date of ...
- 11
0
votes
0
answers
8
views
0
votes
0
answers
109
views
How to Detect and Identify German Compound Words in a Text using Python
I would like to know if there is any python libraries or packages available which can help me detect and extract German compound words. I took a look at nltk and spacy packages and they don't seem to ...
0
votes
1
answer
69
views
How to plot PCA loadings vs spectral region
I am trying to plot the PCA loadings vs spectra. So far I can retrieve the PCA coordinates per observation. In a manual, I saw there was a widget called curves to do this. It does exactly what I need ...
0
votes
0
answers
26
views
How to find optimized combinations of values in input columns that produces the values in output column
I have multiple input columns that produce an output column. I want to find optimized combinations of the values in the input columns that produce the values in the output column.
For eg:
So, the ...
- 1
0
votes
0
answers
19
views
How do you actually train a model to make predictions on new inputed data?
I created a system for my thesis, this system can predict whether something will succeed or fail. I use k-fold cross validation to see the performance, then the model that will be used to predict new ...
0
votes
0
answers
11
views
SubGroup Discovery on unseen data
The technique of subgroup discovery in data mining is known for using Minimum Description Length (MDL) and Rule-Based Induction to calculate the complexity of a learner and divide a dataset into ...
0
votes
1
answer
26
views
Working with functions [closed]
What are some good sources for learning how to deal with functions in Python and R. I have used them a lot of times and they can be a bit challenging. I like to write out a plan of what I want done ...
1
vote
0
answers
126
views
Pytrends package in python is not working correctly
I would like to download data for the keyword "Investment" for all countries from google trends. I wrote the following code:
...
- 11
0
votes
1
answer
318
views
Clustering words with similar meanings
What methods are there to cluster words/word phrases with similar meanings together from a list of words/word phrases?
- 1
0
votes
0
answers
40
views
Does clustering belong to the domain of data mining or to the domain of machine learning?
Question 1. Does clustering belong to the domain of data mining or to the domain of machine learning? Or to both domains?
Question 2. Depending on the answer to Question 1, could you please suggest a ...
- 103
0
votes
1
answer
49
views
Where to find the statement "time series discord is the best overall technique among all techniques" in Chandola et al.(2009)?
Senin et al. (2015) "Time series anomaly discovery with grammar-based compression" claim exactly the following (copied and pasted):
in a recent extensive empirical study by Chandola et al., ...
- 103
0
votes
0
answers
71
views
Using Bayesian statistics in time series forecasting
I would like to forecast demand count time series of taxi fleets at different locations on the map at different points in time. I.e. multivariate demand Time series forecasting.
Given hierarchinal ...
- 101
0
votes
2
answers
79
views
Been stuck on a DS problem. Just need to know whether this problem statement is solvable or not
I've been stuck on the following problem for weeks now. To be clear I'm not asking the community to provide a full solution. Just a few ideas or at least confirmation on whether this problem statement ...
- 173
0
votes
2
answers
149
views
How do I separate periodic data from time series data?
I am currently working on a classification task of gym exercises based on accelerometer data. I am trying modularize window extraction so I can train my model based on metrics within a window (which ...
0
votes
0
answers
26
views
Please give me suggestions to identify scammers from communication data
I have some communication behavior data of carrier subscribers, containing information on calls made, duration and base stations. What do I need to do with this kind of data or what kind of machine ...
- 9
-1
votes
2
answers
359
views
How to generate synthetic dataset with dependent and independent features?
I want to generate a synthetic dataset, in python, composed of 10 features. 5 are dependent among each others and 5 are independent.
Generate dependent features.
...
- 57
1
vote
1
answer
197
views
How to predict what someone will order?
Suppose Prof. X goes to a road side tea-coffee shop everyday at 5pm just after his office. After reaching there he tosses a coin, and places his order tea or coffee. The shop owner Y has been ...
- 155
0
votes
2
answers
59
views
Data Analysis Process
How do you feed data from a data warehouse to Python for ad-hoc analysis?
My day-to-day work is to answer ad-hoc questions, and 95% of the data I need is in our data warehouse. I often query data from ...
2
votes
2
answers
786
views
what qualifies as a data leakage?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
- 2,655
5
votes
1
answer
585
views
Are imbalanced data problems solvable? [closed]
I am working as a data scientist for the past 2 years where I have worked on problems related to binary classification, revenue prediction etc.
In the past two years, I have had 2 problems that ...
- 2,655
0
votes
0
answers
33
views
Can TSNE and other visualisation methods separate multivariate normal blobs?
Consider we have two classes of points. Both of them come from a multivariate normal distribution with an unrestricted covariance matrix.
Let's assume, that the densities of those distributions do not ...
0
votes
0
answers
25
views
How to assign a group to new data using k means?
can we assign a group or cluster to a new data(test data) using unsupervised learning techniques? For expamle k means clustering.
0
votes
1
answer
134
views
Hadoop, Spark and Cloud
It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
- 1
0
votes
1
answer
48
views
Study data science at university [closed]
Is it a waste to study data science in university, despite the advancement of artificial intelligence and automation of analysis?
regards
2
votes
1
answer
3k
views
Can I use LLM to explain codebase?
I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
- 158
0
votes
0
answers
64
views
GitHub Archived Repositories
I'm trying to build a model that observes patterns of source control usage, from how many files are changed per commit, how many contributors there are, even semantic analysis on the commit messages.
...
- 101
1
vote
1
answer
57
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 125
1
vote
0
answers
57
views
Mining association rules between time series
I have a pandas dataframe that represents a time series. My time series is segmented over the phase type that the robot is performing (i.e. I have a column with the phase type per timestamp and the ...
- 45
2
votes
3
answers
2k
views
How to forecast a timeseries with geolocation data?
I have created a dataset with my geolocations from the last three months. The data set contains longitude, latitude, and timestamp, with a frequency of every 5 minutes. Based on this data, I want to ...
- 93
0
votes
1
answer
115
views
Classification of a noisy data
What method can be used to classify data in the following example?
There is a table (hundreds of strings and hundreds of columns). Several columns in this table uniquely allow you to classify each row:...
- 1
1
vote
1
answer
48
views
How to increase retention?
As you might already know there is a concept of retention. Let's say I have created a game and today hundred people have downloaded my game. Let's say tomorrow 47 out of yesterday's hundred people are ...
- 121
0
votes
0
answers
594
views
AttributeError: 'NoneType' object has no attribute 'longitude'
I'm trying to get the location longitude of a address using geopy.
normally this works completely fine.
...
1
vote
1
answer
90
views
revenue forecast using regression - what is the input for future?
I have a dataset with quarter wise revenue for past 3 years from Jan 2020 to Dec 2022. I have 4642 customers.
Each customer has 1 row of data which includes features based on his purchase frequency, ...
- 2,655
1
vote
0
answers
41
views
How to do unsupervised clustering on sentences to find intents
I am working on chatbot for students.
So, I have chatlogs on conversation between student and tutor, which is on mathematical problems (no labels).
First thing I want to find is intents, such as ...
- 33
1
vote
1
answer
30
views
Empirical indications regarding demanded skills and tasks of data science jobs?
I am wondering if there are is any information about the current (and prospected) shares in skills required for advertised/existing data science jobs. This includes of course also the concrete tasks ...
- 111
1
vote
1
answer
39
views
Data Analytics Documantaions
I am working as a data analyst in a company. Me and my colleagues use different tools and software to analyze the data and make the reports (e.g., Excel, Python, R, Alteryx, SQL, Tableau).
Each one ...
- 2,015