Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
581
questions
0
votes
0answers
3 views
Ideas on handling large quantities of “waste” data
Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
0
votes
0answers
10 views
Clarification on an anomalous dataset
Suppose I am concerned with anomaly detection on time series data for a signal, such as vehicle speed data.
Is it a valid approach to train a machine learning model, for example, on datasets of ...
0
votes
1answer
12 views
County-level 2020 election data? Help!
I was using the data from favstats' github where he scraped NYT data, but it's gone now for whatever reason. Does anyone know where I could find this data? I'm in the middle of a huge project and didn'...
-1
votes
0answers
19 views
Normal distribution histogram [closed]
How I can create histogram for normal distribution without using built-in function in python?
The user should enter the value of mean and std first....
-1
votes
1answer
24 views
Can we identify that an academic dataset was used for commercial purpose [closed]
There are many datasets released on the internet. Authors of many of these datasets state that the datasets are strictly for academic usage and not for commercial purposes. Although some datasets are ...
0
votes
1answer
33 views
How to input sets as features
Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on)
This is from the online order processing domain. An order consists of a set of variable ...
0
votes
0answers
10 views
Usage of Doc2Vec as feature extractor for text classification of websites with political articles
I have gathered political articles from polish websites for my engineering thesis. The main goal is to try to predict the website that input text belongs to. So for this few websites I want to create ...
0
votes
1answer
17 views
Hand preprocessing data
I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going ...
0
votes
1answer
17 views
How to increase sales and revenue of a Client?
I was asked this in an interview for a Data Scientist position:
Lets say Holland and Barret came to you and said they'd like to increase their sales and revenue. How will you go about it?
My answer ...
1
vote
0answers
11 views
Marketing Channel Recommender System [closed]
This dataset is collected by a drug making company trying to sell its drug to doctors of different specializations.
The drug company has made promotional activity for its brand
The promotional ...
1
vote
1answer
32 views
adding a feature as “generic”
I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:...
0
votes
0answers
5 views
How to change data source names in Google Data Studio
Google Data Studio has connectors to MySQL, PostgreSQL, etc. but these connections will come with default names.
Couldn't find out how to set the names for data sources in Google Data Studio. Is it ...
1
vote
1answer
14 views
How to conclude the generality of any classification methods?
Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$.
The task $A$ is measured by a consistent measure.
For instance, the task A can be a binary classification. In this ...
0
votes
1answer
15 views
How to sample with guaranteed minimum spacing?
I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
0
votes
1answer
11 views
Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations?
Use cases and Solutions surrounding GPT's have taken NLP world with storm and started the GPT-Best vs GPT- Not So Best war on the internet. There are solutions been derived from API's provided by HF. ...
0
votes
1answer
22 views
Model inclines because of imbalanced data
In life, some events are rare and most cases are normal. So I am wondering, to detect the rare cases, shall we use an imbalanced dataset with more historical rare cases?
Taking the German Credit Data, ...
0
votes
0answers
7 views
Generating prediction when adding new rows into a relational dataset
There is a PostgreSQL table that I regularly update with new book reviews. Meanwhile, I also have trained a Tensorflow model that takes book review text and return a predicted book review category (an ...
2
votes
1answer
45 views
Can we use two independently measured features in a same ML model? [closed]
Two features are measured at different times but belong to the same target. In which conditions or form these features can be modeled together? Or they shouldn't be used in the same model but modeled ...
1
vote
0answers
13 views
How to aggregate Weather Data from county to state level? [closed]
I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident ...
1
vote
0answers
19 views
Generating synthetic data based off existing real data (in Python)
I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
1
vote
0answers
12 views
How much information is produced in our Society? [closed]
I was reading an interesting article about the growth of data in our society:
https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x
Does any of you know if there exists a more ...
2
votes
0answers
25 views
How to improve model performace when model shows a systemic pattern in residues
I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
0
votes
0answers
11 views
Positive records : no behavioral data
0
I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
-1
votes
1answer
34 views
What algorithm should I use to get a mapping between two variables? [closed]
I have a dataset that contains for every row, a list X of x items, that is a subset of X_total, and a list Y of y items, that is a subset of Y_total.
...
1
vote
0answers
13 views
How to use time-sequence data with “meta data” of single value per sample?
I'm, trying to estimate a fish weight by a time-sequence signal of the change in resistance when the fish goes through a gate with electrodes installed.
When the fish pass through the gate there's a ...
1
vote
0answers
20 views
Recommendation Systems User Profile Streaming Data on GCP
I have a recommendation system that recommends articles to different users. I am planning to provide the recommendations in an off-line fashion. Where I already have a table in BigQuery which has the ...
1
vote
1answer
21 views
Is window based sequencing a good idea to obtain more training data for LSTMs?
I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier.
However,...
1
vote
2answers
44 views
Problem with converting string to dummy variables
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
0
votes
1answer
55 views
How to build a database of image data for machine learning?
I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database ...
2
votes
2answers
27 views
Log analysis dataset with labeled cybersecurity issues
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
1answer
14 views
Automating the training and deployment of the model as more data becomes available
My question is about automating the training of the model as more data becomes available.
In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ...
2
votes
1answer
35 views
What's the Seminal Book on data science? [closed]
Can people provide the best textbook they have used for learning data science? I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a ...
0
votes
1answer
76 views
Machine Learning: Predicting target based on a feature
I have a df looks as follow:
-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
2
votes
1answer
35 views
How to account for reduced student capacity when calculating program retention?
I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am ...
1
vote
1answer
55 views
Labelling a Time series dataset
I was asked this during an interview that I did not crack.
You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one ...
0
votes
1answer
27 views
What is the best way to visualize range of data?
If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
...
0
votes
0answers
15 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
0
votes
0answers
9 views
Link Recommendation System
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
1
vote
4answers
33 views
Train a model using multiple data sources
I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is ...
-1
votes
1answer
32 views
Equivalence of two separate data distribution
I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
0
votes
1answer
15 views
Compare two tends with big difference in absolute value
I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners.
So far i retrieved my data, and plotted it.
Obviously, since the various ...
0
votes
0answers
9 views
Data Generation for LSTM with variable time input
So, I have a dataset that observes entities for 60 time periods. At each time period, I collect some X observations per entity.
So I have an LSTM to predict some element in the 60th time period using ...
1
vote
0answers
15 views
Triplet optimization producing a weird diagonal line?
I'm pretty sure this is the right forum for this, or let me know otherwise, I'll happily move this to a better place.
I have a strange problem. I've written an algorithm designed to take three files ...
2
votes
1answer
27 views
Column vector for training example
I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a ...
1
vote
0answers
26 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
0
votes
1answer
17 views
Tagged dataset with photos for race detection
Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as ...
0
votes
0answers
49 views
Machine learning applications in Architecture field
Can there be Machine learning applications in Architecture field?
Example : New York City 100 years back i.e. Year 1920, every ten years after 1920 and today i.e. Year 2020 images of New York City ...
1
vote
0answers
30 views
Association Rules
I'm using association rules for a project and noticed that there is a dearth of papers in the last 10-15 years on the topic although it seemed really popular 15-25 years ago. Is there any reason why?
2
votes
3answers
47 views
Combining Two CSV's in Jupyter Notebook
I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
0
votes
0answers
8 views
Are there labeled multivariate time series data sets where only a subset/partial number of time series is relevant for each class?
I am looking for a data set of multivariate time series data which contains classes that only require a subset of time series for their identification. Assume a human activity recognition data set ...
