Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
861
questions
0
votes
0
answers
15
views
Machine learning features - detect mean shift using expected values
I have an LGBM model that trains on machine learning features and predicts some numerical outcome with new data coming periodically. I would like to make sure that the new input data is "similar&...
- 1
0
votes
1
answer
31
views
Finding points in 3D space - SVM
In the SVM classification, we use planes to classify the labels points if the dataset has 3 input features. We need to use planes when input features are 3. I am describing a toy dataset with 3 input ...
- 359
0
votes
1
answer
30
views
Fraud Pattern Detection Dataset Creation
I have simulated a dataset for clicks on YouTube videos that records each click using dummy data. In the dataset, I collect information such as timestamp, ...
- 195
1
vote
0
answers
35
views
Resources on data science errors?
I am putting together some material on common mistakes that occur in data science.
The problem, is people often don't talk about the mistakes that everyone has made.
Are there any good resources of ...
- 435
1
vote
1
answer
41
views
Converting numeric column to range
I have this data below. I want to convert temperature column into range of data like class interval and get the sum of sales in that range. How can I achieve that?
- 13
6
votes
3
answers
943
views
How to understand data (What questions do you ask yourself)?
I'm curious as to see what questions other people ask themselves when faced with new data. What are some common questions you ask yourself when trying to understand the data or performing EDA? What ...
- 85
0
votes
1
answer
30
views
How to train a regressor model on data that has duplicate subjects but different records for each?
I am working on a dataset that is as follows (just an example):
prop_subj
prop_comp
bed_subj
bath_comp
sqft_subj
sqft_comp
A
B
2
1
1002
1006
A
C
2
2
1002
1075
A
D
2
2
1002
1000
B
G
2
1
1002
978
...
1
vote
1
answer
43
views
Machine learning data leakage
I have a dataset with Information about a transaction that a customer made in the store.
the target is Indication of whether fraud was committed in the transaction.
One of my feature is ...
- 11
0
votes
1
answer
94
views
Sorting column in dataframe in each group in R
I have a timeseries dataset with price variables and want to sort them. The time ist structured into quarter hours. So each group is marked by 4 rows. I want to sort the first two columns within each ...
- 1
0
votes
1
answer
43
views
Do you use categorical data types?
Personally I've never used the categorical data type in pandas, and leave eveything as objects.
I've seen it has the capability to be saved as parquet files, saves data etc...
What are the pros and ...
- 435
0
votes
0
answers
25
views
Contextual Embeddings LSTM DOUBT
I have a simple doubt.
When we use Word2Vec, Its obviously a non contextual embedding because every word has a same representation. When I pass it to my LSTM, We say the hidden states are the ...
- 109
0
votes
0
answers
17
views
Task Distribution
I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
- 1
1
vote
0
answers
11
views
Data filtering framework
I have procurement data that needs to be labeled with product categories. It's tabular data, containing 700k rows and a mix of data types (dates, free text, floats, etc.) The product set we currently ...
- 245
0
votes
0
answers
823
views
train_test_split error: Found input variables with inconsistent numbers of samples
I am trying to split a dataframe into training and test set and I am getting this error: "train_test_split error: Found input variables with inconsistent numbers of samples: [8819, 2795]".
...
- 11
1
vote
0
answers
10
views
Can I know if there's anyway to download Tableau report from server along with Title and Description in excel form?
I would like to download a Tableau report with the title footer and description written in Crosstab as it gets in PDF?
- 11
0
votes
0
answers
20
views
How to create a representative dataset for Object Detection?
Consider that we are building a dataset for a network to localize and classify roadsigns. Furthermore, let's say we only care to classify two very similar types of road signs, a 30 and 40 mph speed ...
0
votes
0
answers
35
views
Equivalent of Presto's 'histogram' function in Azure SQL
I want to get an equivalent function to Presto's 'histogram' which just aggregates counts into a histogram like json, e.g if I have a table called table that looks ...
1
vote
1
answer
66
views
More representative data set OR higher model evaluation metrics?
A colleague and I are working on a churn model and reached an impasse:
Our data set is for a global product. We've been asked to look at the US market only.
When we subset the data to the US only, the ...
- 111
0
votes
0
answers
7
views
Date on visual x axis does not match data set view
My data table reads with a full date (i.e., June 2021), and the visual report won't display the year - only the Month. I'm new to pbi, however still think I should know how to make this work!
Example ...
- 1
0
votes
0
answers
29
views
Why my neural network model is not able to learn a simple linear function?
I'm trying, using a neural network, to predict a simple relation $ f(x) = 1-x $.
I write my function this way: $f(x) = 1 - x^{+} + (-x)^{+} $ or in a more data-scientistic way: $f(x)= 1-Relu(x)+Relu(-...
0
votes
0
answers
13
views
How do you calculate the retention rate given daily installs and daily active users
Example table
Date.
Installs
DAU
13/3/2020
150.
150
14/3/2020
100.
125
15/3/2020
150.
200
16/3/2020
100.
225
- 1
0
votes
1
answer
16
views
Is it a good idea to add predictable features in dataset before training?
For example, let's say a dataset has 100+ columns.
And the ml has to recognize a complex pattern containing 1000 different conditions.
And one of those conditions is,
the integer from the first column ...
- 123
0
votes
0
answers
69
views
Data Scraping Government Site
As I know it may be okay or may not be okay to data scrape a government website. Would the following site be okay to data scrape?
https://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist
I ...
- 1
0
votes
0
answers
30
views
Algorithms for single output based om multiple inputs
I have a data set in which for every single x value, I have two y outputs. Example:
X1=2 gives Y1=3,4;
X2=9 gives Y2=122,89;
X3=19 gives Y3=245,109;
Now, given two y values for each x value, I want to ...
0
votes
1
answer
119
views
Python-SQL-Excel : When to use them?
I am currently learning data science and learned already a lot of Python not just for data science, also for script writing, web scraping etc. Meanwhile i started to look for how people actually earn ...
- 1
0
votes
0
answers
15
views
Where can I find this Data for the recent year period?
Im looking to do some data analysis with the most recent US census/estimates.
Im looking for this exact data in the link below . This data mostly spans time period 2007-2013.
...
- 101
1
vote
0
answers
19
views
Clarity on the scope of a data analyst and data scientist [closed]
I am new to the field of data analyst and I need clarity on the scope of work it covers.
Say there's a dataset consisting of reps and weights of free weights lifted by a group of users over time and ...
- 111
0
votes
0
answers
8
views
Bug created more historic instances than expected and I have to clean the mess
CONTEXT:
We wanted to save historic data from a table in the database (that we'll call X) in a way that it didn't consume much memory.
So we wrote a logic that ...
0
votes
0
answers
26
views
What model to use to forecast on small dataset with many parameters?
I have a dataset of 20 columns/parameters (x's) and not many rows (10-20 historical values for each x) I need to use to predict a y column for the future 5 years. Each row represents one year. What ...
- 1
-1
votes
1
answer
32
views
How best can I determine the target rate for a KPI metric?
I am working on a project where I evaluate the Key Performance Index (KPI) for a solar company. The aim to see how the company is performing generally and see if it is efficient. Some of the metrics ...
- 1
0
votes
1
answer
39
views
Data preprocessing for Multiple Linear Regression Problem
For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log.
...
- 13
1
vote
1
answer
845
views
How do I specify encoding in scikit-learn OrdinalEncoder?
Scikit-learn object OrdinalEncoder() allows the user to create a lineary based encoding principle for ordinal data, however the the codes are encoded randomly. Is there any way I can specify how the ...
- 11
0
votes
1
answer
20
views
Why does the PCA Scores plot (PC1 vs PC2) flips when using extracted variables from the Gaussian?
Imagine you have some multivariate data (1000s of variables) which approximately follows the Gaussian distribution.
You can generate various PCA Scores plots from this data, of course. One option is ...
- 21
1
vote
0
answers
29
views
How to obtain data for warehouse throughput forecasting project?
I am trying to build a machine learning model to predict warehouse throughput. I do not have any domain information or data since I am supposed to build a generic prototype for my clients. I am going ...
- 11
0
votes
1
answer
162
views
Missing Values Correlation
Is it worth it to study missingness correlation between columns?
If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
0
votes
1
answer
56
views
How to deal with imbalanced categorical variables in regression tasks?
I want to predict real estate prices using several Machine Learning algorithms. My dataset contains numerical and categorical predictors. I already eliminated the outliers of numerical variables. Now ...
- 1
0
votes
0
answers
31
views
How can I map the amplitude of an ECG signal in relation to the time stamp?
I'm really new to programming, so can anyone suggest a way to process ECG signals from the MIT-BIH Arrhythmia database that allows me to map the amplitude and time together. Essentially, I want an ...
0
votes
1
answer
39
views
How to Define KPIs for a product solution
I have to find certain KPIs for a given product solutions and describe data tables to track those KPIs.
Context:
Our practitioners conduct treatment X in 3 steps with patient, so it leads to 3 ...
- 3
1
vote
2
answers
551
views
Converting voting district GeoID to approximate zip code?
How do you convert a "GEOID", representing a US voting district, to an approximate ZIP code?
I've extracted the GEOIDs from the JSON published by the New York Times of the results of the ...
- 121
1
vote
1
answer
172
views
Creating a grid type 3D data array from data points [closed]
I have a 3 data column $(X, Y, Z)$ ranges from $(min, max)$. For example,
$X = (0, 5)$, $Y=(0, 3)$, $Z=(0, 2)$. By using them I need to create a numpy array in the form of
$[(0, 0, 0), (0, 0, 1), (0, ...
- 123
0
votes
0
answers
12
views
Selecting Right data and amount of data
Multiple Data Sources are needed to make Credit or Fraud Decision on (loan/credit card) applications. But each data pull costs money to buy. So how do we optimize
a) Number of data sources to pull ...
0
votes
1
answer
25
views
Goodness on test or train set?
I split my data set before on train (80%) and test (20%) splits. Trained logistic regression model on the train set. Now, want to check the goodness of fit using the Chi-square likelihood omnibus test,...
- 11
0
votes
0
answers
25
views
How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]
I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually.
Trying ...
0
votes
0
answers
101
views
How to calculate dataset and feature sparsity/density
I have a dataset with 8 features and 30,000 samples but which is probably a sparse sampling. I would like to quantify how sparse or dense the dataset and individual features are, as described in the ...
- 101
5
votes
4
answers
404
views
Calculating Potential Usefulness of Acquiring Additional Data [closed]
Imagine Anne has a labeled training dataset for a machine learning prediction problem. There is an opportunity to acquire more data from an agent, at a cost. However, before she decides to acquire ...
- 59
0
votes
1
answer
26
views
Data Mining of unresearched data for a master's degree final project
So, I have to start thinking about the topic of my final project in a data science master's degree (business oriented, although I can choose any unrelated field) and one of the requirements is to ...
1
vote
0
answers
10
views
Data collection after the model is built and deployed
I have built a machine learning model which predicts whether a customer will buy a product or not. The model performs well on cross validation tests. Now, I will deploy it in production to recommend ...
- 123
0
votes
0
answers
17
views
Sensor data completeness
We have several sensors collecting data with a specified frequency (usually every 20 minutes but it varies) over a period of many years. The person that programmed them made sure, as a space saving ...
0
votes
0
answers
144
views
Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()
Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
0
votes
0
answers
41
views
Implementation/Optimization of Genetic Algorithm
I have a few queries regarding my implementation of the Genetic Algorithm (GA).
I have a lot of parameters in which I have to find the best combination of these parameters to maximize the value of the ...
