Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Fraud Pattern Detection Dataset Creation

I have simulated a dataset for clicks on YouTube videos that records each click using dummy data. In the dataset, I collect information such as timestamp, ...
0 votes
0 answers
18 views

Resources on data science errors?

I am putting together some material on common mistakes that occur in data science. The problem, is people often don't talk about the mistakes that everyone has made. Are there any good resources of ...
  • 413
1 vote
1 answer
21 views

Converting numeric column to range

I have this data below. I want to convert temperature column into range of data like class interval and get the sum of sales in that range. How can I achieve that?
  • 13
5 votes
3 answers
864 views

How to understand data (What questions do you ask yourself)?

I'm curious as to see what questions other people ask themselves when faced with new data. What are some common questions you ask yourself when trying to understand the data or performing EDA? What ...
  • 75
0 votes
1 answer
27 views

How to train a regressor model on data that has duplicate subjects but different records for each?

I am working on a dataset that is as follows (just an example): prop_subj prop_comp bed_subj bath_comp sqft_subj sqft_comp A B 2 1 1002 1006 A C 2 2 1002 1075 A D 2 2 1002 1000 B G 2 1 1002 978 ...
1 vote
1 answer
25 views

Machine learning data leakage

I have a dataset with Information about a transaction that a customer made in the store. the target is Indication of whether fraud was committed in the transaction. One of my feature is ...
0 votes
1 answer
26 views

Sorting column in dataframe in each group in R

I have a timeseries dataset with price variables and want to sort them. The time ist structured into quarter hours. So each group is marked by 4 rows. I want to sort the first two columns within each ...
  • 1
0 votes
0 answers
24 views

Do you use categorical data types?

Personally I've never used the categorical data type in pandas, and leave eveything as objects. I've seen it has the capability to be saved as parquet files, saves data etc... What are the pros and ...
  • 413
0 votes
0 answers
11 views

Contextual Embeddings LSTM DOUBT

I have a simple doubt. When we use Word2Vec, Its obviously a non contextual embedding because every word has a same representation. When I pass it to my LSTM, We say the hidden states are the ...
  • 109
0 votes
0 answers
13 views

Task Distribution

I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
  • 1
1 vote
0 answers
11 views

Data filtering framework

I have procurement data that needs to be labeled with product categories. It's tabular data, containing 700k rows and a mix of data types (dates, free text, floats, etc.) The product set we currently ...
0 votes
0 answers
58 views

train_test_split error: Found input variables with inconsistent numbers of samples

I am trying to split a dataframe into training and test set and I am getting this error: "train_test_split error: Found input variables with inconsistent numbers of samples: [8819, 2795]". ...
0 votes
0 answers
5 views

Can I know if there's anyway to download Tableau report from server along with Title and Description in excel form?

I would like to download a Tableau report with the title footer and description written in Crosstab as it gets in PDF?
  • 1
0 votes
0 answers
4 views

How to create a representative dataset for Object Detection?

Consider that we are building a dataset for a network to localize and classify roadsigns. Furthermore, let's say we only care to classify two very similar types of road signs, a 30 and 40 mph speed ...
0 votes
0 answers
10 views

Equivalent of Presto's 'histogram' function in Azure SQL

I want to get an equivalent function to Presto's 'histogram' which just aggregates counts into a histogram like json, e.g if I have a table called table that looks ...
1 vote
1 answer
56 views

More representative data set OR higher model evaluation metrics?

A colleague and I are working on a churn model and reached an impasse: Our data set is for a global product. We've been asked to look at the US market only. When we subset the data to the US only, the ...
0 votes
0 answers
5 views

Date on visual x axis does not match data set view

My data table reads with a full date (i.e., June 2021), and the visual report won't display the year - only the Month. I'm new to pbi, however still think I should know how to make this work! Example ...
  • 1
0 votes
0 answers
23 views

Why my neural network model is not able to learn a simple linear function?

I'm trying, using a neural network, to predict a simple relation $ f(x) = 1-x $. I write my function this way: $f(x) = 1 - x^{+} + (-x)^{+} $ or in a more data-scientistic way: $f(x)= 1-Relu(x)+Relu(-...
0 votes
0 answers
10 views

How do you calculate the retention rate given daily installs and daily active users

Example table Date. Installs DAU 13/3/2020 150. 150 14/3/2020 100. 125 15/3/2020 150. 200 16/3/2020 100. 225
0 votes
1 answer
13 views

Is it a good idea to add predictable features in dataset before training?

For example, let's say a dataset has 100+ columns. And the ml has to recognize a complex pattern containing 1000 different conditions. And one of those conditions is, the integer from the first column ...
  • 123
0 votes
0 answers
22 views

Data Scraping Government Site

As I know it may be okay or may not be okay to data scrape a government website. Would the following site be okay to data scrape? https://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist I ...
  • 1
0 votes
0 answers
5 views

Algorithms for single output based om multiple inputs

I have a data set in which for every single x value, I have two y outputs. Example: X1=2 gives Y1=3,4; X2=9 gives Y2=122,89; X3=19 gives Y3=245,109; Now, given two y values for each x value, I want to ...
0 votes
1 answer
27 views

Python-SQL-Excel : When to use them?

I am currently learning data science and learned already a lot of Python not just for data science, also for script writing, web scraping etc. Meanwhile i started to look for how people actually earn ...
0 votes
0 answers
12 views

Where can I find this Data for the recent year period?

Im looking to do some data analysis with the most recent US census/estimates. Im looking for this exact data in the link below . This data mostly spans time period 2007-2013. ...
  • 101
1 vote
0 answers
13 views

Clarity on the scope of a data analyst and data scientist [closed]

I am new to the field of data analyst and I need clarity on the scope of work it covers. Say there's a dataset consisting of reps and weights of free weights lifted by a group of users over time and ...
  • 11
0 votes
0 answers
8 views

Bug created more historic instances than expected and I have to clean the mess

CONTEXT: We wanted to save historic data from a table in the database (that we'll call X) in a way that it didn't consume much memory. So we wrote a logic that ...
0 votes
0 answers
15 views

What model to use to forecast on small dataset with many parameters?

I have a dataset of 20 columns/parameters (x's) and not many rows (10-20 historical values for each x) I need to use to predict a y column for the future 5 years. Each row represents one year. What ...
  • 1
-1 votes
1 answer
21 views

How best can I determine the target rate for a KPI metric?

I am working on a project where I evaluate the Key Performance Index (KPI) for a solar company. The aim to see how the company is performing generally and see if it is efficient. Some of the metrics ...
0 votes
1 answer
19 views

Data preprocessing for Multiple Linear Regression Problem

For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log. ...
1 vote
1 answer
43 views

How do I specify encoding in scikit-learn OrdinalEncoder?

Scikit-learn object OrdinalEncoder() allows the user to create a lineary based encoding principle for ordinal data, however the the codes are encoded randomly. Is there any way I can specify how the ...
  • 11
0 votes
1 answer
10 views

Why does the PCA Scores plot (PC1 vs PC2) flips when using extracted variables from the Gaussian?

Imagine you have some multivariate data (1000s of variables) which approximately follows the Gaussian distribution. You can generate various PCA Scores plots from this data, of course. One option is ...
  • 21
1 vote
0 answers
21 views

How to obtain data for warehouse throughput forecasting project?

I am trying to build a machine learning model to predict warehouse throughput. I do not have any domain information or data since I am supposed to build a generic prototype for my clients. I am going ...
  • 11
0 votes
1 answer
17 views

Missing Values Correlation

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
0 votes
1 answer
47 views

How to deal with imbalanced categorical variables in regression tasks?

I want to predict real estate prices using several Machine Learning algorithms. My dataset contains numerical and categorical predictors. I already eliminated the outliers of numerical variables. Now ...
0 votes
0 answers
12 views

How can I map the amplitude of an ECG signal in relation to the time stamp?

I'm really new to programming, so can anyone suggest a way to process ECG signals from the MIT-BIH Arrhythmia database that allows me to map the amplitude and time together. Essentially, I want an ...
0 votes
1 answer
34 views

How to Define KPIs for a product solution

I have to find certain KPIs for a given product solutions and describe data tables to track those KPIs. Context: Our practitioners conduct treatment X in 3 steps with patient, so it leads to 3 ...
1 vote
2 answers
48 views

Converting voting district GeoID to approximate zip code?

How do you convert a "GEOID", representing a US voting district, to an approximate ZIP code? I've extracted the GEOIDs from the JSON published by the New York Times of the results of the ...
  • 121
0 votes
1 answer
17 views

Creating a grid type 3D data array from data points [closed]

I have a 3 data column $(X, Y, Z)$ ranges from $(min, max)$. For example, $X = (0, 5)$, $Y=(0, 3)$, $Z=(0, 2)$. By using them I need to create a numpy array in the form of $[(0, 0, 0), (0, 0, 1), (0, ...
  • 113
0 votes
0 answers
11 views

Selecting Right data and amount of data

Multiple Data Sources are needed to make Credit or Fraud Decision on (loan/credit card) applications. But each data pull costs money to buy. So how do we optimize a) Number of data sources to pull ...
0 votes
1 answer
19 views

Goodness on test or train set?

I split my data set before on train (80%) and test (20%) splits. Trained logistic regression model on the train set. Now, want to check the goodness of fit using the Chi-square likelihood omnibus test,...
  • 11
0 votes
0 answers
13 views

How to Calculate degrees in un-directed network and plot it? | Using python COLAB [duplicate]

I have loaded txt file (com-dblp.ungraph.txt) which includes DBLP collaboration network to a COLAB notebook. I found 317080 nodes & 1049866 edges in the network by reading the txt manually. Trying ...
0 votes
0 answers
36 views

How to calculate dataset and feature sparsity/density

I have a dataset with 8 features and 30,000 samples but which is probably a sparse sampling. I would like to quantify how sparse or dense the dataset and individual features are, as described in the ...
  • 101
5 votes
4 answers
386 views

Calculating Potential Usefulness of Acquiring Additional Data [closed]

Imagine Anne has a labeled training dataset for a machine learning prediction problem. There is an opportunity to acquire more data from an agent, at a cost. However, before she decides to acquire ...
  • 59
0 votes
1 answer
18 views

Data Mining of unresearched data for a master's degree final project

So, I have to start thinking about the topic of my final project in a data science master's degree (business oriented, although I can choose any unrelated field) and one of the requirements is to ...
1 vote
0 answers
10 views

Data collection after the model is built and deployed

I have built a machine learning model which predicts whether a customer will buy a product or not. The model performs well on cross validation tests. Now, I will deploy it in production to recommend ...
  • 123
0 votes
0 answers
17 views

Sensor data completeness

We have several sensors collecting data with a specified frequency (usually every 20 minutes but it varies) over a period of many years. The person that programmed them made sure, as a space saving ...
0 votes
0 answers
45 views

Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()

Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
0 votes
0 answers
16 views

Implementation/Optimization of Genetic Algorithm

I have a few queries regarding my implementation of the Genetic Algorithm (GA). I have a lot of parameters in which I have to find the best combination of these parameters to maximize the value of the ...
0 votes
0 answers
23 views

Software aid in plotting dataset and data table relations

I am at a complete loss of where to start with this in terms of where to start to look and what tags I should mark this with. I am working on gathering data from different sources with different ...
  • 1
1 vote
1 answer
35 views

NLP logistic regression

A basic doubt I have, Usually when dealing with text data for classic ML algos, We use Tf-idf which uses entire vocabulary for each row and assigns some weighted value. My doubt is can I use 5 feature ...
  • 11

1
2 3 4 5
16