Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
545
questions
0
votes
1answer
18 views
Higher variance in PCA can mean, that data structure is less informative?
I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with ...
0
votes
0answers
11 views
Normalization around the nominal value while avoiding to get too big results
Lets say we measured a signal N times. Each measurement contains T time instance. For example for N=4 measurements and T=5 time instances the data matrix is as follows:
...
0
votes
1answer
20 views
How to adjust/smooth a certain number using constants or rules
Hi, I am handling a dataset with a customer purchase history.
The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage.
There are two ...
2
votes
2answers
27 views
Data Imbalance in Regression Tasks [closed]
Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks.
Particularly,
- What is the procedure to check for ...
3
votes
2answers
154 views
Is it acceptable not to transform() test data after train data is being fit_transform()-ed
We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
1
vote
0answers
19 views
List of naughty faces?
An Australian sporting league simulated live crowds using cardboard cut-outs to fill empty stadiums during COVID-19. Fans were allowed to submit their faces to be used on the cutouts, which resulted ...
0
votes
0answers
10 views
What's the best way to store and then call 3 Million+ records for training?
Good afternoon all,
I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).
Essentially I've built a function that can convert raw data into ...
0
votes
1answer
23 views
Continuous VS Categorical variable
In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are highly correlated.
Which method shall I proceed to identify the feature to be dropped AGE OR ...
0
votes
1answer
11 views
Tools for nutritional value tracking and visualization
I want to track the nutritional value of different flours over time. What I got so far is a
data table like this:
...
0
votes
0answers
7 views
Resources for building data pipelines
I am finding it difficult to find sources which walk you through building a data pipeline from scratch, for example from:
- SQL Server (where I want to do my cleaning/ETL)
- R/Python (where I want to ...
0
votes
1answer
44 views
Data augmentation for multiple output heads in Keras
I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
0
votes
0answers
15 views
Pandas retrieve unwanted things later [closed]
If I'm trying to make some data processing and I want to hide unwanted symbols or html codes during processing stage and I want to popup those unwanted things again on their origin place within the ...
-1
votes
0answers
8 views
how to decide jdbc vs api data ingestion into lake [closed]
We are working on data lake design and one fundamental question is that what are the parameters to consider to choose between direct connection to JDBC vs source data from API or Kafka Stream?
For ...
0
votes
1answer
10 views
Suggestion of dataset
I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary ...
0
votes
0answers
23 views
Comparing the two feature sets
I am working on a classification of two feature sets derived from a dataset. We first obtain two feature matrices derived from two feature extraction methods. Now, I need to compare them. However, the ...
0
votes
1answer
10 views
Getting the earliest date (duplicates due to several call ids and agents)
I am trying to get the earliest call interaction, with only the agent that interacted at that call and the customer id. Using the following code, I still get duplicates:
...
1
vote
2answers
33 views
What's my target variable?
I am beginner in data science. I have this "aids" dataset from "mdhglm" package in R.
dataset = aids, info = Repatead Measures on AIDS Data
...
0
votes
0answers
14 views
Using Google Trends data for comparability across states/regions
I am interested in using Google Trends data in order to analyse some searches across say U.S. States. Say I wanted to compare search intensity for the keyword "Donald Trump" but across different ...
0
votes
1answer
13 views
Vertical lines of overlapping points in Seaborn Swarmplot
When I plot my data to see its distribution via Swarmplot, these two vertical lines that seems to even overlap occur in the figure.
It seems to occur even around values that are not that dense, such ...
0
votes
0answers
6 views
tensorflow feature columns
I am a beginner in TensorFlow. So while going through an article I came across tf.feature_columns being used.
I understand the general use of it,but what I am confused about something.There were some ...
2
votes
1answer
22 views
When training NN, how do data loaders work on large datasets?
How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples ...
1
vote
1answer
38 views
Logistic Regression: Is it viable to use data that is outdated?
TLDR: Want to predict who makes the playoffs (1,0), but there are more playoff spots now than there were in the past, is it okay to use that past data?
I want to use binary logistic regression on MLB ...
0
votes
1answer
42 views
What's an appropriate datastore for variable length sequence data for PyTorch consumption?
I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats.
I need a datastore that can rapidly ...
0
votes
2answers
21 views
CNN training on Different Datasets
I currently want to train a CNN but I have two small datasets that are slightly different because of the camera setup that captured the images. I'm interested in ultimately tuning the neural network ...
1
vote
0answers
11 views
Tableau: Dealing with Date values [closed]
Im a noob at tableau. I have a quick question
I have data that doesn't explicitly list the date. (There is no column labelled "date" )
However there are columns with the following headers:
"Income ...
0
votes
0answers
26 views
Machine learning model for record linkage of unstructured data
I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
1answer
27 views
Building a content-based music recommendation system
I am trying to build a recommendation system in Python that recommends songs based on a playlist.
What I have is two datasets:
1. One dataset consists of 350 songs from my playlist and 13 acoustic ...
6
votes
2answers
100 views
How important is advanced SQL for data science?
Is advanced level SQL required to be competitive as a data scientist? Is it more important for a data analyst to be good in SQL? Is it enough to be able to extract data using simple SQL queries?
I ...
0
votes
1answer
23 views
Avoiding data duplication in multiple databases [closed]
My book Atzeni et al. states that an organization can use multiple databases to keep data for different porpouses.
For example, an university can use only one main database which contains data of ...
6
votes
1answer
61 views
Un-learning a single training example from a trained model
I was going through the paper "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" by google on which suggests best practices for models in production. In a section ...
0
votes
1answer
16 views
How to split pieces of dataframe and create new dataframes based on it?
I have the following dataframe that I need to split bu Product and create a new dataframe for each product contain the series ds and y. How can I do that?
0
votes
1answer
24 views
How do I iterate over my images in dataset?
I am building an autoencoder with help from this site. There I was trying to build an autoencoder for my own custom data. My images are stored in a folder IMG and have names like ...
0
votes
0answers
17 views
How to use data provided in EEM20 forecasting competition?
I am new to competitions. I have gone through few Kaggle competitions and most of the time, they provide train data, test data and other supplementary data. Recently, I came across a new competition (...
0
votes
0answers
12 views
Converting elements in a list to the index value of an element in list of list
I have a list, l, where the list contains random integer numbers and there are 60000 elements in the list. I also have a list of lists, myListofLists, containing 60 lists and in total 2000 elements (...
1
vote
0answers
17 views
Fit model function out defined data range
I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
0
votes
0answers
7 views
0
votes
0answers
16 views
How can insert lots of data into MySQL database?
I have created a MySQL database with four tables, which are linked with each other (via references).
I now want to insert lots of data from Excel into my database. Is there a possibility of inserting ...
0
votes
1answer
26 views
Is it appropriate to run correlation on continuous y value with discrete x value?
I am dealing with temperature data that are measured in different distances away from the forest edge (0m, 10m,30m,50m,80m) and I would like to find out whether the temperature will drop as the ...
0
votes
0answers
26 views
How can I define a function in r to flag holidays by week?
I have a field of dates that mark the end of weeks. I am trying to create a new field using a function that flags (1 or 0) whether that week included any of 6 holidays specified using the timeDate ...
0
votes
0answers
12 views
GEO Choropleth not found
I'm Trying create, from Geo add-on, a Chropleth map in Orange version 3.24.1 for windows . But, this widget just don't appear... some tip?
0
votes
0answers
14 views
Forecasting binary time series
I am working on the next event occurrence prediction task and the data is binary time series with 1 if the event occured and 0 if not. I want to predict whether the event will occur or not on the day ...
0
votes
1answer
38 views
How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?
Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have ...
0
votes
1answer
31 views
Can we have a dataset with slight difference in target values for same value of feature variable?
I am trying to generate a dataset which involves 1 feature variable(X) and 1 target variable(y).
The feature variable ...
3
votes
5answers
220 views
How can I handle a column with list data?
I have a dataset which I processed and created six features:
['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList',
'totalProducts']
And the ...
0
votes
1answer
15 views
Is it acceptable to make new independent variables out of old ones?
I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or ...
0
votes
0answers
14 views
Modeling of changing data for analysia using cube.js
I want to store users object and store all historical changes of their addresses so that I can calculate the count of users in each state at specific date or date range using cube.js
Can someone ...
1
vote
2answers
27 views
How to define tidy data if there is repeated measures?
Some time ago I read R for Data Sciece and there is the following definition of tidy data:
There are three interrelated rules which make a dataset tidy:
Each variable must have its own column.
...
0
votes
0answers
17 views
0
votes
0answers
33 views
np.loadtxt function showing error, could not convert string to float
This is my data file :
I am trying to read this file with loadtxt() and the geometry points get loaded into one NumPy array, so I separate the data into two arrays:...
0
votes
1answer
45 views
Prepare data for an LSTM
-I want to make a python program using the LSTM model to predict an output value that is 1 or 0.
-My data is stored in a .csv file of the form: (Example of line)
Date time ...
