Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
847
questions
0
votes
1
answer
14
views
Is there a standard order of operations for data preparation?
In what order should I do the following given a dataset:
(E)ncoding of Categorical Variables
(N)ormalization
(B)alancing of data
(I)mputation of Missing Values
(R)emoval of Duplicates/Infinity/...
- 15
0
votes
1
answer
15
views
How to cluster based on sensor data? - My first data science job
I'm on my first (real), data, programming job. As everyone can imagine, this can be quite hard and I learn a lot from it, given I am a data science student in university. However, I am completely ...
- 1
1
vote
1
answer
26
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 113
0
votes
0
answers
15
views
Benchmark dataset with both images and tabular data
In the medical sector, there are situations in which an image dataset is associated with a tabular dataset containing different features but the same labels as the image dataset. For example, suppose ...
- 221
-3
votes
1
answer
33
views
What to do if your model's prediciton result wrong because of unlucky?
Have you ever had a situation where your model backtested with very good with historical data, and you also felt that your model was very logical? But when put it into practice case to predict the ...
0
votes
0
answers
5
views
Larger Latent Space Dim for Point Cloud Autoencoder
So I'm trying to follow a paper that uses a AE to learn point clouds. The thing is, the dimension of the point cloud data is 3 (x, y, z), but the dimension of the latent space from what I can tell is ...
- 101
0
votes
1
answer
11
views
What's the word for aggregating by two attributes together, rather than separately?
Let's imagine we have this underlying data:
...
- 111
0
votes
0
answers
12
views
Which is a better tool to build on Azure, SAP or Oracle
I'm trying to understand each platform's pros and cons to help decide how we build out your ERP solution.
I'm new to the cloud, so please bare with me.
We will use this for D.B for ML first, then A.L.
...
- 1
0
votes
1
answer
20
views
I'm Looking for dataset with Name, Father's Name, Mother's name, Age, Gender, Ethnicity, ~Birthplace [closed]
I am new to Data Science and I don't know where to find datasets, I Google searched, Asked ChatGPT and friends, but couldn't find what I was looking for.
My project is to generate new names based on ...
- 1
0
votes
0
answers
14
views
Why Seaborn's histplot takes a longer time to plot when the data is far from the median than when it is closer?
I have recently noticed that, for the some number of data points and for the same precision (int64), when using Seaborn's histplot on data that has huge variance with respect to the median it takes a ...
- 1
0
votes
0
answers
8
views
How to store (lab) data in everyday cases
I'm (let's say half) a data scientist within an engineering group and as I'm the only one familiar with storing, manipulating, using and plotting data (other than only ...
- 490
0
votes
0
answers
12
views
Is it normal for a SARIMA model to produce no residuals?
I'm working on my first time series project where I am required to produce predictions for financial data.
The raw data is below:
Clearly, there is a seasonality and downward trend, I used the ...
- 1
0
votes
0
answers
14
views
reading data from Azure Event Hub with Databricks
I'm having problems reading the data from event hub.
questions :
what am i doing wrong in connecting to event hub using databricks?
is the problem with the message? or is it in the script?
is the ...
- 1
0
votes
0
answers
21
views
Why are all PyTorch dataloader proccesses in S state (interruptible sleep) except one?
Short version: I'm using PyTorch dataloading library to load data in parallel for training a deep learning model. When I look at the CPU usage with htop, I see a ...
- 347
1
vote
1
answer
18
views
Should I drop duplicates over features but no target
I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $).
My point of view is that we should keep those datas, as they can be ...
- 311
2
votes
2
answers
85
views
Timeseries (InfluxDB): How to deal with missing data?
Question Description
We are performing a lot of timeseries queries, these queries sometimes result in issues, they are usually performed through an API (Python) and sometimes result in complete ...
- 141
1
vote
0
answers
11
views
what is a data lake catalog technically?
Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database?
Thanks.
- 11
0
votes
0
answers
25
views
Can Hip-Hop/music trend be estimated?
How can I determine how an "evolutionary" music album affects the development of its genre? Only two perspectives I can come up with: 1. The effect on the number of songs before and after(...
2
votes
1
answer
83
views
Does opting out of having my content used for improvement mean there are no other forms of data retention of my content by OpenAI?
Regarding the use of OpenAI API, Terms of Use at OpenAI mentions:
You can opt out of having Content used for improvement by contacting support@openai.com with your organization ID. Please note that ...
- 5,478
0
votes
0
answers
20
views
1
vote
1
answer
32
views
How to increase retention?
As you might already know there is a concept of retention. Let's say I have created a game and today hundred people have downloaded my game. Let's say tomorrow 47 out of yesterday's hundred people are ...
- 121
0
votes
0
answers
9
views
What is the best way to apply Bootstapped weights to data for statistical analysis?
I'm working with a survey dataset for some statistical analysis. My issue is that there are weight columns, but I have no idea how they did the weighting or what the weights are for. I'm not familiar ...
0
votes
0
answers
15
views
How to scale a vector into a smaller vector?
I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5].
How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
- 11
0
votes
0
answers
12
views
Methods to appropriately capture equal ranges of turnover
Suppose I have a dataset with three columns, turnover, market, count, and turnover is the range of turnover in EUR. Something like:
...
- 1
0
votes
0
answers
126
views
AttributeError: 'NoneType' object has no attribute 'longitude'
I'm trying to get the location longitude of a address using geopy.
normally this works completely fine.
...
0
votes
0
answers
10
views
Is there data on the number of residential units in each building?
I am totally new to conducting data science, so I don't know if this is the place to ask, but I am trying to find data on the number of residential units in each building in the USA for a project I am ...
0
votes
1
answer
10
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
0
answers
32
views
Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX
Problem:
I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0
votes
0
answers
10
views
What is the best way to set up control limits for customer rating data?
I have 2 years of monthly average ratings for a product given by the customers. Obviously there are fluctuations in the average ratings and I wanted to set up limits for this average monthly ratings ...
- 1
0
votes
1
answer
60
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
1
answer
30
views
Models that are good for long answer generation given context and question and what datasets would be the best for training?
Basically I am trying to create a context-needing question and long answer model and I was wondering what model would be best for such tasks, currently I am leaning towards T5, or GPT-NeoX-20B. ...
0
votes
0
answers
10
views
Multicategory choice model resources
I'm a student wanting to do some applied research.
The data I'm working with has 500 consumers, and for each consumer, there are 4 type of goods they bought and each are categorized by company.
For ...
0
votes
0
answers
9
views
Transferring exact same input features to another domain in regression setting?
Assume I have a dataset (D) where each X has a label Y1 under a certain manufacturing process, let's call it Process 1, a regression model learns this task well after training on millions of points.
...
0
votes
0
answers
16
views
How to create synthetic data with custom feature values?
Currently I create a synthetic dataset using the python sdv package and an existing dataset as below:
...
- 21
6
votes
4
answers
116
views
How can I observe my CSV files better? [closed]
I'm running a lot of experiments that give their output as CSV files. An experiment might be running for hours, with a new line being added to the CSV every 10 seconds.
Right now I'm opening these CSV ...
- 255
0
votes
1
answer
19
views
How do I get a big-picture feel for my database?
Background: I was just granted access to my company's SQL Server (MS SQL 2019 via Azure Data Studio). As I have not yet been assigned any specific projects I want start digging.
I've asked my DBA to ...
0
votes
1
answer
25
views
Relationship between size of of data and number of classes
Hi we have a problem on image classification where data is quite less. However we have some flexibility on the classification.We can change the number of classes in our problem by clubbing and ...
- 101
0
votes
1
answer
32
views
What are more advanced categorical encoding methods?
I'm familiar with the common methods:
Label Encoding: {A, B, C} -> [0, 1, 2]
One-Hot Encoding: ...
- 11
0
votes
0
answers
25
views
Is there a difference in result if we apply Polynomial / Kernel Regression on mean of target data, or all data?
Let's say we have some data :
input data X with shape (1, N=100), this will be duplicated 1000 times.
target data Y with shape (S=1000, N=100).
We have 1000 experimental data points, samples.
My ...
0
votes
0
answers
11
views
How to graphic evolution of clusters in time
I am currently working in the definition of the profiles of patients of a specific disease. I have applied an unsupervised clustering technique to my dataset obtaining different patient profiles for ...
0
votes
1
answer
14
views
Whats the best statistical metric for two sets of correlated data?
I have a trained GAN that generates synthetic flight data (9 state variables/features). I plotted the correlation for each pair of state variables similiar to Seaborn's pairplot(). But I have two sets ...
0
votes
0
answers
22
views
Methods for measuring/identifying themes in complaints data that are "interesting"?
all.
I am posting this as more of a discussion post, rather than a question, as I am trying to come up with ideas for a project I have been asked to consider.
For context, I work for a customer ...
0
votes
1
answer
19
views
Azure Devops for Data Science
My team (7 people) is switching to git using Azure Devops. I’m a bit confused on how to set it up for a data science team. I used it for personal projects which is straight forward, do dev in one ...
0
votes
0
answers
10
views
Reshaping data from long to wide format in Stata
I have a dataset on carbon intensity and volume loans across different sectors in the EU and I try to reshape the data set from long format to a wide format as I wrongly first thought that it should'...
- 1
1
vote
1
answer
39
views
How to find correlation between features and labels when features are unordered words?
I have a data set with ~ 85 labels and a target (house price)
I wish to explore the correlation between the labels and the effect it has on the target.
I have some features that number and others that ...
- 11
0
votes
0
answers
29
views
Is Subsymbolic AI machine learning?
I read a few articles regarding different types of AI.https://wiki.pathmind.com/ai-vs-machine-learning-vs-deep-learning. Here it was stated that symbolic A.I is not machine learning. So can all of ...
0
votes
1
answer
67
views
Split dataset into Train/Validation/Test for Object Detection
I have a dataset for Object Detection with YOLO format labels, each imagine can have occurences of different classes and multiple occurences of the same class.
How can the dataset be divided into ...
0
votes
0
answers
10
views
Based on what Categorization is machine learning a part of Ai?
I read an article on microsoft website (https://azure.microsoft.com/en-us/solutions/ai/artificial-intelligence-vs-machine-learning/#introduction) , it said that machine learning is considered a subset ...
0
votes
0
answers
62
views
Orange data mining: instances with missing targets are ignored while scoring
In predictions warning showing "instances with missing targets are ignored while scoring" is coming how to resolve
0
votes
0
answers
16
views
Best ways to visualize the performance of a regression model?
How best to visualize 10,000 observations of continuous labels against predicted values in a way to summarises the overall performance of the model? I want to avoid using performance metrics such as ...
- 1