Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
863
questions
0
votes
0
answers
9
views
Gaussian 'around' a given distribution
I want to find the data points within 0.2 dex of the Main Sequence Relation:
(The main sequence relation is what you can observe in this figure)
Dex is an astronomical jargon which signifies scatter, ...
- 101
0
votes
0
answers
7
views
Is there a statistical test for comparing my dataset percentages with published percentages?
I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%.
Is there a ...
1
vote
0
answers
20
views
Advice in career path
I'm currently in Argentina without any type of degree and knowledge in the field of data, maths, programming. My career goal is to get a starting job from here (Argentina) and eventually with some ...
0
votes
0
answers
26
views
Data Processing Question - Time Series Data
Apologies if this was asked before (links to other answers appreciated!). I am struggling with a conceptual question about how to structure my data for learning using neural networks and was hoping ...
- 1
0
votes
0
answers
11
views
I'm looking for a dataset on enviroment monitoring
I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
- 1
0
votes
2
answers
15
views
How to rank relatedness of two feature in dataset by their distribution?
Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me?
I have an idea like, ...
0
votes
0
answers
18
views
Getting The Data Right For A Model
I'm looking to use a logistic regression model to predict who is most likely to suffer a heart attack within a population.
I have a dependent variable flag for has heart attack along with some other ...
0
votes
0
answers
30
views
Importance of sentinel token placement in T5
There is this paper that I have been trying to reproduce (https://arxiv.org/pdf/2205.11482.pdf) as part of my master's thesis. It uses T5 to learn facts from the training set where either the object ...
- 11
0
votes
0
answers
5
views
How to construct Wiener Filter from powerspectrum?
I am trying to construct a Wiener Filter, to filter the ratio of the peak from the cross-correlation function, between a galaxy spectra and a template spectra, with the peak of the auto-correlation ...
0
votes
0
answers
13
views
Are there free datasets available with several tables? [migrated]
I want to make a presentation with comparison of pandas and pyspark. For this purpose I need some data and ideally a research task that will allow me to:
Show how joins works in both libraries. So at ...
0
votes
0
answers
17
views
Media Mix Modeling: testing LightMMM or Robyn on specific dataset rather than ongoing basis
I'm currently scoping ideas for a student project in data analytics and wanted to build this around Media Mix Modeling through either LightMMM (in Python, unofficial Google library) or Robyn (in R, by ...
- 101
0
votes
2
answers
71
views
Different Algorithms for 50-50 A/B Testing
We are running A/B tests on web app customers, given a customerId. Each customer will see different web-feature designs. Trying to prevent usage of Feature Flags as its not currently setup yet in our ...
- 53
1
vote
2
answers
81
views
Learning from aggregated data
Online and in the literature there seems to be a general consensus that training a machine learning model using aggregated data is harder and/or fundamentally different from training on raw event data....
- 120
0
votes
1
answer
27
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
- 1
0
votes
0
answers
13
views
How to wrap local data reading with an API or custom file format?
Specific problem: people have saved CSV files over the last few years on a network drive. Over time, some columns have been added/modifed and some meaning of the values have changed slightly (e.g. ...
- 193
0
votes
0
answers
13
views
Read from database every time a new record goes through a Spark pipeline
I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
0
votes
1
answer
33
views
Low-dimensional binary classification datasets
If one would like to study aspects of neural networks (say, in an academic paper), and would like to experiment on binary classification of vectors in low-dimensional space (say dim=2 or dim<6); ...
- 1
0
votes
0
answers
14
views
Performing imputation only on the test set?
I'm working on a medical machine learning problem. The key challenge is working with small datasets with quite a lot of missing data.
Experimentally, I've seen complete-case analysis (i.e. dropping ...
0
votes
0
answers
10
views
Best way to regrid data (polar stereographic to regular)
I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
- 91
0
votes
3
answers
51
views
Difference between Data analytics and Data Analysis
What we are referring to when we say Data analytics and Data analysis?
1
vote
2
answers
41
views
Data redundancy between train and test dataset - why is it bad (source needed)
I know that it is not OK to have too similar data in the train and test set (for example two pictures that differ by only one pixel). I'm trying to find a scientifically valid explanation why it is ...
- 13
0
votes
2
answers
24
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
0
votes
0
answers
9
views
is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided
Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion.
The goal is to use this metric:
regression
classification
as a metric
profile_views
...
0
votes
0
answers
27
views
Is there any data structure to model a set of both punctual and continuous events?
I have several dataframes storing information about medical history of patients. The medical history of a patient contains both punctual events, such as tests or interventions, and continuous events, ...
0
votes
1
answer
19
views
Splitting data when combining multiple datasets
I have 13 small datasets from 12 different countries. All datasets have the same outcome and features, though have a different number of observations (ranging from ~50 to ~800). I would like to ...
- 101
0
votes
0
answers
31
views
Orange Data Mining creating a target variable on a table at run time
In Orange Data Mining I have created a table at run time and I need to declare one of the columns as a target. I have seen all widgets, but with no solution. I think to use a Python Script but I don't ...
0
votes
0
answers
11
views
Is there a resource to improve business understanding?
I know that business understanding is a crucial part of any data science project, however, I cannot find any good resource to improve corporate understanding. Do you know any?
- 21
0
votes
2
answers
51
views
Fine-tune GPT on sketch data (stroke-3)
These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it.
I was planning on using some pretrained models that are ...
- 31
0
votes
0
answers
40
views
The weighted average ensemble model does not train on the whole data
I am using custom data generator. I want to apply weighted average ensemble. The training set has 1042 samples, and validation indicates 298 samples. The batch size is 64.
when I run this :
...
- 1
1
vote
1
answer
25
views
Comparing images in N channels
I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
- 125
-1
votes
1
answer
79
views
0
votes
0
answers
78
views
ValueError: Found input variables with inconsistent numbers of samples: [283, 943]
I am trying yo split the data using train_test_split(), but I got this error:
...
- 1
0
votes
1
answer
36
views
Regression with time series data
I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
- 1
0
votes
1
answer
29
views
Is there a standard order of operations for data preparation?
In what order should I do the following given a dataset:
(E)ncoding of Categorical Variables
(N)ormalization
(B)alancing of data
(I)mputation of Missing Values
(R)emoval of Duplicates/Infinity/...
- 15
0
votes
1
answer
39
views
How to cluster based on sensor data? - My first data science job
I'm on my first (real), data, programming job. As everyone can imagine, this can be quite hard and I learn a lot from it, given I am a data science student in university. However, I am completely ...
- 1
1
vote
1
answer
39
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 125
0
votes
0
answers
26
views
Benchmark dataset with both images and tabular data
In the medical sector, there are situations in which an image dataset is associated with a tabular dataset containing different features but the same labels as the image dataset. For example, suppose ...
- 219
-3
votes
1
answer
36
views
What to do if your model's prediciton result wrong because of unlucky?
Have you ever had a situation where your model backtested with very good with historical data, and you also felt that your model was very logical? But when put it into practice case to predict the ...
0
votes
0
answers
10
views
Larger Latent Space Dim for Point Cloud Autoencoder
So I'm trying to follow a paper that uses a AE to learn point clouds. The thing is, the dimension of the point cloud data is 3 (x, y, z), but the dimension of the latent space from what I can tell is ...
- 101
0
votes
1
answer
13
views
What's the word for aggregating by two attributes together, rather than separately?
Let's imagine we have this underlying data:
...
- 111
0
votes
0
answers
13
views
Which is a better tool to build on Azure, SAP or Oracle
I'm trying to understand each platform's pros and cons to help decide how we build out your ERP solution.
I'm new to the cloud, so please bare with me.
We will use this for D.B for ML first, then A.L.
...
- 1
0
votes
1
answer
36
views
I'm Looking for dataset with Name, Father's Name, Mother's name, Age, Gender, Ethnicity, ~Birthplace [closed]
I am new to Data Science and I don't know where to find datasets, I Google searched, Asked ChatGPT and friends, but couldn't find what I was looking for.
My project is to generate new names based on ...
- 1
0
votes
0
answers
57
views
Why Seaborn's histplot takes a longer time to plot when the data is far from the median than when it is closer?
I have recently noticed that, for the some number of data points and for the same precision (int64), when using Seaborn's histplot on data that has huge variance with respect to the median it takes a ...
- 1
0
votes
0
answers
8
views
How to store (lab) data in everyday cases
I'm (let's say half) a data scientist within an engineering group and as I'm the only one familiar with storing, manipulating, using and plotting data (other than only ...
- 500
0
votes
0
answers
23
views
Is it normal for a SARIMA model to produce no residuals?
I'm working on my first time series project where I am required to produce predictions for financial data.
The raw data is below:
Clearly, there is a seasonality and downward trend, I used the ...
- 1
0
votes
0
answers
45
views
reading data from Azure Event Hub with Databricks
I'm having problems reading the data from event hub.
questions :
what am i doing wrong in connecting to event hub using databricks?
is the problem with the message? or is it in the script?
is the ...
- 1
0
votes
0
answers
32
views
Why are all PyTorch dataloader proccesses in S state (interruptible sleep) except one?
Short version: I'm using PyTorch dataloading library to load data in parallel for training a deep learning model. When I look at the CPU usage with htop, I see a ...
- 377
1
vote
1
answer
24
views
Should I drop duplicates over features but no target
I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $).
My point of view is that we should keep those datas, as they can be ...
- 311
2
votes
2
answers
301
views
Timeseries (InfluxDB): How to deal with missing data?
Question Description
We are performing a lot of timeseries queries, these queries sometimes result in issues, they are usually performed through an API (Python) and sometimes result in complete ...
- 141
1
vote
0
answers
13
views
what is a data lake catalog technically?
Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database?
Thanks.
- 11