Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
866
questions
0
votes
0
answers
8
views
Which Frameworks/Libs Best Support Integer-Based Features, Scaling, Training, etc?
Papers such as Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference have interested me in exploring integer-based data science. In particular, I'm thinking of ...
0
votes
0
answers
8
views
Using PowerBI calculated columns vs calculated measures to calculate a frequency flag column
Sorry for the long question.
In PowerBI I have fields called 'user ID' and 'service'. Each row represents an activity, so may not be unique. I want to create a new variable called '5+ flag' that for ...
- 1
0
votes
0
answers
6
views
Terminology: What is it called when the filter criteria is added to the data table?
When we want to focus on a group within our dataset, it is common to filter it.
Suppose we have a data table showing how many points 4 players earned in a game.
Table 1: Game Scores for 4 Players
Age
...
- 123
0
votes
2
answers
29
views
Lack of imagination when creating projects
Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
0
votes
0
answers
11
views
Mapping two unrelated representation spaces with different distributions, roughly preserving similarities
I've got two embedding / representation spaces which are completely unrelated to begin with, yet I wish to find a (=any) mapping between them.
Space A are feature histograms with a somewhat normal ...
- 1
0
votes
0
answers
9
views
How should I save data in deeplearning with nosql (mongodb)?
I usually use file system to manage data for my deep learning model, but one of my boss told me to make nosql database to manage data.
Datasets I use have m rows, and n columns of count matrix and ...
0
votes
0
answers
11
views
Detect Recent Anomalies
I'm trying to see how to approach this problem: I have a dataset of fraud transactions. There are several categorical columns, like country, type, merchant, etc. All of the records are considered ...
- 131
0
votes
0
answers
11
views
(Apparently) Inconsistent Results Between Visual Patterns and Chi-Square Test in Categorical Data Analysis
I'm stuck while working on a categorical data analysis project and would appreciate any insight.
Here are the first 4 rows of the dataset:
successful_upload
feat_1
feat_2
0
T>2
<\$10K
1
0.5&...
0
votes
0
answers
13
views
Data processing for Events that repeat a fixed number of times in each group in machine learning
Assume I have a football teams historical match performance data at player level, with attributes like number of goals score by each player, other performance metrices like passes given, crosses done, ...
- 1
0
votes
0
answers
14
views
I have a data set that having profit column with negative values. what should I do?
I am working on a data set that data set have a profit column with negative values what should I do can we remove it or ignore it .
0
votes
1
answer
42
views
How would normalizing be affected by outliers? And how to avoid it?
I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction.
I'm trying to find the best way to rank all suppliers ...
- 1
0
votes
0
answers
16
views
Are there any publications that deal with decreasing value of data as time decreases toward an event?
I'm looking for research related to the value of data volume as time decreases approaching an event.
For example, large data sets are needed to understand changes in geological and meteorological ...
- 1
0
votes
0
answers
33
views
What is the best way to approach anomaly detection on a data set using machine learning?
I am looking to help on where to start exploring machine learning when it comes to data processing.
Say I have the following csv file with hundreds of thousands of rows of data:
ID
Amount
Overdue (...
- 1
0
votes
0
answers
20
views
Statistical approach for comparing ranks
I'm looking for a statistical approach to compare ranks produced by 2 versions of an ranking algorithm (A & B) against the actual ranks in the system. This is about ranking hospitals and the ...
- 101
0
votes
1
answer
18
views
Converting categorical to the percentage
How do I convert the categorical value to the percentage?|
I have this asset wellness data:
Poor: 3
Warning: 27
Good: 120
How do I convert it to the percentage ...
0
votes
1
answer
43
views
Open retail/ecommerce dataset with customers, transactions, products, loyalty, discounts, promotion information?
Does anyone know of an open retail/e-commerce dataset with information about customers, transactions, products, loyalty, discounts, and promotion information?
I am planning to create 360-degree ...
- 101
0
votes
0
answers
8
views
Data Validity Update
My question is simple. I want to run Orange to predict where will road accidents take place on a highway. So, I will:
Upload the dataset from excel with a few columns and rows of information to be ...
- 1
1
vote
2
answers
37
views
Tips for scraping crypto data in the right way
I am scraping data from crypto site and want to use neural network algorithm for predicting data.
the way i save data is like these:
and there is bunch of other features like open/high/low/close for ...
0
votes
0
answers
20
views
Stacking realization problems
I have two dataframes: x_train with features got from base models and y_train with ground true labels of these features using cross_validation.
...
- 11
0
votes
0
answers
14
views
Multiple coords to one data_vars in xr.dataset
I am trying to create a xarray dataset for a list of events (labeled 1 or 0, so it's an array of integers). To each event three numbers need to be attached - a date, longitude, and latitude.
I provide ...
- 125
0
votes
0
answers
16
views
Gaussian 'around' a given distribution
I want to find the data points within 0.2 dex of the Main Sequence Relation:
(The main sequence relation is what you can observe in this figure)
Dex is an astronomical jargon which signifies scatter, ...
- 101
1
vote
0
answers
29
views
Advice in career path [closed]
I'm currently in Argentina without any type of degree and knowledge in the field of data, maths, programming. My career goal is to get a starting job from here (Argentina) and eventually with some ...
0
votes
0
answers
12
views
I'm looking for a dataset on enviroment monitoring
I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
- 1
0
votes
2
answers
20
views
How to rank relatedness of two feature in dataset by their distribution?
Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me?
I have an idea like, ...
0
votes
0
answers
18
views
Getting The Data Right For A Model
I'm looking to use a logistic regression model to predict who is most likely to suffer a heart attack within a population.
I have a dependent variable flag for has heart attack along with some other ...
0
votes
0
answers
92
views
Importance of sentinel token placement in T5
There is this paper that I have been trying to reproduce (https://arxiv.org/pdf/2205.11482.pdf) as part of my master's thesis. It uses T5 to learn facts from the training set where either the object ...
- 11
0
votes
0
answers
6
views
How to construct Wiener Filter from powerspectrum?
I am trying to construct a Wiener Filter, to filter the ratio of the peak from the cross-correlation function, between a galaxy spectra and a template spectra, with the peak of the auto-correlation ...
0
votes
0
answers
34
views
Media Mix Modeling: testing LightMMM or Robyn on specific dataset rather than ongoing basis
I'm currently scoping ideas for a student project in data analytics and wanted to build this around Media Mix Modeling through either LightMMM (in Python, unofficial Google library) or Robyn (in R, by ...
- 101
0
votes
2
answers
77
views
Different Algorithms for 50-50 A/B Testing
We are running A/B tests on web app customers, given a customerId. Each customer will see different web-feature designs. Trying to prevent usage of Feature Flags as its not currently setup yet in our ...
- 53
1
vote
2
answers
170
views
Learning from aggregated data
Online and in the literature there seems to be a general consensus that training a machine learning model using aggregated data is harder and/or fundamentally different from training on raw event data....
- 120
0
votes
1
answer
54
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
- 1
0
votes
0
answers
13
views
How to wrap local data reading with an API or custom file format?
Specific problem: people have saved CSV files over the last few years on a network drive. Over time, some columns have been added/modifed and some meaning of the values have changed slightly (e.g. ...
- 193
0
votes
0
answers
22
views
Read from database every time a new record goes through a Spark pipeline
I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
0
votes
1
answer
40
views
Low-dimensional binary classification datasets
If one would like to study aspects of neural networks (say, in an academic paper), and would like to experiment on binary classification of vectors in low-dimensional space (say dim=2 or dim<6); ...
- 1
0
votes
0
answers
15
views
Performing imputation only on the test set?
I'm working on a medical machine learning problem. The key challenge is working with small datasets with quite a lot of missing data.
Experimentally, I've seen complete-case analysis (i.e. dropping ...
0
votes
0
answers
15
views
Best way to regrid data (polar stereographic to regular)
I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
- 91
0
votes
3
answers
65
views
Difference between Data analytics and Data Analysis
What we are referring to when we say Data analytics and Data analysis?
1
vote
2
answers
49
views
Data redundancy between train and test dataset - why is it bad (source needed)
I know that it is not OK to have too similar data in the train and test set (for example two pictures that differ by only one pixel). I'm trying to find a scientifically valid explanation why it is ...
- 13
0
votes
2
answers
28
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
0
votes
0
answers
9
views
is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided
Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion.
The goal is to use this metric:
regression
classification
as a metric
profile_views
...
0
votes
0
answers
27
views
Is there any data structure to model a set of both punctual and continuous events?
I have several dataframes storing information about medical history of patients. The medical history of a patient contains both punctual events, such as tests or interventions, and continuous events, ...
0
votes
1
answer
20
views
Splitting data when combining multiple datasets
I have 13 small datasets from 12 different countries. All datasets have the same outcome and features, though have a different number of observations (ranging from ~50 to ~800). I would like to ...
- 101
0
votes
0
answers
66
views
Orange Data Mining creating a target variable on a table at run time
In Orange Data Mining I have created a table at run time and I need to declare one of the columns as a target. I have seen all widgets, but with no solution. I think to use a Python Script but I don't ...
0
votes
0
answers
11
views
Is there a resource to improve business understanding?
I know that business understanding is a crucial part of any data science project, however, I cannot find any good resource to improve corporate understanding. Do you know any?
- 31
0
votes
2
answers
63
views
Fine-tune GPT on sketch data (stroke-3)
These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it.
I was planning on using some pretrained models that are ...
- 31
0
votes
0
answers
47
views
The weighted average ensemble model does not train on the whole data
I am using custom data generator. I want to apply weighted average ensemble. The training set has 1042 samples, and validation indicates 298 samples. The batch size is 64.
when I run this :
...
- 1
1
vote
1
answer
27
views
Comparing images in N channels
I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
- 125
-1
votes
1
answer
190
views
0
votes
0
answers
121
views
ValueError: Found input variables with inconsistent numbers of samples: [283, 943]
I am trying yo split the data using train_test_split(), but I got this error:
...
- 1
0
votes
1
answer
77
views
Regression with time series data
I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
- 1