Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Which Frameworks/Libs Best Support Integer-Based Features, Scaling, Training, etc?

Papers such as Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference have interested me in exploring integer-based data science. In particular, I'm thinking of ...
ezekiel68's user avatar
0 votes
0 answers
8 views

Using PowerBI calculated columns vs calculated measures to calculate a frequency flag column

Sorry for the long question. In PowerBI I have fields called 'user ID' and 'service'. Each row represents an activity, so may not be unique. I want to create a new variable called '5+ flag' that for ...
Molly's user avatar
  • 1
0 votes
0 answers
6 views

Terminology: What is it called when the filter criteria is added to the data table?

When we want to focus on a group within our dataset, it is common to filter it. Suppose we have a data table showing how many points 4 players earned in a game. Table 1: Game Scores for 4 Players Age ...
madprogramer's user avatar
0 votes
2 answers
29 views

Lack of imagination when creating projects

Whenever I find a set of data to carry out a project, I end up running out of ideas and don't know how to proceed with the project. I don't know how to proceed or what information to look for. How do ...
Davi Komura's user avatar
0 votes
0 answers
11 views

Mapping two unrelated representation spaces with different distributions, roughly preserving similarities

I've got two embedding / representation spaces which are completely unrelated to begin with, yet I wish to find a (=any) mapping between them. Space A are feature histograms with a somewhat normal ...
nussbaum's user avatar
0 votes
0 answers
9 views

How should I save data in deeplearning with nosql (mongodb)?

I usually use file system to manage data for my deep learning model, but one of my boss told me to make nosql database to manage data. Datasets I use have m rows, and n columns of count matrix and ...
containletters's user avatar
0 votes
0 answers
11 views

Detect Recent Anomalies

I'm trying to see how to approach this problem: I have a dataset of fraud transactions. There are several categorical columns, like country, type, merchant, etc. All of the records are considered ...
F_M's user avatar
  • 131
0 votes
0 answers
11 views

(Apparently) Inconsistent Results Between Visual Patterns and Chi-Square Test in Categorical Data Analysis

I'm stuck while working on a categorical data analysis project and would appreciate any insight. Here are the first 4 rows of the dataset: successful_upload feat_1 feat_2 0 T>2 <\$10K 1 0.5&...
Maby Esperanza's user avatar
0 votes
0 answers
13 views

Data processing for Events that repeat a fixed number of times in each group in machine learning

Assume I have a football teams historical match performance data at player level, with attributes like number of goals score by each player, other performance metrices like passes given, crosses done, ...
imhaka's user avatar
  • 1
0 votes
0 answers
14 views

I have a data set that having profit column with negative values. what should I do?

I am working on a data set that data set have a profit column with negative values what should I do can we remove it or ignore it .
akshay gaikwad's user avatar
0 votes
1 answer
42 views

How would normalizing be affected by outliers? And how to avoid it?

I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction. I'm trying to find the best way to rank all suppliers ...
Rakuzan's user avatar
0 votes
0 answers
16 views

Are there any publications that deal with decreasing value of data as time decreases toward an event?

I'm looking for research related to the value of data volume as time decreases approaching an event. For example, large data sets are needed to understand changes in geological and meteorological ...
Matt E.'s user avatar
0 votes
0 answers
33 views

What is the best way to approach anomaly detection on a data set using machine learning?

I am looking to help on where to start exploring machine learning when it comes to data processing. Say I have the following csv file with hundreds of thousands of rows of data: ID Amount Overdue (...
Doug's user avatar
  • 1
0 votes
0 answers
20 views

Statistical approach for comparing ranks

I'm looking for a statistical approach to compare ranks produced by 2 versions of an ranking algorithm (A & B) against the actual ranks in the system. This is about ranking hospitals and the ...
museshad's user avatar
  • 101
0 votes
1 answer
18 views

Converting categorical to the percentage

How do I convert the categorical value to the percentage?| I have this asset wellness data: Poor: 3 Warning: 27 Good: 120 How do I convert it to the percentage ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
43 views

Open retail/ecommerce dataset with customers, transactions, products, loyalty, discounts, promotion information?

Does anyone know of an open retail/e-commerce dataset with information about customers, transactions, products, loyalty, discounts, and promotion information? I am planning to create 360-degree ...
exan's user avatar
  • 101
0 votes
0 answers
8 views

Data Validity Update

My question is simple. I want to run Orange to predict where will road accidents take place on a highway. So, I will: Upload the dataset from excel with a few columns and rows of information to be ...
Etzel's user avatar
  • 1
1 vote
2 answers
37 views

Tips for scraping crypto data in the right way

I am scraping data from crypto site and want to use neural network algorithm for predicting data. the way i save data is like these: and there is bunch of other features like open/high/low/close for ...
mohammad ariyan rad's user avatar
0 votes
0 answers
20 views

Stacking realization problems

I have two dataframes: x_train with features got from base models and y_train with ground true labels of these features using cross_validation. ...
XEX's user avatar
  • 11
0 votes
0 answers
14 views

Multiple coords to one data_vars in xr.dataset

I am trying to create a xarray dataset for a list of events (labeled 1 or 0, so it's an array of integers). To each event three numbers need to be attached - a date, longitude, and latitude. I provide ...
Shaz's user avatar
  • 125
0 votes
0 answers
16 views

Gaussian 'around' a given distribution

I want to find the data points within 0.2 dex of the Main Sequence Relation: (The main sequence relation is what you can observe in this figure) Dex is an astronomical jargon which signifies scatter, ...
Ambica Govind's user avatar
1 vote
0 answers
29 views

Advice in career path [closed]

I'm currently in Argentina without any type of degree and knowledge in the field of data, maths, programming. My career goal is to get a starting job from here (Argentina) and eventually with some ...
Santiago Alegre's user avatar
0 votes
0 answers
12 views

I'm looking for a dataset on enviroment monitoring

I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
PiEmmeC's user avatar
0 votes
2 answers
20 views

How to rank relatedness of two feature in dataset by their distribution?

Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me? I have an idea like, ...
Ibrahim Rustamov's user avatar
0 votes
0 answers
18 views

Getting The Data Right For A Model

I'm looking to use a logistic regression model to predict who is most likely to suffer a heart attack within a population. I have a dependent variable flag for has heart attack along with some other ...
HealthAnalyst's user avatar
0 votes
0 answers
92 views

Importance of sentinel token placement in T5

There is this paper that I have been trying to reproduce (https://arxiv.org/pdf/2205.11482.pdf) as part of my master's thesis. It uses T5 to learn facts from the training set where either the object ...
rasgaard's user avatar
0 votes
0 answers
6 views

How to construct Wiener Filter from powerspectrum?

I am trying to construct a Wiener Filter, to filter the ratio of the peak from the cross-correlation function, between a galaxy spectra and a template spectra, with the peak of the auto-correlation ...
user149843's user avatar
0 votes
0 answers
34 views

Media Mix Modeling: testing LightMMM or Robyn on specific dataset rather than ongoing basis

I'm currently scoping ideas for a student project in data analytics and wanted to build this around Media Mix Modeling through either LightMMM (in Python, unofficial Google library) or Robyn (in R, by ...
MFJC's user avatar
  • 101
0 votes
2 answers
77 views

Different Algorithms for 50-50 A/B Testing

We are running A/B tests on web app customers, given a customerId. Each customer will see different web-feature designs. Trying to prevent usage of Feature Flags as its not currently setup yet in our ...
mattsmith5's user avatar
1 vote
2 answers
170 views

Learning from aggregated data

Online and in the literature there seems to be a general consensus that training a machine learning model using aggregated data is harder and/or fundamentally different from training on raw event data....
dendog's user avatar
  • 120
0 votes
1 answer
54 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
Horbeam's user avatar
0 votes
0 answers
13 views

How to wrap local data reading with an API or custom file format?

Specific problem: people have saved CSV files over the last few years on a network drive. Over time, some columns have been added/modifed and some meaning of the values have changed slightly (e.g. ...
gebbissimo's user avatar
0 votes
0 answers
22 views

Read from database every time a new record goes through a Spark pipeline

I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
José Luis Benitez's user avatar
0 votes
1 answer
40 views

Low-dimensional binary classification datasets

If one would like to study aspects of neural networks (say, in an academic paper), and would like to experiment on binary classification of vectors in low-dimensional space (say dim=2 or dim<6); ...
Sasha's user avatar
  • 1
0 votes
0 answers
15 views

Performing imputation only on the test set?

I'm working on a medical machine learning problem. The key challenge is working with small datasets with quite a lot of missing data. Experimentally, I've seen complete-case analysis (i.e. dropping ...
Ben Consterdine's user avatar
0 votes
0 answers
15 views

Best way to regrid data (polar stereographic to regular)

I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
Socorro's user avatar
  • 91
0 votes
3 answers
65 views

Difference between Data analytics and Data Analysis

What we are referring to when we say Data analytics and Data analysis?
Tauqir Ahmed's user avatar
1 vote
2 answers
49 views

Data redundancy between train and test dataset - why is it bad (source needed)

I know that it is not OK to have too similar data in the train and test set (for example two pictures that differ by only one pixel). I'm trying to find a scientifically valid explanation why it is ...
user1633361's user avatar
0 votes
2 answers
28 views

Data imputation for heavily missing features

I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
Hai Nguyen's user avatar
0 votes
0 answers
9 views

is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided

Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion. The goal is to use this metric: regression classification as a metric profile_views ...
Zaahir Ebrahim Dawood's user avatar
0 votes
0 answers
27 views

Is there any data structure to model a set of both punctual and continuous events?

I have several dataframes storing information about medical history of patients. The medical history of a patient contains both punctual events, such as tests or interventions, and continuous events, ...
Agatino Giuliano Mirabella's user avatar
0 votes
1 answer
20 views

Splitting data when combining multiple datasets

I have 13 small datasets from 12 different countries. All datasets have the same outcome and features, though have a different number of observations (ranging from ~50 to ~800). I would like to ...
jpsmith's user avatar
  • 101
0 votes
0 answers
66 views

Orange Data Mining creating a target variable on a table at run time

In Orange Data Mining I have created a table at run time and I need to declare one of the columns as a target. I have seen all widgets, but with no solution. I think to use a Python Script but I don't ...
Mario Capurso's user avatar
0 votes
0 answers
11 views

Is there a resource to improve business understanding?

I know that business understanding is a crucial part of any data science project, however, I cannot find any good resource to improve corporate understanding. Do you know any?
A_cat's user avatar
  • 31
0 votes
2 answers
63 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
ilved17's user avatar
  • 31
0 votes
0 answers
47 views

The weighted average ensemble model does not train on the whole data

I am using custom data generator. I want to apply weighted average ensemble. The training set has 1042 samples, and validation indicates 298 samples. The batch size is 64. when I run this : ...
Zara Nz's user avatar
1 vote
1 answer
27 views

Comparing images in N channels

I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
Shaz's user avatar
  • 125
-1 votes
1 answer
190 views

y should be a 1d array, got an array of shape (60630, 2) instead

...
Wajeeh Rathore's user avatar
0 votes
0 answers
121 views

ValueError: Found input variables with inconsistent numbers of samples: [283, 943]

I am trying yo split the data using train_test_split(), but I got this error: ...
Coco's user avatar
  • 1
0 votes
1 answer
77 views

Regression with time series data

I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
Scholar7's user avatar

1
2 3 4 5
18