Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
1
vote
1answer
63 views

Labelling a Time series dataset

I was asked this during an interview that I did not crack. You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one ...
0
votes
1answer
29 views

What is the best way to visualize range of data?

If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this: ...
0
votes
0answers
17 views

HR employee attrition modeling - making a balanced sample question

I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
0
votes
0answers
9 views

Link Recommendation System

I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
1
vote
4answers
88 views

Train a model using multiple data sources

I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is ...
-1
votes
1answer
33 views

Equivalence of two separate data distribution

I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
0
votes
1answer
21 views

Compare two tends with big difference in absolute value

I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners. So far i retrieved my data, and plotted it. Obviously, since the various ...
0
votes
0answers
12 views

Data Generation for LSTM with variable time input

So, I have a dataset that observes entities for 60 time periods. At each time period, I collect some X observations per entity. So I have an LSTM to predict some element in the 60th time period using ...
1
vote
0answers
16 views

Triplet optimization producing a weird diagonal line?

I'm pretty sure this is the right forum for this, or let me know otherwise, I'll happily move this to a better place. I have a strange problem. I've written an algorithm designed to take three files ...
2
votes
1answer
33 views

Column vector for training example

I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a ...
1
vote
0answers
27 views

Generating artificial data to extend learning set

I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
0
votes
1answer
26 views

Tagged dataset with photos for race detection

Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as ...
0
votes
0answers
50 views

Machine learning applications in Architecture field

Can there be Machine learning applications in Architecture field? Example : New York City 100 years back i.e. Year 1920, every ten years after 1920 and today i.e. Year 2020 images of New York City ...
2
votes
0answers
32 views

Association Rules

I'm using association rules for a project and noticed that there is a dearth of papers in the last 10-15 years on the topic although it seemed really popular 15-25 years ago. Is there any reason why?
2
votes
3answers
530 views

Combining Two CSV's in Jupyter Notebook

I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
0
votes
0answers
8 views

Are there labeled multivariate time series data sets where only a subset/partial number of time series is relevant for each class?

I am looking for a data set of multivariate time series data which contains classes that only require a subset of time series for their identification. Assume a human activity recognition data set ...
1
vote
1answer
39 views

How would you encode missing pixels in image data?

I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
1
vote
2answers
96 views

How to Make Meaningful Conclusions here?

I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question. The question- Suppose 7.5% of the ...
-1
votes
1answer
227 views

Matrix of pairwise cosine similarities from matrix of vectors [closed]

I have a matrix of 200d vectors stored as follows: $ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10 \\ \text{id}_2 & -4 & 6 & \dots & -0.3 ...
2
votes
2answers
134 views

X, Y names in data-science [closed]

I am from CS background and moving towards Data Sciences, I have came to know ML is highly influenced by Statistical Inference/signal processing. The X we use in data science is called input, feature ...
0
votes
0answers
12 views

Free exchange rate data API for all countries at at least daily frequency since at least 2000?

I thought it would be a very common dataset: world exchange rates at daily frequency for every currency pair (or each currency's rate vs USD so triangulation can fill in for every other rate) But ...
1
vote
0answers
52 views

cor() function giving unexpected result for numeric data frame(Updated) [closed]

I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
0answers
24 views

Show overall average on plots of single data

I have data from a survey from multiple locations (1 data set per location per year, each containing about 20-25 numerical scores from 0 to 5). This data is aggregated over topics (e.g. first 5 ...
0
votes
0answers
182 views

User-Item based Recommendation system with data containing binary data

I have a data set which contains about 400,000 unique items present on a platform. The users on this platform can like and save this in their own list. Now, I have about 4000 users with their like ...
0
votes
1answer
188 views

How should I choose n_features in FeatureHasher in sklearn?

How should I choose n_features for FeatureHasher in scikit-learn? Assume that I have 1000 categories in feature "case" and I would like to hash them.
1
vote
1answer
28 views

Correlation with Multi-Dimensional Clustering

I have a dataframe with multiple features, where I'm selecting 3 features to cluster on. Ex. ...
5
votes
1answer
510 views

One Hot Encoding for any kind of dataset

How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how ...
0
votes
1answer
25 views

Exploratory Data Analysis on dataset divided by winners and losers

I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players: ...
2
votes
1answer
450 views

How can I count the number of matching zero elements between two numpy arrays?

I have a function that returns the predicted accuracy of a time-series model. I have two equally-sized numpy arrays, one for the actual direction and one for the predicted direction. I'm classifying ...
2
votes
3answers
1k views

Modality of data

Can anyone please explain in clear words what is generally meant by "modality of data"? I know what modality means with respect to distributions.
2
votes
1answer
199 views

SMOTE oversampling for class imbalanced dataset introduces bias in final distribution

I have a problem statement where percentage of goods (denoted by 0) is 95%, and for bads (denoted by 1) it is 5% only. One way is to do under sampling of goods so that model understands the patterns ...
1
vote
0answers
137 views

How do I efficiently load data from disk during training of deep learning models in pytorch?

I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this? It seems like HDF5 is a common method that people ...
2
votes
4answers
505 views

Scikit-learn: train/test split to include have same representation of two different types of values in a column

I have a dataset of online purchase orders that contains two types of customers: Customers who have an account and thus are known customers with unique customer number. Customers who do not have an ...
1
vote
0answers
79 views

pandas: reshaping the dataset, moving column name as rows [closed]

I currently have the following table: ...
0
votes
1answer
32 views

Historical exchange rate data via World Bank API?

I can see a massive list of indicators available via the World Bank API here (may take 1-2 minutes to load) I found exchange rates for various currencies into USD, but cannot see how to get exchange ...
0
votes
0answers
189 views

Compare two columns and visualize third column based on comparison in Power BI

I am new to Power Bi, learning myself and I got a dataset with some questions. Most of the questions are about comparing two columns and visualize the effect on the third column. For example , lets ...
1
vote
1answer
53 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
1answer
32 views

The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]

A typical OLAP cube looks like this: As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts? Should I use star schema instead when having more than ...
0
votes
0answers
15 views

Determining common columns among many datasets without having to manually inspect them

I have a bunch of datasets (10+) and I want to be able to join/ relate them to one another through columns that may contain similar values. For example, date and location are common columns that could ...
-3
votes
1answer
32 views

Accuracy of a forecasting model for prediction of COVID-19 occurence

My goal is to find the best performing forecasting model for the occurrence of COVID-19 in Toronto. I pre-train the network with data on the occurrence of SARS in ten countries and Toronto. Then I ...
2
votes
0answers
46 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
0
votes
1answer
14 views

Ranking tumors from most malignant to most benign

I have a classification problem classifying tumors as benign or malignant. However, I want to go a step further to provide a ranking of these tumors as most malignant to most benign. Are there any ...
1
vote
1answer
62 views

Random Forest in R with only character variables

I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
-1
votes
2answers
32 views

Data Selection & Visualization Tool

I am looking for a tool that will show neatly all the rows from a large table (100GB+) for complex selection conditions from an sql/text/csv table. DB may be on linux or windows. Looking for easy ...
0
votes
1answer
29 views

Higher variance in PCA can mean, that data structure is less informative?

I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with ...
1
vote
0answers
19 views

Normalization around the nominal value while avoiding to get too big results

Lets say we measured a signal N times. Each measurement contains T time instance. For example for N=4 measurements and T=5 time instances the data matrix is as follows: ...
0
votes
1answer
25 views

How to adjust/smooth a certain number using constants or rules

Hi, I am handling a dataset with a customer purchase history. The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage. There are two ...
2
votes
2answers
79 views

Data Imbalance in Regression Tasks [closed]

Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks. Particularly, - What is the procedure to check for ...
3
votes
2answers
864 views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
0answers
19 views

What's the best way to store and then call 3 Million+ records for training?

Good afternoon all, I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline). Essentially I've built a function that can convert raw data into ...

1 2
3
4 5
13