Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
591
questions
-1
votes
0answers
8 views
unhashable type 'numpy.ndarray' [closed]
I'm testing my code, which works perfectly with a smaller set of data (two columns: 1st is Date, 2nd values). I use the same formatting with the new data and the only differences I can discern at the ...
0
votes
1answer
21 views
Aggregating data [closed]
I have two variables X1 (say inflation) and X2 (say GDP). Iam looking to approximate the economic security (not observable) ...
-2
votes
0answers
27 views
Skills needed for a signal processing data scientist in medical devices industry [closed]
How do I become a signal processing data scientist in medical devices industry?
I have a background in electronics engineering and worked in the semiconductor industry as design engineer. Now I want ...
1
vote
0answers
18 views
Generating synthetic data based off existing real data (in Python)
I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
1
vote
0answers
12 views
How much information is produced in our Society? [closed]
I was reading an interesting article about the growth of data in our society:
https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x
Does any of you know if there exists a more ...
2
votes
0answers
22 views
How to improve model performace when model shows a systemic pattern in residues
I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
0
votes
0answers
11 views
Positive records : no behavioral data
0
I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
-1
votes
1answer
34 views
What algorithm should I use to get a mapping between two variables? [closed]
I have a dataset that contains for every row, a list X of x items, that is a subset of X_total, and a list Y of y items, that is a subset of Y_total.
...
1
vote
0answers
13 views
How to use time-sequence data with “meta data” of single value per sample?
I'm, trying to estimate a fish weight by a time-sequence signal of the change in resistance when the fish goes through a gate with electrodes installed.
When the fish pass through the gate there's a ...
1
vote
0answers
19 views
Recommendation Systems User Profile Streaming Data on GCP
I have a recommendation system that recommends articles to different users. I am planning to provide the recommendations in an off-line fashion. Where I already have a table in BigQuery which has the ...
1
vote
1answer
20 views
Is window based sequencing a good idea to obtain more training data for LSTMs?
I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier.
However,...
1
vote
2answers
44 views
Problem with converting string to dummy variables
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
0
votes
1answer
53 views
How to build a database of image data for machine learning?
I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database ...
2
votes
2answers
24 views
Log analysis dataset with labeled cybersecurity issues
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
1answer
14 views
Automating the training and deployment of the model as more data becomes available
My question is about automating the training of the model as more data becomes available.
In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ...
2
votes
1answer
34 views
What's the Seminal Book on data science? [closed]
Can people provide the best textbook they have used for learning data science? I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a ...
0
votes
1answer
73 views
Machine Learning: Predicting target based on a feature
I have a df looks as follow:
-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
2
votes
1answer
33 views
How to account for reduced student capacity when calculating program retention?
I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am ...
1
vote
1answer
55 views
Labelling a Time series dataset
I was asked this during an interview that I did not crack.
You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one ...
0
votes
1answer
26 views
What is the best way to visualize range of data?
If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
...
0
votes
0answers
15 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
0
votes
0answers
9 views
Link Recommendation System
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
1
vote
4answers
31 views
Train a model using multiple data sources
I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is ...
-1
votes
1answer
31 views
Equivalence of two separate data distribution
I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
0
votes
1answer
14 views
Compare two tends with big difference in absolute value
I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners.
So far i retrieved my data, and plotted it.
Obviously, since the various ...
0
votes
0answers
9 views
Data Generation for LSTM with variable time input
So, I have a dataset that observes entities for 60 time periods. At each time period, I collect some X observations per entity.
So I have an LSTM to predict some element in the 60th time period using ...
1
vote
0answers
15 views
Triplet optimization producing a weird diagonal line?
I'm pretty sure this is the right forum for this, or let me know otherwise, I'll happily move this to a better place.
I have a strange problem. I've written an algorithm designed to take three files ...
2
votes
1answer
27 views
Column vector for training example
I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a ...
1
vote
0answers
24 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
0
votes
1answer
17 views
Tagged dataset with photos for race detection
Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as ...
0
votes
0answers
45 views
How to plot three categorical variables Python/Pandas?
I am exploring data visualization with the Titanic data set from Kaggle and would like to understand the survival based on the family size of the passengers. In short, I would like to be able to plot ...
0
votes
0answers
47 views
Machine learning applications in Architecture field
Can there be Machine learning applications in Architecture field?
Example : New York City 100 years back i.e. Year 1920, every ten years after 1920 and today i.e. Year 2020 images of New York City ...
1
vote
0answers
30 views
Association Rules
I'm using association rules for a project and noticed that there is a dearth of papers in the last 10-15 years on the topic although it seemed really popular 15-25 years ago. Is there any reason why?
2
votes
3answers
38 views
Combining Two CSV's in Jupyter Notebook
I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
0
votes
0answers
8 views
Are there labeled multivariate time series data sets where only a subset/partial number of time series is relevant for each class?
I am looking for a data set of multivariate time series data which contains classes that only require a subset of time series for their identification. Assume a human activity recognition data set ...
1
vote
1answer
20 views
How would you encode missing pixels in image data?
I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
1
vote
2answers
87 views
How to Make Meaningful Conclusions here?
I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question.
The question-
Suppose 7.5% of the ...
0
votes
0answers
21 views
Is there a way to calculate the degree of classification of data points in a set?
I'm trying to write a program in python that takes in list of all stocks on different exchanges and filters them out according to multiple criteria (e.g. marketcap has to be larger than 500m, PE ratio ...
-1
votes
1answer
32 views
Matrix of pairwise cosine similarities from matrix of vectors [closed]
I have a matrix of 200d vectors stored as follows:
$ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10
\\ \text{id}_2 & -4 & 6 & \dots & -0.3
...
2
votes
2answers
49 views
X, Y names in data-science [closed]
I am from CS background and moving towards Data Sciences, I have came to know ML is highly influenced by Statistical Inference/signal processing. The X we use in data science is called input, feature ...
0
votes
0answers
10 views
Free exchange rate data API for all countries at at least daily frequency since at least 2000?
I thought it would be a very common dataset: world exchange rates at daily frequency for every currency pair (or each currency's rate vs USD so triangulation can fill in for every other rate)
But ...
1
vote
0answers
37 views
cor() function giving unexpected result for numeric data frame(Updated) [closed]
I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
0answers
21 views
Show overall average on plots of single data
I have data from a survey from multiple locations (1 data set per location per year, each containing about 20-25 numerical scores from 0 to 5). This data is aggregated over topics (e.g. first 5 ...
0
votes
0answers
55 views
User-Item based Recommendation system with data containing binary data
I have a data set which contains about 400,000 unique items present on a platform. The users on this platform can like and save this in their own list. Now, I have about 4000 users with their like ...
0
votes
1answer
42 views
How should I choose n_features in FeatureHasher in sklearn?
How should I choose n_features in FeatureHasher in sklearn ? assume that I have 1000 categories in feature "case" and I want to hash them
1
vote
0answers
16 views
Correlation with Multi-Dimensional Clustering
I have a dataframe with multiple features, where I'm selecting 3 features to cluster on. Ex.
...
5
votes
1answer
327 views
One Hot Encoding for any kind of dataset
How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how ...
0
votes
1answer
23 views
Exploratory Data Analysis on dataset divided by winners and losers
I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players:
...
0
votes
0answers
12 views
How can a CNN account for spectro-temporal constraints in neural data?
What are there the best ways to leverage the unique "geometrical" constraints of spectro-temporal signal representations (architecture, filter shapes, data augmentation, etc.)? For example, ...
2
votes
1answer
173 views
How can I count the number of matching zero elements between two numpy arrays?
I have a function that returns the predicted accuracy of a time-series model. I have two equally-sized numpy arrays, one for the actual direction and one for the predicted direction. I'm classifying ...
