Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

291 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
2 answers
123 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
dtrinh's user avatar
  • 41
4 votes
0 answers
438 views

Spatial Transformer Networks and Data Augmentation

We are all familiar with the famous Deep Mind paper STN. Upon implementation, such as here, did anyone still use input data augementation such as affine transformations? There are used to make CNN ...
Benedict K.'s user avatar
3 votes
1 answer
108 views

What are descriptors

I am very new to data science, and r programming specifically. Could some explain what a descriptor in data science is? In particular, what are the differences between a 2d and 3d descriptor? Thank ...
AlIgh's user avatar
  • 31
3 votes
0 answers
706 views

What exactly negative/positive value of Captum's Integrated Gradient mean?

I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ... Positive attribution score means that the input in that ...
3ORZ's user avatar
  • 31
3 votes
1 answer
65 views

How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
Mashrafi Iqra's user avatar
3 votes
1 answer
65 views

How do I best visualize this voltage data for a science project

I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
user3687778's user avatar
3 votes
1 answer
799 views

should I shift a dataset to use it for Time series regression with RNN/LSTM?

I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1 so my ...
basilisk's user avatar
  • 243
3 votes
1 answer
91 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
Adrian B.'s user avatar
  • 131
3 votes
0 answers
707 views

Central Probability Interval

So, I'm having problems learning for a data analysis lecture. The topic is the construction of the "central interval" as our lecturer calls it. The concept shall path the way to understand confidence &...
F. Ha's user avatar
  • 31
2 votes
0 answers
38 views

How Should I Create Waveform Datasets

Full disclosure I asked this on StackOverflow and it got taken down as it was more of a how do I do this, not how do I code this question: I am trying to simulate/fake data that I eventually will ...
Jordan Larson's user avatar
2 votes
0 answers
61 views

What are some real world examples of a 2d data set where agglomerative methods work and K means doesnt? (for two real groups of data)

What is 2D dataset with two real groups of data where an agglomerative clustering method will be able to cluster the data correctly, and K-means will not? I thought about a data set that would look ...
Emran Tamimi's user avatar
2 votes
0 answers
49 views

How to compare two one hot encoded data frames based on column names?

I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
Sandhya Indurkar's user avatar
2 votes
1 answer
107 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
Lerner Zhang's user avatar
2 votes
1 answer
28 views

Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., ...
n.baes's user avatar
  • 39
2 votes
1 answer
1k views

Normal vs Uniform Distribution for machine learning

I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
Michael Pulis's user avatar
2 votes
1 answer
903 views

Generating synthetic data based off existing real data (in Python)

I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
Cybernetic's user avatar
2 votes
1 answer
25 views

How to use time-sequence data with "meta data" of single value per sample?

I'm, trying to estimate a fish weight by a time-sequence signal of the change in resistance when the fish goes through a gate with electrodes installed. When the fish pass through the gate there's a ...
Shay's user avatar
  • 23
2 votes
2 answers
257 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
jsbc's user avatar
  • 21
2 votes
0 answers
86 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
David Masip's user avatar
  • 6,051
2 votes
1 answer
200 views

What's an appropriate datastore for variable length sequence data for PyTorch consumption?

I have a large number of sequences - potentially hundreds of thousands - each consisting of between 100 and 10,000 items, which each consist of about 5 floats. I need a datastore that can rapidly ...
Sam's user avatar
  • 121
2 votes
0 answers
130 views

Forecasting binary time series

I am working on the next event occurrence prediction task and the data is binary time series with 1 if the event occured and 0 if not. I want to predict whether the event will occur or not on the day ...
Uchenitsa's user avatar
2 votes
2 answers
2k views

How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

I am trying a use multi linear regression model to predict the salaries of employees. I have a total of 88 dependent features from which 19 are categorical and the rest are continuous. I have managed ...
Ahmed Jyad's user avatar
2 votes
1 answer
81 views

What to do if a specific label of a category appears only a few times?

Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
user avatar
2 votes
0 answers
18 views

How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?

I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
Mateusz 's user avatar
2 votes
2 answers
134 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities'. The labeled data (72,000+ entries with all clusters together) with two columns looks like below. ...
Raady's user avatar
  • 239
2 votes
1 answer
211 views

What should I do with the NaN values on this stock quote data?

I concatenated 3 stock quote data-frames all with date-time indexes. However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
Yoyong's user avatar
  • 37
2 votes
0 answers
90 views

Organizing complex analysis project

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
genericname's user avatar
2 votes
0 answers
75 views

Is data science only for substantive specialist?

Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
Mohammad Rahman's user avatar
2 votes
1 answer
2k views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using the h2o library for Python to build a decision tree and to extract the decision rules out of it. I am using some data for training where ...
Sapiens's user avatar
  • 21
2 votes
0 answers
108 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
Coolio2654's user avatar
2 votes
0 answers
1k views

Issues with pandas chunk merge

I'm trying to solve a kaggle competition Since the data is too much to fit in memory at once, I'm trying to clean, process and save data back to disk. This is the code I'm using: ...
Sany's user avatar
  • 21
2 votes
0 answers
40 views

Please help to identify if there is meaning of linear variables separation by features in phase space?

Description: I have clustering problem with next input: I have truth dataset containing labels for each row of features. I've found that some features can be modified to some phase space, where they ...
Damir Gadiev's user avatar
2 votes
0 answers
517 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
tildekara's user avatar
2 votes
2 answers
1k views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
girl101's user avatar
  • 1,161
2 votes
2 answers
593 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
Jeremy McNees's user avatar
2 votes
0 answers
425 views

How can I find company descriptions for a long list of companies?

I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically. E.g. given a long list of ...
Per Borgen's user avatar
2 votes
0 answers
103 views

Core components of data literacy for working professionals

I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government. I've struggled to develop a concise definition of data literacy, but ...
Austin's user avatar
  • 21
2 votes
0 answers
64 views

Free/open interactive softwares/plugins for end-users' high-dimensional data visualization

Aside from questions about How to visualize data of a multidimensional dataset (TIMIT), the Purpose of visualizing high dimensional data? and High-dimensional data: What are useful techniques to know?,...
Laurent Duval's user avatar
2 votes
0 answers
70 views

Tag categorizer

I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like "Cant update application" is tagged with #Apps #General #CantUpdate. My ...
gogasca's user avatar
  • 749
2 votes
0 answers
56 views

How to select a bunch of optimized data from a larger data set?

How to select a bunch of optimized data from a larger data set? I have a data set containing the products to be sold to the customers. And the products were sold before, it has some information, like ...
user3634601's user avatar
2 votes
0 answers
64 views

What proxies could be used to assess economic value of Stackoverflow for its users?

I'm intrigued by the open data provided by StackExchange, and have been running some really interesting queries on the data.stackexchange.com page (using the Stackoverflow dataset). In particular, I ...
Andrea M's user avatar
2 votes
2 answers
524 views

How to make a dataframe with lists or vectors as its elements

This is something that I have been wondering for ages but have never been able to get an answer on. I am trying to understand how to make a dataframe in R, where each element of the dataframe is ...
Quality's user avatar
  • 121
1 vote
0 answers
23 views

How to build this data pipeline?

I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should ...
Wissem Boujlida's user avatar
1 vote
0 answers
29 views

Determining the start and stop of vehicle movement by GPS

There is a large fleet, slightly more than a million vehicles, which is constantly growing. Each vehicle sends GPS coordinates to the server, as well as events (ignition is on, door is open, parking ...
Abraham Lincoln's user avatar
1 vote
1 answer
35 views

Should I drop duplicates over features but no target

I'm in a debate with someone about a problem where there are duplicates over features (i.e. $ X_1 = X_2 $ but $ Y_1 != Y_2 $). My point of view is that we should keep those datas, as they can be ...
EzrielS's user avatar
  • 323
1 vote
0 answers
18 views

what is a data lake catalog technically?

Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database? Thanks.
SSSOF's user avatar
  • 17
1 vote
1 answer
313 views

How to find correlation between features and labels when features are unordered words?

I have a data set with ~ 85 labels and a target (house price) I wish to explore the correlation between the labels and the effect it has on the target. I have some features that number and others that ...
Noah's user avatar
  • 11
1 vote
0 answers
62 views

Data Leakage (specific case with standardization)

I read a few blog posts and found some books that caution against data leakage when adjusting the training data using any information from the held out datasets. For example, if I were to standardize ...
dzheng1887's user avatar
1 vote
0 answers
35 views

Resources on data science errors?

I am putting together some material on common mistakes that occur in data science. The problem, is people often don't talk about the mistakes that everyone has made. Are there any good resources of ...
GooJ's user avatar
  • 435
1 vote
0 answers
23 views

Data filtering framework

I have procurement data that needs to be labeled with product categories. It's tabular data, containing 700k rows and a mix of data types (dates, free text, floats, etc.) The product set we currently ...
Benjamin B.'s user avatar

1
2 3 4 5 6