Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

224 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
17 views

What are descriptors

I am very new to data science, and r programming specifically. Could some explain what a descriptor in data science is? In particular, what are the differences between a 2d and 3d descriptor? Thank ...
3
votes
1answer
46 views

How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
3
votes
1answer
43 views

How do I best visualize this voltage data for a science project

I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
3
votes
1answer
90 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
3
votes
1answer
1k views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using the h2o library for Python to build a decision tree and to extract the decision rules out of it. I am using some data for training where ...
3
votes
2answers
186 views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
3
votes
0answers
536 views

Central Probability Interval

So, I'm having problems learning for a data analysis lecture. The topic is the construction of the "central interval" as our lecturer calls it. The concept shall path the way to understand confidence &...
2
votes
1answer
45 views

Highly correlated features

What are the conditions to remove highly correlated features in a correlation map? Given the correlation map below, is it OK to remove diagnosis feature and should we remove highly correlated features ...
2
votes
0answers
31 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
2
votes
4answers
34 views

Deep Learning model for classifying skin diseases

I have planned to create a deep learning model that classifies skin diseases(around 5 to 7 diseases). Please suggest me a good deep learning model to go with. I am planning to integrate this model as ...
2
votes
1answer
22 views

Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., ...
2
votes
0answers
44 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
2
votes
2answers
51 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
2
votes
0answers
52 views

Regression dataset with categorical features

I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties: Be a tabular dataset (no images). Have at least 20k rows,...
2
votes
1answer
1k views

How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have ...
2
votes
1answer
109 views

PointCloud value prediction - regression

I have point clouds for 400 objects. I have also some other features of these objects. For example their weights. How I can now train model to predict the weight based on the shape(point cloud)? I ...
2
votes
1answer
249 views

should I shift a dataset to use it for Time series regression with RNN/LSTM?

I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1 so my ...
2
votes
1answer
55 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
2
votes
1answer
50 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities'. The labeled data (72,000+ entries with all clusters together) with two columns looks like below. ...
2
votes
1answer
104 views

What should I do with the NaN values on this stock quote data?

I concatenated 3 stock quote data-frames all with date-time indexes. However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
2
votes
0answers
80 views

Organizing complex analysis project

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
2
votes
0answers
67 views

Is data science only for substantive specialist?

Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
2
votes
0answers
58 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
2
votes
1answer
57 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
2
votes
0answers
37 views

Please help to identify if there is meaning of linear variables separation by features in phase space?

Description: I have clustering problem with next input: I have truth dataset containing labels for each row of features. I've found that some features can be modified to some phase space, where they ...
2
votes
0answers
299 views

Spatial Transformer Networks and Data Augmentation

We are all familiar with the famous Deep Mind paper STN. Upon implementation, such as here, did anyone still use input data augementation such as affine transformations? There are used to make CNN ...
2
votes
0answers
401 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
2
votes
2answers
192 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
2
votes
0answers
283 views

How can I find company descriptions for a long list of companies?

I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically. E.g. given a long list of ...
2
votes
0answers
77 views

Core components of data literacy for working professionals

I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government. I've struggled to develop a concise definition of data literacy, but ...
2
votes
0answers
54 views

Free/open interactive softwares/plugins for end-users' high-dimensional data visualization

Aside from questions about How to visualize data of a multidimensional dataset (TIMIT), the Purpose of visualizing high dimensional data? and High-dimensional data: What are useful techniques to know?,...
2
votes
0answers
66 views

Tag categorizer

I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like "Cant update application" is tagged with #Apps #General #CantUpdate. My ...
2
votes
0answers
52 views

How to select a bunch of optimized data from a larger data set?

How to select a bunch of optimized data from a larger data set? I have a data set containing the products to be sold to the customers. And the products were sold before, it has some information, like ...
2
votes
0answers
58 views

What proxies could be used to assess economic value of Stackoverflow for its users?

I'm intrigued by the open data provided by StackExchange, and have been running some really interesting queries on the data.stackexchange.com page (using the Stackoverflow dataset). In particular, I ...
2
votes
2answers
57 views

How to make a dataframe with lists or vectors as its elements

This is something I have been wondering for ages but I am never able to get an answer. I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
2
votes
3answers
255 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
10 views

Is this set of vectors representing direction in a sequence trainable across LSTM? if not, what are the drawbacks?

I am starting my Neural Network Project for the first time, I have a lot of expectations: The main idea is primarily analyzing the pattern of my writings or drawings when using the mouse given a ...
1
vote
0answers
19 views

Multivariate regression - not enough data?

I have a table with data about 10 agriculture parcels. Each parcel has data in time regard the number of nutrients each parcel has received in each day and in the end I have the total number of ...
1
vote
0answers
12 views

UI-based Tool for Qualitative Evaluation of Data Quality

Dear DS StackExchange community, I'm currently searching the interwebs for a (near-)ready-to-use solution, to perform a qualitative evaluation of extracted features from video data. In my head the ...
1
vote
2answers
21 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
1
vote
0answers
14 views

Clustering dataset with and without estimating means (no EM algorithm)

Given a dataset $D$ of the form $$ D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n) $$ sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
1
vote
1answer
12 views

How to deal with mismatch of timesteps in target variable and features in forecasting problem?

Background Info: I am working with some climate data where I want to predict crop yields with my dataset containing climate- and satellite-derived features. This is a time series regression ...
1
vote
0answers
16 views

Data Pipeline Best Practices

I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
1
vote
0answers
26 views

How to compare two one hot encoded data frames based on column names?

I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
1
vote
0answers
12 views

creating dataset/storing vectors from c++ to python

I am new to the data science field. I need to create dataset which consists of n inputs (vectors of size 5) and n outputs (vectors of size > 10,000). These vectors are generated using c++ code. ...
1
vote
2answers
38 views

How do I assign specific values to categorical variables

I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
1
vote
1answer
50 views

how can I convert tensor shape to numpy array?

This is what my data looks like: I want it to be numpy array, instead of a tensor, so that i can convert it to a dataframe. Is there a way to do so? thanks in advance!
1
vote
0answers
18 views

MCMC algorithm -- understanding some paremeters

I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis. The code is something like this ...
1
vote
0answers
40 views

why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies

Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
1
vote
1answer
12 views

Is there enough datapoints to make a reasonable predcitction?

I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement. Further I would like to create a ...

1
2 3 4 5