Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Ideas on handling large quantities of “waste” data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
0
votes
0answers
10 views

Clarification on an anomalous dataset

Suppose I am concerned with anomaly detection on time series data for a signal, such as vehicle speed data. Is it a valid approach to train a machine learning model, for example, on datasets of ...
0
votes
1answer
12 views

County-level 2020 election data? Help!

I was using the data from favstats' github where he scraped NYT data, but it's gone now for whatever reason. Does anyone know where I could find this data? I'm in the middle of a huge project and didn'...
1
vote
1answer
440 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
1answer
54 views

What to do if a specific label of a category appears only a few times?

Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
9
votes
6answers
603 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
0
votes
2answers
47 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
1
vote
2answers
26 views

Tools for nutritional value tracking and visualization

I want to track the nutritional value of different flours over time. What I got so far is a data table like this: ...
-1
votes
0answers
19 views

Normal distribution histogram [closed]

How I can create histogram for normal distribution without using built-in function in python? The user should enter the value of mean and std first....
0
votes
1answer
6k views

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

X = train_encoded_df.iloc[:, 1: ] y = train_encoded_df["Loan_Status"] print("Precision:",metrics.precision_score(y_test, y_pred)) My training data contains the ...
-1
votes
1answer
24 views

Can we identify that an academic dataset was used for commercial purpose [closed]

There are many datasets released on the internet. Authors of many of these datasets state that the datasets are strictly for academic usage and not for commercial purposes. Although some datasets are ...
0
votes
1answer
33 views

How to input sets as features

Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on) This is from the online order processing domain. An order consists of a set of variable ...
0
votes
0answers
10 views

Usage of Doc2Vec as feature extractor for text classification of websites with political articles

I have gathered political articles from polish websites for my engineering thesis. The main goal is to try to predict the website that input text belongs to. So for this few websites I want to create ...
0
votes
1answer
120 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
0
votes
2answers
114 views

How to cluster categorical and numerical data in the same dataset?

I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some ...
0
votes
1answer
17 views

Hand preprocessing data

I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going ...
0
votes
1answer
17 views

How to increase sales and revenue of a Client?

I was asked this in an interview for a Data Scientist position: Lets say Holland and Barret came to you and said they'd like to increase their sales and revenue. How will you go about it? My answer ...
1
vote
0answers
11 views

Marketing Channel Recommender System [closed]

This dataset is collected by a drug making company trying to sell its drug to doctors of different specializations. The drug company has made promotional activity for its brand The promotional ...
1
vote
0answers
123 views

Self Union All in Tableau Public for Sankey Diagram

I'm following this 2017 YouTube tutorial from which I hope to learn Sankey diagrams. At 2:28, he instructs us to select Convert to Custom SQL, after having opened ...
1
vote
1answer
32 views

adding a feature as “generic”

I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:...
2
votes
1answer
842 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
0
votes
1answer
189 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
0answers
5 views

How to change data source names in Google Data Studio

Google Data Studio has connectors to MySQL, PostgreSQL, etc. but these connections will come with default names. Couldn't find out how to set the names for data sources in Google Data Studio. Is it ...
1
vote
1answer
14 views

How to conclude the generality of any classification methods?

Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$. The task $A$ is measured by a consistent measure. For instance, the task A can be a binary classification. In this ...
1
vote
1answer
24 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
0
votes
1answer
11 views

Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations?

Use cases and Solutions surrounding GPT's have taken NLP world with storm and started the GPT-Best vs GPT- Not So Best war on the internet. There are solutions been derived from API's provided by HF. ...
0
votes
1answer
15 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
0
votes
1answer
22 views

Model inclines because of imbalanced data

In life, some events are rare and most cases are normal. So I am wondering, to detect the rare cases, shall we use an imbalanced dataset with more historical rare cases? Taking the German Credit Data, ...
0
votes
0answers
7 views

Generating prediction when adding new rows into a relational dataset

There is a PostgreSQL table that I regularly update with new book reviews. Meanwhile, I also have trained a Tensorflow model that takes book review text and return a predicted book review category (an ...
1
vote
1answer
46 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below. <...
4
votes
3answers
1k views

Deploying machine learning modules

I am looking to find some resources about what I wan't to do :I wan't to make some GUI of my machine learning models and finally deploy them as a web app.I find R Shiny to be somehow ok , but it ...
1
vote
1answer
436 views

How to determine the weights of features when calculating the weighted average?

I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
3
votes
3answers
77 views

Continuous VS Categorical variable

In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are both highly correlated. ...
0
votes
1answer
69 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
0
votes
1answer
29 views

Higher variance in PCA can mean, that data structure is less informative?

I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with ...
1
vote
1answer
63 views

Find the mode value and frequency in R

I'm trying to come up with a function in R that gives the mode value of a column along with the number of times (or frequency) that the value occurs. I want it to exclude missing (or blank) values, ...
2
votes
1answer
45 views

Can we use two independently measured features in a same ML model? [closed]

Two features are measured at different times but belong to the same target. In which conditions or form these features can be modeled together? Or they shouldn't be used in the same model but modeled ...
2
votes
1answer
158 views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
1
vote
1answer
159 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
1
vote
0answers
13 views

How to aggregate Weather Data from county to state level? [closed]

I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident ...
4
votes
2answers
2k views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
1
vote
1answer
24 views

Project structure - many projects share same large dataset

I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I ...
0
votes
1answer
15 views

Suggestion of dataset

I am implementing my own deep network, but I am not so good at calculus so my network only works for binary data in the moment. I have been searching for big tabular datasets that are for binary ...
1
vote
1answer
21 views

Is window based sequencing a good idea to obtain more training data for LSTMs?

I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier. However,...
1
vote
2answers
102 views

Sentiment Analysis Datasets

I am looking for sentiment analysis data, mostly customer product review. I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
0
votes
1answer
42 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
1
vote
1answer
79 views

How to create a system to detect text structure of a file?

Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
1answer
47 views

difference btw data distribution and frquency distribution

I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2
votes
2answers
27 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
1answer
75 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...

1
2 3 4 5
12