Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
381
questions
0
votes
0answers
15 views
How do I export my data into .h5 format?
I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition.
I need to prepare my own dataset in a format like this:
...
0
votes
0answers
10 views
replace or recover values in big dataset by using another database [on hold]
i am just learning to use base R.
I need to recover /replace values in time series dataset by using another dataset. There are "--" values as missing.
dataset(A)<-c("Day", "Time", ...(18 ...
0
votes
2answers
11 views
Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?
I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data.
Is there a way to include the new data without restarting ...
0
votes
0answers
22 views
How to test model accuracy on new vs. historical data?
I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more ...
0
votes
2answers
32 views
1
vote
1answer
20 views
Technical term for using regular expressions to classify text?
Background
I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext.
He wants to classify the claims as either having been "approved", "denied", ...
0
votes
1answer
20 views
Extracting information from online job postings
Okay, so I'm trying to build a data set about data science job openings.
I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
0
votes
0answers
9 views
How to generate artificial datasets in MATLAB based on behavior of real datasets?
To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
0
votes
0answers
9 views
Power BI RLS through SQL stored procedure
I need dynamic RLS for Power BI - the way that the client has their data structured is that they have SQL RLS through a stored procedure. How can I translate that into power bi?
0
votes
0answers
24 views
While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file
CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar',
'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site-
packages\tabula\tabula-1.0....
2
votes
1answer
46 views
Smart data split (train/eval) for Object Detection
I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images.
...
0
votes
1answer
52 views
ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')
X = train_encoded_df.iloc[:, 1: ]
y = train_encoded_df["Loan_Status"]
print("Precision:",metrics.precision_score(y_test, y_pred))
My training data contains the ...
2
votes
3answers
37 views
Purpose of converting continuous data to categorical data
I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data.
For example, they converted both the Age ...
0
votes
0answers
21 views
preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data
I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
1
vote
1answer
18 views
Twitter Dataset
I have found the following dataset, apparently it is the largest tweet dataset:
https://www.kaggle.com/kazanova/sentiment140
However, I am looking for a dataset of tweets, with columns containing:
...
1
vote
0answers
14 views
Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values
I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...
0
votes
0answers
17 views
Feature vector of linear model
I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this:
...
3
votes
0answers
69 views
XGBoost Huge Dataset ~1TB
Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints.
Is ...
1
vote
1answer
19 views
Read pandas data and convert it to IMMA
I'm working on a project in which my data set is in xarray and I need to write it to IMMA format. I converted it from xarray to pandas and am totally stumped on how to go from pandas to IMMA. Any ...
1
vote
1answer
24 views
how to match a sentence to a cluster of keywords?
I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below.
<...
0
votes
1answer
13 views
Machine learning model using keywords for binary decision
I have a some experience from Uni with convolutional NN and edge detection, but haven't much explored the other types of machine learning models.
I was wondering if there might be one that is suited ...
-2
votes
2answers
36 views
What is Data Science and Machine Learning and what language mostly used to program? [closed]
I am newbie in Data Science, Machine Learning and any related to data science but I want to try it. Unfortunately, googling makes it tedious and complicated so I hope to be answered by anybody who's ...
0
votes
1answer
25 views
Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?
I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
1
vote
1answer
47 views
How to run KNN (or other) on nested features? Image metadata
I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
0
votes
1answer
70 views
Import data from google drive to Kaggle Kernel
I want to import a csv file from google drive . I tried using the link in add dataset tab but it is taking some thing else as "Open". Please see the image.
2
votes
5answers
62 views
30 Years Of Excel Test Data
I am a CS intern at an industrial company that has 30 years of excel files that need to be analyzed. Looking at the data, only a fraction of the files need to be looked at and used. After those files ...
2
votes
1answer
50 views
Survival regression with major event that won't happen
I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "...
0
votes
1answer
25 views
How to club the orders in such a way that maximum number of items are common amongst them?
Consider the following data set:
The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst ...
0
votes
1answer
31 views
Searching prediction from 4 datasets
The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
1
vote
1answer
24 views
Data duplication optimization
I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup...
2
votes
2answers
50 views
Generate timeseries data
Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
0
votes
0answers
6 views
Use “big data” (via batchwise streaming) in boosting or neural nets
I often work with boosting (e.g. lightgbm) and neural nets (e.g. Keras) but I usually work with data that is "small enough" to be loaded in the RAM memory as a whole (except in the case of images ...
1
vote
1answer
35 views
How to create a system to detect text structure of a file?
Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
0answers
25 views
Predict vs. Impute: Filling missing data using Random Forest
I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals ...
0
votes
0answers
3 views
Are there any tools, packages or frameworks that are used to record model parameters, the number of kfols, local cv and so on?
I have been trying to do some experiments or data competitions but I always try to record the parameters of the model, the number of kfolds, local cv and so on.
So I am wondering if there is some ...
1
vote
0answers
49 views
How to reduce position changes after dimensionality reduction?
Disclaimer: I'm a machine learning beginner.
I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
0
votes
0answers
11 views
We know the subspace generated from the data instances, but we cannot constitute the origin space
I was wondering, what if we know the subspace generated F from the data instances, but we cannot constitute the origin space E that can be in higher dimension, and can easily lead us to the true join ...
1
vote
0answers
18 views
System administration skills for data engineering [closed]
As a data scientist student, I will do an internship with a lot of data engineering tasks.
It would be good if I had some initiation skills in sys-admins.
This leads me to my questions:
Why should ...
0
votes
2answers
33 views
Tips for checking data integrity / data sanity?
I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets.
I am interested in ...
0
votes
0answers
17 views
Techniques for Collection/Graph Conversion to Cluster-able Data
Here are my problems; any techniques/papers as to how to approach this problem would be much appreciated. I also apologize for the vagueness of my question title; I do not really know if there is a ...
0
votes
1answer
12 views
3
votes
1answer
35 views
What should I do with the NaN values on this stock quote data?
I concatenated 3 stock quote data-frames all with date-time indexes.
However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
0
votes
0answers
6 views
What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?
Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
1answer
16 views
NAB benchmark (Numenta Anomaly Benchmark)
I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where ...
2
votes
1answer
27 views
How to choose an optimal threshold for binary discretization
We know that we usually do discretizations to continuous features to remove extra information and unwanted regularities, which makes the model robust and well-predicted.
But I am wondering except ...
3
votes
2answers
41 views
How to train ML algorithm with multiple values in target data?
I am new to data science and machine learning and looking for some help.
I am trying to train a machine with following data set:
Here, L3 is the target variable.
As it can be seen that the target ...
3
votes
1answer
38 views
What happens to the left over unpicked data in Random Forest
I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not ...
2
votes
2answers
20 views
why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding
We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data.
And I real a blog which give the difference between Label Encoding and One-Hot Encoding.
So ...
0
votes
0answers
9 views
Automated Spatial data mining tools in python
How can spatial data raster be processed in order to apply random forest prediction algorithm on it using python programming language?
0
votes
0answers
32 views
Deep learning not classify some classes
I want to classify 13 classes, but 2 classes is not classified.
This is the test result.
As you can see in the image, class 3 and class 11 is not detected.
It's simple vector to class classification.
...
