Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
0
votes
0answers
10 views

replace or recover values in big dataset by using another database [on hold]

i am just learning to use base R. I need to recover /replace values in time series dataset by using another dataset. There are "--" values as missing. dataset(A)<-c("Day", "Time", ...(18 ...
0
votes
2answers
11 views

Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?

I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data. Is there a way to include the new data without restarting ...
0
votes
0answers
22 views

How to test model accuracy on new vs. historical data?

I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more ...
0
votes
2answers
32 views

Suggestion of a model for these type of data?

I've got a data set that looks like this ...
1
vote
1answer
20 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
1answer
20 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
0
votes
0answers
9 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
0
votes
0answers
9 views

Power BI RLS through SQL stored procedure

I need dynamic RLS for Power BI - the way that the client has their data structured is that they have SQL RLS through a stored procedure. How can I translate that into power bi?
0
votes
0answers
24 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
2
votes
1answer
46 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
0
votes
1answer
52 views

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

X = train_encoded_df.iloc[:, 1: ] y = train_encoded_df["Loan_Status"] print("Precision:",metrics.precision_score(y_test, y_pred)) My training data contains the ...
2
votes
3answers
37 views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
0
votes
0answers
21 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
1
vote
1answer
18 views

Twitter Dataset

I have found the following dataset, apparently it is the largest tweet dataset: https://www.kaggle.com/kazanova/sentiment140 However, I am looking for a dataset of tweets, with columns containing: ...
1
vote
0answers
14 views

Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values

I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...
0
votes
0answers
17 views

Feature vector of linear model

I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this: ...
3
votes
0answers
69 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
1
vote
1answer
19 views

Read pandas data and convert it to IMMA

I'm working on a project in which my data set is in xarray and I need to write it to IMMA format. I converted it from xarray to pandas and am totally stumped on how to go from pandas to IMMA. Any ...
1
vote
1answer
24 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below. <...
0
votes
1answer
13 views

Machine learning model using keywords for binary decision

I have a some experience from Uni with convolutional NN and edge detection, but haven't much explored the other types of machine learning models. I was wondering if there might be one that is suited ...
-2
votes
2answers
36 views

What is Data Science and Machine Learning and what language mostly used to program? [closed]

I am newbie in Data Science, Machine Learning and any related to data science but I want to try it. Unfortunately, googling makes it tedious and complicated so I hope to be answered by anybody who's ...
0
votes
1answer
25 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
1
vote
1answer
47 views

How to run KNN (or other) on nested features? Image metadata

I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
0
votes
1answer
70 views

Import data from google drive to Kaggle Kernel

I want to import a csv file from google drive . I tried using the link in add dataset tab but it is taking some thing else as "Open". Please see the image.
2
votes
5answers
62 views

30 Years Of Excel Test Data

I am a CS intern at an industrial company that has 30 years of excel files that need to be analyzed. Looking at the data, only a fraction of the files need to be looked at and used. After those files ...
2
votes
1answer
50 views

Survival regression with major event that won't happen

I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "...
0
votes
1answer
25 views

How to club the orders in such a way that maximum number of items are common amongst them?

Consider the following data set: The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst ...
0
votes
1answer
31 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
1
vote
1answer
24 views

Data duplication optimization

I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup...
2
votes
2answers
50 views

Generate timeseries data

Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
0
votes
0answers
6 views

Use “big data” (via batchwise streaming) in boosting or neural nets

I often work with boosting (e.g. lightgbm) and neural nets (e.g. Keras) but I usually work with data that is "small enough" to be loaded in the RAM memory as a whole (except in the case of images ...
1
vote
1answer
35 views

How to create a system to detect text structure of a file?

Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
0answers
25 views

Predict vs. Impute: Filling missing data using Random Forest

I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals ...
0
votes
0answers
3 views

Are there any tools, packages or frameworks that are used to record model parameters, the number of kfols, local cv and so on?

I have been trying to do some experiments or data competitions but I always try to record the parameters of the model, the number of kfolds, local cv and so on. So I am wondering if there is some ...
1
vote
0answers
49 views

How to reduce position changes after dimensionality reduction?

Disclaimer: I'm a machine learning beginner. I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
0
votes
0answers
11 views

We know the subspace generated from the data instances, but we cannot constitute the origin space

I was wondering, what if we know the subspace generated F from the data instances, but we cannot constitute the origin space E that can be in higher dimension, and can easily lead us to the true join ...
1
vote
0answers
18 views

System administration skills for data engineering [closed]

As a data scientist student, I will do an internship with a lot of data engineering tasks. It would be good if I had some initiation skills in sys-admins. This leads me to my questions: Why should ...
0
votes
2answers
33 views

Tips for checking data integrity / data sanity?

I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets. I am interested in ...
0
votes
0answers
17 views

Techniques for Collection/Graph Conversion to Cluster-able Data

Here are my problems; any techniques/papers as to how to approach this problem would be much appreciated. I also apologize for the vagueness of my question title; I do not really know if there is a ...
3
votes
1answer
35 views

What should I do with the NaN values on this stock quote data?

I concatenated 3 stock quote data-frames all with date-time indexes. However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
0
votes
0answers
6 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
1answer
16 views

NAB benchmark (Numenta Anomaly Benchmark)

I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where ...
2
votes
1answer
27 views

How to choose an optimal threshold for binary discretization

We know that we usually do discretizations to continuous features to remove extra information and unwanted regularities, which makes the model robust and well-predicted. But I am wondering except ...
3
votes
2answers
41 views

How to train ML algorithm with multiple values in target data?

I am new to data science and machine learning and looking for some help. I am trying to train a machine with following data set: Here, L3 is the target variable. As it can be seen that the target ...
3
votes
1answer
38 views

What happens to the left over unpicked data in Random Forest

I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not ...
2
votes
2answers
20 views

why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding

We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data. And I real a blog which give the difference between Label Encoding and One-Hot Encoding. So ...
0
votes
0answers
9 views

Automated Spatial data mining tools in python

How can spatial data raster be processed in order to apply random forest prediction algorithm on it using python programming language?
0
votes
0answers
32 views

Deep learning not classify some classes

I want to classify 13 classes, but 2 classes is not classified. This is the test result. As you can see in the image, class 3 and class 11 is not detected. It's simple vector to class classification. ...