Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
0
votes
1answer
5 views
what is step by step if I want to search prediction from 4 datasets?
the fourth data is (train_data, test_data, previous_data, and information_history_data). The goals is searching user's rate on the loan to bank. I confused first step to doing because there many ...
0
votes
0answers
6 views
How do I record and plot pH meter data in real time into an excel sheet? [on hold]
I'm attempting to set up a system where I can record and plot the pH value of a solution in real time. I have a pH meter, a usb-1208ls, my laptop, and many different programs at my disposal. I am ...
1
vote
1answer
7 views
data duplication optimisation
I am working in python3 cleaning data. I have a large number of midi files from scraped from a variety of sources using beautiful soup. Many of them may be duplicates. Some of them have only numbers ...
1
vote
1answer
18 views
Generate timeseries data
Training would be bad if training data is not sufficient. Techniques like SMOTE, ADASYN can be used for oversampling. For Image data, we can blur or change angle to generate more samples from the same ...
0
votes
0answers
3 views
Use “big data” (via batchwise streaming) in boosting or neural nets
I often work with boosting (e.g. lightgbm) and neural nets (e.g. Keras) but I usually work with data that is "small enough" to be loaded in the RAM memory as a whole (except in the case of images ...
1
vote
1answer
27 views
How to create a system to detect text structure of a file?
Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
0answers
14 views
Predict vs. Impute: Filling missing data using Random Forest
I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals ...
0
votes
0answers
3 views
Are there any tools, packages or frameworks that are used to record model parameters, the number of kfols, local cv and so on?
I have been trying to do some experiments or data competitions but I always try to record the parameters of the model, the number of kfolds, local cv and so on.
So I am wondering if there is some ...
1
vote
0answers
45 views
How to reduce position changes after dimensionality reduction?
Disclaimer: I'm a machine learning beginner.
I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
0
votes
0answers
19 views
dummy variables shows nan value? [closed]
I am using dummy on one of my data frame column i.e sex
but after checking the head of my data frame! It displayed sex column with nan values.
Code -
...
-1
votes
0answers
12 views
Extract the unique genres and its count and store in data frame with index key in python [closed]
I am using this, not sure how to take index.
df_genre=pd.DataFrame(df_csv['Genre'].value_counts())
0
votes
0answers
10 views
We know the subspace generated from the data instances, but we cannot constitute the origin space
I was wondering, what if we know the subspace generated F from the data instances, but we cannot constitute the origin space E that can be in higher dimension, and can easily lead us to the true join ...
1
vote
0answers
16 views
System administration skills for data engineering [closed]
As a data scientist student, I will do an internship with a lot of data engineering tasks.
It would be good if I had some initiation skills in sys-admins.
This leads me to my questions:
Why should ...
0
votes
2answers
22 views
Tips for checking data integrity / data sanity?
I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets.
I am interested in ...
0
votes
0answers
17 views
Techniques for Collection/Graph Conversion to Cluster-able Data
Here are my problems; any techniques/papers as to how to approach this problem would be much appreciated. I also apologize for the vagueness of my question title; I do not really know if there is a ...
0
votes
1answer
9 views
3
votes
1answer
27 views
What should I do with the NaN values on this stock quote data?
I concatenated 3 stock quote data-frames all with date-time indexes.
However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
0
votes
0answers
5 views
What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?
Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
0
votes
1answer
10 views
NAB benchmark (Numenta Anomaly Benchmark)
I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where ...
2
votes
1answer
23 views
How to choose an optimal threshold for binary discretization
We know that we usually do discretizations to continuous features to remove extra information and unwanted regularities, which makes the model robust and well-predicted.
But I am wondering except ...
3
votes
2answers
33 views
How to train ML algorithm with multiple values in target data?
I am new to data science and machine learning and looking for some help.
I am trying to train a machine with following data set:
Here, L3 is the target variable.
As it can be seen that the target ...
3
votes
1answer
37 views
What happens to the left over unpicked data in Random Forest
I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not ...
2
votes
2answers
14 views
why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding
We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data.
And I real a blog which give the difference between Label Encoding and One-Hot Encoding.
So ...
0
votes
0answers
8 views
Automated Spatial data mining tools in python
How can spatial data raster be processed in order to apply random forest prediction algorithm on it using python programming language?
0
votes
0answers
30 views
Deep learning not classify some classes
I want to classify 13 classes, but 2 classes is not classified.
This is the test result.
As you can see in the image, class 3 and class 11 is not detected.
It's simple vector to class classification.
...
0
votes
2answers
40 views
Is there any option to represent 160 discrete variables each one with their own color in scatter plot?
Good morning,
I want to know if it is possible to represent with different colors in scatter plot more than 10 parameters instead of putting others and assign a color if the amount of parameters ...
0
votes
0answers
37 views
Practical use of bootstrap in R
I am a beginner and try to perform a bootstrap in R. It would be nice if somebody could help me with the following situation.
I have defined prices and I want to randomly generate quantities (given a ...
4
votes
1answer
40 views
Model for Differing Number of Rows per Observation
Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the ...
0
votes
0answers
24 views
Isolation forest parameters
I am implementing Isolation Forest from scratch, I am in dilemma of how to set contamination value, and what if not used im the code. Is accuracy differ? What other parameters required to run ...
0
votes
1answer
20 views
How should I collect data for an “identifier” rather than classifier?
I'm trying to build a CNN to identify whether or not a picture contains a lion. One of my classes would obviously be filled with pictures of lions, and pictures that have lions.
What about the data I ...
0
votes
0answers
28 views
ways to test simulated data quality
I have a huge data set of logistic objects which I generated by creating a 3D model and rendering it in blender. In order to test the generated data, I've downloaded some images. The simulated data ...
0
votes
0answers
43 views
creating a dataframe out of nested loop data [on hold]
I have three parameter arrays ,Parameter1 = [6,7,8],Parameter2 = [11,12] and Parameter3 which can be calculated using Parameter1 and Parameter2. There is a mapping between parameter 1 and 2.
I want ...
0
votes
2answers
23 views
What fits in a Data Description Report/ Data Exploration Report?
So I am trying to get familiar with Crisp-DM and found the terms "Data Description Report" and "Data Exploration Report", which seem oddly vague in their definition. So far I only found this right ...
4
votes
1answer
257 views
How to handle columns with categorical data and many unique values
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.
I also have another column with 145 nunique values that I could also use ...
3
votes
2answers
32 views
Rearranging data frame from column names to key value pairs
If I have data like this in R:
> a = data.frame(a = c(1, 2),
b = c(3, 4))
> a
a b
1 1 3
2 2 4
but I would like to have it like this:
<...
4
votes
2answers
72 views
Aggregate NumPy array with condition as mask
I have a matrix $b$ with elements:
$$b =
\begin{pmatrix}
0.01 & 0.02 & \cdots & 1 \\
0.01 & 0.02 & \cdots & 1 \\
\vdots& \vdots & \ddots & \vdots \\
0.01 &...
2
votes
1answer
72 views
How to penalize for empty fields in a DataFrame?
I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation ...
1
vote
0answers
16 views
Is the Apriori algorithm suitable for database tuples?
Problem description:
I need to use an association rule algorithm that lets me use database tuples and I think Apriori is a good option, but I am not sure.
Starting point:
I know the Apriori ...
0
votes
1answer
18 views
Date transformation for KNN to get distance [duplicate]
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.
For ...
0
votes
1answer
34 views
Date transformation for KNN
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
0
votes
2answers
26 views
Reading Date Time in R (01.01.2019 00:00)
How do you read a date in R that is given in the following way:
01.01.2019 00:00
While reading it with read.csv("my_file.csv", sep = ";", dec = ",") I am getting ...
1
vote
1answer
26 views
Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?
Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches ...
0
votes
1answer
30 views
Append and replacing data in r
I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs.
Here is my data ...
3
votes
3answers
80 views
How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?
Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same ...
0
votes
1answer
31 views
Convert categorical data in numeric preserve euclidean distance
I m looking how to preserve Euclidean distance with categorical attribute.
Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "...
1
vote
1answer
47 views
Automatic labelling of text data based on predefined entities
I'm new to NLP.
I have a folder containing .txt files which are legal and specific documents. I want to label all these files based on four predefined labels. How can I do that automatically?
0
votes
2answers
38 views
Loops in R programming,
I want to update this Remaining column in this table below:
How do I do this in R programming?
I tried
while (n.Remaining>0)
{ n.remaining <- n.total-n.expense}
Desired Output:
0
votes
1answer
21 views
Can you apply PCA to part of your dataset?
I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
0
votes
1answer
24 views
How to make a dataframe with lists or vectors as its elements
This is something I have been wondering for ages but I am never able to get an answer.
I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
1
vote
1answer
41 views
What kind of data visualization should I use?
I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
