Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
483
questions
1
vote
1answer
63 views
Technical term for using regular expressions to classify text?
Background
I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext.
He wants to classify the claims as either having been "approved", "denied", ...
0
votes
0answers
13 views
How to cluster categorical and numerical data in the same dataset?
I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some ...
0
votes
1answer
27 views
How to preprocess data for Word2Vec?
I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
1
vote
2answers
36 views
Unsupervised encoding of categorical features
I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.
...
0
votes
2answers
243 views
Tips for checking data integrity / data sanity?
I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets.
I am interested in ...
0
votes
1answer
25 views
How can I extract the reason of the legal compensation from a court report?
I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)
Order mister X to ...
1
vote
1answer
74 views
When to stop collecting data?
I have developed an iterative process via which I can collect data in batches. The data are points in a predefined 3D space. I am trying to explore and locate clusters in that 3D space based on my ...
0
votes
0answers
6 views
Virtual Generation of Synthetic Ancient and Dirty English Documents
I have a collection of dirty background image, below is the sample:
I have also a collection of an actual image of a dirty document with text on it, just like below:
My problem with my actual image ...
2
votes
1answer
49 views
When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?
I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.
This completely changed my ...
2
votes
1answer
20 views
How do I best visualize this voltage data for a science project
I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
0
votes
0answers
20 views
How do I read a dat file, for which I don't know its structure?
Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it.
How do I figure out the contents in my dat file. ...
2
votes
1answer
66 views
How to reduce position changes after dimensionality reduction?
Disclaimer: I'm a machine learning beginner.
I'm working on visualizing high dimensional data (text as tdidf vectors) into the 2D-space. My goal is to label/modify those data points and recomputing ...
3
votes
1answer
58 views
What should I do with the NaN values on this stock quote data?
I concatenated 3 stock quote data-frames all with date-time indexes.
However, they differ in starting dates so the resulting data-frame contains NaN values for the stock quotes with more recent ...
3
votes
1answer
21 views
Can numerical discrete finite data be always treated also as categorical?
In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or ...
0
votes
1answer
64 views
1
vote
1answer
13 views
Random Parameters to fix in ML to perform controlled experiments
Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy ...
1
vote
1answer
31 views
What to do about non-responses to demographic survey questions?
Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data?
In ...
0
votes
0answers
11 views
Using a GUI and ML to parse semi structured data from pdfs?
I have an issue where I have several different types of (PDF) documents. These documents are what I would call semi structured, in that there are often lines and borders around data, but the data ...
0
votes
1answer
56 views
Behavioural data required to predict churn
I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all ...
1
vote
0answers
26 views
How to estimate and/or determine how much data is enough to train a model?
To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
0
votes
1answer
24 views
should I shift a dataset to use it for Time series regression with RNN/LSTM?
I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1
so my ...
0
votes
2answers
81 views
Date transformation for KNN
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
1
vote
1answer
52 views
NAB benchmark (Numenta Anomaly Benchmark)
I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where ...
2
votes
1answer
19 views
How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?
I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
2
votes
2answers
281 views
Ways to simulate weather data over several periods (Python or R)?
I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding ...
1
vote
0answers
14 views
SEM (Structural Equation Modelling) with Exploratory Factor Analysis
Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set.
There are no assumed equations to perform SEM on so what would be ...
1
vote
1answer
76 views
Importing Excel format data into R/R Studio and using glmnet package?
I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
2
votes
0answers
34 views
Fixed-Width vs Adaptive Binning
I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
0
votes
1answer
25 views
difference btw data distribution and frquency distribution
I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2
votes
1answer
40 views
How to separate words that are together in a large data set
in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'
i tried this but this is slow
...
0
votes
2answers
38 views
How to make a dataframe with lists or vectors as its elements
This is something I have been wondering for ages but I am never able to get an answer.
I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
1
vote
2answers
1k views
What algorithms are good to predict next numbers?
Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
5
votes
1answer
70 views
8
votes
5answers
462 views
What are some of the best practices for sharing data and models with colleagues?
As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
2
votes
5answers
589 views
Generate timeseries data
Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
0
votes
1answer
42 views
How to obtain and load a good initial data set for object localization?
I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data).
What is a good initial data set to use?
How can ...
0
votes
1answer
24 views
Algorithm for user profiling without distinct profiles
I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
-1
votes
2answers
52 views
Loops in R programming, [closed]
I want to update this Remaining column in this table below:
How do I do this in R programming?
I tried
while (n.Remaining>0)
{ n.remaining <- n.total-n.expense}
Desired Output:
1
vote
0answers
32 views
Dataset for musical Instruments recognition
I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help
thanks and happy ...
1
vote
2answers
28 views
Should I use MongoDB instead of storing data in CSV in python?
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
3
votes
1answer
320 views
Smart data split (train/eval) for Object Detection
I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images.
...
1
vote
3answers
55 views
How to handle “year” variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
0
votes
1answer
37 views
Append and replacing data in r
I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs.
Here is my data ...
0
votes
0answers
27 views
Data Visualization of Feature Distribution Plot
I have been trying my best to plot this type of feature distribution plot using Python. I looked for what they are called to replicate the plot, but couldn't find the name of it?
The Reference of ...
0
votes
0answers
14 views
ValueError while Comparing with Multilevel Index
My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
2
votes
1answer
51 views
Organizing complex analysis project
Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
0
votes
1answer
51 views
Convert categorical data in numeric preserve euclidean distance
I m looking how to preserve Euclidean distance with categorical attribute.
Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "...
2
votes
1answer
35 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
0answers
10 views
access to the didi ditech2016 dataset
a bit off-topic, but I'm trying to locate a famous dataset used in the DiDi DiTech2016 supply-demand challenge. All sources I found point to a single url:
http://research.xiaojukeji.com/competition/...
2
votes
2answers
201 views
Transform single-label data set into multi-label data set
I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
