Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

Python Function to Incorporate Data Transform

I am new to python and I have developed a model and like to create a function for API use. For example: predict(1,2,1) <-- call the function if the first argument is 1 and it has 7 categories, ...
1
vote
0answers
20 views

NearestNeighbors testing

I have used nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features) distances, indices = nbrs.kneighbors(item_features) to find some ...
3
votes
1answer
140 views

Combining time-series data from different devices

I am using four different sensor models (four different devices) to gather some biometric data(eg. heart rate) for the same user, for the same time period. In an ideal scenario, all the devices are ...
1
vote
2answers
176 views

How to cluster categorical and numerical data in the same dataset?

I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some ...
1
vote
1answer
2k views

How to preprocess data for Word2Vec?

I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
2
votes
1answer
93 views

When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould. This completely changed my ...
0
votes
2answers
9k views

How do I read a dat file, for which I don't know its structure?

Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it. How do I figure out the contents in my dat file. ...
3
votes
1answer
33 views

Can numerical discrete finite data be always treated also as categorical?

In many sources, for example here data is classified as being qualitative (categorical) and quantitative (numerical). Where numerical data can be continuous or discrete, and discrete can be finite or ...
1
vote
1answer
93 views

Random Parameters to fix in ML to perform controlled experiments

Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy ...
2
votes
2answers
626 views

Ways to simulate weather data over several periods (Python or R)?

I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding ...
3
votes
1answer
67 views

Estimating and/or determining the amount of data sufficient enough to train a model

To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
3
votes
1answer
42 views

How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?

I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
2
votes
1answer
397 views

Fixed-Width vs Adaptive Binning

I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
0
votes
1answer
51 views

difference btw data distribution and frquency distribution

I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2
votes
1answer
51 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
1
vote
0answers
17 views

SEM (Structural Equation Modelling) with Exploratory Factor Analysis

Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set. There are no assumed equations to perform SEM on so what would be ...
5
votes
1answer
220 views

Machine learning methods for panel (longitudinal) data

I have a panel data set, for example: ...
0
votes
1answer
32 views

How can I extract the reason of the legal compensation from a court report?

I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report) Order mister X to ...
3
votes
1answer
32 views

How do I best visualize this voltage data for a science project

I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
1
vote
0answers
35 views

Dataset for musical Instruments recognition

I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help thanks and happy ...
1
vote
2answers
354 views

Should I use MongoDB instead of storing data in CSV in python?

I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
1
vote
0answers
127 views

Data Visualization of Feature Distribution Plot

I have been trying my best to plot this type of feature distribution plot using Python. I looked for what they are called to replicate the plot, but couldn't find the name of it? The Reference of ...
1
vote
1answer
1k views

What's the best way to train a NER model?

I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used… To do that I want to ...
2
votes
1answer
2k views

Are parquet files compressed? [closed]

Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
2
votes
0answers
88 views

PointCloud value prediction - regression

I have point clouds for 400 objects. I have also some other features of these objects. For example their weights. How I can now train model to predict the weight based on the shape(point cloud)? I ...
0
votes
1answer
40 views

how to know what my data are uninformative and that machine learning will not work with it?

following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I ...
2
votes
1answer
182 views

should I shift a dataset to use it for Time series regression with RNN/LSTM?

I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1 so my ...
2
votes
2answers
99 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
0answers
15 views

How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?

I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
1
vote
1answer
23 views

Help getting corresponding dataframe values

I have two dataframes: ...
1
vote
2answers
266 views

Unsupervised encoding of categorical features

I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding. ...
1
vote
0answers
15 views

subrandomsapler dataloader pytorch lengh does not seems right

...
0
votes
2answers
78 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
26 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
1
vote
2answers
93 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
2
votes
3answers
446 views

How to handle “year” variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
2
votes
1answer
73 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
1answer
115 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
1
vote
2answers
55 views

Clearest way to visualise temporal data?

I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
24 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
1answer
34 views

Free API for historical weather US data?

I am trying to retrieve a free R-Python API that provides historical weather data in US. In fact wunderground API is no longer available. Any suggestion?
2
votes
1answer
56 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
0
votes
0answers
45 views

How to sort of “average” data without smoothing?

I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time. The problem ...
2
votes
1answer
258 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
2
votes
1answer
3k views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
2
votes
1answer
44 views

Is there a way to aggregate frequency distribution for many runs?

In the following plot I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
1
vote
1answer
42 views

Best file format for transfer of EHR data

I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
1
vote
0answers
887 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
7
votes
2answers
2k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
5
votes
2answers
3k views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...

1
3 4
5
6 7
13