Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
27
votes
1answer
52k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
22
votes
4answers
17k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
19
votes
1answer
19k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
15
votes
5answers
12k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
12
votes
2answers
10k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
11
votes
7answers
4k views

Interactive labeling/annotating of time series data

I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
10
votes
3answers
23k views

How to create US state heatmap

I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it ...
10
votes
2answers
7k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
9
votes
4answers
15k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
8
votes
2answers
63k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
8
votes
1answer
8k views

What are the most suitable machine learning algorithms according to type of data? [closed]

I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical). I searched about this ...
8
votes
6answers
2k views

Generate timeseries data

Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
8
votes
1answer
2k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
8
votes
5answers
570 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
8
votes
2answers
82 views

Model for Differing Number of Rows per Observation

Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the ...
7
votes
1answer
10k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...
7
votes
6answers
961 views

Is it advisable to combine two dataset?

I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based ...
7
votes
4answers
385 views

How to handle the CEO expectations from a company that's new to data science?

I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, ...
7
votes
1answer
10k views

How to generate training data for OCR

I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
7
votes
3answers
195 views

Docker for data science

I recently started to read articles about Docker. To me, in data science, Docker is useful because: 1) You have a totally different environment, which protect you against libraries and dependencies ...
7
votes
2answers
1k views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
7
votes
2answers
126 views

How important is advanced SQL for data science?

Is advanced level SQL required to be competitive as a data scientist? Is it more important for a data analyst to be good in SQL? Is it enough to be able to extract data using simple SQL queries? I ...
7
votes
2answers
2k views

IID violation in machine learning

Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix. I train some model (lets say a neural ...
6
votes
4answers
6k views

Why aren't languages like C, C++ used for data analytics instead of R, Python?

I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++. I have not come across many instances (close ...
6
votes
3answers
2k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
6
votes
3answers
937 views

XGBoost Huge Dataset ~1TB

Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints. Is ...
6
votes
1answer
64 views

Un-learning a single training example from a trained model

I was going through the paper "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" by google on which suggests best practices for models in production. In a section ...
5
votes
4answers
337 views

General way to reduce features

Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run. What are some of the best ways to find the most influential columns/...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
8answers
584 views

Skills that school doesn't teach you

I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
5
votes
1answer
2k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
5
votes
1answer
2k views

Purpose of weights in neural networks

I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing. I know, that before the inputs are ...
5
votes
2answers
16k views

Merging large CSV files in Pandas

I have two CSV files(each of the file size is in GBs). I am trying to merge the two CSV files but every time I try to it my computer hangs. Is there no way to merge the files in chunks in pandas ...
5
votes
1answer
327 views

One Hot Encoding for any kind of dataset

How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how ...
5
votes
1answer
6k views

Small data set in machine learning [closed]

I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business. ...
5
votes
1answer
3k views

Can I scrape data from government websites if there is no mention about commercial usage?

I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (...
5
votes
2answers
346 views

How to train ML algorithm with multiple values in target data?

I am new to data science and machine learning and looking for some help. I am trying to train a machine with following data set: Here, L3 is the target variable. As it can be seen that the target ...
5
votes
1answer
1k views

Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

I am creating decision trees modeling data that looks like this. ...
5
votes
1answer
172 views

Machine learning methods for panel (longitudinal) data

I have a panel data set, for example: ...
5
votes
1answer
93 views

Finding aggregated information of data

I am new to data science. I have a dataset of around 200,000 records, having 5 columns. There is a field called, class. For each ...
4
votes
4answers
510 views

How to know for sure if we can learn from a given data or not?

I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?
4
votes
2answers
2k views

Tool to Generate 2D Data via Mouse Clicking

Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or ...
4
votes
2answers
1k views

Recommendations for storing time series data

As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, ...
4
votes
2answers
515 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...
4
votes
3answers
3k views

What are the methods to ensure that the population split for A/B test is random?

Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.
4
votes
3answers
347 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
4
votes
3answers
173 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
4
votes
2answers
143 views

What kind of research can be done with genomic data?

It is well known that science has given us large amounts of free accessible data, such as https://www.1000genomes.org and https://www.ncbi.nlm.nih.gov/genbank. How can we play around with the data and ...
4
votes
1answer
106 views

How to generate bulk graphics using R

I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2. How can I automatically generate one graphic for each ...
4
votes
2answers
629 views

What is the most used format to save data with type information

I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R. ...

1
2 3 4 5
12