Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
2
votes
1answer
51 views

Convert a csv file so that all attributes will become columns with corresponding element [closed]

so I have this dataset in a csv file. I would like to convert it into a matrix form. The data currently looks like this: ...
0
votes
1answer
70 views

Image data augmentation with this function

Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py ...
0
votes
1answer
51 views

Handling imbalanced data by deleting over represented rows vs. adding under represented rows

I am currently working with a very imbalanced data set (frauded credit card data from kaggle, which has 492 rows of frauded cards and over 280,000 rows of non-frauded cards). As much as I know, there ...
1
vote
0answers
197 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
3
votes
2answers
45 views

How to generate data if algo itself is involved in the process with a feedback loop?

I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo. But I am having issues and ...
2
votes
1answer
34 views

Algortihm for making predictions from minimal data

I am working with classification problem. I have a dataset with a lot of features. A lot of them can easily determine class. On production, I want to ask the user to provide me with only part of the ...
9
votes
7answers
4k views

Interactive labeling/annotating of time series data

I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
1
vote
0answers
19 views

Having the target and data row - how to find the correct model

I am coming from a web developer background, and having no data science background. I am looking to skill-up and get some answers for a current task: find a model that returns a predicted value same ...
4
votes
4answers
510 views

How to know for sure if we can learn from a given data or not?

I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?
1
vote
1answer
33 views

Prepare data : many products per row

I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints : A Product is defined by {id, price, continuousVariable1, ..., ...
1
vote
1answer
344 views

Is there an API to extract questions and labels from yahoo answers?

I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it ...
1
vote
0answers
46 views

Time Series Autocorrelation Estimation

How do I find joint PDF f(xi,xj) needed for estimating autocorrelation from 1-D time series data X, using the kernel windowing technique? I am well aware of kernel windowing technique for estimation ...
4
votes
3answers
172 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
11 views

Where to find Facial Visual Object Class

I am trying to make a face/head detection model using YOLO and darkflow. In order to do this I need a set of images peoples face annotated in Visual Object Class, VOC format. I would prefer data set ...
3
votes
0answers
90 views

Deep Learning: Does starting the training on a smaller subset of the data make sense?

I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the ...
2
votes
1answer
85 views

Knowledge of statistics To be a data scientist [closed]

I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science ...
1
vote
0answers
329 views

Good dataset for sentiment Analysis in Tickets for IT Support

I am working on Sentiment Analysis in Tickets for IT Support, But I have only seen the movie reviews and Twitter sentiment analysis which are not related to my work. I found a research paper related ...
1
vote
0answers
26 views

Deciding number of classes

I have a dataset of 1000 labels. I would like to build a custom model from this dataset with three labels, say Dog, Cat, and <...
0
votes
2answers
201 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
0
votes
1answer
60 views

How to give importance to recent data in a large dataset

I am solving a binary classification problem which is also partly a time-series problem. This means that recent data is more relevant than old data. But how can I assign more importance to the recent ...
1
vote
2answers
84 views

How to scale numbers according to proportions

I have some data that I'm trying to represent in a visualization with a week-over-week change. So the source can be something like: ...
1
vote
2answers
265 views

data science python data cleaning

I am preparing a dataset for a model, but somehow the code just doesn't run well. The major error is: ...
1
vote
1answer
30 views

Entropy in a closed box

I have not understood why the maximum number of binary questions needed to reduce uncertainty is essentially log(T). If i have a ball in a box and 10 possible classes, is it enough to ask log(10) ...
0
votes
1answer
320 views

Sparse Data Interpolation

I have a price data set where on some days there are up to five data points and some days none at all. For example: ...
0
votes
1answer
23 views

How do I store a question-and-answer combination for machine learning?

I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the ...
0
votes
3answers
2k views

Retraining a machine learning model [closed]

Created a Machine Learning model on some data. Used this model for predicting test data. Model has learnt only from training data. How this model can be retrained as new data come in for prediction. I ...
0
votes
2answers
74 views

Best ML model for predicting yearly data with many blocks?

I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect. I would like to use a ...
4
votes
2answers
469 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...
1
vote
1answer
318 views

Using DONUT algorithm with keras

I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows: ...
0
votes
1answer
3k views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
1
vote
1answer
37 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished the race in 45 seconds,35 ...
0
votes
1answer
45 views

Are there enough databases for all learning tasks?

this might be a silly question but I guess the answer comes with experience in this field. I'm just wondering if today, with the internet overflowing with data and specifically with images (maybe not ...
2
votes
0answers
50 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
1
vote
1answer
73 views

How handle the add of a new feature to the dataset?

Let's say that you trained a model (eg. Random Forest) on a dataset with ten features (or columns). Now you add one or more features to the dataset. You need the information brought from these new ...
0
votes
1answer
64 views

What are appropriate labels for age categorical labels?

I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
2
votes
0answers
37 views

Please help to identify if there is meaning of linear variables separation by features in phase space?

Description: I have clustering problem with next input: I have truth dataset containing labels for each row of features. I've found that some features can be modified to some phase space, where they ...
0
votes
2answers
2k views

How to plot using facet_wrap, over multiple pages as a .pdf files in r cran

I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in. This is what I have used. ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = ...
2
votes
0answers
241 views

Spatial Transformer Networks and Data Augmentation

We are all familiar with the famous Deep Mind paper STN. Upon implementation, such as https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html , did anyone still use input data ...
0
votes
3answers
7k views

Predictor Variable vs. Target Variable [closed]

[Update] Multiple target variables can exist. If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable?
0
votes
1answer
57 views

Making sense of how R works internally

As what type of "thing" are various objects and functions one deals frequently with in R stored internally? To clarify: For languages like Python it is very easy to understand conceptually how your ...
1
vote
1answer
39 views

Making bigram features from a particular dataset

I have a folder which has a number of files which have a format like these ...
2
votes
1answer
65 views

How Do Bayesian Methods in Machine Learning Help With the Problem of Limited Data? Can This Be Used for Image Classification/Recognition? [closed]

When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who ...
1
vote
0answers
15 views

Parameztrization of a model with data-sheet's sensor information or with empirical data

When working with a 3D laser sensor (LiDAR), the volumetric point density versus distance $\rho_r$ can be theoretically worked out taking into account the physical properties of the laser (number of ...
1
vote
0answers
15 views

Feature building - phone device type

I want to use phone type (e.g., "iPhone 6s", "Lenovo K8 Note", etc.) as a feature in a model. Does anyone know of a way to obtain release date, price, etc of a phone (besides scraping)? That seems ...
1
vote
0answers
29 views

Is it a good idea majoring in data science for undergraduate? [closed]

Is it a good idea majoring in data science for undergraduate? Auckland University offers an undergraduate major in data science, which has courses in computer science and statistics. I read ...
1
vote
0answers
5 views

Where can I find internet data usage stats?

I need stats on how much data the average person uses each day on the internet. Web browsing, emailing, watching netflix, etc. Not just "mobile data" but all data, including cable or fiber or ...
1
vote
0answers
53 views

Features for Neural Networks [closed]

I try to understand my time series data. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. So I have a ...
2
votes
1answer
148 views

How much data is created daily (source of IBM estimation)

I've stumbled upon this IBM text. It states that: Every day, we create 2.5 quintillion bytes of data It's very interesting statistic, but I couldn't find any source for this. Does anyone know ...
6
votes
3answers
2k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
1
vote
1answer
129 views

Training with data of different shapes. Is padding an alternative?

I have a dataset of about 1k samples and I want to apply some unsuspervised techniques in order to clustering and visualization of these data. The data can be interpreted as a table of a spreadsheet ...

1
6 7
8
9 10
12