Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
2
votes
1answer
49 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
1
vote
2answers
17 views

Different types of analysis which can be done with Webdata of users

I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are: Product Usage details Geographic details Session data: Capture what all they did ...
1
vote
2answers
38 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
0
votes
2answers
35 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
0
votes
1answer
38 views

Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case?

So, i am constructing a dataset (from doctor prescription notes and diagnosis) where X (input features) represent the various snowmed codes for illnesses/surgeries performed/chronic conditions etc. ...
2
votes
2answers
4k views

Create a binary-classification dataset (python: sklearn.datasets.make_classification)

I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up. Since the dataset is for a school project, it should be ...
2
votes
0answers
56 views

Beginner Question Related To Data Science Course [closed]

I am Third Year B.Tech student from 3-tier college of India ,Here is no one fellow or collegous who has somebit knowledge about ML or Data science and I am purely sure that I have strong background of ...
7
votes
6answers
467 views

Is it advisable to combine two dataset?

I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based ...
1
vote
1answer
38 views

Why not higher accuracy in Otto data?

On this site of Otto Group Product Classification Challenge, it is shown that best accuracy was possible with RandomForest method, but it was relatively low at 0.83. Accuracy with ANN and with Naive ...
18
votes
1answer
31k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
1
vote
1answer
703 views

How can I sort a data frame by groups? [closed]

I am using R. I have a data frame with "year , price, mileage" columns. I want to group the df by year first and then sort each group by mileage. How can I do this?
2
votes
1answer
45 views

Convert a csv file so that all attributes will become columns with corresponding element [closed]

so I have this dataset in a csv file. I would like to convert it into a matrix form. The data currently looks like this: ...
0
votes
1answer
64 views

Image data augmentation with this function

Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py ...
0
votes
1answer
41 views

Handling imbalanced data by deleting over represented rows vs. adding under represented rows

I am currently working with a very imbalanced data set (frauded credit card data from kaggle, which has 492 rows of frauded cards and over 280,000 rows of non-frauded cards). As much as I know, there ...
1
vote
0answers
142 views

Transposing a dataframe (time-series)

I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting): ...
3
votes
2answers
43 views

How to generate data if algo itself is involved in the process with a feedback loop?

I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo. But I am having issues and ...
2
votes
1answer
33 views

Algortihm for making predictions from minimal data

I am working with classification problem. I have a dataset with a lot of features. A lot of them can easily determine class. On production, I want to ask the user to provide me with only part of the ...
7
votes
5answers
2k views

Interactive labeling/annotating of time series data

I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
1
vote
0answers
18 views

Having the target and data row - how to find the correct model

I am coming from a web developer background, and having no data science background. I am looking to skill-up and get some answers for a current task: find a model that returns a predicted value same ...
4
votes
4answers
506 views

How to know for sure if we can learn from a given data or not?

I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?
1
vote
1answer
33 views

Prepare data : many products per row

I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints : A Product is defined by {id, price, continuousVariable1, ..., ...
1
vote
1answer
266 views

Is there an API to extract questions and labels from yahoo answers?

I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it ...
1
vote
0answers
46 views

Time Series Autocorrelation Estimation

How do I find joint PDF f(xi,xj) needed for estimating autocorrelation from 1-D time series data X, using the kernel windowing technique? I am well aware of kernel windowing technique for estimation ...
4
votes
3answers
167 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
11 views

Where to find Facial Visual Object Class

I am trying to make a face/head detection model using YOLO and darkflow. In order to do this I need a set of images peoples face annotated in Visual Object Class, VOC format. I would prefer data set ...
3
votes
0answers
57 views

Deep Learning: Does starting the training on a smaller subset of the data make sense?

I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the ...
2
votes
1answer
52 views

Knowledge of statistics To be a data scientist [closed]

I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science ...
1
vote
0answers
283 views

Good dataset for sentiment Analysis in Tickets for IT Support

I am working on Sentiment Analysis in Tickets for IT Support, But I have only seen the movie reviews and Twitter sentiment analysis which are not related to my work. I found a research paper related ...
1
vote
0answers
24 views

Deciding number of classes

I have a dataset of 1000 labels. I would like to build a custom model from this dataset with three labels, say Dog, Cat, and <...
0
votes
2answers
144 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
0
votes
1answer
45 views

How to give importance to recent data in a large dataset

I am solving a binary classification problem which is also partly a time-series problem. This means that recent data is more relevant than old data. But how can I assign more importance to the recent ...
1
vote
2answers
65 views

How to scale numbers according to proportions

I have some data that I'm trying to represent in a visualization with a week-over-week change. So the source can be something like: ...
1
vote
2answers
231 views

data science python data cleaning

I am preparing a dataset for a model, but somehow the code just doesn't run well. The major error is: ...
1
vote
1answer
27 views

Entropy in a closed box

I have not understood why the maximum number of binary questions needed to reduce uncertainty is essentially log(T). If i have a ball in a box and 10 possible classes, is it enough to ask log(10) ...
0
votes
1answer
283 views

Sparse Data Interpolation

I have a price data set where on some days there are up to five data points and some days none at all. For example: ...
0
votes
1answer
21 views

How do I store a question-and-answer combination for machine learning?

I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the ...
0
votes
3answers
2k views

Retraining a machine learning model [closed]

Created a Machine Learning model on some data. Used this model for predicting test data. Model has learnt only from training data. How this model can be retrained as new data come in for prediction. I ...
0
votes
2answers
63 views

Best ML model for predicting yearly data with many blocks?

I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect. I would like to use a ...
3
votes
2answers
354 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...
1
vote
1answer
286 views

Using DONUT algorithm with keras

I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows: ...
0
votes
1answer
1k views

How to find 4th point equidistant from 3 other points

For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
1
vote
1answer
34 views

Conversion of Interval data to ordinal data

I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both. Lets say the three participants finished the race in 45 seconds,35 ...
0
votes
1answer
43 views

Are there enough databases for all learning tasks?

this might be a silly question but I guess the answer comes with experience in this field. I'm just wondering if today, with the internet overflowing with data and specifically with images (maybe not ...
2
votes
0answers
50 views

Can Google really bring back billions of results in a blink of an eye (almost) [closed]

I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
1
vote
1answer
51 views

How handle the add of a new feature to the dataset?

Let's say that you trained a model (eg. Random Forest) on a dataset with ten features (or columns). Now you add one or more features to the dataset. You need the information brought from these new ...
0
votes
1answer
62 views

What are appropriate labels for age categorical labels?

I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
2
votes
0answers
36 views

Please help to identify if there is meaning of linear variables separation by features in phase space?

Description: I have clustering problem with next input: I have truth dataset containing labels for each row of features. I've found that some features can be modified to some phase space, where they ...
0
votes
2answers
2k views

How to plot using facet_wrap, over multiple pages as a .pdf files in r cran

I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in. This is what I have used. ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = ...
2
votes
0answers
218 views

Spatial Transformer Networks and Data Augmentation

We are all familiar with the famous Deep Mind paper STN. Upon implementation, such as https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html , did anyone still use input data ...
0
votes
3answers
6k views

Predictor Variable vs. Target Variable [closed]

[Update] Multiple target variables can exist. If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable?

1 4 5 6 7 8 10