Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
0
votes
0answers
12 views
Household Income and Wealth Data for US [on hold]
Does anyone know the best database to get unbiased info about mean, median, etc. on household income/wealth distribution. I have used Pew Research, the Census Bureau, and Sentier Research, but I am ...
1
vote
0answers
20 views
Transposing a dataframe (time-series)
I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting):
...
1
vote
2answers
20 views
How to generate data if algo itself is involved in the process with a feedback loop?
I have an algorithm which would be a rather easy classification task with a set of features and a class output which I would like to solve with a machine learning algo.
But I am having issues and ...
0
votes
0answers
22 views
Releasing data with datajoint
I want to release some data along with a publication. The data are part of a long processing chain stored in MySQL and processed with datajoint. What is the best way to release the data such that the ...
1
vote
1answer
20 views
Algortihm for making predictions from minimal data
I am working with classification problem.
I have a dataset with a lot of features. A lot of them can easily determine class. On production, I want to ask the user to provide me with only part of the ...
2
votes
0answers
21 views
Interactive labeling/annotating of time series data
I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
1
vote
0answers
13 views
Having the target and data row - how to find the correct model
I am coming from a web developer background, and having no data science background.
I am looking to skill-up and get some answers for a current task: find a model that returns a predicted value same ...
0
votes
0answers
7 views
Approach for dealing with missing/non-existent data
Not sure if this is the right place.
I have a set of match results from an international, 32-team competition, stretching back to 1991 (20,000 matches). Each year, there are different teams competing....
0
votes
0answers
7 views
Where can i find Public Datasets related to Astronomy and other fields? [migrated]
I'm planning to do some research. Therefore, Appriciate if you could kindly help me to find out Astronomical datasets.
4
votes
4answers
470 views
How to know for sure if we can learn from a given data or not?
I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?
0
votes
0answers
19 views
Data Science : from handson person to manager
I am a data science hands on person with some experience in SAS/R/python/excel and exposure to techniques like regression, time series, clustering, decision tress etc to name a few. Now I have been ...
1
vote
1answer
25 views
Prepare data : many products per row
I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints :
A Product is defined by {id, price, continuousVariable1, ..., ...
1
vote
1answer
14 views
Is there an API to extract questions and labels from yahoo answers?
I want to work on a ML project which involves the family and relationship category from yahoo answers. I want to extract questions from yahoo answers (only the title) with the label as the category it ...
1
vote
0answers
13 views
Time Series Autocorrelation Estimation
How do I find joint PDF f(xi,xj) needed for estimating autocorrelation from 1-D time series data X, using the kernel windowing technique?
I am well aware of kernel windowing technique for estimation ...
4
votes
3answers
94 views
Understanding data normalisation
So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
1
vote
0answers
7 views
Where to find Facial Visual Object Class
I am trying to make a face/head detection model using YOLO and darkflow.
In order to do this I need a set of images peoples face annotated in Visual Object Class, VOC format. I would prefer data set ...
2
votes
0answers
19 views
Deep Learning: Does starting the training on a smaller subset of the data make sense?
I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the ...
2
votes
1answer
27 views
Knowledge of statistics To be a data scientist [closed]
I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science ...
1
vote
0answers
24 views
Good dataset for sentiment Analysis in Tickets for IT Support
I am working on Sentiment Analysis in Tickets for IT Support, But I have only seen the movie reviews and Twitter sentiment analysis which are not related to my work. I found a research paper related ...
1
vote
0answers
21 views
Deciding number of classes
I have a dataset of 1000 labels. I would like to build a custom model from this dataset with three labels, say Dog, Cat, and <...
0
votes
2answers
23 views
Basics: What is the correct sequence for preparing simple data for ML?
I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one).
I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
0
votes
1answer
19 views
How to give importance to recent data in a large dataset
I am solving a binary classification problem which is also partly a time-series problem. This means that recent data is more relevant than old data. But how can I assign more importance to the recent ...
1
vote
2answers
40 views
How to scale numbers according to proportions
I have some data that I'm trying to represent in a visualization with a week-over-week change. So the source can be something like:
...
2
votes
2answers
92 views
data science python data cleaning
I am preparing a dataset for a model, but somehow the code just doesn't run well.
The major error is:
...
1
vote
1answer
20 views
Entropy in a closed box
I have not understood why the maximum number of binary questions needed to reduce
uncertainty is essentially log(T).
If i have a ball in a box and 10 possible classes, is it enough to ask log(10) ...
0
votes
1answer
17 views
Sparse Data Interpolation
I have a price data set where on some days there are up to five data points and some days none at all. For example:
...
0
votes
0answers
26 views
Selecting representative data
I have a data set of over 10000 datapoints and over 500 variables. Running computations on this is taking forever!! i want to choose a smaller subset of data to run models on but im unsure on how to ...
0
votes
1answer
18 views
How do I store a question-and-answer combination for machine learning?
I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the ...
1
vote
3answers
54 views
Retraining a machine learning model [closed]
Created a Machine Learning model on some data. Used this model for predicting test data. Model has learnt only from training data. How this model can be retrained as new data come in for prediction. I ...
0
votes
2answers
33 views
Best ML model for predicting yearly data with many blocks?
I would like to forewarn by saying I am by no means an expert in this topic and I apologize if I mix terminology, phrased the question wrong, or any information is incorrect.
I would like to use a ...
0
votes
1answer
34 views
What is the meaning of the term “pipeline” within data science?
Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here
People often refer to pipelines when talking about models, data and even layers ...
1
vote
1answer
50 views
Using DONUT algorithm with keras
I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows:
...
0
votes
1answer
25 views
How to find 4th point equidistant from 3 other points
For example, if a=100 , b =200, c=300, then what should be the 4th point equidistant to all the 3 points ie a,b,c. Similarly, is there is any solution for nth point equidistant from n-1 points. The ...
0
votes
1answer
19 views
Conversion of Interval data to ordinal data
I recently learned about Level of measurements and I am really confused in this MCQ where I think the right answer is a and d both.
Lets say the three participants finished the race in 45 seconds,35 ...
0
votes
1answer
35 views
Are there enough databases for all learning tasks?
this might be a silly question but I guess the answer comes with experience in this field. I'm just wondering if today, with the internet overflowing with data and specifically with images (maybe not ...
0
votes
0answers
3 views
What's the best way to represent a zero-filled age variable in DDI?
I am looking at XML produced by the IPUMS website. The files we are downloading contain an AGE variable, and it can be coded from 000 to 115 (age is in years). The XML is in DDI format. An excerpt ...
0
votes
0answers
24 views
What should I start with for starting a career in data science as a fresher with no previous experience in any field?
I have a query:
I just passed out from my engineering college this year with Electronics and communication engineering (ECE) background.
I am particularly looking for a job and thinking of making a ...
2
votes
0answers
39 views
Can Google really bring back billions of results in a blink of an eye (almost) [closed]
I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our ...
0
votes
0answers
14 views
Combining normalized and non-normalized data in Neural Networks
I have tried researching the question in the title without a great deal of success.
When dealing with natural language I am using an LSTM, which well suited to that sort of task. This is a ...
1
vote
1answer
36 views
How handle the add of a new feature to the dataset?
Let's say that you trained a model (eg. Random Forest) on a dataset with ten features (or columns).
Now you add one or more features to the dataset. You need the information brought from these new ...
0
votes
1answer
26 views
What are appropriate labels for age categorical labels?
I am converting some age data to categorical variables. What are some appropriate labels? Some people might take offense to using "Young", "Old" or "Millenial", etc. Is there a "standard" list of ...
2
votes
0answers
34 views
Please help to identify if there is meaning of linear variables separation by features in phase space?
Description:
I have clustering problem with next input:
I have truth dataset containing labels for each row of features.
I've found that some features can be modified to some phase space,
where they ...
0
votes
2answers
105 views
How to plot using facet_wrap, over multiple pages as a .pdf files in r cran
I am using ggplot, to compare 114 unique studies for a particular variable I'm interested in.
This is what I have used.
ggplot(steps, aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = ...
0
votes
0answers
156 views
How to use ggplot, geom_bar to plot counts for categorical variables
I have a huge data set. My response variable is called edu, and I have several group ids under the column named id_study.
There are 113 unique studies for the id_study. There are many categories of ...
1
vote
0answers
52 views
Spatial Transformer Networks and Data Augmentation
We are all familiar with the famous Deep Mind paper STN.
Upon implementation, such as https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html , did anyone still use input data ...
0
votes
3answers
217 views
Predictor Variable vs. Target Variable [closed]
If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable? I don't believe so because if you have 209 variables, and you find your predictor ...
0
votes
1answer
36 views
Making sense of how R works internally
As what type of "thing" are various objects and functions one deals frequently with in R stored internally?
To clarify: For languages like Python it is very easy to understand conceptually how your ...
0
votes
0answers
8 views
Programatically Adjust Parameters of a Thermal Simulation to Fit Measured Data
For a university lab I'm required to estimate the thermodynamic constants of a metal by running a series of experiments and adjusting the parameters of a finite difference model to fit the measured ...
1
vote
1answer
17 views
Making bigram features from a particular dataset
I have a folder which has a number of files which have a format like these
...
2
votes
1answer
52 views
How Do Bayesian Methods in Machine Learning Help With the Problem of Limited Data? Can This Be Used for Image Classification/Recognition? [closed]
When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who ...
