Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
3answers
6k views

Predictor Variable vs. Target Variable [closed]

[Update] Multiple target variables can exist. If I find that a variable is not a predictor variable, does that mean it automatically becomes a target variable?
0
votes
1answer
51 views

Making sense of how R works internally

As what type of "thing" are various objects and functions one deals frequently with in R stored internally? To clarify: For languages like Python it is very easy to understand conceptually how your ...
1
vote
1answer
35 views

Making bigram features from a particular dataset

I have a folder which has a number of files which have a format like these ...
2
votes
1answer
64 views

How Do Bayesian Methods in Machine Learning Help With the Problem of Limited Data? Can This Be Used for Image Classification/Recognition? [closed]

When reading about machine learning, I've often come across information stating that Bayesian methods in machine learning are effective when you only possess a limited amount of data. As someone who ...
1
vote
0answers
15 views

Parameztrization of a model with data-sheet's sensor information or with empirical data

When working with a 3D laser sensor (LiDAR), the volumetric point density versus distance $\rho_r$ can be theoretically worked out taking into account the physical properties of the laser (number of ...
1
vote
0answers
15 views

Feature building - phone device type

I want to use phone type (e.g., "iPhone 6s", "Lenovo K8 Note", etc.) as a feature in a model. Does anyone know of a way to obtain release date, price, etc of a phone (besides scraping)? That seems ...
1
vote
0answers
29 views

Is it a good idea majoring in data science for undergraduate? [closed]

Is it a good idea majoring in data science for undergraduate? Auckland University offers an undergraduate major in data science, which has courses in computer science and statistics. I read ...
1
vote
0answers
5 views

Where can I find internet data usage stats?

I need stats on how much data the average person uses each day on the internet. Web browsing, emailing, watching netflix, etc. Not just "mobile data" but all data, including cable or fiber or ...
1
vote
0answers
53 views

Features for Neural Networks [closed]

I try to understand my time series data. I have temperature values measured with a sampling rate of 10.000 values / 1 second. So I measure in 1 second 10.000 different temperature values. So I have a ...
2
votes
1answer
81 views

How much data is created daily (source of IBM estimation)

I've stumbled upon this IBM text. It states that: Every day, we create 2.5 quintillion bytes of data It's very interesting statistic, but I couldn't find any source for this. Does anyone know ...
4
votes
2answers
1k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
1
vote
1answer
110 views

Training with data of different shapes. Is padding an alternative?

I have a dataset of about 1k samples and I want to apply some unsuspervised techniques in order to clustering and visualization of these data. The data can be interpreted as a table of a spreadsheet ...
1
vote
0answers
220 views

Delete strings with a specific last character in tibco spotfire [closed]

I am using tibco spotfire and I have columns with data like this "0001V", "0002V", "0003N". I know data with the 'N' at the end is not valid so I need to delete the numbers which have that. I think I ...
4
votes
1answer
93 views

How to release datasets with fingerprinting

I intend on monetising some large datasets. These datasets are anonymised and released to (paying) clients via a web api. Are there any standard algorithms such that if the datasets are intentionally ...
0
votes
1answer
214 views

What is difference between a Data Scientist and a Data Analyst?

https://www.datacamp.com/community/tutorials/learn-data-science-infographic https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic These links contain almost everything ...
0
votes
1answer
58 views

What are some data sources with strong positive and negative correlations in feature and sample space?

For example, financial data has in some cases strong positive and negative correlations between different stocks (feature space) and also strong positive and negative correlations between specific ...
1
vote
0answers
126 views

High Correlation between inputs of neural network

I am building a neural network to predict if a video is pornographic or not by analysing the bytes of upload and download at every 0.1 seconds for a total of 25.6 seconds. So, I have 512 input ...
1
vote
2answers
129 views

How to find out the percentage of contribution of a variable for another variable/feature?

I have an e-commerce dataset for a month consisting of five columns where the fifth column is "Revenue". The other four columns are different factors for the revenue such as "Page-Views","Visitors","...
1
vote
1answer
8k views

How to use the same minmaxscaler used on the training data with new data?

Im using the keras LSTM model to make prediction, and the code above is to scale the data: inputs are shaped like (n, 11, 1) and the label is 1D DailyDemand.py ...
1
vote
0answers
33 views

Best masters degree in data science and engineering in Europe [closed]

I have a BSc in Applied Mathematics and I was thinking about an MSc in Data Science or Data Engineering, but I have no ideia what are the best programs available in Europe. Any suggestions of good ...
1
vote
0answers
497 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
8
votes
2answers
52k views

How to delete entire row if values in a column are NaN [closed]

I'd like to drop all the rows containing a NaN values pertaining to a column. Lets assume I have a dataset like this: ...
1
vote
2answers
9k views

Is there a way to replace existing values with NaN

I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column ...
0
votes
2answers
111 views

how to make moving moving graphs that vary in time like a video?

like this, like shown in this video https://www.youtube.com/watch?v=z2DHpW79w0Y&t=7s
0
votes
2answers
77 views

How do I determine if variables are correlated? Is it simply a mathematical calculation?

I'm self learning data science so bear with me as I try and make my question as clear as possible. Lets assume I have a dataset of a dependent variable y and some independent variables x1, x2 and x3 ...
1
vote
0answers
37 views

k nearest neighbors method, temporal trend in error

I have this set of data that looks like this I was asked o build a $k$-nearest neighbors algorithm for it which I just finished building. I have this question in regards to the data that I do not ...
1
vote
1answer
27 views

Ideal gas data needed [closed]

I want to do a quick computation in R that involves estimating the ideal gas constant from experimental data. Ideally I'd have data from a monatomic gas at different pressures, volumes, temperatures ...
5
votes
1answer
2k views

Purpose of weights in neural networks

I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing. I know, that before the inputs are ...
0
votes
2answers
73 views

file name pattern matching and identifying

I have a automation system which manages files sent by multiple client inside S3. It so happens client sends file on a regular basis. Let's not consider the content of the file for time being as it ...
1
vote
4answers
3k views

What is the difference between data singular and data plural ?

I don't understand the difference between these terms and google didn't help much. Please answer the question even if you have any slightest idea.
3
votes
1answer
797 views

Neural Network for Multiple Float Output

I have a dataset containing 40 input columns and 12 output columns (float values). I supposed it's a regression problem and I am wondering of how to choose the ideal architecture of the Neural ...
3
votes
1answer
416 views

What is the allowable limit of oversampling?

Suppose I have 2 classes. One class has 16 samples and the other class has 435 samples. Is it justified to oversample the class with 16 sample to have a 435 number of samples? Or is it better to ...
1
vote
1answer
61 views

I only have microsoft office, how can i write code to manipulate relational data sets?

At work, I am in a role that does not have a lot of computer permissions. I do have the Microsoft office suite. I have skills with programming and would like to be able to use my experience to do more ...
3
votes
1answer
87 views

What is the appropriate name for this dataset?

I'll assume most of us know about cross-validation and are accustomed to using Training and Test sets. However, I've also seen references to a third set that you use to test a completed model. It's ...
5
votes
1answer
887 views

Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

I am creating decision trees modeling data that looks like this. ...
1
vote
0answers
44 views

LaTex Symbol Usage Statistics

Are there any available resources that contain data on LaTeX symbols usage metrics. I want to know what the most used symbols are to determine which I should account for in a project I'm working on.
2
votes
1answer
470 views

Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions?

The problem: I've N classification models (independent), for each of these N models, I've different versions (eg: V0, V1, ..., Vfinal_production,Vexperimental). I'm looking for a way to store my ...
5
votes
1answer
6k views

Small data set in machine learning [closed]

I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business. ...
2
votes
1answer
27 views

Columns with no (or nearly no) differences between rows worth keeping?

I have a dataset which contains multiple columns. On analysis, I found out that there were few columns which contain just a single value. There was 0 in all the rows of these columns. Does it hold any ...
5
votes
1answer
2k views

Can I scrape data from government websites if there is no mention about commercial usage?

I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (...
1
vote
2answers
856 views

What is the definition of Data Scout?

I am looking for a definition of Data Scout. what is the difference between Data Mining and Data Scout?
2
votes
1answer
31 views

When visualizing data that has <1 or <5 ppm how do you display this?

I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists "<1" ppm. What is a good way to visualize this in ...
0
votes
1answer
678 views

How to find the datasets for skill test (like java, python, c c++, etc.) [closed]

I want to extract the skills from a resume using Natural languge processing. To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume?
2
votes
0answers
11k views

How to Process Large JSON Files with Python

I've been using Python for quite some time now, scripting command line tools, etc. I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files. ...
0
votes
1answer
317 views
-1
votes
1answer
208 views

What is the standard procedure for Data Analysis? [closed]

I'm wondering is there a standard procedure like a workflow for doing a project working with analyzing data?, I'm familiar with CRISP-DM and SEMMA for Data Mining projects but was wondering if there ...
2
votes
2answers
1k views

How to deal with large data sets

So i'm very new to this, forgive my silly questions. I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job. I have .csv files containing data ...
-1
votes
1answer
6k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
0
votes
1answer
87 views

Machine Learning algorhtm that can be trained to learn Pig-Latin

Pig-Latin, and other similair argots, were quite popular when i was young (but since i grew up in Sweden the biggest was Rövarspråket). These language games almost always consist of a given set of ...
0
votes
2answers
75 views

Imbalanced class with same rows? [closed]

In my dataset i have 3 classes-> 0,1,2. 0(72k),1(13k)and 2(13K) in brackets are there count. So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted ...

1 6 7 8 9 10