Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
2answers
91 views

What is valued more in the data science job market, statistical analysis or data processing?

Not sure if this question is OT here. If it is, perhaps move it to meta? If not: I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've ...
2
votes
0answers
347 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
-2
votes
1answer
111 views

What is the difference between statistical learning and predictive analytics?

I am new to data science. I am confused about the differences between statistical learning and predictive analytics. To me, it looks like both are the same as we try to work on function of x(...
0
votes
2answers
999 views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
0
votes
1answer
139 views

Where can I find datasets with labeled duplicate text documents?

I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
-1
votes
1answer
181 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
0
votes
1answer
817 views

Difference Between Feature Engineering and Feature Learning

I am playing with features (input data) to improve my model's accuracy. If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data ...
1
vote
1answer
759 views

Unable to immediately continue Jupyer Notebook after closing then reopening the project

When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by ...
0
votes
1answer
57 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
1
vote
1answer
49 views

Recording Data of my learning progress

I'm just starting out with learning about Data Science and programming through various online course. Was thinking a good first project would be a to create a log of what I'm learning and how long ...
1
vote
2answers
193 views

Advice on how to get started for someone a bit later in life? [closed]

I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I ...
10
votes
2answers
5k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
0
votes
1answer
114 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...
1
vote
0answers
35 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
1
vote
0answers
143 views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
1
vote
0answers
24 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
8
votes
1answer
6k views

What are the most suitable machine learning algorithms according to type of data? [closed]

I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical). I searched about this ...
1
vote
2answers
367 views

Best Programming Language for Data Science [closed]

I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language ...
1
vote
1answer
49 views

System to provide guide to students about getting admissions to universities of their choice or some specific courses [closed]

I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear ...
1
vote
1answer
584 views

Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

I have a Data Set of 170 images in total. of each image i want to create 36 images like this. An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of ...
0
votes
1answer
559 views

How is the salary for Data Scientist different among countries?

I would like to know the salary for Data Scientists by countries, but failed to find any resources on it. I know that US gives data scientists an insanely high amoung of money (well, it might not be "...
-1
votes
1answer
3k views

Obtaining datasets of photos of people and car license plates

Where can I legally obtain the following kinds of datasets for training an image recognition algorithm for commercial use? photos of people car license plates The privacy laws in Europe (Germany) ...
1
vote
2answers
565 views

How to choose an appropriate Machine Learning algorithm? [closed]

I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to ...
-1
votes
1answer
70 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
2
votes
1answer
544 views

What can i do after a PCA with the results?

After performing a PCA and studying the proceeding i ask myself what the result is good for in the next step. From the PCA i learned how to visualize the dataset by lowering the dimension, i got handy ...
2
votes
1answer
221 views

How clustering is used in data management? [closed]

Can anyone please tell me how is clustering is used in data management? I was recently asked this question and was left totally stumped.
-1
votes
1answer
231 views

Does the input data representation matter while training CNN for speech recognition?

I am currently doing pattern recognition on spectograms of audio files using convolutional neural networks. The spectograms are made using matplotlib ...
-1
votes
2answers
51 views

what is the proper tool to analyse data and find trends in my case? [closed]

I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for ...
1
vote
1answer
44 views

R Code randomizing selection

new to R, trying to create a comparison group - so far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points). These remaining 55 need to be a ...
0
votes
1answer
2k views

How to shift rows values as columns in pandas?

Input: I have csv file like below as input.... ...
0
votes
1answer
66 views

How should I expose data from my app for data scientists?

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data ...
8
votes
5answers
455 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
1
vote
0answers
89 views

What is preferred upsampling vs. zero padding?

When training a CNNn one option is either to zero pad an image to make it bigger or upsample it... When would one choose what? What criteria is used choosing the which method to use?
1
vote
3answers
73 views

Which graph will be appropriate for the visualization task?

I have some terminal charging values for US and CHINA comes in a pandas DataFrame like the following, ...
1
vote
0answers
178 views

Cook's distance, altering diagnostic plot in R,?

I have a linear model in R, named Q1fit. I am interested in the plot of the residuals vs leverage, with Cook's distance. When I run plot(Q1fit, which=(5)) I get There is no Cook's distance line ...
2
votes
1answer
347 views

Dataset: language audio clips and country labels

I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. Does such a dataset exist? Or is there ...
3
votes
2answers
801 views

Which is better for Data Science, a double major in Math & CS or Physics & CS? [closed]

My dream occupation is to work as a Data Scientist. I'm soon going to be completing my first semester as a physics student, as part of an incredibly prestigious physics program. We get to run an ...
4
votes
1answer
103 views

How to generate bulk graphics using R

I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2. How can I automatically generate one graphic for each ...
14
votes
5answers
10k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
1
vote
1answer
68 views

Uploading huge dataset

I have few questions: Is there a website to upload huge research dataset (over 100GB) for free? Which type of compression (rar, zip ... etc) is good for jpeg images? In case of dataset of 120GB. what ...
0
votes
1answer
106 views

Data preprocessing, relative scale problems in features of same type

I am using Keras NN with theanos backend in Python. In my data i have multiple features of the same type but in different columns (on purpose). Here is an example. ...
2
votes
1answer
110 views

Generating data that look alike my original data

If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features. Is it possible to generate new data from ...
9
votes
4answers
12k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
2
votes
2answers
6k views

What is the difference between data-driven methods and machine learning?

I was wondering (about a more semantic question), is there a difference between data-driven methods and machine learning? Or is it more correct to state that machine learning is a category of data-...
0
votes
2answers
694 views

Connecting Infusionsoft data to Google data studio

I want to create a Google Data studio dashboard from Infusion soft data. The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like ...
2
votes
1answer
136 views

How is a single element of the training set called?

This question is only about the vocabulary. Do / can you say data item data sample recording sample data point something else when you talk about elements of the training / test set? For example: ...
0
votes
2answers
345 views

How synthetic sampling of data helps classifier to predict better result?

I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem. I have the idea how to create the synthetic example to bring noticeable accuracy on a ...
1
vote
3answers
1k views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...