Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
7
votes
1answer
10k views

How to generate training data for OCR

I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
2
votes
2answers
7k views

What is the difference between data-driven methods and machine learning?

I was wondering (about a more semantic question), is there a difference between data-driven methods and machine learning? Or is it more correct to state that machine learning is a category of data-...
-1
votes
1answer
111 views

Extracting paragraph from a document based on Numerals

I am new to the Data Science. The problem I want to solve is relatively simple in terms of the problem statement. Given a Numbered document (usually a pdf) with/without an index, I need to extract ...
1
vote
2answers
1k views

What is the definition of Data Scout?

I am looking for a definition of Data Scout. what is the difference between Data Mining and Data Scout?
2
votes
1answer
31 views

When visualizing data that has <1 or <5 ppm how do you display this?

I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists "<1" ppm. What is a good way to visualize this in ...
2
votes
1answer
222 views

How clustering is used in data management? [closed]

Can anyone please tell me how is clustering is used in data management? I was recently asked this question and was left totally stumped.
0
votes
1answer
902 views

How to find the datasets for skill test (like java, python, c c++, etc.) [closed]

I want to extract the skills from a resume using Natural languge processing. To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume?
3
votes
2answers
2k views

How to deal with large data sets

So i'm very new to this, forgive my silly questions. I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job. I have .csv files containing data ...
-1
votes
1answer
282 views

What is the standard procedure for Data Analysis? [closed]

I'm wondering is there a standard procedure like a workflow for doing a project working with analyzing data?, I'm familiar with CRISP-DM and SEMMA for Data Mining projects but was wondering if there ...
-1
votes
1answer
6k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
0
votes
2answers
77 views

Imbalanced class with same rows? [closed]

In my dataset i have 3 classes-> 0,1,2. 0(72k),1(13k)and 2(13K) in brackets are there count. So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted ...
0
votes
1answer
103 views

Machine Learning algorhtm that can be trained to learn Pig-Latin

Pig-Latin, and other similair argots, were quite popular when i was young (but since i grew up in Sweden the biggest was Rövarspråket). These language games almost always consist of a given set of ...
2
votes
1answer
97 views

Count the number of false positives with respect to the first class [closed]

Can anybody tell me the formula how to find the number of false positives with respect to the first class? where $y$ is the truth / target and $a(x)$ is the prediction
17
votes
1answer
18k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
0
votes
1answer
33 views

How to measure a relationship between interests of topics between two users?

I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example: <...
1
vote
0answers
35 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
-1
votes
1answer
38 views

Is it possible to plan and assign Data Science tasks by complexity levels based on team members' experience?

How would you assign Data Science primary tasks to Data Scientists in a team according to a colleague' seniority? I mean, as always you can expect different things and also level and pace of self-...
-2
votes
1answer
368 views

Data Science Project Manager - Planning [closed]

I'm a project manager and I've been tasked to work with a data scientists. As a project manager it's my role to create a plan which I will monitor and report against. The data scientist has ...
-2
votes
1answer
86 views

Finding relationships from group of variables

Given time-series data of a set of input metrics $M_1$, $M_2$, $M_3$ ... $M_n$ and a set of observed metrics $O_1$, $O_2$, $O_3$ ... $O_m$. For each $M_i$, what is the best way to find equations that ...
0
votes
2answers
104 views

What is valued more in the data science job market, statistical analysis or data processing?

Not sure if this question is OT here. If it is, perhaps move it to meta? If not: I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've ...
2
votes
0answers
368 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
1
vote
1answer
961 views

Unable to immediately continue Jupyer Notebook after closing then reopening the project

When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
1
vote
1answer
49 views

Recording Data of my learning progress

I'm just starting out with learning about Data Science and programming through various online course. Was thinking a good first project would be a to create a log of what I'm learning and how long ...
-1
votes
1answer
210 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
0
votes
1answer
983 views

Difference Between Feature Engineering and Feature Learning

I am playing with features (input data) to improve my model's accuracy. If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data ...
0
votes
1answer
59 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
1
vote
2answers
197 views

Advice on how to get started for someone a bit later in life? [closed]

I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I ...
10
votes
2answers
7k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
-1
votes
1answer
3k views

Obtaining datasets of photos of people and car license plates

Where can I legally obtain the following kinds of datasets for training an image recognition algorithm for commercial use? photos of people car license plates The privacy laws in Europe (Germany) ...
1
vote
1answer
49 views

System to provide guide to students about getting admissions to universities of their choice or some specific courses [closed]

I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear ...
0
votes
1answer
119 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...
0
votes
3answers
3k views

@RISK Vs R. Monte Carlo simulation in R?

Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks? http://www.palisade.com/risk/ @RISK (pronounced “at risk”) ...
1
vote
1answer
616 views

Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

I have a Data Set of 170 images in total. of each image i want to create 36 images like this. An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of ...
1
vote
0answers
25 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
8
votes
1answer
8k views

What are the most suitable machine learning algorithms according to type of data? [closed]

I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical). I searched about this ...
1
vote
2answers
467 views

Best Programming Language for Data Science [closed]

I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language ...
0
votes
1answer
564 views

How is the salary for Data Scientist different among countries?

I would like to know the salary for Data Scientists by countries, but failed to find any resources on it. I know that US gives data scientists an insanely high amoung of money (well, it might not be "...
1
vote
2answers
588 views

How to choose an appropriate Machine Learning algorithm? [closed]

I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to ...
2
votes
2answers
322 views

Data structure design for supporting arbitrary number of columns in table or database

I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data. The task I'm currently struggling is: user needs to ...
0
votes
1answer
67 views

How should I expose data from my app for data scientists?

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data ...
2
votes
2answers
110 views

How many people can use a single Hadoop cluster at one time?

I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
2
votes
1answer
758 views

What can i do after a PCA with the results?

After performing a PCA and studying the proceeding i ask myself what the result is good for in the next step. From the PCA i learned how to visualize the dataset by lowering the dimension, i got handy ...
-1
votes
1answer
71 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
2
votes
2answers
2k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
-1
votes
1answer
253 views

Does the input data representation matter while training CNN for speech recognition?

I am currently doing pattern recognition on spectograms of audio files using convolutional neural networks. The spectograms are made using matplotlib ...
4
votes
2answers
975 views

Recommendations for storing time series data

As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, ...
4
votes
2answers
2k views

Tool to Generate 2D Data via Mouse Clicking

Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or ...
-1
votes
2answers
52 views

what is the proper tool to analyse data and find trends in my case? [closed]

I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for ...
0
votes
1answer
2k views

How to shift rows values as columns in pandas?

Input: I have csv file like below as input.... ...

1
8 9
10
11 12