Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
5
votes
1answer
1k views

Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

I am creating decision trees modeling data that looks like this. ...
1
vote
0answers
44 views

LaTex Symbol Usage Statistics

Are there any available resources that contain data on LaTeX symbols usage metrics. I want to know what the most used symbols are to determine which I should account for in a project I'm working on.
2
votes
1answer
508 views

Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions?

The problem: I've N classification models (independent), for each of these N models, I've different versions (eg: V0, V1, ..., Vfinal_production,Vexperimental). I'm looking for a way to store my ...
5
votes
1answer
6k views

Small data set in machine learning [closed]

I see from reading that the medical industry is using machine learning to apply to small data set and wanted to understand how this worked? As such I created a small data set for my wife's business. ...
2
votes
1answer
27 views

Columns with no (or nearly no) differences between rows worth keeping?

I have a dataset which contains multiple columns. On analysis, I found out that there were few columns which contain just a single value. There was 0 in all the rows of these columns. Does it hold any ...
5
votes
1answer
2k views

Can I scrape data from government websites if there is no mention about commercial usage?

I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (...
1
vote
2answers
900 views

What is the definition of Data Scout?

I am looking for a definition of Data Scout. what is the difference between Data Mining and Data Scout?
2
votes
1answer
31 views

When visualizing data that has <1 or <5 ppm how do you display this?

I have some data (parts per million) where some of it is 1 or greater than one (but has an actual number.) However, some of the data simply lists "<1" ppm. What is a good way to visualize this in ...
0
votes
1answer
725 views

How to find the datasets for skill test (like java, python, c c++, etc.) [closed]

I want to extract the skills from a resume using Natural languge processing. To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume?
2
votes
0answers
11k views

How to Process Large JSON Files with Python [closed]

I've been using Python for quite some time now, scripting command line tools, etc. I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files. ...
0
votes
1answer
333 views
-1
votes
1answer
225 views

What is the standard procedure for Data Analysis? [closed]

I'm wondering is there a standard procedure like a workflow for doing a project working with analyzing data?, I'm familiar with CRISP-DM and SEMMA for Data Mining projects but was wondering if there ...
2
votes
2answers
1k views

How to deal with large data sets

So i'm very new to this, forgive my silly questions. I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job. I have .csv files containing data ...
-1
votes
1answer
6k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
0
votes
1answer
89 views

Machine Learning algorhtm that can be trained to learn Pig-Latin

Pig-Latin, and other similair argots, were quite popular when i was young (but since i grew up in Sweden the biggest was Rövarspråket). These language games almost always consist of a given set of ...
0
votes
2answers
77 views

Imbalanced class with same rows? [closed]

In my dataset i have 3 classes-> 0,1,2. 0(72k),1(13k)and 2(13K) in brackets are there count. So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted ...
-1
votes
1answer
108 views

Extracting paragraph from a document based on Numerals

I am new to the Data Science. The problem I want to solve is relatively simple in terms of the problem statement. Given a Numbered document (usually a pdf) with/without an index, I need to extract ...
2
votes
1answer
91 views

Count the number of false positives with respect to the first class [closed]

Can anybody tell me the formula how to find the number of false positives with respect to the first class? where $y$ is the truth / target and $a(x)$ is the prediction
15
votes
1answer
17k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
0
votes
1answer
32 views

How to measure a relationship between interests of topics between two users?

I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example: <...
-1
votes
1answer
38 views

Is it possible to plan and assign Data Science tasks by complexity levels based on team members' experience?

How would you assign Data Science primary tasks to Data Scientists in a team according to a colleague' seniority? I mean, as always you can expect different things and also level and pace of self-...
17
votes
4answers
14k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
-2
votes
1answer
362 views

Data Science Project Manager - Planning [closed]

I'm a project manager and I've been tasked to work with a data scientists. As a project manager it's my role to create a plan which I will monitor and report against. The data scientist has ...
-2
votes
1answer
84 views

Finding relationships from group of variables

Given time-series data of a set of input metrics $M_1$, $M_2$, $M_3$ ... $M_n$ and a set of observed metrics $O_1$, $O_2$, $O_3$ ... $O_m$. For each $M_i$, what is the best way to find equations that ...
0
votes
2answers
94 views

What is valued more in the data science job market, statistical analysis or data processing?

Not sure if this question is OT here. If it is, perhaps move it to meta? If not: I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've ...
2
votes
0answers
355 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
-2
votes
1answer
111 views

What is the difference between statistical learning and predictive analytics?

I am new to data science. I am confused about the differences between statistical learning and predictive analytics. To me, it looks like both are the same as we try to work on function of x(...
0
votes
2answers
1k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
1answer
146 views

Where can I find datasets with labeled duplicate text documents?

I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
-1
votes
1answer
190 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
0
votes
1answer
870 views

Difference Between Feature Engineering and Feature Learning

I am playing with features (input data) to improve my model's accuracy. If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data ...
1
vote
1answer
832 views

Unable to immediately continue Jupyer Notebook after closing then reopening the project

When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by ...
0
votes
1answer
57 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
1
vote
1answer
49 views

Recording Data of my learning progress

I'm just starting out with learning about Data Science and programming through various online course. Was thinking a good first project would be a to create a log of what I'm learning and how long ...
1
vote
2answers
194 views

Advice on how to get started for someone a bit later in life? [closed]

I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I ...
10
votes
2answers
6k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
0
votes
1answer
118 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...
1
vote
0answers
35 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
1
vote
0answers
143 views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
1
vote
0answers
24 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
8
votes
1answer
6k views

What are the most suitable machine learning algorithms according to type of data? [closed]

I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical). I searched about this ...
1
vote
2answers
397 views

Best Programming Language for Data Science [closed]

I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language ...
1
vote
1answer
49 views

System to provide guide to students about getting admissions to universities of their choice or some specific courses [closed]

I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear ...
1
vote
1answer
596 views

Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

I have a Data Set of 170 images in total. of each image i want to create 36 images like this. An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of ...
0
votes
1answer
560 views

How is the salary for Data Scientist different among countries?

I would like to know the salary for Data Scientists by countries, but failed to find any resources on it. I know that US gives data scientists an insanely high amoung of money (well, it might not be "...
-1
votes
1answer
3k views

Obtaining datasets of photos of people and car license plates

Where can I legally obtain the following kinds of datasets for training an image recognition algorithm for commercial use? photos of people car license plates The privacy laws in Europe (Germany) ...
1
vote
2answers
567 views

How to choose an appropriate Machine Learning algorithm? [closed]

I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to ...
-1
votes
1answer
70 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...