Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
2answers
77 views

Imbalanced class with same rows? [closed]

In my dataset i have 3 classes-> 0,1,2. 0(72k),1(13k)and 2(13K) in brackets are there count. So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted ...
-1
votes
1answer
109 views

Extracting paragraph from a document based on Numerals

I am new to the Data Science. The problem I want to solve is relatively simple in terms of the problem statement. Given a Numbered document (usually a pdf) with/without an index, I need to extract ...
2
votes
1answer
95 views

Count the number of false positives with respect to the first class [closed]

Can anybody tell me the formula how to find the number of false positives with respect to the first class? where $y$ is the truth / target and $a(x)$ is the prediction
17
votes
1answer
18k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
0
votes
1answer
32 views

How to measure a relationship between interests of topics between two users?

I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example: <...
-1
votes
1answer
38 views

Is it possible to plan and assign Data Science tasks by complexity levels based on team members' experience?

How would you assign Data Science primary tasks to Data Scientists in a team according to a colleague' seniority? I mean, as always you can expect different things and also level and pace of self-...
18
votes
4answers
15k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
-2
votes
1answer
367 views

Data Science Project Manager - Planning [closed]

I'm a project manager and I've been tasked to work with a data scientists. As a project manager it's my role to create a plan which I will monitor and report against. The data scientist has ...
-2
votes
1answer
85 views

Finding relationships from group of variables

Given time-series data of a set of input metrics $M_1$, $M_2$, $M_3$ ... $M_n$ and a set of observed metrics $O_1$, $O_2$, $O_3$ ... $O_m$. For each $M_i$, what is the best way to find equations that ...
0
votes
2answers
97 views

What is valued more in the data science job market, statistical analysis or data processing?

Not sure if this question is OT here. If it is, perhaps move it to meta? If not: I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've ...
2
votes
0answers
364 views

Histogram alternatives for two sets of data combined

It's my first question here and I hope it's not too simple for a data science question. I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one ...
-2
votes
1answer
112 views

What is the difference between statistical learning and predictive analytics?

I am new to data science. I am confused about the differences between statistical learning and predictive analytics. To me, it looks like both are the same as we try to work on function of x(...
1
vote
2answers
1k views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
1answer
157 views

Where can I find datasets with labeled duplicate text documents?

I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
-1
votes
1answer
202 views

How to visualise very large graphs with 250M nodes and 500M+ edges? [closed]

I have graph data which has 250 million nodes and over 500 million edges. My files are roughly 230GB is size, this includes the node and edge files. I am having major difficulties visualising the ...
0
votes
1answer
928 views

Difference Between Feature Engineering and Feature Learning

I am playing with features (input data) to improve my model's accuracy. If I have a raw time-series dataframe, does feature engineering mean extracting properties or characteristics of my raw data ...
1
vote
1answer
908 views

Unable to immediately continue Jupyer Notebook after closing then reopening the project

When I reopen a project I am working on in the Jupyter Notebook, and trying to add code, the cell restarts at '1' and the only way I can continue to work with my previous objects present is by ...
0
votes
1answer
59 views

How much data warrants building a pipeline?

If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with ...
1
vote
1answer
49 views

Recording Data of my learning progress

I'm just starting out with learning about Data Science and programming through various online course. Was thinking a good first project would be a to create a log of what I'm learning and how long ...
1
vote
2answers
196 views

Advice on how to get started for someone a bit later in life? [closed]

I am 33 years old and graduated in 2006 from college with a degree in analytic philosophy. For a long time I thought that I was going to get a phd in phil and teach but that dream is long gone, and I ...
10
votes
2answers
6k views

How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression. According to the "rule if ten" I need at least 10 events for each feature to be ...
0
votes
1answer
119 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...
1
vote
0answers
35 views

Find irregular bounding shape of 3D particle distribution

I have some 3D particle data e.g. (x0,y0,z0) (x1,y1,z1) (x2,y2,z2) ... I want to find the irregular bounding shape of the distribution. The image below shows an ...
1
vote
0answers
144 views

Ideal difference in the training accuracy and testing accuracy

In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference ...
1
vote
0answers
24 views

E-Learning related terminology

I want to find a dictionary/ knowledge-base/ ontology that includes e-learning related terminology. Is there any standard dataset?
11
votes
2answers
9k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
8
votes
1answer
7k views

What are the most suitable machine learning algorithms according to type of data? [closed]

I am beginner to data science. I found that some machine learning algorithms perform better, when given particular kind of data(ie - numerical, categorical, text, graphical). I searched about this ...
1
vote
2answers
415 views

Best Programming Language for Data Science [closed]

I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language ...
1
vote
1answer
49 views

System to provide guide to students about getting admissions to universities of their choice or some specific courses [closed]

I basically want to build a system which will provide a student step by step guide or you can say a full route about what courses he/she should take currently or what examinations they need to clear ...
1
vote
1answer
609 views

Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

I have a Data Set of 170 images in total. of each image i want to create 36 images like this. An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of ...
0
votes
1answer
561 views

How is the salary for Data Scientist different among countries?

I would like to know the salary for Data Scientists by countries, but failed to find any resources on it. I know that US gives data scientists an insanely high amoung of money (well, it might not be "...
-1
votes
1answer
3k views

Obtaining datasets of photos of people and car license plates

Where can I legally obtain the following kinds of datasets for training an image recognition algorithm for commercial use? photos of people car license plates The privacy laws in Europe (Germany) ...
1
vote
2answers
581 views

How to choose an appropriate Machine Learning algorithm? [closed]

I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to ...
-1
votes
1answer
70 views

What are the differences between IBM BlueMix and IBM Data Science Experience?

This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered BlueMix is ...
2
votes
1answer
698 views

What can i do after a PCA with the results?

After performing a PCA and studying the proceeding i ask myself what the result is good for in the next step. From the PCA i learned how to visualize the dataset by lowering the dimension, i got handy ...
2
votes
1answer
222 views

How clustering is used in data management? [closed]

Can anyone please tell me how is clustering is used in data management? I was recently asked this question and was left totally stumped.
-1
votes
1answer
251 views

Does the input data representation matter while training CNN for speech recognition?

I am currently doing pattern recognition on spectograms of audio files using convolutional neural networks. The spectograms are made using matplotlib ...
-1
votes
2answers
51 views

what is the proper tool to analyse data and find trends in my case? [closed]

I have a database of hundreds of thousands of loans. I want to find some trends in user behavior, not just some information regarding all loans together but I want to find something similar for ...
1
vote
1answer
50 views

Randomizing selection in R

I am trying to create a comparison group. So far this group contains 45 data points and I need to populate the remaining 55 (for a total of 100 data points). These remaining 55 need to be a ...
0
votes
1answer
2k views

How to shift rows values as columns in pandas?

Input: I have csv file like below as input.... ...
0
votes
1answer
67 views

How should I expose data from my app for data scientists?

I'm the product manager for an online app. I'm currently researching a new feature where our users will be able to access "all their raw application data". This data is likely to be used by data ...
8
votes
5answers
518 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
1
vote
0answers
100 views

What is preferred upsampling vs. zero padding?

When training a CNNn one option is either to zero pad an image to make it bigger or upsample it... When would one choose what? What criteria is used choosing the which method to use?
1
vote
3answers
74 views

Which graph will be appropriate for the visualization task?

I have some terminal charging values for US and CHINA comes in a pandas DataFrame like the following, ...
1
vote
0answers
195 views

Cook's distance, altering diagnostic plot in R,?

I have a linear model in R, named Q1fit. I am interested in the plot of the residuals vs leverage, with Cook's distance. When I run plot(Q1fit, which=(5)) I get There is no Cook's distance line ...
2
votes
1answer
380 views

Dataset: language audio clips and country labels

I'm looking for a dataset containing audio clips of different languages from around the world, including a label indicating which country the clip comes from. Does such a dataset exist? Or is there ...
3
votes
2answers
918 views

Which is better for Data Science, a double major in Math & CS or Physics & CS? [closed]

My dream occupation is to work as a Data Scientist. I'm soon going to be completing my first semester as a physics student, as part of an incredibly prestigious physics program. We get to run an ...
4
votes
1answer
106 views

How to generate bulk graphics using R

I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2. How can I automatically generate one graphic for each ...
14
votes
5answers
11k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...