Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
14
votes
4answers
12k views

Is pandas now faster than data.table?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A-Grouping The data.table benchmarks hasn't been updated since 2014. I heard somewhere that Pandas is ...
0
votes
0answers
44 views

Practical use of bootstrap in R

I am a beginner and try to perform a bootstrap in R. It would be nice if somebody could help me with the following situation. I have defined prices and I want to randomly generate quantities (given a ...
1
vote
0answers
110 views

Isolation forest parameters

I am implementing Isolation Forest from scratch, I am in dilemma of how to set contamination value, and what if not used im the code. Is accuracy differ? What other parameters required to run ...
0
votes
1answer
22 views

How should I collect data for an “identifier” rather than classifier?

I'm trying to build a CNN to identify whether or not a picture contains a lion. One of my classes would obviously be filled with pictures of lions, and pictures that have lions. What about the data I ...
0
votes
0answers
29 views

ways to test simulated data quality

I have a huge data set of logistic objects which I generated by creating a 3D model and rendering it in blender. In order to test the generated data, I've downloaded some images. The simulated data ...
0
votes
2answers
33 views

What fits in a Data Description Report/ Data Exploration Report?

So I am trying to get familiar with Crisp-DM and found the terms "Data Description Report" and "Data Exploration Report", which seem oddly vague in their definition. So far I only found this right ...
4
votes
3answers
237 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
4
votes
1answer
1k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
4
votes
2answers
877 views

Aggregate NumPy array with condition as mask

I have a matrix $b$ with elements: $$b = \begin{pmatrix} 0.01 & 0.02 & \cdots & 1 \\ 0.01 & 0.02 & \cdots & 1 \\ \vdots& \vdots & \ddots & \vdots \\ 0.01 &...
3
votes
2answers
78 views

Rearranging data frame from column names to key value pairs

If I have data like this in R: > a = data.frame(a = c(1, 2), b = c(3, 4)) > a a b 1 1 3 2 2 4 but I would like to have it like this: <...
1
vote
1answer
252 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
2
votes
1answer
75 views

How to penalize for empty fields in a DataFrame?

I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation ...
1
vote
0answers
67 views

Is the Apriori algorithm suitable for database tuples?

Problem description: I need to use an association rule algorithm that lets me use database tuples and I think Apriori is a good option, but I am not sure. Starting point: I know the Apriori ...
0
votes
1answer
28 views

Date transformation for KNN to get distance [duplicate]

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature. For ...
0
votes
2answers
32 views

Reading Date Time in R (01.01.2019 00:00)

How do you read a date in R that is given in the following way: 01.01.2019 00:00 While reading it with read.csv("my_file.csv", sep = ";", dec = ",") I am getting ...
1
vote
1answer
119 views

Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?

Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches ...
0
votes
1answer
37 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
1
vote
1answer
48 views

What kind of data visualization should I use?

I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
1
vote
1answer
225 views

How to use glob() output as input to os.listdir()

I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following: ...
1
vote
1answer
53 views

How could I go about finding the weights or importance of inputs based on outputs?

I have a table who's inputs (sfm, fr, and doc) all affect the outputs (...
0
votes
1answer
15 views

Looking for help calculating a probability formula

How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
1
vote
0answers
25 views

How to label smart meter/plug data (time series)? [closed]

I would like to label smart plug data recorded in different household appliances so that I can train a machine to recognize for example when the oven is turned on. I am new to this topic and I am ...
0
votes
1answer
46 views

How should I tackle this real-life hypermarket problem?

I registered myself in the payback program of the hypermarket I am going to. For every 2$ I get 1 point. I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...). I visit only in weekdays. ...
5
votes
4answers
4k views

Why aren't languages like C, C++ used for data analytics instead of R, Python?

I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++. I have not come across many instances (close ...
2
votes
0answers
40 views

Is data science only for substantive specialist?

Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
1
vote
1answer
231 views

Support Vector Machine Errors

What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!
1
vote
1answer
44 views

How do data scientists test the claimed accuracy of a model?

For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go ...
1
vote
0answers
30 views
0
votes
0answers
14 views

Data Representation for sequential input NN

The essence of the problem I want to model is to go from input sequences(length n) to a "distance matrix"(nxn). Although for this distance matrix we are not actually predicting $n^2$ values because it ...
0
votes
0answers
19 views

Project structure - many projects share same large dataset

I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I ...
0
votes
0answers
14 views

How to apply curve fitting on both completed and active data?

Suppose we have a set of part failure times on which a specific curve (e.g. gamma distribution) is already fitted. Then another set of times are given where the part is still active (not failed yet). ...
0
votes
0answers
12 views

Best Technique for find Similar Elements in Sets with Widely Varying Numbers of Attributes

I have a data set containing millions of items collected from many disparate sources. Each item contains a list of anywhere from fifty to a thousand attributes. The specific attributes available vary ...
1
vote
0answers
23 views

Calculating unaffected data [closed]

Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
0
votes
0answers
65 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
0
votes
0answers
21 views

String Matching columns based on weight

I have the following 2 DataFrames and want to match skills to the key_skills column based on weights. for eg: suppose student 1-{python, Java} job - {python, Ruby, Perl, etc } weight = 1 In the end,...
1
vote
0answers
10 views

To what extent can one link data in a structured or rigorous way, given only partial knowledge base?

I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
0
votes
0answers
19 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
1
vote
0answers
37 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
3
votes
2answers
168 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
0
votes
0answers
19 views

Rate quality of sources

Sorry if the title is misleading, I don't know how to describe the issue correctly, which also makes it hard to google. Say we have historical weather predictions from three different sources. All ...
0
votes
0answers
9 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...
1
vote
0answers
34 views

Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
0
votes
2answers
214 views

Trying to find the correlation between inputs and output

I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code: ...
2
votes
0answers
54 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
0
votes
1answer
867 views

What is the concept of error bars in barplots in seaborn library?

I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in ...
1
vote
2answers
36 views

Continuous values in a dataset

In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?
2
votes
2answers
30 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
1
vote
0answers
28 views

Adding a default date column to a dataframe obtained from a json file

I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format " 2016-1" where 2016 denotes the year ...
0
votes
1answer
33 views

Want suggestions on choosing open source embedded BI tool [closed]

I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities: 1. Can be ...
0
votes
1answer
24 views

Website for Datasets - Miles and Shevlin book, “Applying Regression & Correlation”

I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website ...