Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

How should I collect data for an “identifier” rather than classifier?

I'm trying to build a CNN to identify whether or not a picture contains a lion. One of my classes would obviously be filled with pictures of lions, and pictures that have lions. What about the data I ...
1
vote
0answers
253 views

creating a dataframe out of nested loop data [closed]

I have three parameter arrays ,Parameter1 = [6,7,8],Parameter2 = [11,12] and Parameter3 which can be calculated using Parameter1 and Parameter2. There is a mapping between parameter 1 and 2. I want ...
0
votes
2answers
118 views

What fits in a Data Description Report/ Data Exploration Report?

So I am trying to get familiar with Crisp-DM and found the terms "Data Description Report" and "Data Exploration Report", which seem oddly vague in their definition. So far I only found this right ...
4
votes
1answer
2k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
3
votes
2answers
156 views

Rearranging data frame from column names to key value pairs

If I have data like this in R: > a = data.frame(a = c(1, 2), b = c(3, 4)) > a a b 1 1 3 2 2 4 but I would like to have it like this: <...
4
votes
2answers
1k views

Aggregate NumPy array with condition as mask

I have a matrix $b$ with elements: $$b = \begin{pmatrix} 0.01 & 0.02 & \cdots & 1 \\ 0.01 & 0.02 & \cdots & 1 \\ \vdots& \vdots & \ddots & \vdots \\ 0.01 &...
2
votes
1answer
77 views

How to penalize for empty fields in a DataFrame?

I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation ...
1
vote
0answers
88 views

Is the Apriori algorithm suitable for database tuples?

Problem description: I need to use an association rule algorithm that lets me use database tuples and I think Apriori is a good option, but I am not sure. Starting point: I know the Apriori ...
0
votes
1answer
36 views

Date transformation for KNN to get distance [duplicate]

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature. For ...
0
votes
2answers
109 views

Date transformation for KNN

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
0
votes
2answers
33 views

Reading Date Time in R (01.01.2019 00:00)

How do you read a date in R that is given in the following way: 01.01.2019 00:00 While reading it with read.csv("my_file.csv", sep = ";", dec = ",") I am getting ...
1
vote
1answer
202 views

Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?

Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches ...
0
votes
1answer
51 views

Append and replacing data in r

I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs. Here is my data ...
3
votes
3answers
1k views

How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?

Let's say I used minmaxscaler while creating my model. Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same ...
0
votes
1answer
76 views

Convert categorical data in numeric preserve euclidean distance

I m looking how to preserve Euclidean distance with categorical attribute. Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "...
1
vote
1answer
826 views

Automatic labelling of text data based on predefined entities

I'm new to NLP. I have a folder containing .txt files which are legal and specific documents. I want to label all these files based on four predefined labels. How can I do that automatically?
-1
votes
2answers
52 views

Loops in R programming, [closed]

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
1
vote
1answer
100 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
0
votes
2answers
39 views

How to make a dataframe with lists or vectors as its elements

This is something I have been wondering for ages but I am never able to get an answer. I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
1
vote
1answer
52 views

What kind of data visualization should I use?

I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
1
vote
1answer
142 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
2answers
98 views

Sentiment Analysis Datasets

I am looking for sentiment analysis data, mostly customer product review. I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
1
vote
1answer
53 views

How could I go about finding the weights or importance of inputs based on outputs?

I have a table who's inputs (sfm, fr, and doc) all affect the outputs (...
0
votes
1answer
16 views

Looking for help calculating a probability formula

How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
1
vote
1answer
440 views

How to use glob() output as input to os.listdir()

I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following: ...
1
vote
0answers
25 views

How to label smart meter/plug data (time series)? [closed]

I would like to label smart plug data recorded in different household appliances so that I can train a machine to recognize for example when the oven is turned on. I am new to this topic and I am ...
0
votes
2answers
87 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
1answer
47 views

How should I tackle this real-life hypermarket problem?

I registered myself in the payback program of the hypermarket I am going to. For every 2$ I get 1 point. I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...). I visit only in weekdays. ...
3
votes
1answer
44 views

How to choose tools for web dashboard?

I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that ...
1
vote
1answer
77 views

Alternatives to Heat Maps for Communicating Daily Data

I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's ...
2
votes
1answer
63 views

Organizing complex analysis project

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
4
votes
3answers
311 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
2
votes
0answers
46 views

Is data science only for substantive specialist?

Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
0
votes
1answer
94 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
1
vote
1answer
46 views

How do data scientists test the claimed accuracy of a model?

For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go ...
1
vote
0answers
30 views

is there any package in python similar to tidyrin R? Except Pandas [closed]

is there any package in python similar to tidyr in R? Except Pandas
1
vote
1answer
634 views

Support Vector Machine Errors

What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!
1
vote
1answer
23 views

Project structure - many projects share same large dataset

I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I ...
1
vote
1answer
354 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
2
votes
1answer
736 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
1
vote
1answer
394 views

How to determine the weights of features when calculating the weighted average?

I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
1
vote
0answers
25 views

Calculating unaffected data [closed]

Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
1
vote
0answers
10 views

To what extent can one link data in a structured or rigorous way, given only partial knowledge base?

I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
1
vote
1answer
505 views

How to feed data to Machine Learning Model?

I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression ...
1
vote
1answer
147 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
1
vote
1answer
48 views

Why is correlation between my independent variables helping my linear regression model?

I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
1
vote
0answers
46 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
3
votes
2answers
256 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
1
vote
0answers
38 views

Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
2
votes
2answers
280 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....

1
4 5
6
7 8
12