Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
1
vote
1answer
352 views

Automatic labelling of text data based on predefined entities

I'm new to NLP. I have a folder containing .txt files which are legal and specific documents. I want to label all these files based on four predefined labels. How can I do that automatically?
0
votes
2answers
42 views

Loops in R programming,

I want to update this Remaining column in this table below: How do I do this in R programming? I tried while (n.Remaining>0) { n.remaining <- n.total-n.expense} Desired Output:
0
votes
1answer
33 views

Can you apply PCA to part of your dataset?

I am working with kaggle dataset that has over 130 features composed of 116 categorical and 14 continuous features. I plotted the heatmap for the 14 continuous variables and found that most of them ...
0
votes
1answer
27 views

How to make a dataframe with lists or vectors as its elements

This is something I have been wondering for ages but I am never able to get an answer. I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
1
vote
1answer
46 views

What kind of data visualization should I use?

I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
1
vote
1answer
62 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
2answers
64 views

Sentiment Analysis Datasets

I am looking for sentiment analysis data, mostly customer product review. I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
1
vote
1answer
51 views

How could I go about finding the weights or importance of inputs based on outputs?

I have a table who's inputs (sfm, fr, and doc) all affect the outputs (...
0
votes
1answer
15 views

Looking for help calculating a probability formula

How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
1
vote
1answer
177 views

How to use glob() output as input to os.listdir()

I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following: ...
1
vote
0answers
25 views

How to label smart meter/plug data (time series)? [closed]

I would like to label smart plug data recorded in different household appliances so that I can train a machine to recognize for example when the oven is turned on. I am new to this topic and I am ...
0
votes
2answers
70 views

Dataset Merging

I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
1answer
45 views

How should I tackle this real-life hypermarket problem?

I registered myself in the payback program of the hypermarket I am going to. For every 2$ I get 1 point. I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...). I visit only in weekdays. ...
3
votes
1answer
40 views

How to choose tools for web dashboard?

I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that ...
1
vote
1answer
41 views

Alternatives to Heat Maps for Communicating Daily Data

I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's ...
2
votes
1answer
49 views

Organizing complex analysis project

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
4
votes
3answers
227 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
2
votes
0answers
40 views

Is data science only for substantive specialist?

Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
0
votes
1answer
58 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
1
vote
1answer
44 views

How do data scientists test the claimed accuracy of a model?

For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go ...
1
vote
0answers
29 views

is there any package in python similar to tidyrin R? Except Pandas [closed]

is there any package in python similar to tidyr in R? Except Pandas
0
votes
0answers
14 views

Data Representation for sequential input NN

The essence of the problem I want to model is to go from input sequences(length n) to a "distance matrix"(nxn). Although for this distance matrix we are not actually predicting $n^2$ values because it ...
1
vote
1answer
168 views

Support Vector Machine Errors

What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!
0
votes
0answers
19 views

Project structure - many projects share same large dataset

I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I ...
1
vote
1answer
223 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
2
votes
1answer
447 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
1
vote
1answer
294 views

How to determine the weights of features when calculating the weighted average?

I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
0
votes
0answers
14 views

How to apply curve fitting on both completed and active data?

Suppose we have a set of part failure times on which a specific curve (e.g. gamma distribution) is already fitted. Then another set of times are given where the part is still active (not failed yet). ...
0
votes
0answers
12 views

Best Technique for find Similar Elements in Sets with Widely Varying Numbers of Attributes

I have a data set containing millions of items collected from many disparate sources. Each item contains a list of anywhere from fifty to a thousand attributes. The specific attributes available vary ...
1
vote
0answers
23 views

Calculating unaffected data [closed]

Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
0
votes
0answers
64 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
1
vote
0answers
10 views

To what extent can one link data in a structured or rigorous way, given only partial knowledge base?

I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
0
votes
0answers
19 views

String Matching columns based on weight

I have the following 2 DataFrames and want to match skills to the key_skills column based on weights. for eg: suppose student 1-{python, Java} job - {python, Ruby, Perl, etc } weight = 1 In the end,...
1
vote
1answer
350 views

How to feed data to Machine Learning Model?

I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression ...
1
vote
1answer
116 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
1
vote
1answer
34 views

Why is correlation between my independent variables helping my linear regression model?

I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
0
votes
0answers
18 views

How to efficiently store large amounts of speech to text data

I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task. Most datasets I am using are coming ...
1
vote
0answers
33 views

PyMC3: how to efficiently regress on many variables?

I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
3
votes
2answers
159 views

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
0
votes
0answers
19 views

Rate quality of sources

Sorry if the title is misleading, I don't know how to describe the issue correctly, which also makes it hard to google. Say we have historical weather predictions from three different sources. All ...
0
votes
0answers
8 views

Need suggestion on picking fast aggregation framework or solution

To state the problem, this is what I have in the MariaDB/MySQL. I have the follow table structure, scores ...
0
votes
0answers
15 views

Enrichment analysis Boostrapping fails VS hypergeometric test, why?

I found something that bugs me! Here is the problem: I have a set of ~500 objects, that have tags on them. After the selection of X members of that set, I want to know if any of the tags are ...
1
vote
0answers
34 views

Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
2
votes
2answers
153 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
0
votes
2answers
154 views

Trying to find the correlation between inputs and output

I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code: ...
2
votes
0answers
54 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
0
votes
1answer
783 views

What is the concept of error bars in barplots in seaborn library?

I am using seaborn library to perform EDA on a dataset having categorical variables. I am using barplot function and in that there are some black bars which I guess are error bars as I read in ...
0
votes
1answer
56 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
1answer
53 views

Behavioural data required to predict churn

I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all ...
0
votes
1answer
242 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...