Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
0
votes
1answer
19 views
How to make a dataframe with lists or vectors as its elements
This is something I have been wondering for ages but I am never able to get an answer.
I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
0
votes
1answer
14 views
How to categorize machine learning problems, techniques, solutions and algorithms in one super flowchart, consisting of all possible categories? [on hold]
I am trying to create a flowchart that categorizes everything under the sun that exists in machine learning. From the different techniques to the evaluation metrics to miscellaneous techniques used to ...
1
vote
1answer
36 views
What kind of data visualization should I use?
I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
1
vote
1answer
26 views
Importing Excel format data into R/R Studio and using glmnet package?
I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
0
votes
0answers
8 views
which machine learning algorithm we can use for comparing more than two groups(numeric data) [on hold]
which machine learning algorithm we can use for the comparison of more than two groups of technologies
1
vote
1answer
13 views
Sentiment Analysis Datasets
I am looking for sentiment analysis data, mostly customer product review.
I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
1
vote
1answer
38 views
How could I go about finding the weights or importance of inputs based on outputs?
I have a table who's inputs (sfm, fr, and doc) all affect the outputs (...
0
votes
1answer
14 views
Looking for help calculating a probability formula
How do I put this into a calculator or a excel spreadsheet formula. I have never done this math before, but I want to figure it out.
0
votes
0answers
11 views
Why use CRISP framework over other data science frameworks? [closed]
Why are some frameworks for data science workflow more popular and more used than others? Is there any reason why CRISP is used in a lot of data science projects than others?
1
vote
1answer
34 views
How to use glob() output as input to os.listdir()
I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following:
...
0
votes
0answers
11 views
The Probability distribution of an Adversarial Autoencoder
As I want to gain more knowledge about Adversarial Autoencoders, I'm reading this series on "towardsDataScience" about the subject. Now I do have a decent understanding of statistics (at least about ...
1
vote
0answers
18 views
How to label smart meter/plug data (time series)? [closed]
I would like to label smart plug data recorded in different household appliances so that I can train a machine to recognize for example when the oven is turned on. I am new to this topic and I am ...
0
votes
2answers
37 views
Dataset Merging
I have several datasets (All looks like this). The Problem is, that the same user is on several datasets. I need to merge the different sets in a way, that if the username is the same, the frequencies ...
0
votes
0answers
16 views
Google Data to estimate popularity?
I have Google point of interest data including each poi‘s rating and number of rates. Do you think I can use this somehow to estimate popularity of certain regions? I am not sure if this is really ...
0
votes
0answers
28 views
Working with Stanford twitter datasets in R
I intend to use frequency data and sentiment analysis on the Stanford twitter data. I have a couple issues though. One is the size of the data, which I want to reduce, but how do i do this?
A ...
0
votes
0answers
22 views
Mixed linear Model
I have a dataset with G1 (Grade of first semester)and G2(Grade of second semester)and G3 the final grade. I need to model it using linear mixed model with repeated measure to study the variation ...
0
votes
1answer
38 views
How should I tackle this real-life hypermarket problem?
I registered myself in the payback program of the hypermarket I am going to.
For every 2$ I get 1 point.
I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...).
I visit only in weekdays.
...
0
votes
1answer
15 views
Advice on choosing tools for web dashboard
I am working in a small sales company. Started with making relatively easy, small, static reports in Excel, but then it was required to make a lot of highly-customizable and adaptable reports, that ...
1
vote
1answer
13 views
Alternatives to Heat Maps for Communicating Daily Data
I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's ...
1
vote
0answers
16 views
Organizing complex analysis project
Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
0
votes
2answers
58 views
Train classifier on balanced dataset and apply on imbalanced dataset?
I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
2
votes
1answer
30 views
Is data science only for substantive specialist?
Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
0
votes
0answers
10 views
How to interpolate data from a dataframe and calculate a new one by extracting values at a regular step
My goal would be interpolate values from a data frame and have a new dataframe with an index at a predefined step e new values extracted from the interpolation.
This is what I did but I have only NaN :...
0
votes
0answers
9 views
How to impute opt out data
Consider the following problem:
You have been given some survey data on job satisfaction and have to create a model for the employees with the highest risk of leaving the company.
The questions have ...
0
votes
1answer
26 views
PCA in visual Analytics
I m studying visual analytics and i have a theoretical question about this topic.
My professor introduced this schema in him slide
For connect data to visualisation.
Some topic is very easy to ...
0
votes
0answers
19 views
Create a machine learning model classification using my data
I'm working in gdpr project and i wanna to detect personals informations, the idea is to create a mechine learning able to understand the name of the column and classifiy in category for exemple :
...
1
vote
1answer
36 views
How do data scientists test the claimed accuracy of a model?
For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go ...
1
vote
0answers
16 views
is there any package in python similar to tidyrin R? Except Pandas [closed]
is there any package in python similar to tidyr in R? Except Pandas
0
votes
0answers
12 views
Data Representation for sequential input NN
The essence of the problem I want to model is to go from input sequences(length n) to a "distance matrix"(nxn). Although for this distance matrix we are not actually predicting $n^2$ values because it ...
1
vote
1answer
32 views
Support Vector Machine Errors
What is the difference between SVM classification error, SVM margin error, and SVM total error ? Is there any clear definition for them ? And what is C parameter in SVM ? Its totally confusing me !!!
0
votes
0answers
17 views
Project structure - many projects share same large dataset
I have a bunch of projects for my job that are largely unrelated except they use the same data, which is pretty big on disk in csv format. I want these to exist separately from each other and I ...
1
vote
0answers
22 views
DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?
I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection.
Clustering is done ...
2
votes
1answer
75 views
How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library
Sorry for a long story, but it is a long story. :)
I am using h2o library for python to build a decision tree and to extract a decision rules out of it.
I use some data for training where labels get ...
1
vote
1answer
19 views
How to determine the weights of features when calculating the weighted average?
I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
0
votes
0answers
10 views
How to apply curve fitting on both completed and active data?
Suppose we have a set of part failure times on which a specific curve (e.g. gamma distribution) is already fitted. Then another set of times are given where the part is still active (not failed yet). ...
0
votes
0answers
8 views
Best Technique for find Similar Elements in Sets with Widely Varying Numbers of Attributes
I have a data set containing millions of items collected from many disparate sources. Each item contains a list of anywhere from fifty to a thousand attributes. The specific attributes available vary ...
1
vote
0answers
20 views
Calculating unaffected data [closed]
Some data sets values are missing. Missing values are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data would remain unaffected in %. Why?...
0
votes
0answers
54 views
Finding unaffected data in a dataset
The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
1
vote
0answers
6 views
To what extent can one link data in a structured or rigorous way, given only partial knowledge base?
I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
0
votes
0answers
17 views
String Matching columns based on weight
I have the following 2 DataFrames and want to match skills to the key_skills column based on weights.
for eg:
suppose student 1-{python, Java}
job - {python, Ruby, Perl, etc }
weight = 1
In the end,...
1
vote
1answer
85 views
How to feed data to Machine Learning Model?
I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression ...
1
vote
1answer
84 views
Python equivalent of Wolfram Language Query for JSON?
The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
1
vote
1answer
24 views
Why is correlation between my independent variables helping my linear regression model?
I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
0
votes
0answers
10 views
How to efficiently store large amounts of speech to text data
I am currently facing the problem of having to deal with a lot of data (a few hundred GB and growing) for an automated speech recognition (or speech to text) task.
Most datasets I am using are coming ...
1
vote
0answers
18 views
PyMC3: how to efficiently regress on many variables?
I am sorry ahead of time if this seems like a basic question, but I had difficulty finding resources online addressing this. In PyMC3, when building a basic model of a few variables, it is easy to ...
3
votes
2answers
79 views
Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?
I have seen that a lot of people write code in ipython notebook when doing statistical analysis on data, apart from easy visualization after each step rather than running the whole code every time on ...
0
votes
0answers
19 views
Rate quality of sources
Sorry if the title is misleading, I don't know how to describe the issue correctly, which also makes it hard to google.
Say we have historical weather predictions from three different sources. All ...
0
votes
0answers
7 views
Need suggestion on picking fast aggregation framework or solution
To state the problem, this is what I have in the MariaDB/MySQL.
I have the follow table structure, scores
...
0
votes
0answers
7 views
Enrichment analysis Boostrapping fails VS hypergeometric test, why?
I found something that bugs me!
Here is the problem:
I have a set of ~500 objects, that have tags on them. After the selection of X members of that set, I want to know if any of the tags are ...
1
vote
0answers
15 views
Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)
We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or ...
