Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
864
questions
0
votes
0
answers
17
views
Sensor data completeness
We have several sensors collecting data with a specified frequency (usually every 20 minutes but it varies) over a period of many years. The person that programmed them made sure, as a space saving ...
0
votes
0
answers
144
views
Changing a 0-1 column datatype from int64 to uint8 such as in pandas.get_dummies()
Is it advisable to change the datatype int64 of a 0-1's column to uint8 such as ...
0
votes
0
answers
41
views
Implementation/Optimization of Genetic Algorithm
I have a few queries regarding my implementation of the Genetic Algorithm (GA).
I have a lot of parameters in which I have to find the best combination of these parameters to maximize the value of the ...
0
votes
0
answers
24
views
Software aid in plotting dataset and data table relations
I am at a complete loss of where to start with this in terms of where to start to look and what tags I should mark this with.
I am working on gathering data from different sources with different ...
- 1
1
vote
1
answer
64
views
NLP logistic regression
A basic doubt I have, Usually when dealing with text data for classic ML algos, We use Tf-idf which uses entire vocabulary for each row and assigns some weighted value.
My doubt is can I use 5 feature ...
- 11
0
votes
0
answers
8
views
Purposefully Duplicating Data to weight Binary Classification
I'm working on my first practical machine learning project. It's a binary image classification problem that is intended to sort images into "Pass" or "Fail" categories.
I've gotten ...
- 1
0
votes
0
answers
10
views
Shapiro Wilk test for normality for 6 treatments in R
I am trying to learn R and am struggling with some homework for an agriculture module. I need to check for normality in a dataset on specific growth rate, and I can very easily check for normality ...
1
vote
0
answers
160
views
Averaging multiple train-test splits to estimate the performance with higher variability?
I have a small size data set and I want to assess the effect of a certain type of cases on the overall model performance. For example, is the model biased against people of a certain age group?
Using ...
0
votes
0
answers
109
views
Are there any open datasets for commercial use?
I am creating a bootcamp for data analyst and it's been 2 days I am looking for some good dataset fit for commercial use that I can use to create Tableau and Power BI tutorials. Even on kaggle some ...
- 141
0
votes
0
answers
15
views
Where should I find electrolytic capacitor ageing data
I am trying to get a dataset of Electrolytic capacitors ageing and I am not being able to find one that shows the ripple current and the voltage in order to calculate its Equivalent Series Resistance (...
0
votes
0
answers
23
views
1
vote
1
answer
77
views
Incorrect example of applying Bayes theorem
I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
- 111
1
vote
0
answers
71
views
0
votes
0
answers
64
views
Is there any way to get data sets from mldata.org?
I need data sets for my project."Record of heart sound " which has Recordings of normal and abnormal heartbeats used to recognize heart murmurs etc.
link:
http://mldata.org/repository/data/...
1
vote
1
answer
135
views
How much would the annual data storage costs be for police body-worn camera footage (between 5PB and 120EB)?
Judging from the popular tags, I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways...
I was just wondering what the data ...
- 111
0
votes
1
answer
33
views
Using large files in a OVHcloud AI notebook
I’m trying to train an artificial intelligence model on OVHCloud AI Notebooks with the Common Voice’s dataset from Mozilla.
The problem is that this database has a 70 GB size. I have tried to download ...
- 1
0
votes
0
answers
9
views
How to check the validity of data? if There are repeated Y values against X data
I need to find the best-fitted curve for this data, I am not sure which model should I use? Can someone suggest me a model?
I also have doubts that this dataset is not valid. As there are multiple Y ...
- 101
0
votes
1
answer
76
views
adding a column with numbered observations per id
Good day I have a dataset that is in long form with sheep IDs and their lamb IDs.
There are multiple rows per ewe as they have multiple lambs each with a unique id.
I don't want a count per ewe, but ...
- 1
1
vote
1
answer
102
views
2
votes
1
answer
126
views
too much data to label
I'm working on a Data Science project to flag bots on Instagram. I collected a lot of data (+80k users) and now I have to label them as bot/legit users. I already flagged 20k users with different ...
- 222
0
votes
1
answer
26
views
How do I decide the frequency of data capture for modeling? How does it affect my final model?
I plan to capture data to predict energy consumption in a food processing plant. I want to capture production details such as how much each category of food is produced, what is the machine's output, ...
- 233
2
votes
1
answer
465
views
HyperOpt: Finding the best modeling based on precision or f1 score
I have been using the hyperopt for 2 days now and I am trying to create logistic regression models using the hyperopt and choosing the best combination of parameters by their f1 scores. However, ...
- 21
0
votes
1
answer
165
views
How do I import and export more than one numpy arrays?
I have four numpy arrays X_train, y_train, X_test, y_test...
0
votes
1
answer
170
views
Plotting two pd dataframes with n columns, each df representing an axis coordinate?
I have two dataframes which each have n columns (changes depending input), where each dataframe represents an axis coordinate. Each column represents separate lines I want to plot.
Example dfs (1st ...
- 1
1
vote
1
answer
38
views
the right laptop for data analysis? (using a big Data Lake)
Does working with data visualization from a big Data Lake require a powerful laptop? I've been offered a P15 because my boss believes my work will be requiring heavy computations, which normally would ...
- 11
1
vote
1
answer
548
views
how many rows does pandas' interpolate consider? [closed]
How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers:
is it just the row before the NaNs and the row right after?
Or is it the whole DataFrame (how does that ...
- 11
1
vote
1
answer
131
views
Probability Distribution on the Test Set
How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted?
Here x axis ...
- 11
4
votes
1
answer
255
views
How to interpret standard deviation calculated by excel
I have some percentage values in my excel spreadsheet like:
When i use the STDEV function to calculate the standard deviation with these values, i get ...
- 395
0
votes
1
answer
18
views
Extracting Public Page Posts from Facebook
I am a data science student working on my capstone research project. I am trying to collect posts from a number of public organization pages from facebook. I am looking at the graphs api and it does ...
- 101
1
vote
1
answer
24
views
Merging 2 datasets
In the given task, they've provides us with 2 datasets namely the test & train dataset. So, I was wondering if we could merge those 2 datasets into 1 data-frame and delete the duplicates. Would ...
2
votes
0
answers
37
views
How Should I Create Waveform Datasets
Full disclosure I asked this on StackOverflow and it got taken down as it was more of a how do I do this, not how do I code this question:
I am trying to simulate/fake data that I eventually will ...
0
votes
1
answer
59
views
Can I include a quotient as dependent variable and independent variables with same denominator in a linear model? How do we interpret such models?
I want to create a model in a food processing plant where my dependent variable is Electricity (KWhr) consumption per kg. Plant produce different food items with varying electricity consumption. I'm ...
- 233
0
votes
1
answer
569
views
Study names not showing in forest plot (R)
Context: I followed an existing R script to create a forest plot in R using the forest function from the metafor package.
Problem: The plot is showing, however, the study names are not showing on it ...
- 39
0
votes
1
answer
192
views
Dataset with Multiple Choice Questions for fine tuning
I hope it's allowed to ask here, but I am looking for a dataset (the format is not that important) that is similar to SQuAD, but it also contains false answers to the questions. I wanna use it to fine ...
3
votes
1
answer
192
views
How to merge with smallest Euclidean distance?
Given a main left dataset, how can I merge with right dataset with smallest Euclidean distance (...
- 130
2
votes
3
answers
263
views
What is exactly the difference between Validation data and Testing data
I asked this question on stack overflow and was told that this is a better place for it.
I am confused with the terms validation and testing, is validating the model same as testing it? is it possible ...
- 23
0
votes
2
answers
63
views
Association Mining / rules are the statements appears to be true
I have a problem. I don't know how I cloud explain which of the three statements appears to be true and if my calculation below is invalid?`
The following table summarizes the results of a medical ...
- 89
1
vote
3
answers
87
views
Association Mining - is buying Independent?
I have a problem. I can't not solve this exerciese. What is the best way to solve this exerciese? What are the approaches for this kind of exerciese?
The following table summarizes transactions in a ...
- 89
0
votes
0
answers
28
views
Prediction based on predicted data
may I ask if anyone of you knows well references on the prediction method based on the predicted data? By prediction based on the predicted data, I mean for example I have constructed a predictive ...
- 19
0
votes
1
answer
53
views
Associationg Rules/ Mining is this association rule strong
I have a problem. I can't not solve this exerciese. What is the best way to solve this exerciese? What are the approaches for this kind of exerciese?
The following table summarizes transactions in a ...
- 89
0
votes
2
answers
370
views
How to convert String data to something more meaningful for regression model training
I have a bunch of data of employees and their salaries.
I would like to build a regression model that predicts
The columns in question are countries, employment_status, job_title, education
All of ...
0
votes
1
answer
40
views
Predicting Antennae Coverage using SPLAT
I have been trying to predict the network coverage based on antennae characteristics (such as height, lat and long etc).
Came across this package which seems to be dormant now (https://github.com/NZRS/...
- 23
1
vote
0
answers
22
views
Repeating values caught with a binary classifier
If my machine is broken, it starts to repeat certain channels. Thing is if there are no out-liars, it is difficult to tell it's broken as we would expect all data points to be around the same value. I ...
- 11
1
vote
1
answer
121
views
Knit with R markdown [closed]
I have tried running some code on RStudio Desktop, within the Chunk the codes runs smoothly, but when knitting to view as html, I get the message that something is wrong with a line of code. What can ...
0
votes
1
answer
144
views
Cross Validation with One Class Classification in Python
I'm trying to do cross-validation with One Class Classification - I'm using the PyOD lib - but I don't know if I'm doing it right. The precision is too low and I'm also not able to bring up the mean ...
1
vote
0
answers
24
views
How can one ensure the quality of collectively created ground truth labels?
Here is the situation: I have a small tool to assign (one or multiple) labels to image data. In the tool, you can not only assign a label(s) to the images but also assign a score for the certainty of ...
- 11
0
votes
1
answer
34
views
Issue with sarcasm detection
I am working on the Reddit dataset for sarcasm detection but the sarcastic data points(1) are showing zero percent recall, precision, and accuracy however nonsarcastic are showing 100% recall and 50 ...
1
vote
0
answers
15
views
Where to get all major stock market events/regimes and dates?
I am looking for a dataset that gets all events and regimes in the stock market (i.e. an event like 9/11 and a regime like Bush's presidency). I was doing some searching and found I could probably do ...
- 111
0
votes
0
answers
585
views
showing this error ValueError: shapes (1,400) and (10000,10) not aligned: 400 (dim 1) != 10000 (dim 0)
I am trying to run a network and get this error .I am a noob,can someone help me.
...
- 1
0
votes
1
answer
41
views
Data Anonymization for all domains?
I am using a dataset from Marketing and sales department. The dataset contains customer name (company name), company address, pincode, no of orders placed, revenue generated from that customer etc.
My ...
- 2,507