Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
764
questions
0
votes
0
answers
13
views
Are there any open datasets for commercial use?
I am creating a bootcamp for data analyst and it's been 2 days I am looking for some good dataset fit for commercial use that I can use to create Tableau and Power BI tutorials. Even on kaggle some ...
- 141
-3
votes
0
answers
13
views
Which regulations should not be overlooked? [closed]
i made some research about this subject. im literly facinated about that. I dont have PHD and doesn't work on this industry.
Im young man interesting, this is my questions.
italicIt is possible to ...
- 1
0
votes
0
answers
8
views
Wave Function in Python
How to apply Wave Function to a Data Set in Python to derive frequency distribution and probability amplitude?
0
votes
0
answers
14
views
How can I statistically measure/determine if A performs better than B?
Hi Data Science Community!
I am a new Data Intern and I have been stuck on this question for a while.
Here is a sample dataset I am working with:
Customer
Manufacturer A Spending
Manufacturer B ...
0
votes
1
answer
11
views
How do i conduct t-test for comparing the accuracy of two binary classifiers?
I am using two binary classifiers that predicts the accuracy of samples over a dataset. Accuracy is defined as ratio of correct vs incorrect predictions. Do i need to take accuracies sampled over ...
- 11
0
votes
0
answers
8
views
Where should I find electrolytic capacitor ageing data
I am trying to get a dataset of Electrolytic capacitors ageing and I am not being able to find one that shows the ripple current and the voltage in order to calculate its Equivalent Series Resistance (...
0
votes
0
answers
19
views
Are experiments using confidence interval can be said a statistical test
I am working on some algorithm that is comparing results with other model using confidence interval , 90%. Can this be said a statistical test ?
I read a article where it said about statistical test ...
- 11
0
votes
0
answers
14
views
0
votes
0
answers
17
views
Approaches on grouping/clustering network device data
So I come from more of a computer science background, and recently have been trying to find a solution to a data-centered problem. I would like to try experimenting different data-science methods on ...
- 1
0
votes
0
answers
29
views
What kind of statistical test can be performed in a recommender system dataset that predicts the ratings for the movies?
The dataset consists of 1000s of users and users and each row of the dataset consist of user_id,movie_id and ratings the user provides to the movie.
eg. 1,56,5
In my experiment i am calculating the ...
- 11
1
vote
1
answer
45
views
Incorrect example of applying Bayes theorem
I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
- 111
0
votes
0
answers
5
views
Creating synthetic data to increase the training data for ML model
I was wondering if I can create more data from the available dataset to train my model and what are the pitfalls of such a practice?
I have a modest number of examples in my dataset, but I believe ...
- 131
0
votes
1
answer
43
views
Data preprocessing methods
Data Cleaning
Data Imbalance solving (Classification)
Data Smoothing (decreasing noise)
Creating-deleting features from original data
Data Transformation (Box-cox,Log Transform)
Making Dataset ...
- 23
-1
votes
0
answers
14
views
What are the biggest risks about the ethical dilemmas in data science?
What is your opinion about ethical dilemmas in data science?, what are some things that may happen when the privacy rights are not respected?
0
votes
0
answers
6
views
Missclassfication of Hand Generated Signals
I have two types of time series accelerometer data from two kinds of machines - one is very fast (Type A) and another one is relatively slow in terms of number of peaks/movement (Type B).
I have tried ...
- 11
0
votes
0
answers
8
views
stratified segment-grouped k-fold cross-validation
I have a music numerical data (2282 rows × 173 columns) to predict the target sad, happy, angry, relaxed. Now one of the attribute is segment_id and I want to group the data according to segment_id ...
- 11
0
votes
0
answers
16
views
How to sample a dataframe or numpy array with a particular interval?
I have the following dataframe :
...
- 1
0
votes
0
answers
12
views
Looking for NLP datasets with small vocabularies
I'm looking for NLP datasets/corpora with small vocabulary sizes -- less than 5K unique words, but smaller still is better. I've tried looking for e.g. datasets of children's books but I've not found ...
- 1
1
vote
0
answers
59
views
0
votes
0
answers
10
views
Is there any way to get data sets from mldata.org?
I need data sets for my project."Record of heart sound " which has Recordings of normal and abnormal heartbeats used to recognize heart murmurs etc.
link:
http://mldata.org/repository/data/...
0
votes
0
answers
10
views
Has anyone joined FHIR resources?
I have FHIR-formatted data (Fast Healthcare Interoperability Resources), that I've converted to CSV using the npm package ndjson-to-csv, and on first glance, this seems useable for ML experimentation, ...
- 101
1
vote
0
answers
26
views
How much would the annual data storage costs be for police body-worn camera footage (between 5PB and 120EB)?
Judging from the popular tags, I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways...
I was just wondering what the data ...
- 111
0
votes
1
answer
15
views
Using large files in a OVHcloud AI notebook
I’m trying to train an artificial intelligence model on OVHCloud AI Notebooks with the Common Voice’s dataset from Mozilla.
The problem is that this database has a 70 GB size. I have tried to download ...
- 1
0
votes
0
answers
7
views
Where can I find trimodal data? (soft question)
I'm looking for a particular kind of data which is usually called trimodal data in order to fit them onto a new probability distribution I created with other colleagues. In essence, it's data with ...
- 101
0
votes
0
answers
13
views
Data privacy breach example (data anonymisation)
I remember that I read a story where journalists were able to figure out the health records of some individual (I think it was some senator, but not sure) by using different data sets. That was an ...
- 9
0
votes
0
answers
9
views
How to check the validity of data? if There are repeated Y values against X data
I need to find the best-fitted curve for this data, I am not sure which model should I use? Can someone suggest me a model?
I also have doubts that this dataset is not valid. As there are multiple Y ...
- 101
0
votes
0
answers
7
views
Calculating Demand confidence
I recently was asked to work on a business problem related to sales using Data Science. This required me to come up with a Demand confidence number that would specify how confident we are on ...
0
votes
1
answer
11
views
adding a column with numbered observations per id
Good day I have a dataset that is in long form with sheep IDs and their lamb IDs.
There are multiple rows per ewe as they have multiple lambs each with a unique id.
I don't want a count per ewe, but ...
- 1
1
vote
1
answer
25
views
2
votes
1
answer
83
views
too much data to label
I'm working on a Data Science project to flag bots on Instagram. I collected a lot of data (+80k users) and now I have to label them as bot/legit users. I already flagged 20k users with different ...
- 83
0
votes
1
answer
18
views
How do I decide the frequency of data capture for modeling? How does it affect my final model?
I plan to capture data to predict energy consumption in a food processing plant. I want to capture production details such as how much each category of food is produced, what is the machine's output, ...
- 109
1
vote
1
answer
27
views
HyperOpt: Finding the best modeling based on precision or f1 score
I have been using the hyperopt for 2 days now and I am trying to create logistic regression models using the hyperopt and choosing the best combination of parameters by their f1 scores. However, ...
- 11
0
votes
1
answer
18
views
How do I import and export more than one numpy arrays?
I have four numpy arrays X_train, y_train, X_test, y_test...
0
votes
1
answer
18
views
Plotting two pd dataframes with n columns, each df representing an axis coordinate?
I have two dataframes which each have n columns (changes depending input), where each dataframe represents an axis coordinate. Each column represents separate lines I want to plot.
Example dfs (1st ...
- 1
0
votes
0
answers
14
views
Clustering by resource use in different categories
Task
The task is to cluster units by similar resource use.
For example we might have data such as:
Resource A
Resource B
Resource C
Resource D
Resource E
Total
Unit 1
500
0
0
0
100
600
Unit 2
0
1,...
1
vote
1
answer
27
views
the right laptop for data analysis? (using a big Data Lake)
Does working with data visualization from a big Data Lake require a powerful laptop? I've been offered a P15 because my boss believes my work will be requiring heavy computations, which normally would ...
- 11
1
vote
1
answer
58
views
how many rows does pandas' interpolate consider? [closed]
How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers:
is it just the row before the NaNs and the row right after?
Or is it the whole DataFrame (how does that ...
- 11
0
votes
0
answers
22
views
Is it absolutely necessary to shuffle a large training data set every epoch?
There have been few questions on shuffling the training dataset every epoch. The general consensus is that it helps so you should do it.
My question is whether this gain from shuffling every epoch ...
- 101
0
votes
0
answers
10
views
I have two datasets x(t) and y(t) with mismatching time frames. How do I get y(x)?
I intend to use python to solve this problem. I looked at merge() and merge_asof() but they dont seem to solve my specific problem.
They timeframes have non-uniform delta-t and they seldomly match ...
0
votes
0
answers
9
views
Power BI: Add slicer from measure
I have a list of working hours by date for every employee at a company and I have created a measure to calculate the total working hours. In the report I used a date slider so that the user can see ...
- 1
0
votes
0
answers
11
views
How to predict on a different time scale to the feature set in ML models?
Is there a technique, or selection of techniques, for developing robust machine learning forecasts on a different timescale to the feature set using a directional classifier model?
For example, say I ...
- 1
0
votes
0
answers
7
views
0
votes
0
answers
11
views
create a model to identify the skills priroty
I have created a dataset that has row information as [jobtitle, (skill1, skill1_score), (...), (skilln, skilln_score)].
To create the dataset, I have extracted job titles and job descriptions that ...
- 199
1
vote
1
answer
33
views
Probability Distribution on the Test Set
How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted?
Here x axis ...
- 11
4
votes
1
answer
184
views
How to interpret standard deviation calculated by excel
I have some percentage values in my excel spreadsheet like:
When i use the STDEV function to calculate the standard deviation with these values, i get ...
- 395
0
votes
1
answer
10
views
Extracting Public Page Posts from Facebook
I am a data science student working on my capstone research project. I am trying to collect posts from a number of public organization pages from facebook. I am looking at the graphs api and it does ...
- 101
0
votes
0
answers
12
views
How to mathematically define the "singular points" in a graph
In the graph, we can see several "singular points" (red circle) which show a significant change. But I don't know how to mathematically describe such points and find them out. Here the "...
- 125
0
votes
0
answers
8
views
Discover the geographical position of weather historical data using weather historical predictions
I have too little experience in neural networks and I wanted to solve a problem that I don't know how to raise.
On the one side, I have a 1 year hourly historical data of the temperature in a specific ...
- 101
1
vote
1
answer
17
views
Merging 2 datasets
In the given task, they've provides us with 2 datasets namely the test & train dataset. So, I was wondering if we could merge those 2 datasets into 1 data-frame and delete the duplicates. Would ...
2
votes
0
answers
30
views
How Should I Create Waveform Datasets
Full disclosure I asked this on StackOverflow and it got taken down as it was more of a how do I do this, not how do I code this question:
I am trying to simulate/fake data that I eventually will ...
