Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Are there any open datasets for commercial use?

I am creating a bootcamp for data analyst and it's been 2 days I am looking for some good dataset fit for commercial use that I can use to create Tableau and Power BI tutorials. Even on kaggle some ...
user avatar
  • 141
-3 votes
0 answers
13 views

Which regulations should not be overlooked? [closed]

i made some research about this subject. im literly facinated about that. I dont have PHD and doesn't work on this industry. Im young man interesting, this is my questions. italicIt is possible to ...
user avatar
  • 1
0 votes
0 answers
8 views

Wave Function in Python

How to apply Wave Function to a Data Set in Python to derive frequency distribution and probability amplitude?
user avatar
0 votes
0 answers
14 views

How can I statistically measure/determine if A performs better than B?

Hi Data Science Community! I am a new Data Intern and I have been stuck on this question for a while. Here is a sample dataset I am working with: Customer Manufacturer A Spending Manufacturer B ...
user avatar
0 votes
1 answer
11 views

How do i conduct t-test for comparing the accuracy of two binary classifiers?

I am using two binary classifiers that predicts the accuracy of samples over a dataset. Accuracy is defined as ratio of correct vs incorrect predictions. Do i need to take accuracies sampled over ...
user avatar
0 votes
0 answers
8 views

Where should I find electrolytic capacitor ageing data

I am trying to get a dataset of Electrolytic capacitors ageing and I am not being able to find one that shows the ripple current and the voltage in order to calculate its Equivalent Series Resistance (...
user avatar
0 votes
0 answers
19 views

Are experiments using confidence interval can be said a statistical test

I am working on some algorithm that is comparing results with other model using confidence interval , 90%. Can this be said a statistical test ? I read a article where it said about statistical test ...
user avatar
0 votes
0 answers
14 views

How to do binning in matrix data

I have some data like ...
user avatar
0 votes
0 answers
17 views

Approaches on grouping/clustering network device data

So I come from more of a computer science background, and recently have been trying to find a solution to a data-centered problem. I would like to try experimenting different data-science methods on ...
user avatar
  • 1
0 votes
0 answers
29 views

What kind of statistical test can be performed in a recommender system dataset that predicts the ratings for the movies?

The dataset consists of 1000s of users and users and each row of the dataset consist of user_id,movie_id and ratings the user provides to the movie. eg. 1,56,5 In my experiment i am calculating the ...
user avatar
1 vote
1 answer
45 views

Incorrect example of applying Bayes theorem

I have been reading the book "The Data Science Design Manual" (by Steven S. Skiena) and I came across an example that explained how the Bayes theorem can be applied that confused me and made ...
user avatar
  • 111
0 votes
0 answers
5 views

Creating synthetic data to increase the training data for ML model

I was wondering if I can create more data from the available dataset to train my model and what are the pitfalls of such a practice? I have a modest number of examples in my dataset, but I believe ...
user avatar
  • 131
0 votes
1 answer
43 views

Data preprocessing methods

Data Cleaning Data Imbalance solving (Classification) Data Smoothing (decreasing noise) Creating-deleting features from original data Data Transformation (Box-cox,Log Transform) Making Dataset ...
user avatar
  • 23
-1 votes
0 answers
14 views

What are the biggest risks about the ethical dilemmas in data science?

What is your opinion about ethical dilemmas in data science?, what are some things that may happen when the privacy rights are not respected?
user avatar
0 votes
0 answers
6 views

Missclassfication of Hand Generated Signals

I have two types of time series accelerometer data from two kinds of machines - one is very fast (Type A) and another one is relatively slow in terms of number of peaks/movement (Type B). I have tried ...
user avatar
  • 11
0 votes
0 answers
8 views

stratified segment-grouped k-fold cross-validation

I have a music numerical data (2282 rows × 173 columns) to predict the target sad, happy, angry, relaxed. Now one of the attribute is segment_id and I want to group the data according to segment_id ...
user avatar
0 votes
0 answers
16 views

How to sample a dataframe or numpy array with a particular interval?

I have the following dataframe : ...
user avatar
  • 1
0 votes
0 answers
12 views

Looking for NLP datasets with small vocabularies

I'm looking for NLP datasets/corpora with small vocabulary sizes -- less than 5K unique words, but smaller still is better. I've tried looking for e.g. datasets of children's books but I've not found ...
user avatar
  • 1
1 vote
0 answers
59 views

Proof related to Ward's Method

According to Ward's Method that says :
user avatar
0 votes
0 answers
10 views

Is there any way to get data sets from mldata.org?

I need data sets for my project."Record of heart sound " which has Recordings of normal and abnormal heartbeats used to recognize heart murmurs etc. link: http://mldata.org/repository/data/...
user avatar
0 votes
0 answers
10 views

Has anyone joined FHIR resources?

I have FHIR-formatted data (Fast Healthcare Interoperability Resources), that I've converted to CSV using the npm package ndjson-to-csv, and on first glance, this seems useable for ML experimentation, ...
user avatar
1 vote
0 answers
26 views

How much would the annual data storage costs be for police body-worn camera footage (between 5PB and 120EB)?

Judging from the popular tags, I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways... I was just wondering what the data ...
user avatar
  • 111
0 votes
1 answer
15 views

Using large files in a OVHcloud AI notebook

I’m trying to train an artificial intelligence model on OVHCloud AI Notebooks with the Common Voice’s dataset from Mozilla. The problem is that this database has a 70 GB size. I have tried to download ...
user avatar
0 votes
0 answers
7 views

Where can I find trimodal data? (soft question)

I'm looking for a particular kind of data which is usually called trimodal data in order to fit them onto a new probability distribution I created with other colleagues. In essence, it's data with ...
user avatar
  • 101
0 votes
0 answers
13 views

Data privacy breach example (data anonymisation)

I remember that I read a story where journalists were able to figure out the health records of some individual (I think it was some senator, but not sure) by using different data sets. That was an ...
user avatar
  • 9
0 votes
0 answers
9 views

How to check the validity of data? if There are repeated Y values against X data

I need to find the best-fitted curve for this data, I am not sure which model should I use? Can someone suggest me a model? I also have doubts that this dataset is not valid. As there are multiple Y ...
user avatar
  • 101
0 votes
0 answers
7 views

Calculating Demand confidence

I recently was asked to work on a business problem related to sales using Data Science. This required me to come up with a Demand confidence number that would specify how confident we are on ...
user avatar
0 votes
1 answer
11 views

adding a column with numbered observations per id

Good day I have a dataset that is in long form with sheep IDs and their lamb IDs. There are multiple rows per ewe as they have multiple lambs each with a unique id. I don't want a count per ewe, but ...
user avatar
  • 1
1 vote
1 answer
25 views

Replacing count by percentage in R

I have the following code : ...
user avatar
  • 25
2 votes
1 answer
83 views

too much data to label

I'm working on a Data Science project to flag bots on Instagram. I collected a lot of data (+80k users) and now I have to label them as bot/legit users. I already flagged 20k users with different ...
user avatar
  • 83
0 votes
1 answer
18 views

How do I decide the frequency of data capture for modeling? How does it affect my final model?

I plan to capture data to predict energy consumption in a food processing plant. I want to capture production details such as how much each category of food is produced, what is the machine's output, ...
user avatar
  • 109
1 vote
1 answer
27 views

HyperOpt: Finding the best modeling based on precision or f1 score

I have been using the hyperopt for 2 days now and I am trying to create logistic regression models using the hyperopt and choosing the best combination of parameters by their f1 scores. However, ...
user avatar
  • 11
0 votes
1 answer
18 views

How do I import and export more than one numpy arrays?

I have four numpy arrays X_train, y_train, X_test, y_test...
user avatar
0 votes
1 answer
18 views

Plotting two pd dataframes with n columns, each df representing an axis coordinate?

I have two dataframes which each have n columns (changes depending input), where each dataframe represents an axis coordinate. Each column represents separate lines I want to plot. Example dfs (1st ...
user avatar
  • 1
0 votes
0 answers
14 views

Clustering by resource use in different categories

Task The task is to cluster units by similar resource use. For example we might have data such as: Resource A Resource B Resource C Resource D Resource E Total Unit 1 500 0 0 0 100 600 Unit 2 0 1,...
user avatar
1 vote
1 answer
27 views

the right laptop for data analysis? (using a big Data Lake)

Does working with data visualization from a big Data Lake require a powerful laptop? I've been offered a P15 because my boss believes my work will be requiring heavy computations, which normally would ...
user avatar
1 vote
1 answer
58 views

how many rows does pandas' interpolate consider? [closed]

How does pandas' DataFrame.interpolate() work in relation to the amount of rows it considers: is it just the row before the NaNs and the row right after? Or is it the whole DataFrame (how does that ...
user avatar
0 votes
0 answers
22 views

Is it absolutely necessary to shuffle a large training data set every epoch?

There have been few questions on shuffling the training dataset every epoch. The general consensus is that it helps so you should do it. My question is whether this gain from shuffling every epoch ...
user avatar
0 votes
0 answers
10 views

I have two datasets x(t) and y(t) with mismatching time frames. How do I get y(x)?

I intend to use python to solve this problem. I looked at merge() and merge_asof() but they dont seem to solve my specific problem. They timeframes have non-uniform delta-t and they seldomly match ...
user avatar
0 votes
0 answers
9 views

Power BI: Add slicer from measure

I have a list of working hours by date for every employee at a company and I have created a measure to calculate the total working hours. In the report I used a date slider so that the user can see ...
user avatar
  • 1
0 votes
0 answers
11 views

How to predict on a different time scale to the feature set in ML models?

Is there a technique, or selection of techniques, for developing robust machine learning forecasts on a different timescale to the feature set using a directional classifier model? For example, say I ...
user avatar
  • 1
0 votes
0 answers
7 views

What exactly is the "currency" dimension in data profiling?

...
user avatar
  • 133
0 votes
0 answers
11 views

create a model to identify the skills priroty

I have created a dataset that has row information as [jobtitle, (skill1, skill1_score), (...), (skilln, skilln_score)]. To create the dataset, I have extracted job titles and job descriptions that ...
user avatar
  • 199
1 vote
1 answer
33 views

Probability Distribution on the Test Set

How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted? Here x axis ...
user avatar
  • 11
4 votes
1 answer
184 views

How to interpret standard deviation calculated by excel

I have some percentage values in my excel spreadsheet like: When i use the STDEV function to calculate the standard deviation with these values, i get ...
user avatar
0 votes
1 answer
10 views

Extracting Public Page Posts from Facebook

I am a data science student working on my capstone research project. I am trying to collect posts from a number of public organization pages from facebook. I am looking at the graphs api and it does ...
user avatar
  • 101
0 votes
0 answers
12 views

How to mathematically define the "singular points" in a graph

In the graph, we can see several "singular points" (red circle) which show a significant change. But I don't know how to mathematically describe such points and find them out. Here the "...
user avatar
0 votes
0 answers
8 views

Discover the geographical position of weather historical data using weather historical predictions

I have too little experience in neural networks and I wanted to solve a problem that I don't know how to raise. On the one side, I have a 1 year hourly historical data of the temperature in a specific ...
user avatar
1 vote
1 answer
17 views

Merging 2 datasets

In the given task, they've provides us with 2 datasets namely the test & train dataset. So, I was wondering if we could merge those 2 datasets into 1 data-frame and delete the duplicates. Would ...
user avatar
2 votes
0 answers
30 views

How Should I Create Waveform Datasets

Full disclosure I asked this on StackOverflow and it got taken down as it was more of a how do I do this, not how do I code this question: I am trying to simulate/fake data that I eventually will ...
user avatar

1
2 3 4 5
16