Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
452
questions
1
vote
1answer
22 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
0answers
10 views
Anonymizing the CSV data using Python Faker
To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations.
Is it possible to anonymize the ...
0
votes
0answers
12 views
Excel date-time auto fill error
I have data of 10 metrics for a 24 hour period which is in 5 minute intervals. I merged all the 10 data metrics into one file and converted it into a .xlsx file.
After opening the excel file I ...
0
votes
0answers
14 views
Difference between Big Data and a Data Warehouse?
I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
1answer
24 views
Stata: Trying to merge two data sets with common year observations
I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
1
vote
2answers
32 views
Clearest way to visualise temporal data?
I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
20 views
Transforming Data for Classification
I have a dataset of the type:
Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views
Space and time fourier transform of data
I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
13 views
Free API for historical weather US data?
I am trying to retrieve a free R-Python API that provides historical weather data in US.
In fact wunderground API is no longer available.
Any suggestion?
2
votes
1answer
47 views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
0
votes
0answers
11 views
Free historical news data base?
I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
31 views
How to sort of “average” data without smoothing?
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem ...
2
votes
1answer
37 views
When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?
I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
0
votes
0answers
30 views
pd.qcut bins error!
Hi Data Science community!
I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error.
...
1
vote
1answer
17 views
Is there a way to aggregate frequency distribution for many runs?
In the following plot
I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
1
vote
1answer
14 views
Best file format for transfer of EHR data
I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
0
votes
0answers
27 views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
6
votes
2answers
147 views
Data anonymization in Python
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
0
votes
2answers
32 views
Oversampling/Undersampling only train set only or both train and validation set
I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
10 views
Most reliable source for Literacy data
I'm searching for the most reliable source for granular American literacy population data.
I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
1
vote
0answers
17 views
How can I measure data anonymity when present summary statistics
I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below:
...
0
votes
0answers
8 views
Reflections about types of bias in comparative analysis
I am working on an analysis on proposal rejections across two different periods in time and I was wondering what reflections I would have to make about the risk of bias.
The reason for this is ...
0
votes
2answers
62 views
Class Imbalance and Cost-Sensitive Learning XGBoost
I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is ...
1
vote
1answer
137 views
can I use z-score normalization even if it doesn't make sense for my data to be negative?
I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
0answers
12 views
Do I include a spatial features inside my training data for a SOM
I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
0answers
20 views
GPS Data Preprocessing Recommandations?
GPS Data Preprocessing
I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
0answers
18 views
Dataset creation software
Hello I am looking for a dataset for printed english characters but I am not able to find anything concrete. Is there any software which can generate multiple images from one image. I can build my ...
0
votes
1answer
12 views
Should I have “normal” sampled data in my dataset?
I am busy working on a project to find the reasons why kids in normal households are doing badly in school.
I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
1answer
20 views
Computing Jaccard Similarity between two documents
Data Mining:
Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) =
D1 = the quick brown fox jumps over the lazy dog
D2 = jeff typed the quick brown dog jumps over the lazy fox ...
1
vote
2answers
39 views
Enormous dataset, how to proceed?
So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level ...
0
votes
1answer
37 views
SQL and JSON database - R
I have a database (PostgresSQL) which looks like this:
""results"":
{""WEBS"":
{""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}
I would ...
1
vote
2answers
55 views
What is the difference between a data-driven model and an empirical model?
Are they the same?
Empirical models, per Wikipedia, are
any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system ...
-1
votes
1answer
17 views
How to interact two variables in python?
I have about 8 features as my predictors in a logistic model I am trying to fit in python. One of the features is TotalAward (Financial Aid) and the second is NEED. I am attempting to predict the ...
1
vote
1answer
14 views
Growth function of a 6-dimensional linear classifier
In our course, we are dealing with a d-dimensional classification problem
($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the ...
0
votes
0answers
17 views
real-valued data normalization for regression
We already know that non-real-valued fields of a data set to be modeled by neural network or some other machine learning approach must be transformed into real value. Some categorical or nominal type ...
1
vote
1answer
21 views
Meaningful and Non-Meaningful Data?
I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?
0
votes
0answers
15 views
Averaging over Irregular Intervals
Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation?
Specifically, suppose we are ...
2
votes
1answer
46 views
How to correct mislabeled data in dataset?
I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly ...
1
vote
1answer
27 views
1
vote
0answers
14 views
How to use SOFT dataset in Orange
I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
20 views
How has data generation evolved over the years (and centuries)?
I would like to know how data generation has evolved through the years.
My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...
0
votes
1answer
32 views
Data Visualisation Techniques for Multi Labelled data
I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: ...
1
vote
1answer
21 views
Where to find height dataset, or datasets in General
Hi there smart people,
I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest.
For example, I ...
1
vote
0answers
28 views
Clustering data by multiple values
I am trying to find a way to cluster/group students by their knowledge of different subjects.
Given following as an example:
...
1
vote
1answer
30 views
What to do about non-responses to demographic survey questions?
Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data?
In ...
1
vote
0answers
10 views
How to create trade flows-map chart?
I would like to know if there is a relatively easy way to create trade flows-map charts(preferably without coding), showing i.e. exports from Country A to Countries B,C,D. I have tried doing it so far ...
0
votes
0answers
21 views
Are application assignments in Datascience common practice?
this might seem off topic, but I hope this question will be tolerated.
I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And ...
0
votes
1answer
41 views
How to predict consumer purchase in next 6 months?
I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...
0
votes
0answers
21 views
Study resources for data manipulation interview questions
A colleague of mine is studying for tech roles, and they're asked to solve a consistent type of problem during the phone screenings: practicing manipulating data (sets, hash tables/dictionaries, ...
0
votes
2answers
75 views
Conditional Statement to update columns based on range
I have a dataframe called clients that has 5000+ rows with the following:
...
