Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
443
questions
2
votes
1answer
42 views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
0
votes
0answers
11 views
Free historical news data base?
I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
27 views
How to sort of “average” data without smoothing?
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem ...
2
votes
1answer
33 views
When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?
I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
0
votes
0answers
22 views
pd.qcut bins error!
Hi Data Science community!
I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error.
...
1
vote
1answer
16 views
Is there a way to aggregate frequency distribution for many runs?
In the following plot
I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
1
vote
1answer
14 views
Best file format for transfer of EHR data
I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
0
votes
0answers
20 views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
6
votes
2answers
126 views
Data anonymization in Python
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
0
votes
1answer
23 views
Oversampling/Undersampling only train set only or both train and validation set
I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
10 views
Most reliable source for Literacy data
I'm searching for the most reliable source for granular American literacy population data.
I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
1
vote
0answers
16 views
How can I measure data anonymity when present summary statistics
I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below:
...
0
votes
0answers
7 views
Reflections about types of bias in comparative analysis
I am working on an analysis on proposal rejections across two different periods in time and I was wondering what reflections I would have to make about the risk of bias.
The reason for this is ...
0
votes
2answers
54 views
Class Imbalance and Cost-Sensitive Learning XGBoost
I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is ...
1
vote
1answer
137 views
can I use z-score normalization even if it doesn't make sense for my data to be negative?
I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
0answers
12 views
Do I include a spatial features inside my training data for a SOM
I want to prepare my data for data mining, using a SOM to find data patterns. I have data about households in South Africa. The data is per individual per household. The census asked questions such as ...
0
votes
0answers
20 views
GPS Data Preprocessing Recommandations?
GPS Data Preprocessing
I got gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends ...
0
votes
0answers
17 views
Dataset creation software
Hello I am looking for a dataset for printed english characters but I am not able to find anything concrete. Is there any software which can generate multiple images from one image. I can build my ...
0
votes
1answer
12 views
Should I have “normal” sampled data in my dataset?
I am busy working on a project to find the reasons why kids in normal households are doing badly in school.
I have a dataset of which consists of kids that live in environments where the family is ...
1
vote
1answer
20 views
Computing Jaccard Similarity between two documents
Data Mining:
Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) =
D1 = the quick brown fox jumps over the lazy dog
D2 = jeff typed the quick brown dog jumps over the lazy fox ...
1
vote
2answers
39 views
Enormous dataset, how to proceed?
So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level ...
0
votes
1answer
37 views
SQL and JSON database - R
I have a database (PostgresSQL) which looks like this:
""results"":
{""WEBS"":
{""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}
I would ...
1
vote
2answers
44 views
What is the difference between a data-driven model and an empirical model?
Are they the same?
Empirical models, per Wikipedia, are
any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system ...
-1
votes
1answer
16 views
How to interact two variables in python?
I have about 8 features as my predictors in a logistic model I am trying to fit in python. One of the features is TotalAward (Financial Aid) and the second is NEED. I am attempting to predict the ...
1
vote
1answer
11 views
Growth function of a 6-dimensional linear classifier
In our course, we are dealing with a d-dimensional classification problem
($\chi = \mathbb{R}^{d}$ as our input space, and $y = \{-1,+1\}$). Our hypothesis class $H$ consists of all hypotheses of the ...
0
votes
0answers
17 views
real-valued data normalization for regression
We already know that non-real-valued fields of a data set to be modeled by neural network or some other machine learning approach must be transformed into real value. Some categorical or nominal type ...
1
vote
1answer
20 views
Meaningful and Non-Meaningful Data?
I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?
0
votes
0answers
15 views
Averaging over Irregular Intervals
Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation?
Specifically, suppose we are ...
2
votes
1answer
43 views
How to correct mislabeled data in dataset?
I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly ...
1
vote
1answer
27 views
1
vote
0answers
14 views
How to use SOFT dataset in Orange
I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
18 views
How has data generation evolved over the years (and centuries)?
I would like to know how data generation has evolved through the years.
My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...
0
votes
1answer
28 views
Data Visualisation Techniques for Multi Labelled data
I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: ...
1
vote
1answer
20 views
Where to find height dataset, or datasets in General
Hi there smart people,
I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest.
For example, I ...
1
vote
0answers
28 views
Clustering data by multiple values
I am trying to find a way to cluster/group students by their knowledge of different subjects.
Given following as an example:
...
1
vote
1answer
30 views
What to do about non-responses to demographic survey questions?
Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data?
In ...
1
vote
0answers
10 views
How to create trade flows-map chart?
I would like to know if there is a relatively easy way to create trade flows-map charts(preferably without coding), showing i.e. exports from Country A to Countries B,C,D. I have tried doing it so far ...
0
votes
0answers
20 views
Are application assignments in Datascience common practice?
this might seem off topic, but I hope this question will be tolerated.
I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And ...
0
votes
1answer
37 views
How to predict consumer purchase in next 6 months?
I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...
0
votes
0answers
19 views
Study resources for data manipulation interview questions
A colleague of mine is studying for tech roles, and they're asked to solve a consistent type of problem during the phone screenings: practicing manipulating data (sets, hash tables/dictionaries, ...
0
votes
2answers
68 views
Conditional Statement to update columns based on range
I have a dataframe called clients that has 5000+ rows with the following:
...
1
vote
0answers
14 views
Plotting ROC curve after re-sampling the data set
I am a beginner in Machine Learning. I am working on a project to compare the performance of three ML algorithms which are: LR, SVM and ANN on a dataset containing medical records.
After exploring ...
0
votes
0answers
32 views
Self Union All in Tableau Public for Sankey Diagram
I'm following this 2017 youtube tutorial from-which I hope to learn Sankey diagrams.
At 2:28 he instructs us to select Convert to Custom SQL, after having opened ...
7
votes
3answers
143 views
Docker for data science
I recently started to read articles about Docker.
To me, in data science, Docker is useful because:
1) You have a totally different environment, which protect you against libraries and dependencies ...
-1
votes
1answer
27 views
Inputting Data for Machine Learning in PyTorch
I have scoured the internet and documentation, but have not yet been able to find a simple explanation of how to load data and/or simply type data into pytorch.
I am just learning this library, and I ...
0
votes
1answer
42 views
How to input a 3d model into ML algorithm?
I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly.
I would like to drop csv with ...
0
votes
3answers
73 views
what are effects of working with categorical dataset
I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over ...
0
votes
0answers
21 views
Temperature in Farenheit : Interval or ratio data
According to the Level of Measurement classification, there are four types of data: nominal, ordinal, interval and ratio.
The temperature in Fahrenheit and Celsius is often classified as interval ...
0
votes
1answer
22 views
Algorithm for user profiling without distinct profiles
I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
2
votes
1answer
42 views
Data science degree or Computer Science degree? [closed]
I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for ...
