Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
421
questions
0
votes
0answers
13 views
Averaging over Irregular Intervals
Given a sample of data where the samples are taken over irregular intervals, what is the most sensible way of going about calculating the mean and Standard deviation?
Specifically, suppose we are ...
2
votes
1answer
31 views
How to correct mislabeled data in dataset?
I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly ...
1
vote
1answer
24 views
-2
votes
0answers
14 views
time series data modeling for deep learning [closed]
what is the best format to feed the input data, which are time series with varying density over time, to a deep learning network, while at any iteration we want to feed a batch of data including a ...
-2
votes
0answers
16 views
Kaggle Lung Cancer Dataset MISSING (Data Science Bowl 2017)
For a couple of days now, I have been looking for the Kaggle Lung Cancer Dataset (Data Science Bowl 2017) containing 2D, 3D, and labels of CT scans for lung cancer, but I did not succeed yet. It is ...
1
vote
0answers
11 views
How to use SOFT dataset in Orange
I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
17 views
How has data generation evolved over the years (and centuries)?
I would like to know how data generation has evolved through the years.
My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...
0
votes
1answer
23 views
Data Visualisation Techniques for Multi Labelled data
I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: ...
1
vote
1answer
17 views
Where to find height dataset, or datasets in General
Hi there smart people,
I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest.
For example, I ...
0
votes
0answers
19 views
Clustering data by multiple values
I am trying to find a way to cluster/group students by their knowledge of different subjects.
Given following as an example:
...
1
vote
1answer
27 views
What to do about non-responses to demographic survey questions?
Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data?
In ...
1
vote
0answers
8 views
How to create trade flows-map chart?
I would like to know if there is a relatively easy way to create trade flows-map charts(preferably without coding), showing i.e. exports from Country A to Countries B,C,D. I have tried doing it so far ...
0
votes
0answers
17 views
Are application assignments in Datascience common practice?
this might seem off topic, but I hope this question will be tolerated.
I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And ...
0
votes
1answer
23 views
How to predict consumer purchase in next 6 months?
I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...
0
votes
0answers
10 views
Study resources for data manipulation interview questions
A colleague of mine is studying for tech roles, and they're asked to solve a consistent type of problem during the phone screenings: practicing manipulating data (sets, hash tables/dictionaries, ...
0
votes
2answers
40 views
Conditional Statement to update columns based on range
I have a dataframe called clients that has 5000+ rows with the following:
...
1
vote
0answers
13 views
Plotting ROC curve after re-sampling the data set
I am a beginner in Machine Learning. I am working on a project to compare the performance of three ML algorithms which are: LR, SVM and ANN on a dataset containing medical records.
After exploring ...
0
votes
0answers
13 views
Neural Network : possibility to get simple relationships from first layer?
Working on a data set with a lot of similar variables (for exemple : spending in something, spending in something else...), and suspected additive relationships, I was wondering about their agregation....
0
votes
0answers
19 views
Self Union All in Tableau Public for Sankey Diagram
I'm following this 2017 youtube tutorial from-which I hope to learn Sankey diagrams.
At 2:28 he instructs us to select Convert to Custom SQL, after having opened ...
0
votes
0answers
2 views
TPOT freezes while optimizing in Ubuntu 18.04 [migrated]
I have started working with tpot with dask. But the optimization always freezes at 0%. I have tried almost all the solutions suggested. But still no luck. Here is my code -
...
7
votes
3answers
135 views
Docker for data science
I recently started to read articles about Docker.
To me, in data science, Docker is useful because:
1) You have a totally different environment, which protect you against libraries and dependencies ...
0
votes
0answers
7 views
KeyError in python [migrated]
I am trying to count the number of times each string in my list is repeated but it keeps giving me KeyError:
...
-1
votes
1answer
26 views
Inputting Data for Machine Learning in PyTorch
I have scoured the internet and documentation, but have not yet been able to find a simple explanation of how to load data and/or simply type data into pytorch.
I am just learning this library, and I ...
0
votes
1answer
39 views
How to input a 3d model into ML algorithm?
I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly.
I would like to drop csv with ...
0
votes
3answers
63 views
what are effects of working with categorical dataset
I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over ...
0
votes
0answers
21 views
Temperature in Farenheit : Interval or ratio data
According to the Level of Measurement classification, there are four types of data: nominal, ordinal, interval and ratio.
The temperature in Fahrenheit and Celsius is often classified as interval ...
0
votes
1answer
19 views
Algorithm for user profiling without distinct profiles
I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
2
votes
1answer
39 views
Data science degree or Computer Science degree? [closed]
I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for ...
0
votes
0answers
26 views
reverse engineer console game FPS sensitivity for replication
Would it be possible to reverse engineer a FPS games sensitivity/dead zone/acceleration curve/ and other data by recording the games screen and running a controller through a pc to record joystick ...
0
votes
0answers
10 views
Filter a signal at a frequency, which is proportional to another signal
I want to filter a signal at a frequency, which is proportional to a frequency of another signal. This other signal is perfectly measureable and changes its frequency from time to time, so this filter ...
2
votes
1answer
29 views
Can a Logsitic Regression model continue making predictions after removing predictions from the data set?
I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then ...
1
vote
1answer
19 views
Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?
I'm looking at telecom customers data. Two of the variables I'm looking at currently are:
Monthly Charges - The total amount charged to the customer monthly.
Is Senior Citizen - Whether the customer ...
0
votes
0answers
8 views
Is it possible to map micro clusters in density based stream clustering into the grid in grid based stream clustering?
In hybrid micro-cluster and grid based stream clustering, the incoming data points are first map in the micro cluster and if it do not fit in any cluster then it is mapped to the grid. Upon reaching a ...
2
votes
2answers
42 views
Training vs test data set for supervised learning in real life scenario
In the tutorials, I have noticed only similar data has been used with models training and prediction.
I was wondering how cases where you can't find training data that is similar to your final use ...
1
vote
0answers
13 views
Python is reading my data with NANS and Infs, but they don't have any
I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
0answers
43 views
Best strategy to build Classifier for Mixed Data with class imbalance
I have a dataset which contains : 94 numeric features + 56 categorical features
I am trying to build a classifier to predict Target (disease/healthy).
2. Rows : 1812
3. Class imbalance ( Majority ...
-1
votes
1answer
27 views
What are the tools I should know to start a career in Data Science? [closed]
Based on what I have seen on the internet, here is a list of things I should be knowing:
Python
SQL
mySQL
MS Excel
Statistics
Machine Learning
Probability
Calculus
Linear Algebra
R
MongoDB
Java
Scala
...
1
vote
1answer
23 views
Is it possible retrive hourly data from daily weather data?
I'm a beginner, so sorry if my question could be basic. Reading on internet I've found example written in python that makes the reverse of my questions (covert from daily to monthly).
My problem is ...
0
votes
1answer
14 views
Data driven project for a self-catering accommodation business
I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
15 views
How much can be learned from a dataset?
Is there a way to quantify or characterize some upper limits for how much one can "learn" from a dataset?
I got puzzled, when the neural state machine got me thinking components of AGI and lead me ...
0
votes
1answer
24 views
VlookUp: how to loop it? [closed]
I am not an experienced user of Excel because I usually use python but I am obliged to use it this time.
This is the big picture of my case.
The problem is:
I have a sheet with a column with some ...
0
votes
0answers
26 views
Working with a biased sample of the data
I have a project to build an automatic fraud detection system for an energy company. I have a huge amount of unlabeled data and only an small and biased labeled one. My labeled data is based on ...
0
votes
0answers
20 views
How do I export my data into .h5 format?
I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition.
I need to prepare my own dataset in a format like this:
...
0
votes
2answers
18 views
Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?
I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data.
Is there a way to include the new data without restarting ...
0
votes
0answers
31 views
How to test model accuracy on new vs. historical data?
I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more ...
0
votes
2answers
32 views
1
vote
1answer
23 views
Technical term for using regular expressions to classify text?
Background
I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext.
He wants to classify the claims as either having been "approved", "denied", ...
-1
votes
1answer
24 views
Extracting information from online job postings
Okay, so I'm trying to build a data set about data science job openings.
I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
0
votes
0answers
14 views
How to generate artificial datasets in MATLAB based on behavior of real datasets?
To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
0
votes
0answers
13 views
Power BI RLS through SQL stored procedure
I need dynamic RLS for Power BI - the way that the client has their data structured is that they have SQL RLS through a stored procedure. How can I translate that into power bi?
