Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
1
vote
0answers
17 views

How to use SOFT dataset in Orange

I have a dataset which I obtained from NCBI, but it is a SOFT file. When I upload the SOFT file to the file widget in Orange, it cannot be uploaded. Is there a way to upload SOFT files in Orange?
1
vote
1answer
20 views

How has data generation evolved over the years (and centuries)?

I would like to know how data generation has evolved through the years. My end goal would be to generate a logarithmic line plot showing the ridiculous increase in data generation. Please, does ...
0
votes
1answer
40 views

Data Visualisation Techniques for Multi Labelled data

I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: ...
1
vote
1answer
24 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
1
vote
0answers
30 views

Clustering data by multiple values

I am trying to find a way to cluster/group students by their knowledge of different subjects. Given following as an example: ...
1
vote
1answer
30 views

What to do about non-responses to demographic survey questions?

Can anyone point me to some think pieces on what to do with census type data where at least some of the people surveyed do not self-identify a race/ethnicity, gender, or other demographic data? In ...
1
vote
0answers
10 views

How to create trade flows-map chart?

I would like to know if there is a relatively easy way to create trade flows-map charts(preferably without coding), showing i.e. exports from Country A to Countries B,C,D. I have tried doing it so far ...
0
votes
0answers
21 views

Are application assignments in Datascience common practice?

this might seem off topic, but I hope this question will be tolerated. I was recently asked to help a friend with an assignment in the process of applying for a new job as data scientist. And ...
0
votes
1answer
58 views

How to predict consumer purchase in next 6 months?

I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...
0
votes
0answers
22 views

Study resources for data manipulation interview questions

A colleague of mine is studying for tech roles, and they're asked to solve a consistent type of problem during the phone screenings: practicing manipulating data (sets, hash tables/dictionaries, ...
0
votes
2answers
96 views

Conditional Statement to update columns based on range

I have a dataframe called clients that has 5000+ rows with the following: ...
1
vote
0answers
15 views

Plotting ROC curve after re-sampling the data set

I am a beginner in Machine Learning. I am working on a project to compare the performance of three ML algorithms which are: LR, SVM and ANN on a dataset containing medical records. After exploring ...
0
votes
0answers
39 views

Self Union All in Tableau Public for Sankey Diagram

I'm following this 2017 youtube tutorial from-which I hope to learn Sankey diagrams. At 2:28 he instructs us to select Convert to Custom SQL, after having opened ...
7
votes
3answers
145 views

Docker for data science

I recently started to read articles about Docker. To me, in data science, Docker is useful because: 1) You have a totally different environment, which protect you against libraries and dependencies ...
-1
votes
1answer
35 views

Inputting Data for Machine Learning in PyTorch

I have scoured the internet and documentation, but have not yet been able to find a simple explanation of how to load data and/or simply type data into pytorch. I am just learning this library, and I ...
0
votes
1answer
45 views

How to input a 3d model into ML algorithm?

I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly. I would like to drop csv with ...
0
votes
3answers
79 views

what are effects of working with categorical dataset

I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over ...
0
votes
0answers
22 views

Temperature in Farenheit : Interval or ratio data

According to the Level of Measurement classification, there are four types of data: nominal, ordinal, interval and ratio. The temperature in Fahrenheit and Celsius is often classified as interval ...
0
votes
1answer
22 views

Algorithm for user profiling without distinct profiles

I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
2
votes
1answer
42 views

Data science degree or Computer Science degree? [closed]

I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for ...
0
votes
0answers
33 views

reverse engineer console game FPS sensitivity for replication

Would it be possible to reverse engineer a FPS games sensitivity/dead zone/acceleration curve/ and other data by recording the games screen and running a controller through a pc to record joystick ...
0
votes
0answers
10 views

Filter a signal at a frequency, which is proportional to another signal

I want to filter a signal at a frequency, which is proportional to a frequency of another signal. This other signal is perfectly measureable and changes its frequency from time to time, so this filter ...
2
votes
1answer
30 views

Can a Logsitic Regression model continue making predictions after removing predictions from the data set?

I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then ...
1
vote
1answer
21 views

Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

I'm looking at telecom customers data. Two of the variables I'm looking at currently are: Monthly Charges - The total amount charged to the customer monthly. Is Senior Citizen - Whether the customer ...
0
votes
0answers
8 views

Is it possible to map micro clusters in density based stream clustering into the grid in grid based stream clustering?

In hybrid micro-cluster and grid based stream clustering, the incoming data points are first map in the micro cluster and if it do not fit in any cluster then it is mapped to the grid. Upon reaching a ...
2
votes
2answers
43 views

Training vs test data set for supervised learning in real life scenario

In the tutorials, I have noticed only similar data has been used with models training and prediction. I was wondering how cases where you can't find training data that is similar to your final use ...
1
vote
0answers
21 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
0answers
71 views

Best strategy to build Classifier for Mixed Data with class imbalance

I have a dataset which contains : 94 numeric features + 56 categorical features I am trying to build a classifier to predict Target (disease/healthy). 2. Rows : 1812 3. Class imbalance ( Majority ...
-1
votes
1answer
29 views

What are the tools I should know to start a career in Data Science? [closed]

Based on what I have seen on the internet, here is a list of things I should be knowing: Python SQL mySQL MS Excel Statistics Machine Learning Probability Calculus Linear Algebra R MongoDB Java Scala ...
1
vote
1answer
27 views

Is it possible retrive hourly data from daily weather data?

I'm a beginner, so sorry if my question could be basic. Reading on internet I've found example written in python that makes the reverse of my questions (covert from daily to monthly). My problem is ...
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
1
vote
0answers
15 views

How much can be learned from a dataset?

Is there a way to quantify or characterize some upper limits for how much one can "learn" from a dataset? I got puzzled, when the neural state machine got me thinking components of AGI and lead me ...
0
votes
1answer
38 views

VlookUp: how to loop it? [closed]

I am not an experienced user of Excel because I usually use python but I am obliged to use it this time. This is the big picture of my case. The problem is: I have a sheet with a column with some ...
0
votes
0answers
28 views

Working with a biased sample of the data

I have a project to build an automatic fraud detection system for an energy company. I have a huge amount of unlabeled data and only an small and biased labeled one. My labeled data is based on ...
0
votes
0answers
21 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
0
votes
2answers
28 views

Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?

I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data. Is there a way to include the new data without restarting ...
0
votes
0answers
33 views

How to test model accuracy on new vs. historical data?

I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more ...
0
votes
2answers
37 views

Suggestion of a model for these type of data?

I've got a data set that looks like this ...
1
vote
1answer
35 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
-1
votes
1answer
28 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
0
votes
0answers
17 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
0
votes
0answers
15 views

Power BI RLS through SQL stored procedure

I need dynamic RLS for Power BI - the way that the client has their data structured is that they have SQL RLS through a stored procedure. How can I translate that into power bi?
0
votes
0answers
83 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
2
votes
1answer
217 views

Smart data split (train/eval) for Object Detection

I am looking for a smart way of splitting object detection data (images with labelled objects inside them) while taking into account the distribution of the objects themselves and not just the images. ...
0
votes
1answer
853 views

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

X = train_encoded_df.iloc[:, 1: ] y = train_encoded_df["Loan_Status"] print("Precision:",metrics.precision_score(y_test, y_pred)) My training data contains the ...
2
votes
3answers
260 views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
1
vote
0answers
54 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
1
vote
1answer
29 views

Twitter Dataset

I have found the following dataset, apparently it is the largest tweet dataset: https://www.kaggle.com/kazanova/sentiment140 However, I am looking for a dataset of tweets, with columns containing: ...
1
vote
0answers
16 views

Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values

I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...