Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
-1
votes
1answer
49 views

Inputting Data for Machine Learning in PyTorch

I have scoured the internet and documentation, but have not yet been able to find a simple explanation of how to load data and/or simply type data into pytorch. I am just learning this library, and I ...
0
votes
1answer
82 views

How to input a 3d model into ML algorithm?

I have a machine learning model that uses csv with measured data about buildings: width, length, height etc. I use it to predict some features and it works properly. I would like to drop csv with ...
0
votes
3answers
101 views

what are effects of working with categorical dataset

I am working on classification problem where the dataset contains 90% of features as categorical. It is binary classification problem, and the class is heavily imbalanced. I performed Smote over ...
1
vote
1answer
49 views

Alternatives to Heat Maps for Communicating Daily Data

I am currently using the heat map below to communicate the number of messages sent on a messing platform over time. I don't think this is the most effective way to communicate the information as it's ...
0
votes
0answers
28 views

Temperature in Farenheit : Interval or ratio data

According to the Level of Measurement classification, there are four types of data: nominal, ordinal, interval and ratio. The temperature in Fahrenheit and Celsius is often classified as interval ...
1
vote
2answers
17 views

Different types of analysis which can be done with Webdata of users

I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are: Product Usage details Geographic details Session data: Capture what all they did ...
-1
votes
1answer
32 views

Extracting information from online job postings

Okay, so I'm trying to build a data set about data science job openings. I want to extract information about what kind of minimum education level is expected in each job posting and also how much ...
1
vote
1answer
46 views

Data science degree or Computer Science degree? [closed]

I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for ...
0
votes
0answers
34 views

reverse engineer console game FPS sensitivity for replication

Would it be possible to reverse engineer a FPS games sensitivity/dead zone/acceleration curve/ and other data by recording the games screen and running a controller through a pc to record joystick ...
0
votes
0answers
10 views

Filter a signal at a frequency, which is proportional to another signal

I want to filter a signal at a frequency, which is proportional to a frequency of another signal. This other signal is perfectly measureable and changes its frequency from time to time, so this filter ...
2
votes
1answer
36 views

Can a Logsitic Regression model continue making predictions after removing predictions from the data set?

I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then ...
2
votes
0answers
11k views

How to Process Large JSON Files with Python

I've been using Python for quite some time now, scripting command line tools, etc. I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files. ...
2
votes
2answers
44 views

Training vs test data set for supervised learning in real life scenario

In the tutorials, I have noticed only similar data has been used with models training and prediction. I was wondering how cases where you can't find training data that is similar to your final use ...
2
votes
1answer
58 views

Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

I'm looking at telecom customers data. Two of the variables I'm looking at currently are: Monthly Charges - The total amount charged to the customer monthly. Is Senior Citizen - Whether the customer ...
0
votes
0answers
9 views

Is it possible to map micro clusters in density based stream clustering into the grid in grid based stream clustering?

In hybrid micro-cluster and grid based stream clustering, the incoming data points are first map in the micro cluster and if it do not fit in any cluster then it is mapped to the grid. Upon reaching a ...
1
vote
0answers
28 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
1
vote
0answers
100 views

Best strategy to build Classifier for Mixed Data with class imbalance

I have a dataset which contains : 94 numeric features + 56 categorical features I am trying to build a classifier to predict Target (disease/healthy). 2. Rows : 1812 3. Class imbalance ( Majority ...
-1
votes
1answer
31 views

What are the tools I should know to start a career in Data Science? [closed]

Based on what I have seen on the internet, here is a list of things I should be knowing: Python SQL mySQL MS Excel Statistics Machine Learning Probability Calculus Linear Algebra R MongoDB Java Scala ...
0
votes
1answer
14 views

Data driven project for a self-catering accommodation business

I want to apologise in advance for my ignorance but I'm hitting my head in a wall here as I'm not sure how to proceed with my project and I've got no experience in that field even though I'm doing a ...
0
votes
1answer
91 views

VlookUp: how to loop it? [closed]

I am not an experienced user of Excel because I usually use python but I am obliged to use it this time. This is the big picture of my case. The problem is: I have a sheet with a column with some ...
1
vote
0answers
16 views

How much can be learned from a dataset?

Is there a way to quantify or characterize some upper limits for how much one can "learn" from a dataset? I got puzzled, when the neural state machine got me thinking components of AGI and lead me ...
0
votes
0answers
29 views

Working with a biased sample of the data

I have a project to build an automatic fraud detection system for an energy company. I have a huge amount of unlabeled data and only an small and biased labeled one. My labeled data is based on ...
0
votes
0answers
22 views

How do I export my data into .h5 format?

I am trying to make my data useful for to use in this model, i.e for Hierarchical Novelty Detection for Visual Object Recognition. I need to prepare my own dataset in a format like this: ...
1
vote
1answer
32 views

Read pandas data and convert it to IMMA

I'm working on a project in which my data set is in xarray and I need to write it to IMMA format. I converted it from xarray to pandas and am totally stumped on how to go from pandas to IMMA. Any ...
0
votes
2answers
31 views

Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?

I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data. Is there a way to include the new data without restarting ...
0
votes
0answers
37 views

How to test model accuracy on new vs. historical data?

I created an XG Boost model to predict churn using a dataset of customers who were sold during 2018. The accuracy of the model is 89%. Does it make more sense to re-pull the 2018 dataset, where more ...
0
votes
0answers
20 views

How to generate artificial datasets in MATLAB based on behavior of real datasets?

To understand: Let's consider a scenario. I have real datasets of 2 buildings showing different behaviors of occupants. Now to expand the analysis for multiple buildings, I want to generate simulated ...
0
votes
0answers
19 views

Power BI RLS through SQL stored procedure

I need dynamic RLS for Power BI - the way that the client has their data structured is that they have SQL RLS through a stored procedure. How can I translate that into power bi?
1
vote
1answer
416 views

How to feed data to Machine Learning Model?

I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression ...
0
votes
1answer
30 views

How to club the orders in such a way that maximum number of items are common amongst them?

Consider the following data set: The above table shows the quantity of each item used in the orders SO1 SO2 etc. I need to club the orders in such a way that maximum number of items are common amongst ...
0
votes
0answers
94 views

While reading the pages of a PDF file using Python, I get the following error. There are 300 pages in pdf file

CalledProcessError: Command '['java', '-Dfile.encoding=UTF8', '-jar', 'C:\Users\105051884\AppData\Local\Continuum\anaconda3\lib\site- packages\tabula\tabula-1.0....
0
votes
1answer
2k views

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

X = train_encoded_df.iloc[:, 1: ] y = train_encoded_df["Loan_Status"] print("Precision:",metrics.precision_score(y_test, y_pred)) My training data contains the ...
2
votes
3answers
637 views

Purpose of converting continuous data to categorical data

I was reading through a notebook tutorial working with the Titanic dataset, linked here, and noticed that they highly favored ordinal data to continuous data. For example, they converted both the Age ...
1
vote
0answers
63 views

preprocessing : Predicting with Multiple+Multivariate+Multitrend time series data

I am trying to predict the value of a variable in a multivariate time series; of which I have multiple time datasets (one system = one dataset containing 10 variables in time and average 120,000 rows) ...
1
vote
1answer
37 views

Twitter Dataset

I have found the following dataset, apparently it is the largest tweet dataset: https://www.kaggle.com/kazanova/sentiment140 However, I am looking for a dataset of tweets, with columns containing: ...
1
vote
0answers
16 views

Reducing the dimensions of data who's predominant categorical feature, its layer, has depths that overlaps with other samples layer values

I am working with a data set of soil types with multiple layers of varying depths and sizes with multiple features. There are $1-9$ layers each with differing dimensions, for example, a soil type ...
2
votes
5answers
81 views

30 Years Of Excel Test Data

I am a CS intern at an industrial company that has 30 years of excel files that need to be analyzed. Looking at the data, only a fraction of the files need to be looked at and used. After those files ...
0
votes
0answers
19 views

Feature vector of linear model

I read this paper that applies logistic regression to a dataset generated from a simulation they created. The dataset contains a set of binary vectors (called challenges) that looks like this: ...
-2
votes
2answers
42 views

What is Data Science and Machine Learning and what language mostly used to program? [closed]

I am newbie in Data Science, Machine Learning and any related to data science but I want to try it. Unfortunately, googling makes it tedious and complicated so I hope to be answered by anybody who's ...
0
votes
1answer
16 views

Machine learning model using keywords for binary decision

I have a some experience from Uni with convolutional NN and edge detection, but haven't much explored the other types of machine learning models. I was wondering if there might be one that is suited ...
14
votes
5answers
10k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
-2
votes
1answer
111 views

What is the difference between statistical learning and predictive analytics?

I am new to data science. I am confused about the differences between statistical learning and predictive analytics. To me, it looks like both are the same as we try to work on function of x(...
-3
votes
1answer
1k views

I have 11 years experience on .net, should i learn data science? [closed]

I have 11 years experience on .net, should i learn data science? I have 11 years experience on .net and getting paid rs 14L in india noida, Should my experience get consider in Data Science and get ...
1
vote
1answer
58 views

How to run KNN (or other) on nested features? Image metadata

I have looked everywhere and I can't find a straight solution. I have a set of metadata from images and their elements (name, heigh, width, position{top, right}). I ...
0
votes
1answer
2k views

Import data from google drive to Kaggle Kernel

I want to import a csv file from google drive . I tried using the link in add dataset tab but it is taking some thing else as "Open". Please see the image.
2
votes
1answer
54 views

Survival regression with major event that won't happen

I would like to do some survival regression about the duration before the "death" of an individual. The final purpose is to know, given an individual, how long it should take before he'll most likely "...
1
vote
1answer
25 views

Data duplication optimization

I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup...
3
votes
2answers
354 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...
1
vote
0answers
186 views

creating a dataframe out of nested loop data [closed]

I have three parameter arrays ,Parameter1 = [6,7,8],Parameter2 = [11,12] and Parameter3 which can be calculated using Parameter1 and Parameter2. There is a mapping between parameter 1 and 2. I want ...
3
votes
2answers
41 views

why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding

We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data. And I real a blog which give the difference between Label Encoding and One-Hot Encoding. So ...

1 2 3 4 5 10