Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
587
questions
1
vote
1answer
160 views
Importing Excel format data into R/R Studio and using glmnet package?
I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
2answers
37 views
Problem with converting string to dummy variables
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
1
vote
0answers
12 views
Is window based sequencing a good idea to obtain more training data for LSTMs?
I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier.
However,...
1
vote
2answers
46 views
How to make a dataframe with lists or vectors as its elements
This is something I have been wondering for ages but I am never able to get an answer.
I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector ...
0
votes
1answer
42 views
How to build a database of image data for machine learning?
I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database ...
2
votes
3answers
35 views
Combining Two CSV's in Jupyter Notebook
I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
1
vote
2answers
3k views
What algorithms are good to predict next numbers?
Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are ...
0
votes
1answer
105 views
How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?
Basically I am trying a use Multi Linear Regression Model to predict the salaries of employees. I have a total of 88 dependent feature from which 19 are categorical and the rest are continuous. I have ...
2
votes
2answers
21 views
Log analysis dataset with labeled cybersecurity issues
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
0answers
20 views
How start data science? [closed]
emphasized textHow start data science?
what tool i use and What are the requirements/I want to learn with myself \how to get started
What channels should I follow in order to learn?
0
votes
2answers
358 views
0
votes
1answer
12 views
Automating the training and deployment of the model as more data becomes available
My question is about automating the training of the model as more data becomes available.
In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ...
-1
votes
1answer
14 views
Changing of non-zero values to 1 [closed]
I want to change all non-zero values to 1 as an integer. However, I cannot change the non-zero values. Could you help me to do this?
1
vote
2answers
540 views
What is the difference between a data-driven model and an empirical model?
Are they the same?
Empirical models, per Wikipedia, are
any kind of (computer) modelling based on empirical observations rather than on mathematically describable relationships of the system ...
0
votes
1answer
105 views
How to obtain and load a good initial data set for object localization?
I'm looking for a good data set for training a CNN based network to do object localization (i.e. a data set with class labels and bounding box data).
What is a good initial data set to use?
How can ...
0
votes
1answer
30 views
Standardizing features by one specific feature
I am working on a project with a dataset that looks something like the following:
...
2
votes
1answer
32 views
How to account for reduced student capacity when calculating program retention?
I work at a nonprofit youth center that has 2 distinctive programs throughout the year. Our afterschool program runs during the school year, and our summer camp program runs during the summer.
I am ...
2
votes
1answer
33 views
What's the Seminal Book on data science? [closed]
Can people provide the best textbook they have used for learning data science? I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a ...
7
votes
2answers
80 views
Model for Differing Number of Rows per Observation
Looking to build a response model (click or no click) on marketing data which displays varying number of offers to a person. I don't want to model which offer they click but do they click any of the ...
1
vote
2answers
43 views
How to define tidy data if there is repeated measures?
Some time ago I read R for Data Sciece and there is the following definition of tidy data:
There are three interrelated rules which make a dataset tidy:
Each variable must have its own column.
Each ...
1
vote
1answer
37 views
Labelling a Time series dataset
I was asked this during an interview that I did not crack.
You are given data of total (aggregated) power consumption, and the TV power consumption of three households. Each household has data for one ...
0
votes
1answer
63 views
Machine Learning: Predicting target based on a feature
I have a df looks as follow:
-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times....
0
votes
1answer
25 views
What is the best way to visualize range of data?
If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
...
0
votes
0answers
15 views
HR employee attrition modeling - making a balanced sample question
I have dataset 1 (stayers) consisting of 1500 record of HR data demographic data of employees (11 features) who currently are in the company. Dataset 2 (leavers) contains 180 records -same features- ...
0
votes
1answer
35 views
Algorithm for user profiling without distinct profiles
I am trying to design an algorithm that takes in a new user with the variables department, location, job_role etc. and I want a machine-learning algorithm to decide ...
1
vote
4answers
31 views
Train a model using multiple data sources
I have to train a classification model to predict if a customer will buy a product or not. I have multiple (eg. 3 or 4) data sources. The variable distributions among the different data sources is ...
0
votes
0answers
8 views
Link Recommendation System
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
-1
votes
1answer
30 views
Equivalence of two separate data distribution
I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
0
votes
1answer
13 views
Compare two tends with big difference in absolute value
I'm studying the Spotify streams trend for different nations in order to show the impact of the pandemic on music listeners.
So far i retrieved my data, and plotted it.
Obviously, since the various ...
0
votes
0answers
9 views
Data Generation for LSTM with variable time input
So, I have a dataset that observes entities for 60 time periods. At each time period, I collect some X observations per entity.
So I have an LSTM to predict some element in the 60th time period using ...
1
vote
0answers
13 views
Triplet optimization producing a weird diagonal line?
I'm pretty sure this is the right forum for this, or let me know otherwise, I'll happily move this to a better place.
I have a strange problem. I've written an algorithm designed to take three files ...
0
votes
1answer
87 views
Convert categorical data in numeric preserve euclidean distance
I m looking how to preserve Euclidean distance with categorical attribute.
Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "...
0
votes
2answers
941 views
Connecting Infusionsoft data to Google data studio
I want to create a Google Data studio dashboard from Infusion soft data.
The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like ...
1
vote
1answer
20 views
Column vector for training example
I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a ...
0
votes
1answer
16 views
Tagged dataset with photos for race detection
Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as ...
0
votes
1answer
55 views
Append and replacing data in r
I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs.
Here is my data ...
2
votes
1answer
54 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
3
votes
1answer
41 views
How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?
I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
2
votes
0answers
64 views
Is data science only for substantive specialist?
Right now there is a lot of hype for data science. Everyone is learning it. There are lots of courses and boot camps being offered. Unlike software engineering, however, are data scientists required ...
1
vote
0answers
22 views
Generating artificial data to extend learning set
I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size small, I wish to extend it. I am kind of ...
2
votes
1answer
65 views
Organizing complex analysis project
Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric ...
0
votes
2answers
156 views
how to make moving moving graphs that vary in time like a video? [closed]
like this, like shown in this video.
7
votes
1answer
10k views
How to generate training data for OCR
I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
0
votes
0answers
47 views
Machine learning applications in Architecture field
Can there be Machine learning applications in Architecture field?
Example : New York City 100 years back i.e. Year 1920, every ten years after 1920 and today i.e. Year 2020 images of New York City ...
0
votes
1answer
27 views
How should I choose n_features in FeatureHasher in sklearn?
How should I choose n_features in FeatureHasher in sklearn ? assume that I have 1000 categories in feature "case" and I want to hash them
5
votes
4answers
337 views
General way to reduce features
Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run.
What are some of the best ways to find the most influential columns/...
5
votes
4answers
3k views
Missing Values in Data [duplicate]
I have experienced that most of the datasets contain missing values, which make our task bit challenging.
Please let me know how to fill up those missing values in an efficient way? and is there any ...
0
votes
0answers
32 views
How to plot three categorical variables Python/Pandas?
I am exploring data visualization with the Titanic data set from Kaggle and would like to understand the survival based on the family size of the passengers. In short, I would like to be able to plot ...
0
votes
1answer
27 views
Call routing using AI
I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can ...
0
votes
1answer
12 views
Ranking tumors from most malignant to most benign
I have a classification problem classifying tumors as benign or malignant. However, I want to go a step further to provide a ranking of these tumors as most malignant to most benign. Are there any ...
