Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
673
questions
0
votes
0answers
18 views
How to interpret 3 outputs of Precision?
Say im running a classification Machine learning algorithm,
of 2 classes 0 & 1.
A 0 label is detecting a visitor/row did not Convert. while 1 label is detecting a visitor/row did Convert.
When the ...
0
votes
0answers
74 views
Why am I getting a different answer in Principal Component Analysis dimensional reduction?
Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8).
Compute the principal component using PCA Algorithm.
Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
18 views
What are some examples of ordered data?
Tan,Steinbech,Kumar's book says
For some types of data, attributes have relationships that involve order in time or space.
I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
7 views
How to clean/analyze relative time series data
I'm relatively new to data science, and I decided to stretch a bit and try a project with time series data. I downloaded the Human Activity Recognition from Continuous Ambient Sensor Data Data Set, ...
0
votes
0answers
9 views
Dictionary of life sciences or medical terminologies
I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
1
vote
0answers
14 views
Clustering dataset with and without estimating means (no EM algorithm)
Given a dataset $D$ of the form
$$
D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n)
$$
sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
0
votes
1answer
12 views
How do you normalise the train+validation sets together?
This question is somewhat related to: Is it correct to join training and validation set before inferring on test-set?
As far as I understand, normalisation in general is done in the following way:
...
0
votes
1answer
37 views
Is it a good idea to use parallel coordinates for visualising outliers? [closed]
I tried using parallel coordinates to visualize outliers. Is it fundamentally correct?
-1
votes
0answers
23 views
Predicting Estimated time of arrival (ETA) [closed]
My background is in Data Analysis and Engineering.
I am a newbie in ML and want to do a side project in the logistic company I work for.
I chose to work on an ETA of packages delivered to customers.
...
0
votes
0answers
25 views
Training data for image digit recognition
I am creating a digit image recognition algorithm. For this I would like to collect sample data to train the model. Does python have any inbuilt libraries for this? Where could I get good training ...
0
votes
0answers
10 views
Most Hot Freelance Data Science Project Niches
I almost finish my final year in my information system major for computer science's bachelor degree, i also take big data and machine learning as my specialization. And then while i finding and apply ...
0
votes
2answers
24 views
Assign value of existing column to new columns in pandas based on conditions [closed]
I am trying to create a new column in pandas data frame based on existing columns.
column1
column2
column3
y1
y2
y3
100
200
300
2020
2021
2022
100
200
300
2021
2022
2023
100
200
300
2019
2020
2021
...
0
votes
0answers
18 views
How to do exploratory data analysis on a database
In all courses, the focus is mainly on building models from an already clean dataset that is in a csv file.
If I have a database, Postgres, with a table that has ~10 million rows, what is the best way ...
0
votes
3answers
47 views
Convert .data file to .csv
I'm using a data called 'adults.data', I need to work with that data as a '.csv' file. Can anyone guide me on how can I change the format?
I tried opening the file in excel and then save it as csv, ...
1
vote
1answer
10 views
How to deal with mismatch of timesteps in target variable and features in forecasting problem?
Background Info:
I am working with some climate data where I want to predict crop yields with my dataset containing climate- and satellite-derived features.
This is a time series regression ...
2
votes
1answer
28 views
0
votes
0answers
10 views
Why do images in Open CV template matching get matched to the templates more than once?
The following code reads the set of input tiff files under tiffile and template tiff files under files and performs template matching on them. After the image gets matched, opencv contour extracts the ...
2
votes
1answer
29 views
Visualization suggestion:-
I have a data frame like below:-
Here I have 194 countries and the columns are fan_out values which is in percent of the total population.
Like for country AD, the total fan_out value is 2.24 -06 % of ...
2
votes
2answers
26 views
How to input a list into my model and not have it care about order
I'm trying to predict a list of numbers, e.g:
[23,55,198,200,64]
The data I have includes multiple things, along with:
The numbers from the previous run (These ...
0
votes
0answers
16 views
When training a cGAN on a X (B/W image) to Y (RGB image) “paired” dataset, how much will slight differences in the images matter in terms of results?
Right now I am working on a colorization project with GANs, and had originally settled on using a CycleGAN because I considered structuring my dataset in an unpaired manner. I've since been able to ...
0
votes
1answer
48 views
Does the abstraction of a class affects the performance of neural networks?
For example, if I have 3 audio classes including
Ambulance Siren
Police Car Siren
Firetruck Siren
assuming these 3 classes could be distinguished by humans. If I just want the model to classify all ...
0
votes
0answers
8 views
how to prevent feedback loop while you can't use another data source
we have a harassment prediction model which trained and deployed many years ago. each day this model predicts some conversation as harassment and annotators label these predictions. (they only label ...
0
votes
0answers
26 views
VGG16 (pytorch) training issues using a very large dataset vs a smaller dataset
I've constructed a simple VGG16 layer model from the original Simonyan & Zisserman paper for use on a DBT (Digital Breast Tomo.) data challenge. As a starter model, I chose to make the 3x244x244 ...
0
votes
0answers
11 views
How would I perform a regression of this nature?
I have two dataframes. One is a dataframe containing the injury history of all players before 2013, the rows are the names of each player, and the columns in this dataframe are upper leg, lower leg, ...
0
votes
0answers
9 views
Identify the top clients
I am asking the metric to identify the top clients for the ecommerce data.
Usually, is the metric based on revenue, or average order value, or average spending?
Also, how can we take top 10, 20, or ...
0
votes
0answers
33 views
How to adjust the unadjusted charts in Pandas
I would like to adjust the unadjusted charts in Pandas. Suppose I have a data frame of the price of a stock symbol that there is a price gap between two candlesticks due to a capital increase. Sites ...
0
votes
0answers
7 views
Will training with yolov4 backup weights early cause weak prediction or is a true backup?
I am training a dataset in yolov4 using the repo from AlexeyAB darknet.
In his repo, backup weights are created every so iterations but you originally train with a pretrained weights file.
I was ...
1
vote
0answers
15 views
Data Pipeline Best Practices
I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
0
votes
1answer
12 views
European soccer championship: Where can I find historic data on the market value of the teams
I want to write a simple model predicting the performance of the teams in the european soccer championship. The results of soccer matches are known to be strongly correlated with the market value of ...
1
vote
0answers
26 views
How to compare two one hot encoded data frames based on column names?
I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
2
votes
1answer
43 views
Highly correlated features
What are the conditions to remove highly correlated features in a correlation map?
Given the correlation map below, is it OK to remove diagnosis feature and should we remove highly correlated features ...
0
votes
0answers
9 views
Data Warehouse design schema for sales (items, shopping carts, ..)
I have to design some OLAP-Cubes based on an OLTP database for the sales.
At first, that sounds very basic and common but I struggle with the item <-> shopping cart thing. One cart can contain ...
2
votes
2answers
55 views
How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?
I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...
0
votes
0answers
15 views
What is a good way to extract blocks of texts from documents in various formats?
I have lots of documentation in pdf, ppt, xls files, which contain tables with text, pictures, headlines etc. The goal is to extract blocks of texts where the information is continuous, so to say.
The ...
0
votes
0answers
16 views
How to encode high dimensional dynamic categorical data?
Example: Facebook newsfeed ranking. This is typically done by relevance scoring each post using a regression model based on a number of features.
Typical features might include information from the ...
0
votes
0answers
10 views
Bidding strategy without some information (vertical search)
I am facing the following problem which I wish to do some classification. Suppose when a customer search for car insurance, the features one enters are
Currently insured or not (Yes, No, Unknown)
...
1
vote
1answer
16 views
Using ggplot2 to create a bar chart [closed]
So I'm trying to create a simple bar chart of Survive vs Not Survive for the common Titanic data set in R. I keep getting just the number of No's and Yes's, and not the frequencies or counts ...
1
vote
0answers
20 views
Working with others: Tidy data vs Pretty data [closed]
When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data?
There are notes of what I want ...
1
vote
0answers
12 views
creating dataset/storing vectors from c++ to python
I am new to the data science field.
I need to create dataset which consists of n inputs (vectors of size 5) and n outputs (vectors of size > 10,000). These vectors are generated using c++ code.
...
1
vote
1answer
31 views
Merge two datasets based on date ranges. R
My goal is to merge two datasets using date ranges. Dataset1 contains patients stays in a hospital overtime. Dataset2 contains room information overtime. My goal is to identify what type of room the ...
0
votes
0answers
23 views
StratifiedShuffleSplit
I deal with a cars data, but the data is non representative, there is a huge number of some cars and small number of other car as follows:
]1
I am using ...
0
votes
0answers
13 views
Grape dataset for smart viticulture
I'm involved in a smart viticulture project. I need some datasets for machine learning models. Datasets about grape disease or ripeness related to climate variables or grape.
0
votes
0answers
11 views
Finding the wiggles pattern in the original dataset. (Wiggles appear after performing division by another dataset)
I have multiple measurements regarding scientific observations. The problem is that there is a subtle noise pattern caused by the instrument - the wiggles. These wiggles are invisible when looking at ...
0
votes
0answers
11 views
Data generation ( Low KMO index)
I tried to generate a dataset with a respect to a correlation matrix R using cholesky decomposition. The problem is that i always get an KMO index that is very low ( smaller than 0.5 ).
Is there any ...
1
vote
2answers
23 views
How do I assign specific values to categorical variables
I have a Pandas data frame with columns within a survey with the following categorical values - "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to ...
0
votes
1answer
21 views
Data Analytics how to read ECDF graph
Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be ...
1
vote
0answers
28 views
Is scraping job posts and gathering skills tags illegal? [closed]
I want to scrape a freelancer site (freelancer.com) and gather their public projects skill tags. Is this illegal to do? And is publishing it illegal?
0
votes
0answers
8 views
Database best-practice when sunsetting a feature
Say, your product team decided to sunset a feature of your product, and there are columns in your database related to that feature. Is it a good practice to keep the columns (so you don't lose this ...
-1
votes
1answer
44 views
How to aggregate data inserted by users to avoid outliers?
I'm developing a new application based on machine learning. In this application users can insert new data to improve the prediction system.
As you may guess, users could insert data that doesn't make ...
1
vote
1answer
37 views
how can I convert tensor shape to numpy array?
This is what my data looks like:
I want it to be numpy array, instead of a tensor, so that i can convert it to a dataframe. Is there a way to do so?
thanks in advance!
