Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
3
votes
0answers
17 views

What are descriptors

I am very new to data science, and r programming specifically. Could some explain what a descriptor in data science is? In particular, what are the differences between a 2d and 3d descriptor? Thank ...
2
votes
1answer
38 views

How create a representative small subset from a huge dataset, for local development?

​ I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
0
votes
0answers
21 views

How to mathematically quantify the quality of a corpus?

I am working on a text classification project. I have around 60,000 text samples of 40intents. By calculating the frequency of ...
0
votes
1answer
17 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
0
votes
0answers
13 views

Best way to store large amounts of time series data? Relational database (SQL) or NoSQL route. Also additional python/pandas question inside

so I'm working on a project and I'm sort of stuck as to how to store my data. I have a concept I want to propose but am unsure whether it is possible, if it is not I would appreciate any help ...
0
votes
0answers
9 views

Find the formula for these data series to create model

Thank you for taking the time. I'm working on creating a model my team can use to see the maximum number of miles that can be added to a delivery trip in a measured area based on the number of stops (...
0
votes
0answers
10 views

How to find the lag that shows highest correlation between two time series variables?

I have two variables in my time series data (X1,X2). I need to find the correlation between these two variables at different lags and identify the lag that shows the highest correlation. For e.g. I ...
1
vote
0answers
10 views

Is this set of vectors representing direction in a sequence trainable across LSTM? if not, what are the drawbacks?

I am starting my Neural Network Project for the first time, I have a lot of expectations: The main idea is primarily analyzing the pattern of my writings or drawings when using the mouse given a ...
0
votes
0answers
5 views

How to model multinomail logistic regression to get desired outcome?

I want to run multinomial logistic regression in SPSS. Dependent variable: Code=0 Cognitively normal(CN) Code=1 Mild cognitive normal(MCI) Code=3 Alzheimer’s(AD) Independent variable: Genotype.1=0 ...
1
vote
0answers
19 views

Multivariate regression - not enough data?

I have a table with data about 10 agriculture parcels. Each parcel has data in time regard the number of nutrients each parcel has received in each day and in the end I have the total number of ...
0
votes
1answer
13 views

Encoding when different number of records for each month-One hot or different type?

I am working on a dataset which is pretty small: 1169 records. There is a column called month and it takes the values 'Jan','Feb', or 'March'. The number of records for each month is different. I have ...
0
votes
1answer
28 views

Preprocessing , EDA , and Feature Engineering

What is the difference between EDA, Feature Engineering, and Preprocessing? The main purpose is to make the raw data suitable for modeling. In EDA, we are cleaning the data and so does the ...
0
votes
0answers
10 views

Can you POST to a rest API from databricks?

I am looking to post a JSON file to a rest api via databricks. I am having trouble trying to figure out how to do this.
0
votes
1answer
21 views

Creating a complex featureset for regression modeling

I am currently working a on project that requires me to convert all of the categorical variables to continuous (or binary) variables to build a regression model. The problem is that I have more than ...
0
votes
1answer
16 views

Machine Learning A-Z: Help connecting with datasets

I'm in Machine Learning A-Z lecture 19 where we access our first dataset. I've connected to the Google Colab folder in my gmail account, but when I try to access the the file I get the below error ...
0
votes
0answers
16 views

How to generate a positive skewed array?

As the title suggests, I would like to specify the task that I really need help with: Given three possible distributions of an array: I know that given a skewed dataset (the left and the right), then ...
1
vote
0answers
12 views

UI-based Tool for Qualitative Evaluation of Data Quality

Dear DS StackExchange community, I'm currently searching the interwebs for a (near-)ready-to-use solution, to perform a qualitative evaluation of extracted features from video data. In my head the ...
0
votes
0answers
19 views

What analytics model/classifier should be used to predict price?

I've go dataset with more than 10 features - football skills describing players. There is also price value in each row. I would like to predict price by specifying ...
0
votes
1answer
30 views

How to interpret 3 outputs of Precision?

Say im running a classification Machine learning algorithm, of 2 classes 0 & 1. A 0 label is detecting a visitor/row did not Convert. while 1 label is detecting a visitor/row did Convert. When the ...
0
votes
0answers
94 views

Why am I getting a different answer in Principal Component Analysis dimensional reduction?

Problem-: Consider the two dimensional patterns (2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8). Compute the principal component using PCA Algorithm. Use PCA Algorithm to transform the pattern (2, 1) ...
1
vote
2answers
21 views

What are some examples of ordered data?

Tan,Steinbech,Kumar's book says For some types of data, attributes have relationships that involve order in time or space. I want some real life examples of such data, can you provide me. I couldn't ...
0
votes
0answers
7 views

How to clean/analyze relative time series data

I'm relatively new to data science, and I decided to stretch a bit and try a project with time series data. I downloaded the Human Activity Recognition from Continuous Ambient Sensor Data Data Set, ...
0
votes
0answers
11 views

Dictionary of life sciences or medical terminologies

I'm exploring available open-source dictionaries with medical terminologies. I found this but it's limited. Currently focusing on how to make use of NIH. However, the challenge is that I'm running ...
1
vote
0answers
14 views

Clustering dataset with and without estimating means (no EM algorithm)

Given a dataset $D$ of the form $$ D = \{ (x_0,y_0), (x_1,y_1),\ldots,(x_{n},y_n) $$ sampled from a Gaussian mixture model with identity covariance matrices, I want to understand what are my options ...
0
votes
1answer
13 views

How do you normalise the train+validation sets together?

This question is somewhat related to: Is it correct to join training and validation set before inferring on test-set? As far as I understand, normalisation in general is done in the following way: ...
0
votes
1answer
41 views

Is it a good idea to use parallel coordinates for visualising outliers? [closed]

I tried using parallel coordinates to visualize outliers. Is it fundamentally correct?
0
votes
0answers
25 views

Training data for image digit recognition

I am creating a digit image recognition algorithm. For this I would like to collect sample data to train the model. Does python have any inbuilt libraries for this? Where could I get good training ...
0
votes
0answers
11 views

Most Hot Freelance Data Science Project Niches

I almost finish my final year in my information system major for computer science's bachelor degree, i also take big data and machine learning as my specialization. And then while i finding and apply ...
0
votes
2answers
28 views

Assign value of existing column to new columns in pandas based on conditions [closed]

I am trying to create a new column in pandas data frame based on existing columns. column1 column2 column3 y1 y2 y3 100 200 300 2020 2021 2022 100 200 300 2021 2022 2023 100 200 300 2019 2020 2021 ...
0
votes
0answers
22 views

How to do exploratory data analysis on a database

In all courses, the focus is mainly on building models from an already clean dataset that is in a csv file. If I have a database, Postgres, with a table that has ~10 million rows, what is the best way ...
0
votes
3answers
87 views

Convert .data file to .csv

I'm using a data called 'adults.data', I need to work with that data as a '.csv' file. Can anyone guide me on how can I change the format? I tried opening the file in excel and then save it as csv, ...
1
vote
1answer
12 views

How to deal with mismatch of timesteps in target variable and features in forecasting problem?

Background Info: I am working with some climate data where I want to predict crop yields with my dataset containing climate- and satellite-derived features. This is a time series regression ...
2
votes
1answer
31 views

Are there any popular English corpus?

Are there any popular English corpus?
0
votes
0answers
10 views

Why do images in Open CV template matching get matched to the templates more than once?

The following code reads the set of input tiff files under tiffile and template tiff files under files and performs template matching on them. After the image gets matched, opencv contour extracts the ...
2
votes
1answer
31 views

Visualization suggestion:-

I have a data frame like below:- Here I have 194 countries and the columns are fan_out values which is in percent of the total population. Like for country AD, the total fan_out value is 2.24 -06 % of ...
2
votes
2answers
26 views

How to input a list into my model and not have it care about order

I'm trying to predict a list of numbers, e.g: [23,55,198,200,64] The data I have includes multiple things, along with: The numbers from the previous run (These ...
0
votes
0answers
16 views

When training a cGAN on a X (B/W image) to Y (RGB image) "paired" dataset, how much will slight differences in the images matter in terms of results?

Right now I am working on a colorization project with GANs, and had originally settled on using a CycleGAN because I considered structuring my dataset in an unpaired manner. I've since been able to ...
0
votes
1answer
48 views

Does the abstraction of a class affects the performance of neural networks?

For example, if I have 3 audio classes including Ambulance Siren Police Car Siren Firetruck Siren assuming these 3 classes could be distinguished by humans. If I just want the model to classify all ...
0
votes
0answers
8 views

how to prevent feedback loop while you can't use another data source

we have a harassment prediction model which trained and deployed many years ago. each day this model predicts some conversation as harassment and annotators label these predictions. (they only label ...
0
votes
0answers
26 views

VGG16 (pytorch) training issues using a very large dataset vs a smaller dataset

I've constructed a simple VGG16 layer model from the original Simonyan & Zisserman paper for use on a DBT (Digital Breast Tomo.) data challenge. As a starter model, I chose to make the 3x244x244 ...
0
votes
0answers
11 views

How would I perform a regression of this nature?

I have two dataframes. One is a dataframe containing the injury history of all players before 2013, the rows are the names of each player, and the columns in this dataframe are upper leg, lower leg, ...
0
votes
0answers
9 views

Identify the top clients

I am asking the metric to identify the top clients for the ecommerce data. Usually, is the metric based on revenue, or average order value, or average spending? Also, how can we take top 10, 20, or ...
0
votes
0answers
35 views

How to adjust the unadjusted charts in Pandas

I would like to adjust the unadjusted charts in Pandas. Suppose I have a data frame of the price of a stock symbol that there is a price gap between two candlesticks due to a capital increase. Sites ...
0
votes
0answers
9 views

Will training with yolov4 backup weights early cause weak prediction or is a true backup?

I am training a dataset in yolov4 using the repo from AlexeyAB darknet. In his repo, backup weights are created every so iterations but you originally train with a pretrained weights file. I was ...
1
vote
0answers
16 views

Data Pipeline Best Practices

I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
0
votes
1answer
12 views

European soccer championship: Where can I find historic data on the market value of the teams

I want to write a simple model predicting the performance of the teams in the european soccer championship. The results of soccer matches are known to be strongly correlated with the market value of ...
1
vote
0answers
26 views

How to compare two one hot encoded data frames based on column names?

I have two datasets with shapes (329, 159) and (26,24). Both of them are one-hot encoded. The columns in the smaller dataset are present in the larger dataset. The smaller dataset has scores that I ...
2
votes
1answer
45 views

Highly correlated features

What are the conditions to remove highly correlated features in a correlation map? Given the correlation map below, is it OK to remove diagnosis feature and should we remove highly correlated features ...
0
votes
0answers
12 views

Data Warehouse design schema for sales (items, shopping carts, ..)

I have to design some OLAP-Cubes based on an OLTP database for the sales. At first, that sounds very basic and common but I struggle with the item <-> shopping cart thing. One cart can contain ...
2
votes
2answers
58 views

How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?

I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...

1
2 3 4 5
14