Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
0 votes
1 answer
12 views

Classification Problematics : Feature Number Variance & Feature Repetition

I have a harsh case study (in my mind). The problem is I need make binary classification on Quality of Service (good or bad). I have a feedback on quality on groups of devices belonging to company. I ...
secuf's user avatar
  • 1
0 votes
0 answers
8 views

I want to import my states legislation into a database. I need advice in parsing large HTML, TXT into an importable form. It is Large PDF initialy

My data consists of large pdf files containing state legislation. I can convert the pdf to xml, txt or html, and I can manually go through, identify the tables and write a script to extract only the ...
RBB's user avatar
  • 1
0 votes
0 answers
78 views

Best practice for fine tuning LLM

I have a dataset that i have collect for specific topic the dataset is in these format : raw text (similar to shake spare dataset) where it has no label or input, just text Question and answer ...
Mustafa Alahmid's user avatar
0 votes
1 answer
160 views

Can bert uncased predict text classification on foreign data?

I am trying to do the fake news/real news classification and used a pre-trained bert uncased model as transfer learning and it gave a solid 81% accuracy. But the problem is while doing sanity checks, ...
Jasmin Wilson's user avatar
0 votes
0 answers
7 views

Dataframe shape for multi-point prediction using LSTM & time-serise data

Can I ask a very simple question about LSTM time-series data prediction? I have 1258 data points which are Amazon's historical stock price by day unit. And I build the below data structure in order to ...
orde.r's user avatar
  • 113
0 votes
0 answers
113 views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python - from_gerator fails

I know this question has answers, but they are not working. why this fails ? (and why tensorflow has troubles in reading numpy!!) ...
user305883's user avatar
0 votes
0 answers
27 views

How do I test with custom images for a model that was trained on quick draw npy dataset

I have been trying to test my CNN model on a custom doodle of an apple that I drew. But even when I preprocess the image to have the same shape with the training data, the model gives wrong prediction ...
Marcuss's user avatar
2 votes
1 answer
42 views

Smart Selection of Training Data for Fine-tuning Language Models in Small Domains

Background I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$. If the size of $D$ is $N$, ...
Mr.Robot's user avatar
0 votes
0 answers
17 views

Do Hospital Episode Statistics (HES) data (NHS UK) contain labs?

Can anyone tell me if HES data contain lab results such as blood tests commissioned and their result? That can be in any one of the HES datasets. Thank you.
user2822693's user avatar
0 votes
1 answer
39 views

Spam tweets dataset needed [closed]

Where can I find labelled spam tweets dataset that has at least 50000 rows. I've searched everywhere; but there are very few datasets available, and those datasets are way too small. I am in urgent ...
Kingster20's user avatar
0 votes
1 answer
189 views

Open retail/ecommerce dataset with customers, transactions, products, loyalty, discounts, promotion information?

Does anyone know of an open retail/e-commerce dataset with information about customers, transactions, products, loyalty, discounts, and promotion information? I am planning to create 360-degree ...
exan's user avatar
  • 101
2 votes
2 answers
256 views

Training GPT Model with Swagger Documents, Need Help with Model Fit

We are currently developing an application that performs actions based on user input using 'gpt-3.5-turbo-0613'. For example, if a user enters the prompt "create a user with the name Potter and ...
mostwelcome's user avatar
1 vote
2 answers
41 views

Tips for scraping crypto data in the right way

I am scraping data from crypto site and want to use neural network algorithm for predicting data. the way i save data is like these: and there is bunch of other features like open/high/low/close for ...
mohammad ariyan rad's user avatar
0 votes
0 answers
15 views

Multiple coords to one data_vars in xr.dataset

I am trying to create a xarray dataset for a list of events (labeled 1 or 0, so it's an array of integers). To each event three numbers need to be attached - a date, longitude, and latitude. I provide ...
Shaz's user avatar
  • 125
1 vote
0 answers
25 views

Is there a statistical test for comparing my dataset percentages with published percentages?

I have a clinical dataset which is showing the percentage of A is 57% and B is 35% for where an tumours is initially found. However the literature is showing it is A is 73% and B is 23%. Is there a ...
Matt Smith's user avatar
-1 votes
2 answers
135 views

How to generate synthetic dataset with dependent and independent features?

I want to generate a synthetic dataset, in python, composed of 10 features. 5 are dependent among each others and 5 are independent. Generate dependent features. ...
I Sui's user avatar
  • 57
0 votes
2 answers
99 views

Is there any rationale for performing SMOTE-ENN before train-test-split?

I have created a classification model for predicting data, and the problem is that the two classes are highly imbalanced I have a problem. I have created a classification model for predicting data, ...
Hello is me's user avatar
0 votes
1 answer
58 views

How to handle ID variables when splitting data for machine learning?

I'm new to machine learning, and I'm working with some international head-to-head sports competition data. I used relational data creation techniques in tidyverse to join several data sources to ...
Emr8980's user avatar
1 vote
0 answers
38 views

Accounting for differences in average teacher salaries, adjusted for inflation and the cost of living in each state

My aim is to determine whether teacher salaries in all 50 of the United States over between 2013 and 2023—adjusted for inflation and the cost of living—differ significantly. I would like to ask what ...
Nazarene's user avatar
0 votes
0 answers
19 views

Problem with the size of test data?

I am working on my thesis, which has two different research questions: Evaluate transformer models while incorporating non textuel features Evaluate the importance of data quality in tranformer ...
Aymen Ba's user avatar
0 votes
0 answers
13 views

I'm looking for a dataset on enviroment monitoring

I'm looking for a dataset related to environmental monitoring, made up of values obtained from various types of sensors (such as temperature, pressure, CO2...etc) for the purpose of a classification ...
PiEmmeC's user avatar
0 votes
2 answers
180 views

How to create a plot of specific row with every column using python and which package to choose matplotlib or seaborn?

I'm trying to create a plot for the ranking of each country from 2002 to 2023. I created this dataset by loading each csv file from the respective years, which contains the Countries and Ranking ...
SAUBHIK KUMAR's user avatar
0 votes
1 answer
60 views

Data transformation pipeline error

When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
Amy's user avatar
  • 1
0 votes
1 answer
22 views

Domain Constraints

Consider a dataset with 5 numerical features: A, B, C, D and E. Can we train a deep learning/machine learning model which can learn constraints between (A,B) and (C,D,E) i.e I want to learn a function ...
Vivek Kumar's user avatar
0 votes
1 answer
62 views

Keras MLP not working

For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a ...
alex martinez's user avatar
0 votes
0 answers
13 views

Scaling datasets for multi-dataset time series

Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
Hadar's user avatar
  • 137
0 votes
0 answers
19 views

Training a Face Recognition model using benchmark datasets

Is it recommend (or conventional) to train, for instance a face recognition method, using benchmark datasets such as XQLFW, RFW, CALFW, etc.? I would like to fine tune my model so it is more robust to ...
Carreira's user avatar
0 votes
0 answers
12 views

How to analyze 2 NLP datasets and understand their differences

I have 2 test sets (let's call them test set #1 and test set #2), and 2 Language models (let's call them model A and model B). The issue I'm facing is that model A is performing better than model B on ...
Astariul's user avatar
  • 1,004
0 votes
1 answer
130 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
Horbeam's user avatar
0 votes
1 answer
30 views

how to prepare pandas dataframe from Solar Flare Data Set?

I am looking to download and use Solar Flare Data Set (https://archive.ics.uci.edu/ml/datasets/Solar+Flare) but unable to figure out how to get the pandas dataframe from the file that is being ...
Sachin's user avatar
  • 101
0 votes
0 answers
20 views

How to manage and store datasets used to train Computer Vision models?

I am not sure if this is the right place to ask this question but I am looking for an insightful answer. I have been working as a Data Scientist for one year now. Currently at the company that I work ...
Medine Kamberi's user avatar
0 votes
0 answers
27 views

Grouping data or clustering data

I have frame data of 10 subjects gathered from a timeseries analysis. Each subject have 100 frames except the reference frame (timeseries length is 100+1 = 101). Particular frame property is ...
user149345's user avatar
0 votes
1 answer
48 views

Low-dimensional binary classification datasets

If one would like to study aspects of neural networks (say, in an academic paper), and would like to experiment on binary classification of vectors in low-dimensional space (say dim=2 or dim<6); ...
Sasha's user avatar
  • 1
0 votes
0 answers
44 views

Multiprocessing data loading in colab

I want to convert mozilla common voice dataset from mp3 to wav. But this dataset is large and convertion takes many time. How can I make this convertion in colab with multiprocessing to decrease time ...
randomuser228's user avatar
0 votes
1 answer
24 views

A guide to learn data analysis

I'm new to data analysis and I need to do a data analysis project using clustering methods for a course in R. I have no idea how to start and choose my data set. I'm looking for some resources. Is ...
user115608's user avatar
0 votes
0 answers
16 views

Best way to regrid data (polar stereographic to regular)

I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
Socorro's user avatar
  • 91
1 vote
1 answer
41 views

Can we create tensorflow or tflite model for object detection without uploading our dataset images to cloud?

I need to create a custom Tensor flow lite model for object detection to integrate in an android app. But I have a constraint that the dataset images to be used is confidential and cant be uploaded ...
Roohi Zuwairiyah's user avatar
0 votes
0 answers
22 views

Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?

According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset. I know a little about the reason, i.e. there are ...
EvilRoach's user avatar
  • 101
0 votes
1 answer
121 views

Does the Transformer model has memory to store the state accross different data injection sequences(segments)?

I've trained a transformer model based on the pytorch tutorial: https://pytorch.org/tutorials/beginner/transformer_tutorial.html, But I found I've difficulties to understant this model's input and ...
Clock ZHONG's user avatar
0 votes
0 answers
9 views

is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided

Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion. The goal is to use this metric: regression classification as a metric profile_views ...
Zaahir Ebrahim Dawood's user avatar
0 votes
1 answer
172 views

Plot multiple catplots from a dataframe

I have a dataframe and I want to plot multiple catplots. I want to use a loop but i do not know how to code one for this. Any examples? ...
ahmike's user avatar
  • 1
0 votes
0 answers
16 views

Having trouble manipulating data to make better dashboard

I have a test dataset which has has 39 columns. Index is given below: Sample data is as follows: Some columns are omitted due to privacy concerns. The colored cells in the index belong to one type. ...
Chinmay Datar's user avatar
1 vote
1 answer
44 views

I want to forecast a NetCDF data

I have a chlorophyl-a and sea surface temp (SST) in a NetCDF format, the data is 4 dimensional mattrix, each data is saved inside certain longitude and lattitude value, if the data plotted into ...
P_dog's user avatar
  • 11
0 votes
0 answers
33 views

How should data from closed down sites or companies be licensed?

There are sites that are closed down but exists on archive.org, e.g. Oyshee aka Rakuten Recipe (English) closed down, but the parent company Rakuten still exists DP Review will close down in April ...
alvas's user avatar
  • 2,340
1 vote
1 answer
27 views

Comparing images in N channels

I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
Shaz's user avatar
  • 125
0 votes
0 answers
19 views

Time series : add constants for each time series

I have N time series of shape (30*36) (time step, feature). For each time series 500 parameters that can be seen as the history of the time series have to be added. ...
Ketchup's user avatar
  • 29
0 votes
0 answers
25 views

How to label a transactions dataset made from scratch?

I have a question regarding creating a transactions dataset from scratch. I've created customer profiles and am generating transactions based on these profiles. The way I do this is based on the ...
pnav32's user avatar
  • 1
0 votes
1 answer
20 views

What is the best way to store large data when training a model?

This is rather a technical question, when training the model I import data locally to Google Colab runtime which does not seem to be efficient, when the runtime ...
Bionix1441's user avatar
0 votes
0 answers
15 views

How to estimate this variable in an MILP formulation

This is my first question being asked here. I've thought about different methods to do it, but to no avail. I want to estimate a variable that is either 0 or a positive number. Then I want to use this ...
Mohammad Rajabdorri's user avatar
0 votes
0 answers
9 views

Implementing a dataset to Computer Vision Article

I want to implement the PIE dataset in the AgentFormer arch. AgentFormer uses ETH and nuScene datasets. I successfully run these datasets on this arch. However, I couldn't take a good way with the PIE ...
Zehra N.'s user avatar

1
2
3 4 5
30