Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
1answer
20 views

Microsoft custom vision vs Tensorflow model?

I am planning to implement my own image classifier model using TensorFlow instead of a custom vision platform. what is the biggest difference between custom vision(https://www.customvision.ai/) vs ...
0
votes
0answers
9 views

What makes a high quality image dataset?

I am constructing a custom dataset for image classification. It is a simple task in the sense that only one class needs to be detected. However it is crucial that the dataset is high quality, and I am ...
2
votes
2answers
24 views

How to determine if dataset is a suitable representation of the context?

How can I determine if the data I have collected is a good enough representation of the context? For example, I am working on an object detection system and have been building an image dataset. How ...
1
vote
3answers
22 views

Deep Learning model for classifying skin diseases

I have planned to create a deep learning model that classifies skin diseases(around 5 to 7 diseases). Please suggest me a good deep learning model to go with. I am planning to integrate this model as ...
0
votes
0answers
12 views

Customer Segmentation - Can I use an ML model for this problem?

I have a seller segmentation problem where I need to segment the top and bottom N customers based on three variables: sales, Average selling price and product category. However, the thresholds for the ...
2
votes
0answers
41 views

SVD Data Requirements [closed]

I'm very very new to data-science, so go easy on me. I want to incorporate an SVD based game recommendation model (using the Surprise Python Lib) for a full-stack project I'm working on, but before I ...
0
votes
0answers
20 views

how to know if there is a bias in data collection methods

I am collecting data for machine learning models I want to build for some application. I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
0
votes
1answer
47 views

Does shuffling the training data matter in a K Nearest Neighbors Classifier model?

I am new to machine learning and I have a couple of questions about a project. So, I created a classifier using the MNIST data set for a ML project that I was working on. I augmented the data by ...
0
votes
0answers
15 views

What is the azure solution for storing and querying Netcdf data on the fly?

We currently run a live web application which utilises standard azure mounted file storage to store 2tb worth of metocean data in netcdf format. On here we run python queries directly on the data e.g. ...
1
vote
1answer
31 views

What are the soft skills required in data science? [closed]

Most discussions center around hard skills, e.g., coding, learning specific languages. I'm curious what are the soft skills one needs to thrive in data science. Veterans, what would you tell the you ...
0
votes
1answer
27 views

Methods of disaggregating data to smaller units?

I have a relatively straightforward question that I know poses some difficult challenges. Let's say I have a state-level rate of X. I would like to disaggregate the state-level rate to the county-...
-1
votes
1answer
26 views

Why isn't all data periodic? [closed]

Let's say we have 2 classes (-1 and 1), 1 feature (x), and an arbitrary amount of data points. Why can't we always find a frequency and phase that fit a sine wave or a square wave to our data points? ...
0
votes
0answers
17 views

How to do parameter estimation on relatively few samples?

Context: I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes. My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
1answer
32 views

How to split data into training, validation, test data sets if the data is non-stationary?

When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-...
2
votes
1answer
17 views

Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., ...
28
votes
8answers
5k views

Why is it wrong to train and test a model on the same dataset?

What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?
0
votes
0answers
21 views

Is there a data structure that works similarly to a (binary) tree?

Im currently working with some data, which is part of a very large data set (height, about 6000 elements per timestep - timestep every minute over days at a time). I need to differentiate between ...
0
votes
1answer
30 views

Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?

I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. ...
0
votes
1answer
63 views

How can I generate reasonable dummy/artificial data from a pre-existing time-series data?

I have a dataset like this, basically all numerical time-series data. I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for ...
0
votes
0answers
59 views

Normal vs Uniform Distribution for machine learning

I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
-1
votes
1answer
24 views

Similar values cleaning [closed]

can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one Duplicates with different values: Geico val1 NaN =====>>...
0
votes
0answers
16 views

Ideas on handling large quantities of “waste” data

Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
-1
votes
1answer
61 views

Can we identify that an academic dataset was used for commercial purpose [closed]

There are many datasets released on the internet. Authors of many of these datasets state that the datasets are strictly for academic usage and not for commercial purposes. Although some datasets are ...
0
votes
1answer
38 views

How to input sets as features

Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on) This is from the online order processing domain. An order consists of a set of variable ...
0
votes
0answers
15 views

Usage of Doc2Vec as feature extractor for text classification of websites with political articles

I have gathered political articles from polish websites for my engineering thesis. The main goal is to try to predict the website that input text belongs to. So for this few websites I want to create ...
0
votes
1answer
17 views

Hand preprocessing data

I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going ...
0
votes
1answer
20 views

How to increase sales and revenue of a Client?

I was asked this in an interview for a Data Scientist position: Lets say Holland and Barret came to you and said they'd like to increase their sales and revenue. How will you go about it? My answer ...
1
vote
0answers
13 views

Marketing Channel Recommender System [closed]

This dataset is collected by a drug making company trying to sell its drug to doctors of different specializations. The drug company has made promotional activity for its brand The promotional ...
1
vote
1answer
32 views

adding a feature as “generic”

I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:...
0
votes
0answers
6 views

How to change data source names in Google Data Studio

Google Data Studio has connectors to MySQL, PostgreSQL, etc. but these connections will come with default names. Couldn't find out how to set the names for data sources in Google Data Studio. Is it ...
1
vote
1answer
15 views

How to conclude the generality of any classification methods?

Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$. The task $A$ is measured by a consistent measure. For instance, the task A can be a binary classification. In this ...
0
votes
1answer
16 views

How to sample with guaranteed minimum spacing?

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of ...
0
votes
1answer
15 views

Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations?

Use cases and Solutions surrounding GPT's have taken NLP world with storm and started the GPT-Best vs GPT- Not So Best war on the internet. There are solutions been derived from API's provided by HF. ...
0
votes
1answer
23 views

Model inclines because of imbalanced data

In life, some events are rare and most cases are normal. So I am wondering, to detect the rare cases, shall we use an imbalanced dataset with more historical rare cases? Taking the German Credit Data, ...
0
votes
0answers
9 views

Generating prediction when adding new rows into a relational dataset

There is a PostgreSQL table that I regularly update with new book reviews. Meanwhile, I also have trained a Tensorflow model that takes book review text and return a predicted book review category (an ...
2
votes
1answer
45 views

Can we use two independently measured features in a same ML model? [closed]

Two features are measured at different times but belong to the same target. In which conditions or form these features can be modeled together? Or they shouldn't be used in the same model but modeled ...
2
votes
0answers
19 views

How to aggregate Weather Data from county to state level? [closed]

I am doing a prediction of forest fire risk using the weather data and the fire incident data. I have data related to different county and the weather data on the day of the incident. As the incident ...
1
vote
0answers
37 views

Generating synthetic data based off existing real data (in Python)

I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
1
vote
0answers
12 views

How much information is produced in our Society? [closed]

I was reading an interesting article about the growth of data in our society: https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2012.00584.x Does any of you know if there exists a more ...
2
votes
0answers
31 views

How to improve model performace when model shows a systemic pattern in residues

I'm working on a regression model using Boosting algorithms (CatBoost, XGBoost, and LightGBM). All models give similar accuracy of 0.2 RMSE (Target varies from 0 to 1). I obtained the following plots ...
0
votes
0answers
12 views

Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
-1
votes
1answer
37 views

What algorithm should I use to get a mapping between two variables? [closed]

I have a dataset that contains for every row, a list X of x items, that is a subset of X_total, and a list Y of y items, that is a subset of Y_total. ...
1
vote
0answers
13 views

How to use time-sequence data with “meta data” of single value per sample?

I'm, trying to estimate a fish weight by a time-sequence signal of the change in resistance when the fish goes through a gate with electrodes installed. When the fish pass through the gate there's a ...
1
vote
0answers
20 views

Recommendation Systems User Profile Streaming Data on GCP

I have a recommendation system that recommends articles to different users. I am planning to provide the recommendations in an off-line fashion. Where I already have a table in BigQuery which has the ...
1
vote
1answer
34 views

Is window based sequencing a good idea to obtain more training data for LSTMs?

I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier. However,...
1
vote
2answers
51 views

Problem with converting string to dummy variables

I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
1
vote
1answer
169 views

How to build a database of image data for machine learning?

I want to build a database of image data for machine learning. But how should this be done? I'm assuming people don't just dump all of their image data into a folder? Do they use a relational database ...
2
votes
2answers
39 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
1answer
18 views

Automating the training and deployment of the model as more data becomes available

My question is about automating the training of the model as more data becomes available. In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ...
2
votes
1answer
52 views

What's the Seminal Book on data science? [closed]

Can people provide the best textbook they have used for learning data science? I know this is somewhat difficult because you have no basis of comparison but if you can just give your opinion on a ...

1
2
3 4 5
13