Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
863
questions
0
votes
0
answers
25
views
Can Hip-Hop/music trend be estimated?
How can I determine how an "evolutionary" music album affects the development of its genre? Only two perspectives I can come up with: 1. The effect on the number of songs before and after(...
2
votes
1
answer
151
views
Does opting out of having my content used for improvement mean there are no other forms of data retention of my content by OpenAI?
Regarding the use of OpenAI API, Terms of Use at OpenAI mentions:
You can opt out of having Content used for improvement by contacting support@openai.com with your organization ID. Please note that ...
- 5,553
0
votes
0
answers
30
views
1
vote
1
answer
34
views
How to increase retention?
As you might already know there is a concept of retention. Let's say I have created a game and today hundred people have downloaded my game. Let's say tomorrow 47 out of yesterday's hundred people are ...
- 121
0
votes
0
answers
9
views
What is the best way to apply Bootstapped weights to data for statistical analysis?
I'm working with a survey dataset for some statistical analysis. My issue is that there are weight columns, but I have no idea how they did the weighting or what the weights are for. I'm not familiar ...
0
votes
0
answers
20
views
How to scale a vector into a smaller vector?
I have a vector A of 6 integers [1000,900,800,700,600,500] and each number corresponds to another vector B [0,1,2,3,4,5].
How can I get a vector A_new of 5 integers if I scale the vector B to [0,0.25,...
- 11
0
votes
0
answers
12
views
Methods to appropriately capture equal ranges of turnover
Suppose I have a dataset with three columns, turnover, market, count, and turnover is the range of turnover in EUR. Something like:
...
- 1
0
votes
0
answers
231
views
AttributeError: 'NoneType' object has no attribute 'longitude'
I'm trying to get the location longitude of a address using geopy.
normally this works completely fine.
...
0
votes
0
answers
10
views
Is there data on the number of residential units in each building?
I am totally new to conducting data science, so I don't know if this is the place to ask, but I am trying to find data on the number of residential units in each building in the USA for a project I am ...
- 101
0
votes
1
answer
14
views
Total Retention Rate Calculated from Categories
I am calculating retention for 3 categories and then total, and I am trying to double check my total, but my check formula isn't working.
I am comparing the last 14 days (let's call it Period 1) to ...
0
votes
0
answers
63
views
Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX
Problem:
I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0
votes
0
answers
10
views
What is the best way to set up control limits for customer rating data?
I have 2 years of monthly average ratings for a product given by the customers. Obviously there are fluctuations in the average ratings and I wanted to set up limits for this average monthly ratings ...
- 1
0
votes
1
answer
86
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
1
answer
42
views
Models that are good for long answer generation given context and question and what datasets would be the best for training?
Basically I am trying to create a context-needing question and long answer model and I was wondering what model would be best for such tasks, currently I am leaning towards T5, or GPT-NeoX-20B. ...
0
votes
0
answers
10
views
Multicategory choice model resources
I'm a student wanting to do some applied research.
The data I'm working with has 500 consumers, and for each consumer, there are 4 type of goods they bought and each are categorized by company.
For ...
0
votes
0
answers
9
views
Transferring exact same input features to another domain in regression setting?
Assume I have a dataset (D) where each X has a label Y1 under a certain manufacturing process, let's call it Process 1, a regression model learns this task well after training on millions of points.
...
0
votes
0
answers
30
views
How to create synthetic data with custom feature values?
Currently I create a synthetic dataset using the python sdv package and an existing dataset as below:
...
- 21
6
votes
4
answers
144
views
How can I observe my CSV files better? [closed]
I'm running a lot of experiments that give their output as CSV files. An experiment might be running for hours, with a new line being added to the CSV every 10 seconds.
Right now I'm opening these CSV ...
- 255
0
votes
1
answer
21
views
How do I get a big-picture feel for my database?
Background: I was just granted access to my company's SQL Server (MS SQL 2019 via Azure Data Studio). As I have not yet been assigned any specific projects I want start digging.
I've asked my DBA to ...
0
votes
1
answer
32
views
Relationship between size of of data and number of classes
Hi we have a problem on image classification where data is quite less. However we have some flexibility on the classification.We can change the number of classes in our problem by clubbing and ...
- 101
0
votes
1
answer
49
views
What are more advanced categorical encoding methods?
I'm familiar with the common methods:
Label Encoding: {A, B, C} -> [0, 1, 2]
One-Hot Encoding: ...
- 11
0
votes
0
answers
25
views
Is there a difference in result if we apply Polynomial / Kernel Regression on mean of target data, or all data?
Let's say we have some data :
input data X with shape (1, N=100), this will be duplicated 1000 times.
target data Y with shape (S=1000, N=100).
We have 1000 experimental data points, samples.
My ...
0
votes
0
answers
11
views
How to graphic evolution of clusters in time
I am currently working in the definition of the profiles of patients of a specific disease. I have applied an unsupervised clustering technique to my dataset obtaining different patient profiles for ...
0
votes
1
answer
16
views
Whats the best statistical metric for two sets of correlated data?
I have a trained GAN that generates synthetic flight data (9 state variables/features). I plotted the correlation for each pair of state variables similiar to Seaborn's pairplot(). But I have two sets ...
0
votes
0
answers
23
views
Methods for measuring/identifying themes in complaints data that are "interesting"?
all.
I am posting this as more of a discussion post, rather than a question, as I am trying to come up with ideas for a project I have been asked to consider.
For context, I work for a customer ...
0
votes
1
answer
23
views
Azure Devops for Data Science
My team (7 people) is switching to git using Azure Devops. I’m a bit confused on how to set it up for a data science team. I used it for personal projects which is straight forward, do dev in one ...
0
votes
0
answers
14
views
Reshaping data from long to wide format in Stata
I have a dataset on carbon intensity and volume loans across different sectors in the EU and I try to reshape the data set from long format to a wide format as I wrongly first thought that it should'...
- 1
1
vote
1
answer
88
views
How to find correlation between features and labels when features are unordered words?
I have a data set with ~ 85 labels and a target (house price)
I wish to explore the correlation between the labels and the effect it has on the target.
I have some features that number and others that ...
- 11
0
votes
0
answers
40
views
Is Subsymbolic AI machine learning?
I read a few articles regarding different types of AI.https://wiki.pathmind.com/ai-vs-machine-learning-vs-deep-learning. Here it was stated that symbolic A.I is not machine learning. So can all of ...
0
votes
1
answer
125
views
Split dataset into Train/Validation/Test for Object Detection
I have a dataset for Object Detection with YOLO format labels, each imagine can have occurences of different classes and multiple occurences of the same class.
How can the dataset be divided into ...
0
votes
0
answers
11
views
Based on what Categorization is machine learning a part of Ai?
I read an article on microsoft website (https://azure.microsoft.com/en-us/solutions/ai/artificial-intelligence-vs-machine-learning/#introduction) , it said that machine learning is considered a subset ...
0
votes
0
answers
119
views
Orange data mining: instances with missing targets are ignored while scoring
In predictions warning showing "instances with missing targets are ignored while scoring" is coming how to resolve
0
votes
0
answers
22
views
Best ways to visualize the performance of a regression model?
How best to visualize 10,000 observations of continuous labels against predicted values in a way to summarises the overall performance of the model? I want to avoid using performance metrics such as ...
- 1
0
votes
0
answers
5
views
I am trying to create a score that will group each feature (columns) in my dataset, between 1- 20 where the highest group will be 20 and lowest is 1
#here are the list of columns i am working with
'transaction_volume', 'transaction_value', 'avg_transamount',
'avg_running_balance', 'min_amount_transacted', 'max_amount_Transacted',
'deposit_volume', ...
- 1
0
votes
1
answer
250
views
How do you Specify which Columns Pandas reads in?
I have a huge dataset with ~470,000 rows and ~1400 columns. I only require 184 of the available columns. I'd like to limit the columns I read in to improve my dataset's loading times.
How do you ...
- 557
0
votes
0
answers
11
views
Basketball-related model divided the in-sample data in half for use as two "out-of-sample" data sets. Is this an acceptable methodology?
so I'm an amateur in terms of data science skills, so I was wondering about this. I was digging through the methodology for a popular NBA player impact model called RAPTOR and came across this passage ...
0
votes
1
answer
28
views
What is the name for 'cheating' with data, i.e. letting machine learning models use out-of-sample data?
What is the term for abusing data and machine learning methods to get better results than normally (with proper training and test set)? Some of my co-workers call it data-mining but I do not believe ...
- 143
0
votes
1
answer
1k
views
How do I use FAISS searching with Haystack?
I have a private SQL database with text entries in it. I want to create a search engine to query these entries. I have been looking into haystack (https://haystack.deepset.ai/overview/intro) with the ...
- 103
0
votes
0
answers
19
views
How to create test data for multi input model without label with tf.data api
I am having problem creating test data with tf.data api for a model that takes 2 inputs.
model
...
- 135
1
vote
0
answers
40
views
Data Leakage (specific case with standardization)
I read a few blog posts and found some books that caution against data leakage when adjusting the training data using any information from the held out datasets. For example, if I were to standardize ...
- 11
0
votes
0
answers
9
views
Data augmentation useful for this or not?
I have a training set of medical breast x-ray images. Approximately half of them are flipped along the horizontal axis since some are from the right side and some are from the left side. See the ...
- 1
0
votes
0
answers
13
views
How to classify and organize very complex data for easy future reference?
I've studied binomial nomenclature in college, but forgot all about it!
I use Joplin to organize my life and business, but it's getting out of hand complicated and I can't find stuff.
Struggling on ...
- 101
0
votes
0
answers
19
views
Understanding test predictions from audio data
Solving - I am working on solving a spoken digit recognition problem. The inputs are audio files of digits from 0 to 9. Data is balanced.
Model Arch - As a simple model, I used librosa.load() and took ...
- 1
0
votes
0
answers
76
views
Time Series data Augmentation in pytorch forecasting
I have count time series of demand data and some covariates like weather information every hour. I have used 168 hours (7 days) for encoder and 24 hours (next day) for decoder in DeepAR pytorch ...
- 101
0
votes
0
answers
29
views
can someone please explain what is "data mismatch"?
I have searched for relevant blogs but can't seem to find satisfactory answers. Can someone please explain what is "data mismatch". Thanks
- 1
0
votes
0
answers
71
views
Unable to connect to data connection power BI
I am trying to connect to azure databricks but getting the following error.
I have checked the path and the spark odbc driver is present there still it is throwing error.
Details: "ODBC: ERROR [...
- 157
0
votes
0
answers
236
views
What is the difference between Covariate Shift, Label Shift, Concept Shift, Concept Drift, and Prior Probability Shift?
As a beginner in MLOps, I was overwhelmed by some confusing definitions.
As far as I understand, when we have a classifier or regressor with y = f(X) function:
<...
- 33
0
votes
1
answer
53
views
What is the next after finding best distribution for my data?
I found the distribution of my data with "distfit" library for python. But what now? The best distribution that describes my data is "weibull" distribution. But I don't know what ...
- 53
0
votes
1
answer
362
views
pd.get_dummies One-Hot Encoding but Keep the Order of Appearence
So I have this data which which has the 12 months as below:
When I one-hot encode it using pd.get_dummies the result is this:
You see that the new columns are ...
0
votes
1
answer
31
views
From what percentage should I remove a column that has empty values?
I have a problem. I am currently cleaning and preprocessing my data.
Suppose I have the following data set
...
- 89