Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
3 views

How to drop the previous rows of a database based on a matching value in a column?

So I am currently trying to sort through a data frame containing attribute classes and values of teams. However, my data has multiple rows of different classes and values of the same Team ID/Attribute ...
0
votes
0answers
7 views

Causal Inference where the treatment assignment is randomized

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
0
votes
0answers
7 views

Aggregation based on previous month from eventdate

I'm Stuck on a problem it would be great if you could help me :) I created a dataframe with pandas: looks like that: HostName Date A 2021-01-01 12:30 B 2021-01-01 12:42 B 2021-02-01 12:30 A 2021-...
0
votes
0answers
11 views

How to merge dataframe column cell data into single column in mean

I'm using pandas dataframe unable to get the expected output as described in image? Current Output: Expected Output:
0
votes
0answers
6 views
0
votes
0answers
11 views

NameError: name 'start' is not defined and ValueError: operands could not be broadcast together with shapes

I have declared "start" variable as a global member. Still it is giving the error: ...
0
votes
0answers
4 views

Implement NestedCV into Neural Networks

I have a regression task for which I am using ML models. My input features are 64. I implement NestedCV to get best ML models and hyperparameters. I have recently learned Neural Networks and want to ...
0
votes
1answer
12 views

Categorical Variables (Y or N) in KNN Classification?

Practicing KNN and I just had a query about pre-processing, as I understand KNN doesn't work with categorical features. I've read into one-hot-encoding (dummy variables) which I suppose if I applied ...
0
votes
0answers
14 views

custom convolutional layer in Keras but model.predict() fails

I've developed a custom convolutional layer. I can use to build a model and train it (model.fit works), but model.predict() ...
0
votes
0answers
20 views

If a feature is highly correlated to the target variable, would it also rank high when performing feature importance using a Random Forest Classifier?

I am trying to understand the relationship between pairwise correlation of variables in a dataset AND the feature importance of these variables using a Random Forest Classifier.
1
vote
0answers
10 views

Create A count Features right

I have a table like that: PC USER Period A U1 2020/01 A U2 2020/02 A U3 2020/03 A U4 2020/04 B U1 2020/01 B U2 2020/02 B U2 2020/03 for every row show the max connection: PC USER Period Max ...
0
votes
0answers
18 views

Dual Optimization SVM in Python using Numpy

I need to implement the dual function of SVMs optimization problem with numpy in Python and I am pretty stuck since I am not a Python or a numpy pro at all. Dual function evaluation What I want to ...
1
vote
1answer
30 views

Select one best model according to accuracy, precision, recall, f1 score and roc score

I have two classifiers that classify the same dataset with these results: ...
1
vote
0answers
12 views

Appropriate objective for a data science project

I've done a couple of DA projects found on the internet and I want to start one myself. I'm interested in baseball data and I think I can exploit a lot of that data but also want to delimit the topic. ...
0
votes
0answers
8 views

Is there an easy way to find the number of subscribers of a channel using python?

I will get a list of discord channels and I want to get the number of subscribers. For example, givens this : https://discord.com/invite/NqfVS7M How do I get the number of subscribers?
1
vote
0answers
41 views

Predicting single floats based on set of 2 feature arrays each of 100 values

I am trying to predict audio to video desynchronization based on set of two arrays of lenght 100 which consist of coresponding audio and video samples. The problem is that my labels are single floats (...
0
votes
0answers
13 views

How can I combine the information from these two dataframes?

If one of my dataframes gives me some info about items: ...
0
votes
0answers
15 views

How to perform entity level train-val-test split for NER task?

A normal and stratified split option is provided by sklearn method that can be used for ML problems like multi-class classification. This is relatively easier to do as (1) one sample has one class, ...
0
votes
2answers
44 views

Call function from one class in the same class

I have 2 classes model and impute. I am defining a function mode_impute inside ...
1
vote
1answer
25 views

Help starting ML project in pythin(novice)

I am starting a machine learning project (for fun!), but I am not sure where to start from... I am fairly new to ML so any hints are appreciated. I have a relatively large data-set where each input is ...
0
votes
0answers
22 views

Error:DQN expects a model that has one dimension for each action, in this case (1, 2, 1, 0)

I am building an RL agent for which the model is defined: ...
0
votes
1answer
17 views

Applying standardization using ImageDataGenerator

I have a multiclass image dataset ( 8 classes) that is divided as follows, the main folder is called training and I have 8 subfolders with each subfolder for one class. I know how to perform data ...
0
votes
0answers
15 views

NaNs not being ignored despite program being told to do so; Linear Regression

I have gone through the process of filtering a set of data that I have by removing excess numbers and values that are invalid. However, there is a lot of data that occasionally has ' ' in it, and ...
0
votes
0answers
15 views

ImageDataGenerator for image preprocessing

I'm trying to do a superresolution network, but I am having trouble importing my own data. I have two types of images: resized images (smaller), original images. The first one is going to be used as ...
2
votes
0answers
25 views

Should hexadecimal addresses of a dataset be cleaned?

I am working on fraud detection on blockchains. To be more specific, I fetched a big number of transactions that took place on the blockchain, labeled them to spam / non spam using an appropriate API ...
0
votes
1answer
27 views

Custom preprocessing using piplines

I have searched a lot for this issue but unfortunately came up with nothing. Usually in a ML model, during preprocessing, we use Pipelines and ...
0
votes
0answers
4 views

REGRESSION PROBLEM_ Generating y from X and error

Good Morning. I am working on an assignment where i need to perform the following tasks: 1.Create 500 observations based on standard normal distribution and store them in variable X 2.Create 500 ...
1
vote
1answer
34 views

Date time conversion in a CSV column

I am new to data science. I am attempting to write a program using regression techniques, and all of my values are numerical, except for the date and time (UTC), which are written in this format: HH:...
2
votes
1answer
23 views

Which Scikit-learn application should I use in order to process my data? (Python)

I am writing a program that will take a multitude of data, such as chlorophyll levels, water temperature, nitrogen/phosphorous levels, etc., in order to predict the growth of algae in a body of water. ...
1
vote
1answer
44 views

Why is val accuracy 100% within 2 epochs and incorrectly predicting new images? (1,000 images per class when training)

My CNN tensorflow model reports 100% validation accuracy within 2 epochs. But it incorrectly predicts on single new images. (It is multiclass problem. I have 3 classes). How to resolve this? Can you ...
0
votes
1answer
26 views

Should I add string feature columns?

If my dataframe looks like this: ...
1
vote
1answer
17 views

Preprocessing for the final model to be deployed

Typically for a ML workflow, we import the data (X and y), split the X and ...
1
vote
0answers
10 views

Bad accuracy on predicting user ratings upon his reviews

I am trying to predict user ratings based on books he rated(ratings range between 1-10). So I encoded the summary of the books and then tried to train the model with the text encoding(practically the ...
0
votes
0answers
15 views

How to retrieve flatten PCA analysed image?

I am trying to retrieve each of the images after PCA. What I have done, First, I flatten all the images and concatenate them vertically. Suppose, image size is 80 x 80, then flattening it creates 1 x ...
1
vote
1answer
31 views

I am getting the error in SimpleImputer

...
0
votes
1answer
12 views

Data captured outside Pandas DataFrame

I'm new to Python for starters, when I got this DataFrame from Yahoo Finance, the Date shows up outside the DataFrame. This is leading me to get KeyError: 'Date' when I try to write this line of code ...
0
votes
0answers
5 views

Correct implementation and usage of NestedCV and train and test data?

I have some confusion regarding NestedCV and the correct usage/split of the data in order to reduce/prevent overfitting/bias/leakage. Below I have implemented my understanding:- ...
0
votes
1answer
18 views

Splitting string column into several columns

In Python, I want to split a string column into multiple columns. The problem is, the strings are not identical or the same length. Example of data: ...
0
votes
1answer
22 views

How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
2
votes
0answers
15 views

Why does a linear regression in time series forecasting does not provide a line in predictions?

I'm reading the TensorFlow Time Series forecasting Tutorial 1 trying to perform my own time series prediction. However, specifically on single-shot models section for multiple time steps, the Linear ...
1
vote
1answer
21 views

error: (-215:Assertion failed) [closed]

I was testing OpenCV with their code provided within the documentation. Unfortunately, I'm facing an unexpected error. I think, it might be due to the newer version that I've. I've tried the fix of ...
1
vote
1answer
22 views

How to preprocess heavy MRI images?

I have a large MRI dataset for an image segmentation task that cannot directly fit in memory in Colab, you can access the data with the link I put at the end. They are brain MRI images: 484 training ...
0
votes
3answers
19 views

Creating numeric word representation of input sentences resulting in MemoryError

I am trying to use CountVectorizer to obtain word numerical word representation of data which is essentialy list of 160000 English sentences: ...
2
votes
1answer
18 views

Generate fake model predictions according to desired precision/recall values

Lets assume I generate a random set of target labels for a binary classification with N elements and a certain frequency of the positive class (1), e.g. 10%: ...
0
votes
0answers
14 views

Threshold moving for classification - how can i assess the cost?

I have a binary classifier. When i used my model to make predictions about 4k out of 10k were predicted to be "Rich". I am predicting affluence. Normally in classification the cut off to ...
0
votes
2answers
15 views

How do I remove float object in dataframe? Ex: "winter 2021" to "winter"

I have the premiered column where I want to remove the year and only keep the season. Example: "winter 2021" to become "winter" .
1
vote
1answer
18 views

Defining a function in Pandas [closed]

Hello Data Scientists, Disclaimer: I am a Beginner learning Python and at the same time just trying out a lot of things. I am doing some data extraction on a file with with weekly deaths in ...
0
votes
0answers
8 views

Pytorch torchvision: Efficient way of calculating the mean and stds for images in the train set from datasets.ImageFolder

Let us say I have the loading images from my local files using the pytorch torchvision datasets.ImageFolder as follows: ...

1
2 3 4 5
118