Skip to main content

Questions tagged [numpy]

NumPy is one of the fundamental packages used for scientific computing in the Python ecosystem. Specifically, NumPy is used for numerical computing and uses n-dimensional arrays.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Efficiently Implement a Data Transformation || Pytorch || coding advice

I have created the following class and function to create a 'custom' dataset. The use case was to sample every nth image from a data set to reduce the size of the data set while still maintaining ...
0 votes
2 answers
875 views

Python make_scorer giving incorrect outputs for Root Mean Square Logarithmic Error

I want to generate a grid search for which I need the scoring parameter based on which the search will take place. I have defined the following function to provide ...
0 votes
3 answers
4k views

How to add noise to supervised (binary-classifier)?

Note: The question is not about validating/testing a trained model. Say i have an unlabeled features set, I want to approximate the true labels (for the sake of argument lets assume it's a binary ...
0 votes
1 answer
158 views

Remove outlier from image array in Python

Could someone please suggest me what would be the best way to remove such huge number of outlier data from the image. The regular clipping between data range in numpy array would simply reduce the ...
3 votes
1 answer
233 views

tensorflow pseudo inverse doesn't work for complex matrices!

The Tensorflow documentation here says that: tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in default value of rcond''. However, tf.linalg.pinv requires the matrix to ...
1 vote
1 answer
469 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
1 vote
2 answers
1k views

Trying to convert a .txt file into .npy file to split and use for train/validation/test

I have a large data set in .txt form, and was trying to split it into three for testing, validation, and training by transforming the data into a ...
1 vote
1 answer
61 views

How does time needed for training differ between different batch sizes?

I've constructed a CNN in Python using Numpy, which is trained with mini batch gradient descent for MNIST digit classification. When training with a batch size of 1, the time needed for 5 epochs is ...
0 votes
1 answer
525 views

How can I convert a numpy array of tensors to tensor of tensors?

It is my first GRU model so pardon the stupidity. I am trying to learn by training a simple GRU network on variable length sequences. The sequences are numpy arrays of tensors. The length of numpy ...
0 votes
0 answers
17 views

Calculation of Covariance Matrices for a QDA classifier in Python Numpy

For a school project, I have to design a QDA classifier for 28x28 pixel images of letters in sign language. I have been given 27455 images for training, which have to be flattened to a 784 pixel ...
0 votes
1 answer
57 views

Sequence to sequence RNN model, maximum number of training size

So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or ...
2 votes
1 answer
253 views

Not able to encode multiple categorical columns at once

I have written the following code for encoding categorical features of the dataframe( named 't') - ...
0 votes
1 answer
1k views

Python: calculate the weighted average correlation coefficient

I am calculating the volatility (standard deviation) of returns of a portfolio of assets using the variance-covariance approach. Correlation coefficients and asset volatilities have been estimated ...
0 votes
0 answers
19 views

Generating a Sample T-Distribution in Python

I'm new to DataScience. I'm going through the stats module and the ways to create a sample T-distribution. Using the below line to generate the sample. However, my question is how is the array ...
0 votes
1 answer
625 views

How to group by one column and find second occurance of values greater than a threshold

The Problem I have a pandas dataframe that contains series of people, the week number that a visit occurred, and their systolic and diastolic blood pressures. ...
0 votes
0 answers
8 views

Add a custom loss in a public github repository on distillation

I’m trying to implement a custom loss in a public repository regarding knowledge distillation. The link to the repository is the following: " https://github.com/DefangChen/SimKD " The main ...
0 votes
1 answer
91 views

Counting values from diffrent columns

I'm working with a pandas dataframe that has some columns like these: ...
0 votes
1 answer
1k views

ValueError: Found input variables with inconsistent numbers of samples: [6, 366]

I'm trying to split my x and y into train and test data for my ML model but it's giving me this error: ...
1 vote
2 answers
440 views

Reshaping Numpy arrays

I am having a problem trying to reshape my numpy array. i have a 2 dimensional array and i want to make it a 3D array. I have looked everywhere but even if i find some answering my question they just ...
2 votes
1 answer
569 views

Integration of NLP and Angular application

I'm doing a small POC in which I've trained my Machine Learning model (Naive Bayes) and is saved in ".pkl" (pickle) format. Now my next task is to develop a web application which asks the user to ...
0 votes
1 answer
80 views

Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
0 votes
1 answer
22 views

Why does largest eigen value used for eigen vector calcuation?

Why is the largest eigenvalue used for eigenvector calculation? What is the significance of using a largest absolute value for the whitening of a picture?
0 votes
1 answer
340 views

How to convert postgresql tabular data to a numpy array?

I'm completely new to data science and I have a problem that I need help in resolving. I have time series data (with 87 million rows currently, though that will grow) with x, y coordinates, a ...
0 votes
1 answer
201 views

Cross Validation with One Class Classification in Python

I'm trying to do cross-validation with One Class Classification - I'm using the PyOD lib - but I don't know if I'm doing it right. The precision is too low and I'm also not able to bring up the mean ...
-1 votes
1 answer
24 views

TypeError: unhashable type: 'slice' K-Means; Custom code for K-Means

Problem Statement The goal is to have the K-Means customer code run for clusters and not use scikit-learn libraries. Learning exercise. This K-means has the standard predict, fix, centroids, cluster ...
1 vote
1 answer
405 views

How to find average lag time with variance & confidence of two time series

I have two variables as time series, one a consequent of the other, I would like to find the average time delay it takes the dependent variable to act on the independent variable. Additionally, I ...
1 vote
2 answers
401 views

Pre-process data images before training OneClassSVM and decrease number of features

I want to train a OneClassSVM() using sklearn, and I have a set of around 800 images in my training set. I am using opencv to read the images and resize them to constant dimensions (960x540) and then ...
2 votes
1 answer
736 views

Model prediction on meshgrid in python

Suppose I have data with two independent variable $X_1$, $X_2$ and one dependent variable say $y$, as follows: $X_1$: $x_{1,1}$, $x_{1,2}$ , $x_{1,3}$ $X_2$: $x_{2,1}$, $x_{2,2}$, $x_{2,3}$ $y$: $...
9 votes
2 answers
389 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
0 votes
1 answer
480 views

Pandas filtering with np.where as filter not working as expected

working on Kaggle's SpaceShip Titanic dataset and I was doing some analysis of the data. Call tt the pd.DataFrame containing the ...
0 votes
2 answers
492 views

Solve a non-linear system, in Python, with the GAUSS-NEWTON algorithm? (Jacobian matrix J, etc.)

I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
0 votes
1 answer
251 views

How do I import and export more than one numpy arrays?

I have four numpy arrays X_train, y_train, X_test, y_test...
0 votes
1 answer
1k views

How to revert np.log(data) and data.diff()?

I have used np.log(data) and then applied data.diff() to transform my data in timeseries model. I have the predictions. How do I ...
2 votes
1 answer
1k views

Keras intermediate layer (attention model) output

I have a model with this summary: ...
1 vote
1 answer
596 views

Vectorize scipy.stats.norm.logpdf

I am tryint to trying to train a Bayesian NN and at some point I need to compute log-likelihoods for some data points, according to a multivariate diagonal gaussian distribution with parameters (mu, ...
0 votes
0 answers
11 views

Passing the Parallel API tests in PettingZoo for custom multi-agent environment

from pettingzoo.test import ( parallel_api_test, parallel_seed_test, max_cycles_test, performance_benchmark, ) I have a custom multiagent ...
1 vote
2 answers
331 views

When should we use jaccard score?

I am a newbie in Machine Learning, I trained a binary classifier for bank loan prediction through Logistic Regression. I measured the accuracy of it with two methods: accuracy score and jaccard index. ...
1 vote
0 answers
233 views

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int) in Python - from_gerator fails

I know this question has answers, but they are not working. why this fails ? (and why tensorflow has troubles in reading numpy!!) ...
0 votes
0 answers
25 views

Python Implementation of ODE for matrix operations

I am trying to develop a model and I am stuck as to how this should be implemented in python using ODE. My entire population is divided into regions and part of the model takes this into consideration ...
1 vote
1 answer
36 views

Keras Model Produces Different Outputs, But Outputs Map To The Same Vector

I have a neural network that produces output vectors from input vectors. These output vectors are different depending on what input vector it is being asked to predict for. I have a dictionary that ...
1 vote
1 answer
93 views

Data Skewness is nan or inf

I have checked the Skewness of my data before applying a Log transformation using the next code : print("Skewness: %f" % df['Wind Speed (km/h)'].skew()) ...
0 votes
0 answers
14 views

Array column in dataframe as input for classification models

Please consider the following dataframe: My challenge is with the column 'geometry_coordnates'. I've got a column 'geometry_coordnates' that is an array of numbers. It can be a representation of a ...
0 votes
0 answers
18 views

df['col'] refactoring

I made a big mistake by using df.iterrows() in a computationally heavy project. Now I need to convert it to np loop and need to rename all df['col'] occurrences to np[i]. How can I do it not manually, ...
0 votes
0 answers
162 views

Batch processing for a dictionary of dictionaries Python

I have a very large dictionary of the form vf = {key1:{...}, key2:{...}, key3:{...}, ...}, which is stored as an npz file. Trying to load it at once causes a memory error, so I tried to load it in ...
0 votes
1 answer
74 views

why np.cos returns erroneous values?

I'm debugging my python program. I have np.cos(param_t_10[0]) returns 5.9168e-06 where ...
0 votes
0 answers
117 views

How to deal with "Could not broadcast input array from shape (1141,2) into shape (1141,)" to get Out-of-Bag error while using Random Forest

I have a dataset that consists of 171 features and 39 labels. I captured both features and labels of the dataset through slicing: ...
0 votes
1 answer
30 views

Trying to write a code that detects a maximum value before a drop off in slope

I'm trying to write code for a force test that will output the maximum force before structural failure occurs. I'm a bit of a novice to python, so the issue here might be something simple that I'm ...
0 votes
1 answer
62 views

Why most keras examples use shape=(None,) instead of shape=(None)?

I'm reading several keras tutorials and found that many examples are written like this: keras.Input(shape=(None,), dtype="int64", name="english")...
0 votes
0 answers
38 views

What is numpy.dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]) in numpy.dot() documentation?

I was trying to understand difference between np.dot and np.matmul and in the docs of np.dot() there is this code dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]). ...
0 votes
0 answers
14 views

How to solve MemoryError problem?

I have audio signals which I need to convert them to melspectrogram. I am using Deepfilternet to remove background noise. when I use the output of Deepfilternet for next phase like padding, it showed ...

1
2 3 4 5
8