Questions tagged [numpy]

NumPy is a scientific and numerical computing extension to the Python programming language.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Handling numpy data in highly frequently updated environment

Having the following environment: Where the input rows (150 million per day) are getting directed into 500 independent DataFrames (300k rows per DataFrame). In ...
1
vote
0answers
24 views

Stratified K fold same index present in both test and valid set

I am trying to do a stratified k fold cross validation for my dataset and want to keep an isolated 10% test set from the dataset and use the remaining for training and validation. Below is the code ...
-2
votes
1answer
24 views
1
vote
1answer
13 views

Random Parameters to fix in ML to perform controlled experiments

Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy ...
3
votes
1answer
59 views

tensorflow pseudo inverse doesn't work for complex matrices!

The Tensorflow documentation here says that: tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in default value of rcond''. However, tf.linalg.pinv requires the matrix to ...
1
vote
0answers
37 views

How to apply a groupby rolling function to create multiple columns in the dataframe

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo and the link to the data is here and the example here. Some Sample of data ...
2
votes
1answer
25 views

Combine 100 2D arrays into one 3D array

I have 100 2D lists a0 to a99, it is easy to convert them into arrays (I verified already): ...
2
votes
2answers
282 views

Ways to simulate weather data over several periods (Python or R)?

I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding ...
1
vote
1answer
32 views

Implementing “full convolution” to find gradient w.r.t the convolution layer inputs

I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this: So, I wrote this function: ...
5
votes
4answers
614 views

Convert Numpy array with 'n' and 'y' into integer array of 0 and 1

I have a NumPy array of strings: 'n', 'y', wanna convert it into integer array of 0, 1, how to convert it? ...
2
votes
1answer
36 views

Machine learning dataframe dimension concept vs NumPy dimension

From Machine Learning for Absolute Beginners: A Plain English Introduction: Contained in each column is a feature. A feature is also known as variable, a dimension or an attribute - but they all ...
-1
votes
1answer
72 views

NumPy — ImportError: PyInit__multiarray_umath [closed]

I'm facing the below error while importing NumPy library. I have tried all of the suggestions advised by Python in its error log, but no luck. Strangely, when I use Python's OS library and print the "...
0
votes
0answers
14 views

ValueError while Comparing with Multilevel Index

My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
0
votes
0answers
12 views

Input shape error for multi label segmentation

I am trying to perform multi-class segmentation and when I try to feed the model with my input images and model it get this error: ...
1
vote
1answer
24 views

Pandas dataframe with multiple observations per model

I currently have a pandas dataframe with the following format ...
0
votes
1answer
63 views

Trouble in calculating the covariance matrix

I'm trying to calculate the covariance matrix for a dummy dataset using the following formula, but it's not matching with the actual result. Let's say the dummy dataset contains three features, #...
0
votes
0answers
13 views

Group a dataframe or numpy array with cumulative value effectively

Suppose we have an array x = [1, 2, 1, 3, 2, 4, 0, 1, 1, 3] I want to divide it into sub-groups sequentially and in each sub-group the summation is just larger ...
1
vote
0answers
22 views

How to do Back Propagation Updation for below code?

Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code: Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be ...
1
vote
1answer
22 views

How should I improve my Vectorized Gradient descent linear regression model?

I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like: It's Not Working properly as I am getting negative R Squared error I don't ...
0
votes
0answers
29 views

How the Pandas Dataframe and Numpy masking technique works

I'm trying to understand how the masking technique works. I've a dataframe with some values (say one_row_incidents) some of the are not unique. By grouping them .by the duplicate index, I can get the ...
2
votes
0answers
15 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
0
votes
1answer
27 views

empty fields to be filled with previous values

In the above sample data, I have empty fields and now the task is to fill the fields with previous values. my columns are dates and the values are a number of items present for that particular article ...
5
votes
0answers
69 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
0
votes
0answers
17 views

What types of variables are these? And what types of tests could I run?

So, I recently made a sheet of data collected from the 2010 Census on Cincinnati neighborhoods. The data is from demographics about the neighborhoods of the city such as median household income and ...
1
vote
0answers
46 views

Reshape Time series data for Conv2d Block

I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial. ...
0
votes
0answers
35 views

Data Analysis on normally distributed salary with Outliers

I have created normally distributed data with 2 outliers ...
0
votes
0answers
38 views

sess.run in losses.py

now I want to use sess.run in losses.py so that complex codes can be written by python style for customizing error function as below; ...
1
vote
0answers
11 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
0
votes
0answers
51 views

For loop and assignment procedure in losses.py for customizing loss function of ML model

I want to customize a loss function for neural networks. A computation of the loss function requires the inclusion of a for loop like below. However, it is super difficult to implement because ...
0
votes
0answers
18 views

Fastest way to generate mini-batches from structured data

To generate mini-batches for training, I use a for loop to extract slices of a 2-D numpy array. Here is some example code I wrote to demonstrate this. Basically I loop through a 2-D numpy array ...
1
vote
0answers
33 views

Method for 'Simple' Clustering?

In a recent adult education class I took the prof shared a spreadsheet that was written in VBA code and takes ages to execute, I'm sure ...
2
votes
1answer
59 views

Numpy arithmetic operation between two columns

Below is the numpy array. I need to perform two operations on this array. Add one column with value [column 1] - [column 3]. Add another column with value [column 1] - [previous value of column1]. I ...
1
vote
1answer
39 views

Linear regression compute theta

I'm trying to compute the theta for a regression linear exercice. ...
0
votes
0answers
37 views

Backpropagation incorrect only for hidden layer bias node

I am trying to implement a neural network in Python, however I am having some trouble when it comes to the implementation of backpropagation. I have been checking my results with this example and ...
0
votes
1answer
135 views

Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization

I'm new to programming with Python, and so far it's been a headache to create a build environment- and need your support and expertise in this area. Background I'm running a MacBook, and using ...
0
votes
0answers
53 views

IndexError: too many indices for array

Attempting to load a dataset and print out a prediction for each array, yet I keep getting an IndexError.... ...
3
votes
0answers
11 views

Feeding uint8 type to NN

I’m facing some (RAM) memory issues to train a neural network. I have an input array consisting of grayscale images encoded as the type numpy.uint8 (therefore the whole range 0-255 can be covered). ...
1
vote
1answer
688 views

how to check all values in particular column has same data type or not?

I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. ...
2
votes
0answers
15 views

How do I plot between 2 variables in a summation equation?

I am new to Python and using Jupyter notebook. Following is the summation equation that I have: $$y(t) = \sum_{i=1}^{t-1} a_{i}y(t-i)$$ How can I plot between variables y and t by formulating this ...
0
votes
2answers
136 views

What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm. ...
3
votes
1answer
38 views

How to deal with memory exceptions while processing large amount of data?

Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps: Take backup of the file Extract new records of previous day ...
1
vote
0answers
127 views

Would writing a decision tree algorithm in Pytorch or Tensorflow be faster than with Numpy?

Since these libraries can turn CPU arrays into GPU tensors, could you parallelize (and therefore accelerate) the calculations for a decision tree? I am considering making a decision tree class written ...
2
votes
1answer
43 views

Data sets that have strings and numerical data all in one column

I m new to the field of Data Science, but I have a great passion for it. Here's a code I wrote for pre processing a data set. It works ...
1
vote
0answers
37 views

ValueError when reshaping to 4 dim array for the Keras API

DL beginner here. I'm trying to implement LeNet using Keras and apply it on good ol' MNIST. ...
0
votes
1answer
64 views

Subtract Rows of Matrix from rows of another matrix numpy

I have two matrix V_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a ...
1
vote
1answer
39 views

normal conda install vs forge install

In order to install some libraries, we just use: conda install pandas While in other cases we use: ...
0
votes
0answers
40 views

Consecutive multiplication of 3*3 matrix with the 3*1 matrix until a specific condition is met ( pagerank algorithm problem)

I have a 3*3 transition matrix and another 3*1 matrix. On multiplying these two, I get another 3*1 matrix. I must multiply the new 3*1 matrix with the original 3*3 matrix and this will give me another ...
0
votes
1answer
201 views

Getting a transition matrix from a Adjacency matrix in python

I have a 3*3 Adjacency matrix and I'm trying to sum the elements of each column and divide each column element by that sum to get the transition matrix. Can you please help me code this part? Thanks ...
1
vote
1answer
124 views

np.shape[::-1] reversal of dims and np.moveaxis splitting array

I cannot find anything in the documentation but it was used in some starter code for a class I am taking at school. Upon testing per below, it seems to reverse the order of the dimensions of an numpy ...
2
votes
0answers
57 views

Neural network cost is constant never changing during training

I am trying to build a binary classifier to predict a pulsar star with Single Hidden layer Neural Network. But the cost on training dataset after almost 100 iterations has no change, following is the ...