Questions tagged [scipy]

SciPy is a python ecosystem of open-source packages for scientific computation. Some of the core packages include NumPy, SciPy, Matplotlib, IPython, SymPy, and pandas.

Filter by
Sorted by
Tagged with
0
votes
1answer
36 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
0
votes
0answers
32 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
0
votes
0answers
25 views

Scipy kstest problem

I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture ...
0
votes
0answers
19 views

how to detect bifurcation, trifurcation, and crossover on image with python opencv

how to detect bifurcation, trifurcation, and crossover from the branch point with OpenCV when the input like this Expect output : I'm trying to use binary_hit_or_miss but I don't know how to display ...
0
votes
1answer
167 views

numpy array center of mass

I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array. For example let's say we have an ...
0
votes
0answers
14 views

What are practical benefits of the various distance metrics in scipy?

I’m looking for a information about distance metrics. Python’s scipy gives many different metrics, and I’d like to know more about their practical use. In particular, I’m trying to understand the ...
0
votes
0answers
12 views

Rating Sports Teams: How to perform least squares minimization with a constraint?

I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like: ...
2
votes
1answer
337 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
1
vote
1answer
62 views

How to find the best fitting parametric distribution for an empirical dataset (stock returns)?

Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged. Instead, I ...
1
vote
1answer
82 views

what is criterion in flcuster of scipy package?

Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?
0
votes
1answer
82 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
1
vote
1answer
35 views

How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
0
votes
0answers
16 views

Python: significance test (claim), one sample, test the proportion - simpler code

We have two sheets with pictures of dogs and dog owners. On one sheet - owners and their dogs are associated correctly. The other sheet shows random associations between owners and dogs. 61 students ...
3
votes
0answers
344 views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
3
votes
1answer
38 views

Smaller alternatives to sklearn that doesn't require scipy?

I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies. Sklearn, if you include its dependencies of numpy and scipy is a huge package. Are there any ...
1
vote
0answers
43 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
3
votes
0answers
693 views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
0
votes
1answer
622 views

Fitting a pandas dataframe to a Poisson Distribution

I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using: ...
1
vote
0answers
21 views

Parallelization of a MIMO linear filter

I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation: ...
1
vote
2answers
401 views

Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
0
votes
1answer
31 views

Why i am getting t statistics different

import scipy.stats as st st.t.ppf(0.95, df=9) Output: 1.8331129326536335 As per this, I should get 2.262 with dof = 9.
1
vote
0answers
22 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
1
vote
0answers
11 views

Scipy.optimize getting back x0 for optimization of input to recurrent neural network

I need help with this optimization problem which is either not getting solved at all or is taking a copious amount of time. I am trying to find optimized input to an RNN (GRU) model of a process ...
1
vote
0answers
146 views

Scipy Sparse vstack memory error

I have a bunch of scipy matrices (of the same #columns) loaded from disk. I want to combine them into one scipy sparse matrix. I am using scipy sparse vstack method. I am able to load the ...
2
votes
1answer
52 views

Which is the best algorithm for entity extraction for unstructured document

I have unstructured documents from which I have to extract the information like let buyer name, seller name, expiry date, buying date etc. I had planned to use spacy(Custom entity recolonization(...
2
votes
1answer
23 views

optimizing a linear optimization function with linear constarints and binary variables

I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization. I have an ojective function that needs to be maximized ...
1
vote
1answer
224 views

How to perform convolution with kernel bigger than image?

In this question I've seen an example of convolution by the kernel with the shape bigger than initial image's one: ...
2
votes
0answers
59 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
1
vote
1answer
504 views

Fastest batch perspective transform for image augmentation

I need to do some augmentation for my training images for a neural networks. The problem is that even when loading batches in parallel, the augmentation is taking longer to perform than the network ...
3
votes
2answers
55 views

How to interpret ANOVA results?

I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier. The target is a categorical variable with three different values. I also have a lot ...
1
vote
1answer
1k views

How do I force specified coefficients in a Linear Regression model to be positive?

Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive. Some additional context: I have a data frame with a some explanatory variables and a response ...
1
vote
0answers
526 views

Two different results from seaborn and matplotlib for a kde plot. What's the reason?

I'm plotting kde plots for a set of data with seaborn and scipy.stats.gaussian_kde: My understanding is that under the hood, seaborn uses scipy (see here). It also seems as though the bandwidth ...
0
votes
1answer
30 views

Probabilities of a Poisson distribution not making sense

I am trying to find probabilities of orders a restaurant gets on Sunday's. For last 6 months average orders are 1000 without any big anomalies like 700 or 1300. This is a case of poisson distribution &...
5
votes
1answer
89 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
2
votes
2answers
531 views

Fitting a distribution to data

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
2
votes
2answers
632 views

Can I use scipy.optimize module with PyTorch?

I need to use scipy.optimize module after encoding some data with pytorch. However, scipy.optimize does not take torch.tensor as its input. How can I use scipy.optimize module with data with gradient ...
2
votes
1answer
2k views

Which method to use to remove trend from time series?

From what I understand, differencing is necessary to remove the trend and seasonality of a time series. So I assumed it basically does the same thing as signal.detrend from the scipy library. But I ...
2
votes
1answer
2k views

scipy.optimize.minimize(method=’trust-constr’) doesn't terminate on xtol condition

I have set up an optimization problem with linear equality constraints as follows ...
1
vote
1answer
2k views

Scipy curve_fit and method “dogbox”

I am trying to duplicate this papers feature engineering for user activity. They take 14 days of accumulated user activity and keep the parameters (2 parameters) that fit a sigmoid to it. I would like ...
1
vote
1answer
2k views

How to measure the correlation between categorical variables and a continuous variable

I have the following list of the names of the categorical variables in my dataset: ...
2
votes
0answers
195 views

Orange3 / python script / scipy interpn

I tried to create a learner using the Orange canvas script from the Python script widget, but I have not been able to find an effective example. What I would like to do is: 1) Create a learner based ...
0
votes
1answer
3k views

Python: multiplication of sparse matrices slower in csr_matrix than numpy

I am trying to multiply a sparse matrix with itself using numpy and scipy.sparse.csr_matrix. The size of matrix is 128x256. Its 93% values are 0. Ironically the multiplication using numpy is faster ...
1
vote
0answers
35 views

Area Under Curve 5 Year Prediction [closed]

I have a dataset that includes a ratio(y-axis) and tenure(x-axis). My goal is to take the training set and establish a learning curve by tenure (1 to 15 years) which looks like this: Actual Group ...
1
vote
0answers
42 views

Semantic Segmentation using Scipy or Scikit-image

Many days ago I saw a user using scipy function to get region of interest (RoI) in an image using different colors. I believe it was some sort of filter. Now I'm trying to do the same but I'm not able ...
0
votes
2answers
4k views

How do I properly write scipy.stats.binom.cdf() details

I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$). How do I properly ...
6
votes
3answers
130 views

What is the best algorithm/solution for predicting the following?

I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country: If I were to take a random sample ...
2
votes
0answers
3k views

How to make scipy.optimize.basinhopping find the global optimal point

Question Try to find the global optimal point of the function (reading Python for finance 2nd edition - Chapter 11. Mathematical Tools). ...
1
vote
1answer
2k views

How to understand the return values of scipy.interpolate.splrep

Background Continuation of Spline interpolation - why cube with 2nd derivative as following Cubic Spline Interpolation in youtube. The example in the youtube is below. Implemented using scipy....
1
vote
1answer
2k views

What does sklearn's pairwise_distances with metric='correlation' do?

I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed. Then I tried digging through the function itself, but its ...
1
vote
0answers
27 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?