Questions tagged [scipy]

SciPy is a python ecosystem of open-source packages for scientific computation. Some of the core packages include NumPy, SciPy, Matplotlib, IPython, SymPy, and pandas.

Filter by
Sorted by
Tagged with
0 votes
1 answer
14 views

Why does largest eigen value used for eigen vector calcuation?

Why is the largest eigenvalue used for eigenvector calculation? What is the significance of using a largest absolute value for the whitening of a picture?
EMKAY's user avatar
  • 105
0 votes
2 answers
425 views

Solve a non-linear system, in Python, with the GAUSS-NEWTON algorithm? (Jacobian matrix J, etc.)

I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
Zollikofen4's user avatar
0 votes
0 answers
24 views

Python Implementation of ODE for matrix operations

I am trying to develop a model and I am stuck as to how this should be implemented in python using ODE. My entire population is divided into regions and part of the model takes this into consideration ...
Longestail's user avatar
0 votes
0 answers
49 views

How to calculate Pairwise Correlation Difference?

I was trying to implement the pairwise correlation difference (PCD) found in the following paper. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1 PCD measures the ...
Sumon Jafrul's user avatar
1 vote
1 answer
44 views

Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?

I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
zipline86's user avatar
  • 359
0 votes
1 answer
30 views

Optimization problem of find a pair of values (x, y) such that they produce an output z

Problem I am trying to create a process that determines the rent and purchase price of a home that gives me a capitalization rate of 0.08 for instance. Background Normally I calculate the ...
Olek's user avatar
  • 11
0 votes
0 answers
25 views

Interpolation techniques predicting negative values in a column with positive values

I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
Apoorva's user avatar
  • 275
0 votes
0 answers
61 views

Matrix Factorization with SVD/Gradient Descent - Dealing with new users

I'm working on implementing matrix factorization for a recommender system. I implemented matrix factorization, once with stochastic gradient descent and once with scipy's svds() function. The thing ...
0diraf's user avatar
  • 1
0 votes
1 answer
75 views

Cramer von mises test requires too much resources

I am doing some data analysis, and I am using the cramer von mises test to check if two samples are derived from the same distribution. I am using the ...
ptushev's user avatar
  • 11
0 votes
0 answers
7 views

Understanding RSV in WrapCauchy

I am trying to predict using wrapcauchy. This was the best fit when running the fit_transform() method. Unfortunately, I am running into some issues. I keep receiving the error: ...
isabella's user avatar
0 votes
0 answers
109 views

Interpreting chi-square statistic values and also scipy.stats.chisquare giving unreasonable value

I have some data from the S&P500 of daily returns. I'm not sure if I can show my graph, as I will be using it in my undergraduate paper, but it looks essentially the same as the histogram here: ...
probablysid's user avatar
0 votes
1 answer
148 views

How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
hokge's user avatar
  • 1
0 votes
1 answer
91 views

How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
janjilecek's user avatar
3 votes
1 answer
752 views

Incremental clustering algorithm

I am looking for an incremental clustering algorithm. By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
Sirion's user avatar
  • 131
0 votes
1 answer
142 views

How do I print data after fitting data into a pipeline?

I was using 3 functions form scipy: TFIDF vectorizer, Multioutput classifier and Linear SVC. The code goes like this. ...
User123456's user avatar
0 votes
1 answer
97 views

How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?

I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following: from this plot I see that ...
HelpNeederStudent's user avatar
3 votes
1 answer
823 views

Detect max extreme peaks/valleys with min 5% vertical delta

I have been using scipy.signal.argrelextrema but I'm not getting the desired results. My goal is to: identify the peaks/valleys in a 1D timeseries that are a ...
Mark's user avatar
  • 53
1 vote
0 answers
169 views

How to calculate the KL divergence for two multivariate pandas dataframes

I am training a Gaussian-Process model iteratively. In each iteration, a new sample is added to the training dataset (Pandas DataFrame), and the model is re-trained and evaluated. Each row of the ...
guest001's user avatar
0 votes
1 answer
135 views

Optimize a non-linear function in Python

I am trying to optimize a function using scipy.optimize, but it does not converge. I have a trading strategy with a default stop-loss based on the lowest price over 20 days. I want to optimize this ...
Pier-Olivier Marquis's user avatar
1 vote
1 answer
1k views

Using scipy.minimize to find the maximum likelihood estimates for multivariate gaussian

Let's say I have a 100x2 normally distributed array of data. ...
Bazoya's user avatar
  • 31
1 vote
0 answers
49 views

Evaluate Dendrogram Statistical Significance

I have N=21 objects and each one has about 80 possible not NaN descriptors. I carried out a hierarchical clustering on the objects and I obtained this dendrogram. I want some kind of 'confidence' ...
Mirko's user avatar
  • 111
1 vote
0 answers
31 views

Find the right balance between price of a property and agent fee

I would like to know when buying a property when is better for an estate agent to get a higher fee from me compared to the seller if we get a deal with a lower amount. As an example, let's say that: ...
TPPZ's user avatar
  • 111
0 votes
1 answer
259 views

Feature Selection: How to select categorical features in a regression problem

I am reviewing information for feature selection based in filter methods. I got info (link1, link2, link3, link4, link5) for: Numerical input, numerical output Categorical input, categorical output ...
user140259's user avatar
0 votes
3 answers
4k views

p-value of chi squared test is exactly 0.0

I need to do a chi square test of two of my dataset's categorical variables. This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
Michele Papucci's user avatar
1 vote
1 answer
358 views

Why does the 1st derivative appear to lag the slope of the fit in Scipy's Savitzky-Golay filter?

I have a simple script that performs the Savitzky-Golay filter on a toy dataset of forex prices from yahoo finance: ...
quant's user avatar
  • 121
1 vote
1 answer
49 views

Create Period column based on a date column where the first month is 1, second 2, etc

I have a dataset with many project's monthly expendituries (cost curve), like this one: Project Date Expenditure(USD) Project A 12-2020 500 Project A 01-2021 1257 Project A 02-2021 125889 Project ...
conradoov's user avatar
1 vote
1 answer
558 views

Vectorize scipy.stats.norm.logpdf

I am tryint to trying to train a Bayesian NN and at some point I need to compute log-likelihoods for some data points, according to a multivariate diagonal gaussian distribution with parameters (mu, ...
Vellyxenya's user avatar
1 vote
0 answers
14 views

Similarity between binary vector with hierarchal structure

I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category. Each of those categories has a different size e.g. ...
Amit be's user avatar
  • 111
2 votes
1 answer
887 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
johnnydoe's user avatar
  • 141
0 votes
2 answers
90 views

What is '[0]'in boxcox transformation with scipy.stats.boxcox?

While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
Mohamad Alaa Al Zainy's user avatar
0 votes
1 answer
34 views

Multiple Hypotheses in python

I want to write a method to test multiple hypotheses for a pair of schools (say TAMU and UT Austin). I want to consider all possible pairs of words (Research Thesis Proposal AI Analytics), and ...
stacky's user avatar
  • 1
0 votes
0 answers
108 views

Why the error between the measured data and model data is not minimizing in Python?

I want to fit the non-linear experimental data with the model function by estimating some parameters in the function. The model function I have is: ...
N_T's user avatar
  • 1
1 vote
0 answers
54 views

How to create a graph network (Networkx) from the solution of Ordinary differential equations?

I have a list of Nodes. Suppose, N =["1","2","3","4"] # there can be different number of nodes And I have solved the ...
Atom Store's user avatar
2 votes
0 answers
32 views

What would be a good randomization environment for data science?

I would like to know if there are any best practices to optimize random environment. Currently I use this simple structure in my config : ...
Al_P's user avatar
  • 21
0 votes
1 answer
794 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
rb173's user avatar
  • 71
0 votes
0 answers
171 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
Lizardinablizzard's user avatar
0 votes
0 answers
64 views

Scipy kstest problem

I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture ...
Eglantine46's user avatar
0 votes
1 answer
4k views

numpy array center of mass

I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array. For example let's say we have an ...
physics_2015's user avatar
2 votes
1 answer
966 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
Sam.H's user avatar
  • 121
1 vote
1 answer
517 views

How to find the best fitting parametric distribution for an empirical dataset (stock returns)?

Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged. Instead, I ...
develarist's user avatar
1 vote
1 answer
2k views

what is criterion in flcuster of scipy package?

Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?
preetk's user avatar
  • 21
0 votes
1 answer
432 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
Mostafa Atallah's user avatar
1 vote
1 answer
207 views

How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
Josh Purtell's user avatar
3 votes
0 answers
1k views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
dayum's user avatar
  • 131
3 votes
1 answer
370 views

Smaller alternatives to sklearn that doesn't require scipy?

I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies. Sklearn, if you include its dependencies of numpy and scipy is a huge package. Are there any ...
coderboi's user avatar
1 vote
0 answers
68 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
Laurent's user avatar
  • 53
3 votes
0 answers
2k views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
learner's user avatar
  • 359
0 votes
1 answer
2k views

Fitting a pandas dataframe to a Poisson Distribution

I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using: ...
HaneenSu's user avatar
1 vote
0 answers
58 views

Parallelization of a MIMO linear filter

I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation: ...
marco's user avatar
  • 11
1 vote
2 answers
2k views

Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
user421473's user avatar