Questions tagged [scipy]
SciPy is a python ecosystem of open-source packages for scientific computation. Some of the core packages include NumPy, SciPy, Matplotlib, IPython, SymPy, and pandas.
96
questions
0
votes
0
answers
11
views
Custom linkage matrix, Dendrogram labels aren't correct
I created my own linkage function, that produces a linkage matrix with the right format of scipy (n-1x4). When I try to use the dendrogram function with the labels I used for the linkage, the ...
0
votes
0
answers
55
views
why there is discrepancy in the result of code giving different results in matlab and python?
I want to solve the 1-D transient heat transfer equation
Define a 1-D geometry(a line) in y-direction
**dT/dt = (k/(rhocp))d²T/dy²
I.C. : @t=0, temperature of all elements is 25°C
B.C. : The geometry ...
0
votes
1
answer
12
views
How set binary random projection to features, num_samples for X-train, x_test, y_train to match knn distances L dimension
Binary Random Projection of Features, Samples
Creating a binary random projection that will be used in a kNN Hanning function for hamming distances on nearest neighbors that will be processed by ...
0
votes
1
answer
23
views
Why does largest eigen value used for eigen vector calcuation?
Why is the largest eigenvalue used for eigenvector calculation?
What is the significance of using a largest absolute value for the whitening of a picture?
- 105
0
votes
2
answers
626
views
Solve a non-linear system, in Python, with the GAUSS-NEWTON algorithm? (Jacobian matrix J, etc.)
I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
- 59
0
votes
0
answers
27
views
Python Implementation of ODE for matrix operations
I am trying to develop a model and I am stuck as to how this should be implemented in python using ODE. My entire population is divided into regions and part of the model takes this into consideration ...
0
votes
0
answers
133
views
How to calculate Pairwise Correlation Difference?
I was trying to implement the pairwise correlation difference (PCD) found in the following paper.
https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1
PCD measures the ...
1
vote
1
answer
108
views
Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?
I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
- 379
1
vote
1
answer
36
views
Optimization problem of find a pair of values (x, y) such that they produce an output z
Problem
I am trying to create a process that determines the rent and purchase price of a home that gives me a capitalization rate of 0.08 for instance.
Background
Normally I calculate the ...
- 21
0
votes
1
answer
120
views
Cramer von mises test requires too much resources
I am doing some data analysis, and I am using the cramer von mises test to check if two samples are derived from the same distribution. I am using the ...
- 11
0
votes
1
answer
229
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
1
answer
213
views
How to correctly compute the correlation index of a column value from table in Python 3?
I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
3
votes
1
answer
1k
views
Incremental clustering algorithm
I am looking for an incremental clustering algorithm.
By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
- 131
0
votes
1
answer
193
views
How do I print data after fitting data into a pipeline?
I was using 3 functions form scipy: TFIDF vectorizer, Multioutput classifier and Linear SVC.
The code goes like this.
...
- 21
0
votes
1
answer
108
views
How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?
I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following:
from this plot I see that ...
3
votes
1
answer
1k
views
Detect max extreme peaks/valleys with min 5% vertical delta
I have been using scipy.signal.argrelextrema but I'm not getting the desired results. My goal is to: identify the peaks/valleys in a 1D timeseries that are a ...
- 53
1
vote
0
answers
224
views
How to calculate the KL divergence for two multivariate pandas dataframes
I am training a Gaussian-Process model iteratively. In each iteration, a new sample is added to the training dataset (Pandas DataFrame), and the model is re-trained and evaluated. Each row of the ...
- 11
0
votes
1
answer
153
views
Optimize a non-linear function in Python
I am trying to optimize a function using scipy.optimize, but it does not converge. I have a trading strategy with a default stop-loss based on the lowest price over 20 days. I want to optimize this ...
1
vote
1
answer
2k
views
Using scipy.minimize to find the maximum likelihood estimates for multivariate gaussian
Let's say I have a 100x2 normally distributed array of data.
...
- 31
1
vote
0
answers
62
views
Evaluate Dendrogram Statistical Significance
I have N=21 objects and each one has about 80 possible not NaN descriptors.
I carried out a hierarchical clustering on the objects and I obtained this dendrogram.
I want some kind of 'confidence' ...
- 111
1
vote
0
answers
31
views
Find the right balance between price of a property and agent fee
I would like to know when buying a property when is better for an estate agent to get a higher fee from me compared to the seller if we get a deal with a lower amount.
As an example, let's say that:
...
- 111
0
votes
1
answer
278
views
Feature Selection: How to select categorical features in a regression problem
I am reviewing information for feature selection based in filter methods. I got info (link1, link2, link3, link4, link5) for:
Numerical input, numerical output
Categorical input, categorical output
...
- 111
0
votes
3
answers
5k
views
p-value of chi squared test is exactly 0.0
I need to do a chi square test of two of my dataset's categorical variables.
This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
1
vote
1
answer
422
views
Why does the 1st derivative appear to lag the slope of the fit in Scipy's Savitzky-Golay filter?
I have a simple script that performs the Savitzky-Golay filter on a toy dataset of forex prices from yahoo finance:
...
- 121
1
vote
1
answer
50
views
Create Period column based on a date column where the first month is 1, second 2, etc
I have a dataset with many project's monthly expendituries (cost curve), like this one:
Project
Date
Expenditure(USD)
Project A
12-2020
500
Project A
01-2021
1257
Project A
02-2021
125889
Project ...
- 13
1
vote
1
answer
680
views
Vectorize scipy.stats.norm.logpdf
I am tryint to trying to train a Bayesian NN and at some point I need to compute log-likelihoods for some data points, according to a multivariate diagonal gaussian distribution with parameters (mu, ...
- 11
1
vote
0
answers
14
views
Similarity between binary vector with hierarchal structure
I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category.
Each of those categories has a different size e.g. ...
- 111
2
votes
1
answer
1k
views
Stemming/lemmatization for German words
I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
- 141
0
votes
2
answers
110
views
What is '[0]'in boxcox transformation with scipy.stats.boxcox?
While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
0
votes
1
answer
37
views
Multiple Hypotheses in python
I want to write a method to test multiple hypotheses for a pair of schools (say TAMU and UT Austin). I want to consider all possible pairs of words (Research Thesis Proposal AI Analytics), and ...
- 1
0
votes
0
answers
124
views
Why the error between the measured data and model data is not minimizing in Python?
I want to fit the non-linear experimental data with the model function by estimating some parameters in the function. The model function I have is: ...
- 1
1
vote
0
answers
57
views
How to create a graph network (Networkx) from the solution of Ordinary differential equations?
I have a list of Nodes.
Suppose,
N =["1","2","3","4"] # there can be different number of nodes
And I have solved the ...
- 103
2
votes
0
answers
34
views
What would be a good randomization environment for data science?
I would like to know if there are any best practices to optimize random environment. Currently I use this simple structure in my config :
...
- 21
0
votes
1
answer
960
views
How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?
np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
- 71
0
votes
0
answers
188
views
2D Z-score/Mahalanobis distance that includes a penalty for uncertainty
I have some 2D points and I want to assess their performance against the target point.
When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
0
votes
0
answers
68
views
Scipy kstest problem
I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture ...
0
votes
1
answer
4k
views
numpy array center of mass
I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array.
For example let's say we have an ...
- 229
2
votes
1
answer
979
views
Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here.
I first chose columns to work with:
...
- 121
1
vote
1
answer
680
views
How to find the best fitting parametric distribution for an empirical dataset (stock returns)?
Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged.
Instead, I ...
- 214
1
vote
1
answer
2k
views
what is criterion in flcuster of scipy package?
Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?
- 21
0
votes
1
answer
529
views
Softmax regression cost function code [closed]
I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()
The code is related to calculating the ...
1
vote
1
answer
237
views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
- 178
3
votes
0
answers
1k
views
Scipy minimization failing with inequality constraints or bounds [closed]
I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
- 131
3
votes
1
answer
553
views
Smaller alternatives to sklearn that doesn't require scipy?
I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies.
Sklearn, if you include its dependencies of numpy and scipy is a huge package.
Are there any ...
- 133
1
vote
0
answers
68
views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
- 51
3
votes
0
answers
3k
views
Converting pandas dataframe to scipy sparse arrays
Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning.
Now, if my pandas' data frame ...
- 359
0
votes
1
answer
2k
views
Fitting a pandas dataframe to a Poisson Distribution
I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using:
...
- 85
1
vote
0
answers
59
views
Parallelization of a MIMO linear filter
I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation:
...
- 11
1
vote
2
answers
2k
views
Differential Evolution optimal tolerance parameter
I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
- 11
0
votes
1
answer
66
views
Why i am getting t statistics different
import scipy.stats as st
st.t.ppf(0.95, df=9)
Output: 1.8331129326536335
As per this, I should get 2.262 with dof = 9.
