Questions tagged [scipy]
SciPy is a python ecosystem of open-source packages for scientific computation. Some of the core packages include NumPy, SciPy, Matplotlib, IPython, SymPy, and pandas.
92
questions
0
votes
0
answers
16
views
Interpolation techniques predicting negative values in a column with positive values
I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
- 275
0
votes
0
answers
18
views
Matrix Factorization with SVD/Gradient Descent - Dealing with new users
I'm working on implementing matrix factorization for a recommender system. I implemented matrix factorization, once with stochastic gradient descent and once with scipy's svds() function. The thing ...
- 1
0
votes
1
answer
54
views
Cramer von mises test requires too much resources
I am doing some data analysis, and I am using the cramer von mises test to check if two samples are derived from the same distribution. I am using the ...
- 11
0
votes
0
answers
6
views
Understanding RSV in WrapCauchy
I am trying to predict using wrapcauchy. This was the best fit when running the fit_transform() method.
Unfortunately, I am running into some issues. I keep receiving the error:
...
- 1
0
votes
0
answers
68
views
Interpreting chi-square statistic values and also scipy.stats.chisquare giving unreasonable value
I have some data from the S&P500 of daily returns. I'm not sure if I can show my graph, as I will be using it in my undergraduate paper, but it looks essentially the same as the histogram here:
...
0
votes
0
answers
17
views
Scipy WrappedCauchy isn't wrapping when loc != 0
I want to plot the PDF of a wrapped Cauchy distribution, but when I set a value for the location, the entire distribution gets shifted, rather than wrapped.
When I run this:
...
- 53
0
votes
1
answer
101
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
1
answer
39
views
How to correctly compute the correlation index of a column value from table in Python 3?
I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
3
votes
1
answer
378
views
Incremental clustering algorithm
I am looking for an incremental clustering algorithm.
By incremental I mean an algorithm that builds clusters starting from an initial dataset and that is able to progressively ingest new items/...
- 131
0
votes
1
answer
91
views
How do I print data after fitting data into a pipeline?
I was using 3 functions form scipy: TFIDF vectorizer, Multioutput classifier and Linear SVC.
The code goes like this.
...
- 21
0
votes
1
answer
68
views
How can I obtain the mean of a Poisson distribution given the first improbable point of the distribution?
I generated a Poisson distribution with mean equal to 3 and 10000 samples by using np.random.poisson(3,10000). The plot is the following:
from this plot I see that ...
3
votes
1
answer
524
views
Detect max extreme peaks/valleys with min 5% vertical delta
I have been using scipy.signal.argrelextrema but I'm not getting the desired results. My goal is to: identify the peaks/valleys in a 1D timeseries that are a ...
- 53
1
vote
0
answers
121
views
How to calculate the KL divergence for two multivariate pandas dataframes
I am training a Gaussian-Process model iteratively. In each iteration, a new sample is added to the training dataset (Pandas DataFrame), and the model is re-trained and evaluated. Each row of the ...
- 11
0
votes
1
answer
108
views
Optimize a non-linear function in Python
I am trying to optimize a function using scipy.optimize, but it does not converge. I have a trading strategy with a default stop-loss based on the lowest price over 20 days. I want to optimize this ...
1
vote
1
answer
969
views
Using scipy.minimize to find the maximum likelihood estimates for multivariate gaussian
Let's say I have a 100x2 normally distributed array of data.
...
- 31
1
vote
0
answers
41
views
Evaluate Dendrogram Statistical Significance
I have N=21 objects and each one has about 80 possible not NaN descriptors.
I carried out a hierarchical clustering on the objects and I obtained this dendrogram.
I want some kind of 'confidence' ...
- 11
1
vote
0
answers
30
views
Find the right balance between price of a property and agent fee
I would like to know when buying a property when is better for an estate agent to get a higher fee from me compared to the seller if we get a deal with a lower amount.
As an example, let's say that:
...
- 111
0
votes
1
answer
226
views
Feature Selection: How to select categorical features in a regression problem
I am reviewing information for feature selection based in filter methods. I got info (link1, link2, link3, link4, link5) for:
Numerical input, numerical output
Categorical input, categorical output
...
- 111
0
votes
3
answers
3k
views
p-value of chi squared test is exactly 0.0
I need to do a chi square test of two of my dataset's categorical variables.
This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
1
vote
1
answer
293
views
Why does the 1st derivative appear to lag the slope of the fit in Scipy's Savitzky-Golay filter?
I have a simple script that performs the Savitzky-Golay filter on a toy dataset of forex prices from yahoo finance:
...
- 121
1
vote
1
answer
46
views
Create Period column based on a date column where the first month is 1, second 2, etc
I have a dataset with many project's monthly expendituries (cost curve), like this one:
Project
Date
Expenditure(USD)
Project A
12-2020
500
Project A
01-2021
1257
Project A
02-2021
125889
Project ...
- 13
1
vote
1
answer
412
views
Vectorize scipy.stats.norm.logpdf
I am tryint to trying to train a Bayesian NN and at some point I need to compute log-likelihoods for some data points, according to a multivariate diagonal gaussian distribution with parameters (mu, ...
- 11
1
vote
0
answers
13
views
Similarity between binary vector with hierarchal structure
I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category.
Each of those categories has a different size e.g. ...
- 111
1
vote
1
answer
656
views
Stemming/lemmatization for German words
I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
- 131
0
votes
2
answers
79
views
What is '[0]'in boxcox transformation with scipy.stats.boxcox?
While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
0
votes
1
answer
30
views
Multiple Hypotheses in python
I want to write a method to test multiple hypotheses for a pair of schools (say TAMU and UT Austin). I want to consider all possible pairs of words (Research Thesis Proposal AI Analytics), and ...
- 1
0
votes
0
answers
92
views
Why the error between the measured data and model data is not minimizing in Python?
I want to fit the non-linear experimental data with the model function by estimating some parameters in the function. The model function I have is: ...
- 1
1
vote
0
answers
52
views
How to create a graph network (Networkx) from the solution of Ordinary differential equations?
I have a list of Nodes.
Suppose,
N =["1","2","3","4"] # there can be different number of nodes
And I have solved the ...
- 103
2
votes
0
answers
30
views
What would be a good randomization environment for data science?
I would like to know if there are any best practices to optimize random environment. Currently I use this simple structure in my config :
...
- 21
0
votes
1
answer
626
views
How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?
np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
- 71
0
votes
0
answers
158
views
2D Z-score/Mahalanobis distance that includes a penalty for uncertainty
I have some 2D points and I want to assess their performance against the target point.
When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
0
votes
0
answers
63
views
Scipy kstest problem
I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture ...
0
votes
1
answer
3k
views
numpy array center of mass
I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array.
For example let's say we have an ...
- 129
2
votes
1
answer
959
views
Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here.
I first chose columns to work with:
...
- 21
1
vote
1
answer
370
views
How to find the best fitting parametric distribution for an empirical dataset (stock returns)?
Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged.
Instead, I ...
- 214
1
vote
1
answer
1k
views
what is criterion in flcuster of scipy package?
Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?
- 21
0
votes
1
answer
357
views
Softmax regression cost function code [closed]
I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()
The code is related to calculating the ...
1
vote
1
answer
166
views
How should I sample from a mixture distribution?
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
- 178
3
votes
0
answers
1k
views
Scipy minimization failing with inequality constraints or bounds [closed]
I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
- 131
3
votes
1
answer
305
views
Smaller alternatives to sklearn that doesn't require scipy?
I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies.
Sklearn, if you include its dependencies of numpy and scipy is a huge package.
Are there any ...
- 33
1
vote
0
answers
67
views
Chi Square Test Goodness of Fit
I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
- 53
3
votes
0
answers
2k
views
Converting pandas dataframe to scipy sparse arrays
Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning.
Now, if my pandas' data frame ...
- 359
0
votes
1
answer
2k
views
Fitting a pandas dataframe to a Poisson Distribution
I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using:
...
- 85
1
vote
0
answers
57
views
Parallelization of a MIMO linear filter
I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation:
...
- 11
1
vote
2
answers
1k
views
Differential Evolution optimal tolerance parameter
I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
- 11
0
votes
1
answer
43
views
Why i am getting t statistics different
import scipy.stats as st
st.t.ppf(0.95, df=9)
Output: 1.8331129326536335
As per this, I should get 2.262 with dof = 9.
1
vote
0
answers
27
views
Should the weights be rotated when using SciPy full convolution?
I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
- 125
1
vote
0
answers
19
views
Scipy.optimize getting back x0 for optimization of input to recurrent neural network
I need help with this optimization problem which is either not getting solved at all or is taking a copious amount of time.
I am trying to find optimized input to an RNN (GRU) model of a process ...
- 21
1
vote
0
answers
294
views
Scipy Sparse vstack memory error
I have a bunch of scipy matrices (of the same #columns) loaded from disk.
I want to combine them into one scipy sparse matrix.
I am using scipy sparse vstack method.
I am able to load the ...
- 3,755
2
votes
1
answer
77
views
Which is the best algorithm for entity extraction for unstructured document
I have unstructured documents from which I have to extract the information like
let buyer name, seller name, expiry date, buying date etc. I had planned to use spacy(Custom entity recolonization(...
- 113