Questions tagged [scipy]

SciPy is a python ecosystem of open-source packages for scientific computation. Some of the core packages include NumPy, SciPy, Matplotlib, IPython, SymPy, and pandas.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Evaluate Dendrogram Statistical Significance

I have N=21 objects and each one has about 80 possible not NaN descriptors. I carried out a hierarchical clustering on the objects and I obtained this dendrogram. I want some kind of 'confidence' ...
user avatar
  • 1
0 votes
0 answers
12 views

How to sample a dataframe or numpy array with a particular interval?

I have the following dataframe : ...
user avatar
  • 1
0 votes
0 answers
7 views

the mean and standard deviation aren't the same as those of the input data i provided after sampling

I have a log-normal mean and a standard deviation. after i converted them to the underlying normal distribution's parameters mu and sigma, I sampled from the log-normal distribution however when i ...
user avatar
0 votes
0 answers
6 views

What kind of hypothesis testing in Python can be used to validate that 4 job titles are significantly different based on their skillset?

I have 4 job titles, for each of which I scraped hundreds of job descriptions and classified them by if they contain words related to a predefined list of skills. For each job description, I now have ...
user avatar
1 vote
0 answers
25 views

Find the right balance between price of a property and agent fee

I would like to know when buying a property when is better for an estate agent to get a higher fee from me compared to the seller if we get a deal with a lower amount. As an example, let's say that: ...
user avatar
  • 111
0 votes
1 answer
42 views

Feature Selection: How to select categorical features in a regression problem

I am reviewing information for feature selection based in filter methods. I got info (link1, link2, link3, link4, link5) for: Numerical input, numerical output Categorical input, categorical output ...
user avatar
0 votes
3 answers
315 views

p-value of chi squared test is exactly 0.0

I need to do a chi square test of two of my dataset's categorical variables. This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
user avatar
1 vote
1 answer
76 views

Why does the 1st derivative appear to lag the slope of the fit in Scipy's Savitzky-Golay filter?

I have a simple script that performs the Savitzky-Golay filter on a toy dataset of forex prices from yahoo finance: ...
user avatar
  • 111
1 vote
1 answer
23 views

Create Period column based on a date column where the first month is 1, second 2, etc

I have a dataset with many project's monthly expendituries (cost curve), like this one: Project Date Expenditure(USD) Project A 12-2020 500 Project A 01-2021 1257 Project A 02-2021 125889 Project ...
user avatar
1 vote
1 answer
126 views

Vectorize scipy.stats.norm.logpdf

I am tryint to trying to train a Bayesian NN and at some point I need to compute log-likelihoods for some data points, according to a multivariate diagonal gaussian distribution with parameters (mu, ...
user avatar
1 vote
0 answers
11 views

Similarity between binary vector with hierarchal structure

I have dataset of binary vectors, where each vector composed from several small vector coming from a different parent category. Each of those categories has a different size e.g. ...
user avatar
  • 111
0 votes
1 answer
111 views

Stemming/lemmatization for German words

I have a huge dataset of German words and their frequency in a text corpus (so words like "der", "die", "das" have a very high frequency, whereas terminology-like words ...
user avatar
0 votes
0 answers
72 views

Desired error not necessarily achieved due to precision loss

I am trying to maximize a function but I am getting a message of precision loss. ...
user avatar
  • 1
0 votes
2 answers
38 views

What is '[0]'in boxcox transformation with scipy.stats.boxcox?

While performing a boxcox transformation on a series of data, I noticed a certain [0], which I don't quite understand
user avatar
0 votes
1 answer
27 views

Multiple Hypotheses in python

I want to write a method to test multiple hypotheses for a pair of schools (say TAMU and UT Austin). I want to consider all possible pairs of words (Research Thesis Proposal AI Analytics), and ...
user avatar
  • 1
0 votes
0 answers
64 views

Why the error between the measured data and model data is not minimizing in Python?

I want to fit the non-linear experimental data with the model function by estimating some parameters in the function. The model function I have is: ...
user avatar
  • 1
1 vote
0 answers
36 views

How to create a graph network (Networkx) from the solution of Ordinary differential equations?

I have a list of Nodes. Suppose, N =["1","2","3","4"] # there can be different number of nodes And I have solved the ...
user avatar
2 votes
0 answers
27 views

What would be a good randomization environment for data science?

I would like to know if there are any best practices to optimize random environment. Currently I use this simple structure in my config : ...
user avatar
  • 21
0 votes
1 answer
287 views

How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on ...
user avatar
  • 71
0 votes
0 answers
120 views

2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

I have some 2D points and I want to assess their performance against the target point. When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that ...
user avatar
0 votes
0 answers
46 views

Scipy kstest problem

I am fitting mixture models to data and assessing how mixtures with more or less components will fit the data. To do this, I am going to plot the cdf of the empirical data and the cdf of my mixture ...
user avatar
0 votes
1 answer
2k views

numpy array center of mass

I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array. For example let's say we have an ...
user avatar
2 votes
1 answer
784 views

Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I am trying to plot a Dendrogram to cluster data but this error is stopping me. My datea is here. I first chose columns to work with: ...
user avatar
  • 21
1 vote
1 answer
169 views

How to find the best fitting parametric distribution for an empirical dataset (stock returns)?

Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged. Instead, I ...
user avatar
1 vote
1 answer
765 views

what is criterion in flcuster of scipy package?

Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?
user avatar
  • 21
0 votes
1 answer
175 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
user avatar
1 vote
1 answer
65 views

How should I sample from a mixture distribution?

Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf ...
user avatar
3 votes
0 answers
769 views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
user avatar
  • 131
3 votes
1 answer
114 views

Smaller alternatives to sklearn that doesn't require scipy?

I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies. Sklearn, if you include its dependencies of numpy and scipy is a huge package. Are there any ...
user avatar
1 vote
0 answers
56 views

Chi Square Test Goodness of Fit

I want to use a chi square test but I'm unsure if I'm using it right. The KickStarter website shows the frequency of main categories projects. It is updated once a day. I got a data set of KickStarter ...
user avatar
  • 43
3 votes
0 answers
2k views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
user avatar
  • 339
0 votes
1 answer
1k views

Fitting a pandas dataframe to a Poisson Distribution

I have a simple dataframe df2 that consist of indices and one column of values. I want to fit this dataframe to a poisson distribution. Below is the code I am using: ...
user avatar
1 vote
0 answers
43 views

Parallelization of a MIMO linear filter

I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation: ...
user avatar
  • 11
1 vote
2 answers
738 views

Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
user avatar
0 votes
1 answer
35 views

Why i am getting t statistics different

import scipy.stats as st st.t.ppf(0.95, df=9) Output: 1.8331129326536335 As per this, I should get 2.262 with dof = 9.
user avatar
1 vote
0 answers
23 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
user avatar
  • 125
1 vote
0 answers
14 views

Scipy.optimize getting back x0 for optimization of input to recurrent neural network

I need help with this optimization problem which is either not getting solved at all or is taking a copious amount of time. I am trying to find optimized input to an RNN (GRU) model of a process ...
user avatar
  • 21
1 vote
0 answers
206 views

Scipy Sparse vstack memory error

I have a bunch of scipy matrices (of the same #columns) loaded from disk. I want to combine them into one scipy sparse matrix. I am using scipy sparse vstack method. I am able to load the ...
user avatar
2 votes
1 answer
67 views

Which is the best algorithm for entity extraction for unstructured document

I have unstructured documents from which I have to extract the information like let buyer name, seller name, expiry date, buying date etc. I had planned to use spacy(Custom entity recolonization(...
user avatar
2 votes
1 answer
47 views

optimizing a linear optimization function with linear constarints and binary variables

I am new to optimizations and trying to solve a problem, which I feel falls in the umbrella of optimization. I have an ojective function that needs to be maximized ...
user avatar
  • 515
1 vote
1 answer
500 views

How to perform convolution with kernel bigger than image?

In this question I've seen an example of convolution by the kernel with the shape bigger than initial image's one: ...
user avatar
2 votes
1 answer
85 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
user avatar
  • 21
1 vote
1 answer
750 views

Fastest batch perspective transform for image augmentation

I need to do some augmentation for my training images for a neural networks. The problem is that even when loading batches in parallel, the augmentation is taking longer to perform than the network ...
user avatar
3 votes
2 answers
80 views

How to interpret ANOVA results?

I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier. The target is a categorical variable with three different values. I also have a lot ...
user avatar
  • 317
1 vote
1 answer
2k views

How do I force specified coefficients in a Linear Regression model to be positive?

Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive. Some additional context: I have a data frame with a some explanatory variables and a response ...
user avatar
1 vote
0 answers
746 views

Two different results from seaborn and matplotlib for a kde plot. What's the reason?

I'm plotting kde plots for a set of data with seaborn and scipy.stats.gaussian_kde: My understanding is that under the hood, seaborn uses scipy (see here). It also seems as though the bandwidth ...
user avatar
  • 111
0 votes
1 answer
53 views

Probabilities of a Poisson distribution not making sense

I am trying to find probabilities of orders a restaurant gets on Sunday's. For last 6 months average orders are 1000 without any big anomalies like 700 or 1300. This is a case of poisson distribution &...
user avatar
5 votes
1 answer
97 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
user avatar
2 votes
2 answers
706 views

Fitting a distribution to data

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
user avatar
4 votes
2 answers
1k views

Can I use scipy.optimize module with PyTorch?

I need to use scipy.optimize module after encoding some data with pytorch. However, scipy.optimize does not take torch.tensor as its input. How can I use scipy.optimize module with data with gradient ...
user avatar