Questions tagged [scipy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
0answers
11 views

Error when drawing random numbers from a custom continuous distribution using scipy.rv_continuous

I am trying to generate a sample of random numbers from a custom distribution $$ p(x) = x^{n}e^{-xtn}. $$ After reading the tutorial on scipy's website, I wrote a subclass which I called kbayes: <...
1
vote
1answer
59 views

Fastest batch perspective transform for image augmentation

I need to do some augmentation for my training images for a neural networks. The problem is that even when loading batches in parallel, the augmentation is taking longer to perform than the network ...
1
vote
1answer
20 views

How to interpret ANOVA results?

I am trying to identify what attributes are not relevant in my dataset to remove them before fitting a classifier. The target is a categorical variable with three different values. I also have a lot ...
1
vote
1answer
43 views

How do I force specified coefficients in a Linear Regression model to be positive?

Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive. Some additional context: I have a data frame with a some explanatory variables and a response ...
0
votes
0answers
9 views

From stationary value to original value

...
1
vote
0answers
45 views

Two different results from seaborn and matplotlib for a kde plot. What's the reason?

I'm plotting kde plots for a set of data with seaborn and scipy.stats.gaussian_kde: My understanding is that under the hood, seaborn uses scipy (see here). It also seems as though the bandwidth ...
0
votes
1answer
22 views

Probabilities of a Poisson distribution not making sense

I am trying to find probabilities of orders a restaurant gets on Sunday's. For last 6 months average orders are 1000 without any big anomalies like 700 or 1300. This is a case of poisson distribution &...
0
votes
0answers
16 views

How to use skimage.feature.hog to calculate hog of training data where examples are stacked vertically

I have a $n*1024$ dimensional $2D$ array where n is number of examples which contain n images($32*32$) stacked vertically. I would like to calculate the hog of these images ...
4
votes
0answers
33 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
1
vote
2answers
74 views

Fitting a distribution to data

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation ...
0
votes
0answers
18 views

KDTree classified as AI /ML?

We are currently using a KDTree to find similar documents based on keywords identified via TF-IDF. After the tree is built, we are querying the tree to find nearest points. Would this fall in the ...
0
votes
0answers
13 views

Distributed optimization with a common variable among individual agents

I am new to optimization techniques and have a doubt in my approach. Consider I have an agent which tries to optimize on 3 variables C1,x11,x12 to minimize power. I have 60 such agents which ...
1
vote
0answers
70 views

Can I use scipy.optimize module with PyTorch?

I need to use scipy.optimize module after encoding some data with pytorch. However, scipy.optimize does not take torch.tensor as its input. How can I use scipy.optimize module with data with gradient ...
2
votes
1answer
54 views

Which method to use to remove trend from time series?

From what I understand, differencing is necessary to remove the trend and seasonality of a time series. So I assumed it basically does the same thing as signal.detrend from the scipy library. But I ...
1
vote
1answer
285 views

scipy.optimize.minimize(method=’trust-constr’) doesn't terminate on xtol condition

I have set up an optimization problem with linear equality constraints as follows ...
1
vote
1answer
178 views

Scipy curve_fit and method “dogbox”

I am trying to duplicate this papers feature engineering for user activity. They take 14 days of accumulated user activity and keep the parameters (2 parameters) that fit a sigmoid to it. I would like ...
0
votes
0answers
43 views

How to perform a 1-way ANOVA right after One-Hot-Encoding

I am at the phase of dimensionality reduction. I am trying to figure out which categorical columns I should keep for my model and which I should discard. Because some of my categorical columns have ...
0
votes
1answer
42 views

How to measure the correlation between categorical variables and a continuous variable

I have the following list of the names of the categorical variables in my dataset: ...
2
votes
0answers
102 views

Orange3 / python script / scipy interpn

I tried to create a learner using the Orange canvas script from the Python script widget, but I have not been able to find an effective example. What I would like to do is: 1) Create a learner based ...
0
votes
1answer
705 views

Python: multiplication of sparse matrices slower in csr_matrix than numpy

I am trying to multiply a sparse matrix with itself using numpy and scipy.sparse.csr_matrix. The size of matrix is 128x256. Its 93% values are 0. Ironically the multiplication using numpy is faster ...
1
vote
0answers
32 views

Area Under Curve 5 Year Prediction

I have a dataset that includes a ratio(y-axis) and tenure(x-axis). My goal is to take the training set and establish a learning curve by tenure (1 to 15 years) which looks like this: Actual Group ...
1
vote
0answers
29 views

Semantic Segmentation using Scipy or Scikit-image

Many days ago I saw a user using scipy function to get region of interest (RoI) in an image using different colors. I believe it was some sort of filter. Now I'm trying to do the same but I'm not able ...
0
votes
1answer
669 views

How do I properly write scipy.stats.binom.cdf() details

I need to calculate the probability of my random variable being $\le 0$. It's a binomial distribution, $10000$ trials, probability of success is $\frac{10}{19}$ (roughly $0.53$). How do I properly ...
6
votes
3answers
124 views

What is the best algorithm/solution for predicting the following?

I have a dataset that comprises 76 countries, and 6 columns of distinct quantitative variables, which are the mean values of that variable relative to each country: If I were to take a random sample ...
2
votes
0answers
844 views

How to make scipy.optimize.basinhopping find the global optimal point

Question Try to find the global optimal point of the function (reading Python for finance 2nd edition - Chapter 11. Mathematical Tools). ...
1
vote
1answer
477 views

How to understand the return values of scipy.interpolate.splrep

Background Continuation of Spline interpolation - why cube with 2nd derivative as following Cubic Spline Interpolation in youtube. The example in the youtube is below. Implemented using scipy....
1
vote
1answer
588 views

What does sklearn's pairwise_distances with metric='correlation' do?

I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed. Then I tried digging through the function itself, but its ...
1
vote
0answers
21 views

How can I transform (pre-process) pure count data for PCA analysis?

If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?
0
votes
0answers
117 views

B-spline interpolation for image

I am looking B-spline image interpolation implementation using python library. Like: bicubic_interpolation = cv2.resize(image, (512, 512), interpolation = cv2.INTER_CUBIC)
0
votes
1answer
486 views

Machine learning with sklearn vs. scipy stats

I've created 50 random x and y points (with slope of y = 2x-1). First, I used Linear Regression from sklearn to fit the model onto my dataset where I got a slope of ...
1
vote
1answer
534 views

t - table and degrees of freedom

I am interested if someone can review this process and give me some tips. (I don't have any data science coworkers or friends to collaborate with...) When the script (.py file) below runs, it will ...
1
vote
0answers
26 views

Bayes posteriograms

My main objective is to predict the posterior probability of an individual belonging to one of the classes, using Bayes theorem. The information I have is: value of the data point mean and stdev of ...
1
vote
0answers
84 views

Two-sample K–S test Kolmogorov–Smirnov test in Python

For my evaluation, I have three different time-series data of the following format with different characterstics where the first column is timestamp and the second column is the value. ...
2
votes
1answer
23 views

Normally distribute occurence or counts

I am creating a mock of sales data. One of the columns is salesperson_id where each id can occur more than once (a salesperson can have multiple sales). I want to ...
0
votes
1answer
57 views

Faster 3D Matrix Operation - Python

I am working with 3D matrix in Python, for example, given matrix like this with size of 2x3x4: ...
0
votes
1answer
18 views

Restricting a weight vector (optimization parameter) to be in a certain domain using python ML library linear regression model

Sorry if the title is a bit long, but basically I'm trying to predict values $$ \hat{y}_i \in [-1,1]$$ using a simple model i.e. something like OLS or ridge regression, I'd like to know if anyone ...
1
vote
0answers
75 views

python calculating degree days

I am trying to write a program in python to calculate growing degree-day for a specific insect out of average hourly temperature. The way you calculate growing degree day from hourly temperature is by ...
1
vote
0answers
470 views

Optimizing an averaged perceptron algorithm using numpy and scipy instead of dictionaries

So I'm trying to write an averaged perceptron algorithm (page 48 here for the equation) in python. Instead of storing the historical weights, I simply accumulate the weights and then multiply ...
4
votes
4answers
6k views

Using Machine Learning to Predict Temperature [closed]

I am a beginner in ML and I want to create a smart thermostat, that after collecting enough data from the interaction with the user, it will start to set the home temperature by itself. What I got so ...
0
votes
2answers
126 views

Are Hadoop and Python SciPy used for the same? [closed]

Right now I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis. Specifically,...
6
votes
2answers
3k views

Understanding scipy sparse matrix types

I am trying to select the best scipy sparse matrix type to use in my algorithm. In that, I should initialize data in a vij way, then I should use it to perform matrix vector multiplication. Eventually ...
1
vote
0answers
138 views

Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables

Need lesser memory consumption than doing sklearn.decomposition.TruncatedSVD on 99999 variables. So I know that I can use TruncatedSVD to do the dimensionality reduction. However, my problem is about ...
2
votes
0answers
897 views

Clustering with cosine similarity with specific threshold (in Python)

Given : 1, either given the origin data matrix X , such as X = [[1,1,1,1],[1,1,-1,-1],[1,1,1,-1]] or given the cosine similarity matrix of the original data X , ...
1
vote
1answer
265 views

How to calculate VIF in python 3.6 with scipy 1.0.0?

I am trying to import from statsmodels.stats.outliers_influence import variance_inflation_factor This is working fine upto Scipy 0.19. But , with Python 3.6.3 ,...
1
vote
1answer
39k views

ValueError: operands could not be broadcast together with shapes while using two sample independent t test

I am trying to perform two sample t test. My data set consists of 744 rows and 186 columns for which I have calculated total sum and mean. I need to perform two sample t test. My csv looks like this ...
4
votes
1answer
465 views

Compute parameters of a PDF (probability density function) for which no closed form expression is available

I would like to compute parameters such as mean, variance, quantiles, etc. for a PDF which is only given as a piece of code. That is, it can only be evaluated numerically at given points; no closed-...