Hot answers tagged

11

Ok, I was looking for an answer and now I have it clearer: Scipy documentation does not elaborate too much on the explanation, but wikipedia article is much more clear. For those who are looking for an answer, there are two major groups of sparse matrices: a) Sparse types used to construct the matrices: DOK (Dictionary Of Keys): a dictionary that maps (row,...


6

I would not recommend going ahead with the data that you know might be wrong. Looking at your current data, the reason you got a bad result with linear regression is because the relation between them is not linear for the current data. For eg. There is high variation in response (i.e. user_set_temp) for same value of your predictor (i.e. environment_temp). ...


5

Your problem could be solved either by direct numeric integration or by MCMC. Numeric integration can be performed most easily by scipy: import numpy as np import scipy.stats import scipy.integrate def weird_density(x): """ The function I want to sample """ return scipy.stats.lognorm.pdf(x, 1.0) def quad_quantile(fun, q, precision=1e-10, ...


4

f_val is the F Statistic value. Mathematically it is $ F = \frac{MS_{Between}}{MS_{Within}}$ The null hypothesis for your ANOVA is $H_0: \mu_{Entire home/apt} = \mu_{Private room} = \mu_{Shared room} $ which means all means ($\mu_i$ s) are equal and there is no need of grouping using explanatory variable vs $H_A:$ At least one $\mu_i$ is different. ...


3

Sorry, but on the surface, this sounds like a terrible idea to me: if linear regression gives you negative coefficients for some explanatory variables that you think should be positive, then it means that either your data is "wrong" (typically noisy or too small) or your intuition is misguided. I can't see any good reason why one would use a data-driven ...


3

The Chi-squared test measures the relationship between two categorical variables. To measure the relationship between a categorical feature and a continuous feature, you can use an ANOVA test. As an aside, you don't need to use a for loop for your encoding - get_dummies has the parameter columns which allows the user to specify which columns are encoded.


3

Very interesting question indeed. I will dare to give you an answer, but I could be changing it if I find a better reasoning about the problem after watching the data. Your problem is perhaps a problem of multi(many)-label classification, where your label is the country and your data is what you have as "patience", "risktaking", etc. You could work with ...


2

The answer to this question would be : The objects created by the xs method of the Pandas DataFrame look like two-dimensional arrays. These must be flattened to look like one-dimensional arrays when passed to ttest_ind. The values attribute of the Pandas objects gives a numpy array, and the ravel() method flattens the array to one-dimension. It would go ...


2

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access file stores as you would in a local file store (with some modification) and modify things via Java API. Furthermore, ON TOP OF the file system there exist a ...


2

Here is one way to do it: from collections import Counter from numpy.random import binomial # Generate mock sales data by randomly sampling if a salesperson makes a sale p = 0.5 # Control "flatness" of distribution n_records = 14 salesperson_id = binomial(n=100, p=p, size=n_records).tolist() # Binomial is a reasonable approximation for a discrete normal ...


2

Yes fitting the data and finding the best fitting line is called training the model. If you look at the source code of scikit-learn linear regression you can find the its using scipy linalg.lstsq module for finding the coefficients and intercept (most cases). See the source code for more details . Machine learning is fancy word for Application of ...


2

Here, you do not time only the time taken to make the matrix multiplication but also the time taken to convert your matrix from dense to sparse. If you convert your matrix before the timing starts, you will see that multiplication with scipy is indeed more than twice faster import time, numpy, scipy from scipy.sparse import csr_matrix import numpy as np W =...


2

Looking at the description I will go for a 'distance based' approach. Fist, I would produce prototypes of the six variables per country (the summarised table may be an example but there could be more than one single prototype per country). Then, in order to assign a datum to a given country, I would follow a k-nearest or similar approch. Neverthless, ...


2

If you wish an easy to understand implementation that has the opportunity to make the matrix bigger as well as being able to row/column operation, I suggest you coo_matrix. coo_matrix is efficient and fast to construct but, arithmetic operations are not efficient on this matrix. Instead, you can easily convert coo_matrix to csc_matrix/csr_matrix that are ...


2

My understanding of your problem: you have realizations of $X$ (property value at first point in time) and $Y$ (value at second point in time). Want to fit a distribution for $Y-X$--more generally, a "model for the data." One way of doing this would be plot a histogram of these values and then come up with plausible distributions that might fit the data, ...


2

Detrend does a least squares fit (linear or constant) and subtracts this from your data points. You can look this up in the docs. Simply taking the difference between consecutive data points will in general lead to other results. In general the regression based detrending seems to be more reasonable. You could also think about using random sample consensus ...


2

The mixture distribution can be obtained in the following way. Let $f(x)=w_1p_1(x) + w_2p_2(x) + ... + w_np_n(x)$, where $p_i$ are density functions and $w_i>0$. Note that $f(x)$ is a density function if the sum of all weights is one. Then, we use the following two-stage process. Stage 1. Draw a random variable $X$ (selector, if I remember correctly), ...


2

First of all, if you want to find the best distribution that fits your data you just iteratively fit your data to the longlist of distributions. Scipy supports most of them. After fitting, you can either use KS-test to find which distribution fitted best or you can use fit error to decide. This solution does what you want, also other solutions in that post ...


1

Use scipy.optimize.minimize of the negative objective function. Use the link to play and modify as you wish but here is application in the nutshell. Lets say you have your data > array([[ 0.00749589, 0.01255155, 0.02396251, 0.04750988, 0.09495377] > [ 0.01255155, 0.02510441, 0.04794055, 0.09502834, 0.18996269], > [ 0.02396251, ...


1

Is your question how can we perform convolution operation with kernel bigger than input image? If yes then answer is padding. TL;DR you increase the size of the original image with boundary pixels so that kernel can "fit"


1

Image augmentations heavily relies on your DataGenerator and DataLoader design, mostly along with the hardware resources that you are using. Apart from that, here is a quick comparison chart to help you with the transformations along with the libraries links. The numbers represent the number of images it processes per second. Here are the links to all the ...


1

This seems quite right to me. A poisson(1000) law has a standard deviation of $\sqrt{1000}$ which is something around $31.6$. At such values, the Poisson law pretty much behaves like a normal distribution, so the probability of getting a value greater than 1100 would be very close to 0 (around 0.1%). Perhaps what you need is not a Poisson law, but a ...


1

Q1: What impact does a linear trend have on the correlation between non-spurious time series? The 4 main measures of correlation are Pearson, Kendall rank, Spearman and Point-biserial (the latter of which is not applicable for this type of problem). For simplicity, I'll only explain how it affects measuring Pearson correlation. Let's assume $X$ represents a ...


1

Cool question. My understanding is that the Kolmogorov Smirnov test is what you're after here. You could use the distribution functions in scipy to generate various kinds of distributions and use the K-S test to assess the similarity between your distribution of value variances and each of the distributions you select for a test. K-S test documentation ...


1

Here's a workaround I'm currently using: # model definition class MLP(Module): # define model elements def __init__(self): super(MLP, self).__init__() self.hidden1 = Linear(2, 200) self.hidden2 = Linear(200, 100) self.hidden3 = Linear(100, 1) self.activation1 = LeakyReLU() self.activation2 = LeakyReLU() ...


1

>>> from scipy.stats import binom >>> binom.cdf(0, 10000,10/19) 0.0 Binomial random variable are nonnegative. Hence you want to compute the probability that it is equal to zero which is $$\left(\frac{10}{19} \right)^{10000}\approx 10^{-2787.54}\approx 0$$


1

The SVM.SVC() model is a good choice to go with, though obviously there is more than one option. https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC The 'Target' values I set as the countries included in the dataset, and then labeled using label_encoder as quantitative variables (1-76). After defining the model, I was ...


1

Y = pdist(X, 'correlation') Computes the correlation distance between vectors u and v. This is $$ 1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2} $$ where $\overline{u}$ is the mean of the elements of vector $u$, and $x\dot{}y$ is the dot product of $x$ and $y$. The correlation between any vector which have ONLY ...


1

See this post on machinelearningmastery to use scipy to find critical t value. This code below seemed to work. from scipy.stats import t #savings calc diff = actualKwh - total #compute precision of results, starting with standard error stdErr = totalStd / math.sqrt(len(total.index)) # define probability & degrees of freedom p = 0.95 degf = len(data....


1

It is faster if you use the built-in functions of numpy (instead of reimplementing them yourself): import numpy as np from scipy.stats import entropy np.apply_along_axis(func1d=entropy, axis=2, arr=matrix)


Only top voted, non community-wiki answers of a minimum length are eligible