Hot answers tagged

12

Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas. If we assume that your dataframe is called df and the column you want to filter based AVG, then Q1 = df['AVG'].quantile(0.25) Q3 = df['AVG'].quantile(0.75) IQR = Q3 - ...


12

It is quite trivial (X=='y').astype(int) Should do the trick. It simply converts your array to True or False according to your requirements and then astype will impose the required datatype. By default int will give you 1 for True and 0 for False.


11

I got your problem like this way: You want to show labels on x and y axis on seaborn heatmap. So for that sns.heatmap() function has two parameter they are xticklabels for x-axis and yticklabels for y-axis labels. follow below code snippet import seaborn as sns # for data visualization flight = sns.load_dataset('flights') # load flights datset from GitHub ...


10

There may be column of type object in your dataFrame. Remove it or convert it into float,int etc. if possible From Documentation of Seaborn: data : rectangular dataset 2D dataset that can be coerced into an ndarray. If a Pandas DataFrame is provided, the index/column information will be used to label the columns and rows. so you can try sns.heatmap(...


9

Given you have images stretched out as columns in a table with ~48,500 rows, I am assuming you have the raw images that are 220x220 in dimension. You can use a function available via OpenCV called inpaint, which will restore missing pixel values (for example black pixels of degraded photos). Here is an image example. Top-left shows the image with missing ...


8

I suggest you to visit this link also for this case would work np.reshape((-1,2))


7

It works for me. >>> import numpy as np >>> dx = np.matrix([[ 1.6, 3.6, 0.4, 14.4, 25.6], ... [10. , 10. , 0.4, 14.4, 3.6], ... [ 0.4, 0. , 0. , 1.6, 10. ], ... [ 6.4, 0. , 3.6, 1.6, 0.4], ... [14.4, 0. , 25.6, 0.4, 6.4]]) >>> 10 * dx matrix([[ 16., 36., 4., 144., 256.], [ 100., 100., 4., 144., 36.], ...


7

You have already computed that, but you've not bound the output to a variable, also called name in python. Try the following snippet: result = np.linalg.norm(v1,ord=2,axis=1,keepdims=True) print(result) Based on the edit, I update the answer. As you may find answers to your question, a typical way to find what you need is something like the following ...


6

The number of rows in your training data is not part of the input shape of the network because the training process feeds the network one sample per batch (or, more precisely, batch_size samples per batch). And in input_shape, the batch dimension is not included for the first layer. You can read more on this here. So, the input shape for your problem will ...


6

You could use the following code: X[X=='y'] = 1 X[X=='n'] = 0 This replaces the indexes of 'y' with 1 and of 'n' with 0. Generally the X=='y' returns a Boolean array which contains True where the 'y' and False everywhere else and so on.


5

I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options: discard missing values (the entire row): you end up with less but more consistent training data if you really need these rows, then fill missing values ...


5

nazz's answer doesn't work in all cases and is not a standard way of doing the scaling you try to perform (there are an infinite number of possible ways to scale to [-1,1] ). I assume you want to scale each column separately: 1) you should divide by the absolute maximum: arr = arr - arr.mean(axis=0) arr = arr / np.abs(arr).max(axis=0) 2) But if the ...


5

I would import the datasets in pandas separately, mold them as you please, and then you can use the pd.concat function. This will assume that the instances are aligned by the automatically assigned index in pandas. If there is more data in one list than the other, the missing values will be NaN. df1 = pd.DataFrame(data=[1,2,3]) df2 = pd.DataFrame(data=['a','...


5

You can compare the predictions with the expected results directly, using simple comparisons, in this case just ==. This returns boolean values - True or False, which you can sum up because True == 1 and False = 0. Here is an example for your case using some randomly generated dummy data: In [1]: import numpy as np ...


4

You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the fit_generator method rather than fit to pass in an iterator that feeds samples to your model from disk rather than loading all of that data at once. Here's a tutorial that discusses this more.


4

Found the answer. Thank you @Aditya import seaborn as sns sns.lmplot('Time', 'Amount', dataset, hue='Class', fit_reg=False) fig = plt.gcf() fig.set_size_inches(15, 10) plt.show() where Time and Amount are the two features I needed to plot. Class is the column of the dataset that has the dependent binary class value. And this is the plot I got as required.


4

One approach is to plot the data as a scatter plot with a low alpha, so you can see the individual points as well as a rough measure of density. from sklearn.datasets import load_iris iris = load_iris() features = iris.data.T plt.scatter(features[0], features[1], alpha=0.2, s=100*features[3], c=iris.target, cmap='viridis') plt.xlabel(iris....


4

Suppose you have an array arr. You can normalize it like this: arr = arr - arr.mean() arr = arr / arr.max() You first subtract the mean to center it around $0$, then divide by the max to scale it to $[-1, 1]$.


4

I still would apply numpy's covaranice function using numpy.apply_along_axis import numpy as np x = np.array([[0, 2], [1, 1], [2, 0]]).T np.apply_along_axis(func1d=np.cov, arr=x, axis=0)


4

You should implement a generator and feed it to model.fit_generator(). Your generator may look like this: def batch_generator(X, Y, batch_size = BATCH_SIZE): indices = np.arange(len(X)) batch=[] while True: # it might be a good idea to shuffle your data before each epoch np.random.shuffle(indices) for i in ...


4

This is expected and is not related to SMOTE sampling. The computational complexity of non-linear SVM is on the order of $O(n^2)$ to $O(n^3)$ where $n$ is the number of samples. This means that if it takes 0.8 seconds for 7.5K data points, it should take [3, 48] minutes for 115K, $$[(115/7.5)^{2} \times 0.8, (115/7.5)^{3} \times 0.8]s=[3,48]m,$$and from 16 ...


4

On a side note, I don't think correlation is the correct measure of relation for you to be using, since Survived is technically a binary categorical variable. "Correlation" measures used should depend on the type of variables being investigated: continuous variable v continuous variable: use "traditional" correlation - e.g. Spearman's rank correlation or ...


4

The problem is that train_test_split(X, y, ...) returns numpy arrays and not pandas dataframes. Numpy arrays have no attribute named columns If you want to see what features SelectFromModel kept, you need to substitute X_train (which is a numpy.array) with X which is a pandas.DataFrame. selected_feat= X.columns[(sel.get_support())] This will return a ...


4

If you need to remove outliers and you need it to work with grouped data, without extra complications, just add showfliers argument as False in the function call. It's inherited from matplotlib. showfliers=False


4

You can get the type of the entries of your column with map: df['ABC'].map(type) So to filter on all values, which are not stored as str, you can use: df['ABC'].map(type) != str If however you just want to check if some of the rows contain a string, that has a special format (like a date), you can check this with a regex like: df['ABC'].str.match('[0-9]{...


4

What about creating a Pandas DataFrame and adding a new column such as "Temp_simulated" and simulate the temperature?


4

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this: df['counter'] = df.diff().ne(0).cumsum() Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the ...


3

I think I know what you are going for. Here is some example code. # Import pandas import pandas as pd # Load data df_a = pd.read_csv('gdp.csv') df_b = pd.read_csv('le.csv') # Use melt to pull the columns from each df into separate rows tied to the key column. df_a = df_a.melt(id_vars='country', var_name='year', value_name='gdp') df_b = df_b.melt(...


3

MemoryError is exactly what it means, you have run out of memory in your RAM for your code to execute. When this error occurs it is likely because you have loaded the entire data into memory. For large datasets you will want to use batch processing. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access ...


3

If your purpose is to deal with some data analysis from your arrays and have a nice view of it, try this using pandas: import pandas as pd df = pd.DataFrame({'distA' : np.array([0.2, 0.3, 0.4]), 'distB' : np.array([0.3, 0.4, 0.5])}) df In the case you want to make computation with vectors, matrixes, tensors, etc .. NumPy is probably a ...


Only top voted, non community-wiki answers of a minimum length are eligible