# Tag Info

5

You can compare the predictions with the expected results directly, using simple comparisons, in this case just ==. This returns boolean values - True or False, which you can sum up because True == 1 and False = 0. Here is an example for your case using some randomly generated dummy data: In [1]: import numpy as np ...

4

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this: df['counter'] = df.diff().ne(0).cumsum() Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the ...

4

If you need to remove outliers and you need it to work with grouped data, without extra complications, just add showfliers argument as False in the function call. It's inherited from matplotlib. showfliers=False

3

There are multiple possiblities from dusk, to others model etc. Here are my 2 favorites, not to loose you in the number of possibilities: www.h5py.org/ "It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. ...

3

For this sort of problem you can use np.where and multiple boolean expressions to get your answer. #1 We need to test if the column is equal to your target values, 0 or 1 second, we have to ensure the column value is not equal to the column value above. 3rd, for any value that is not equal to our input value we return a nan as its easier to use with numeric ...

3

numpy.corrcoef will gives you Pearson Correlation but your Features are Categorical. You should calculate Crammer'v. You can get details/code in this answer as both the questions are a bit similar DS.SE On Plot What Erwan has suggested seems good. Also, try to plot(line-plot) between days-of-week and spam/total ratio(i.e. Stanardizing for total volume) ...

3

(started as a comment but it turned out to be longer than expected) With a dataset like this a simple barplot could be very insightful: on the X axis the days of the week, on the Y axis the frequency, with two bars (spam/not spam using different color) for each day. A slightly more advanced version: two boxplots, one for weekdays the other for weekends. A ...

2

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("xxx"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["xxx"]). so the code below will work. ## Important: i have passed the columns a string to CV and list of columns to OHE transformer=...

2

One reason why you aren't getting fitted values close to the true values could be the initial values of the parameters used. It's likely what you have found is a local maxima. You have to try a number of initial starts and then pick the one with that gives the highest likelihood.

2

I don't know if it's the best way, but I'd write a function and apply it to the df['size']: def rename_size(size): if size not in ['M','S','L']: return np.nan else: return size df['size']= df['size'].apply(rename_size)

2

Firstly the function returns the variables u and indices: u contains the unique elements sorted. In other words no element is repeated (the number 2 does not appear twice) and the elements will be listed from the smallest to the largest value indices is the same size as a and basically it contains the index in u you should used to recover a. So when they ...

2

I replicate your code using a toy data set and I did not find anything wrong with your implementation: import numpy as np from sklearn.preprocessing import StandardScaler from sklearn.datasets import load_breast_cancer from sklearn.metrics import accuracy_score import matplotlib.pyplot as plt plt.style.use("seaborn-whitegrid") import warnings ...

2

import numpy as np a = np.array(['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF']) f = lambda x: 1 if x == "COLLECTION" else 0 np.fromiter(map(f,a),dtype=np.int) Alternative: np.where(a == "COLLECTION",1,0)

2

The easiest way of achieving this would probably to split the string column using the fraction character and then dividing the first value by the second value: import pandas as pd df = pd.DataFrame({"col": ["250/500", "100/300", "500/1000"]}) df["result"] = df["col"].str.split("/").apply(...

1

Thank you for the above answer! That definitely works. However, I found a more efficient way in terms of computation using np.rolling df['D'] = df['A'].rolling(min_periods=1, window=3).mean() df['B'] = np.where(df['B'].isnull,df['D'],df['B']) np.rolling helps to compute the cumulative sum of previous n values. np.where helps to apply some output based on a ...

1

You can reshape a numpy array simply by: newxdata = xdata.reshape((60000,28*28)) for example. Or simply: newxdata = xdata.reshape((len(xdata),-1)) Note that reshape is a numpy function which can used also as: import numpy as np newxdata = np.reshape(xdata, (60000,-1)) To speed up your loop you could alternatively use libraries like multiprocessing.Pool ...

1

Update Accordingly: Your question was not clear before, therefore, sorry for the irrelevant solution. To achieve that, I don't think there is a public library function, however, you can build your solution using some beautiful "ready" functions. Two solutions come into my mind: First one's time complexity is O(N*M) N is your prediction (list a in ...

1

There are two ways to correct this you passed three lists to np.array, and there are not all same size demo_matrix = np.array(([13,35,74,48],[23,37,37,38],[73,39,93,39])) demo_matrix[2,3] #demo_matrix (list of lists) array([[13, 35, 74, 48], [23, 37, 37, 38], [73, 39, 93, 39]]) you used 35.74, I guess it might be typing mistake If you ...

1

Could probably be more elegant, but here's an idea: mean[:, :, 0:2] = np.mean(imgs[:, :, :, 0:2], axis=0)

1

You should look at the documentation. You do not pass the column names as an argument. subset_df = df[['col1', 'col2']] subset_df.corr() That should solve this for you.

1

You need $2734 \times 132\times 126\times 1=45,471,888$ values in order to reshape into that tensor. Since you have $136,415,664$ values, the reshaping is impossible. If your fourth dimension is $4$, then the reshape will be possible.

1

This is a problem that I have also run into before, right now your ytrain is a one dimensional array (advisable to avoid). Check this answer. expanding(adding) additional dimension while assigning ytrain should fix your problem x = np.array([1, 2]) x.shape (2,) y = np.expand_dims(x, axis=1) y array([[1], [2]]) y.shape (2, 1)

1

tl;dr: refresh your linear algebra and check for built-in functions. In addition to brevity/elegance, vectorization often dramatically improves performance. Tools like MATLAB and numpy use high-level vectorized wrappers for low-level optimized implementations, which in the end are composed of your traditional loop operations. Where MATLAB and Python are ...

1

Normalization/Standardization is suggested because it makes the convergence easy and faster. Andrew Ng has explained the process and the reason with a bowl shape 2-dimensional loss space in his course. If you are not doing that, you should be very slow with your LR and spend a lot more epochs. I found this combination working lr = LogisticRegression(...

1

This will create a matrix. Rows/Cols represent the IDs. You can check the result like a lookup table i.e. (ID1, ID91) - Can look at either (0,90) Or (90, 0) import numpy as np, pandas as pd from numpy.linalg import norm x = np.random.random((8000,200)) cosine = np.zeros((200,200)) for i in range(200): for j in range(200): c_tmp = np.dot(x[i], ...

1

Keras needs you to pass one more dimension than it says in the error message: the batch dimension. I.e. If you have a model that requires each sample to have an input shape of (4,) and you have 1000 training samples you need to feed it with an array of (1000, 4). In your case since you want to feed it with just one sample you need to pass a shape of (1, 4). ...

1

Your question leaves a lot unexplained, but the error you're receiving probably came from the last line vectorized_text = features.transform("Heathcare is good").toarray(). The variable features is itself an array (it came from features = ... .toarray()). I'm assuming you meant to replace it with vectorized_text = tfidf.transform("Heathcare ...

1

what is the difference between slope of line and slope of curve It is really a matter of perspective. The slope of a line is the same over the entire span of that line, i.e. until the line changes direction. The slope of a curve is like the slope of millions of tiny lines all connected, so the slope is only the same value over tiny spans. So we can only ...

1

Try the following code. As you want all first value of zeroth dimension. test=test[:,0,0]

1

Here is a quick solution that does not do it in-place but takes up extra space: def transform_series(x, chunk_size): df = pd.DataFrame() for i in range(chunk_size): df[f'column_{i+1}'] = x[i::chunk_size].reset_index(drop=True) return df input_series = pd.Series([10,11,12,13,14,15]) transformed_df = transform_series(input_series, ...

Only top voted, non community-wiki answers of a minimum length are eligible