I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing:
- I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there is an example bellow)
[6.19926266e-02 4.74323332e-02 1.22703509e-02 1.02023669e-02 1.45243444e-02 2.16958560e-02 3.42147425e-02 9.73318636e-01 7.03395926e-05 1.93824657e-04 5.42396388e-04 9.58181568e-04 1.64594641e-03 2.38841982e-03 3.74831865e-03 8.31570302e-04 7.34657951e-05 2.03203264e-04 6.39308710e-04 1.13481213e-03 9.09725379e-04 1.56779133e-03 1.78818870e-03 3.70455178e-04 6.25240791e-05 1.71941225e-04 3.67328990e-04 4.62678203e-04 4.90814040e-04 4.39231662e-04 3.72018287e-04 8.12813087e-05 1.40679185e-04 8.59706124e-05 1.65688820e-04 2.53222534e-04 3.04804911e-04 3.09494208e-04 5.93978766e-05 1.56310205e-06 ....]
- I stored my dataframe as pickle because when I was storing as csv, the column was converting into string. My columns stored in the dataframe:
0 [0.0026778684, 0.003117677, 0.00040434036, 0.0... 1 [0.061992627, 0.047432333, 0.012270351, 0.0102... 2 [0.0, 0.0, 0.0, 4.3830705e-06, 1.3149212e-05, ... 3 [0.30314153, 0.04477268, 0.01840577, 0.0319251... 4 [0.2563626, 0.03259786, 0.018686974, 0.0198365... ... 1287 [0.11471527, 0.032394826, 0.012400794, 0.01131... 1288 [0.002138354, 0.001044489, 0.0007786191, 0.001... 1289 [0.056204572, 0.026556363, 0.02082041, 0.01966... 1290 [0.051759016, 0.0058623934, 0.0054726205, 0.00... 1291 [0.0, 5.4140626e-05, 4.4114586e-05, 4.8125003e... Name: F3, Length: 1292, dtype: object
- I can use this column if I create a list and use np.hstack in each row (see the code bellow) and then I can apply the MinMaxScaler on the list I create and use in my machine learning model and this works:
mli =  for i in range(0,len(data)): mli.append(np.hstack([data['F3'][i],data['F5'][i],data['F6'][i],data['F7'][i]])) X = scaler.fit_transform(mli)
PS: 'F5', 'F6', 'F7' have the same structure as F3
I need to apply this column to a machine learning model, which I'm using make_column_transformer because I have text, number and array data to apply transformations on it, so I need to use the column as an input data, and when I try to fit_transform (MinMaxScaler) I have the error:
TypeError Traceback (most recent call last) TypeError: only size-1 arrays can be converted to Python scalars ValueError: setting an array element with a sequence.
I need to use at least one column numpy.ndarray as input to the MinMaxScaler, has anyone tried to use an array like this, or convert into something that would work?
One example of exactly what I want to do:
features = make_column_transformer( (transf,'timagem'),(transf,'legenda'), (scaler, 'F3'),remainder ='drop')
transf = Tdidf Transformer timagem, legenda = text input scaler = MinMax F3 = Exemple of code (1) above