0
$\begingroup$

I am trying to make a submission, so I have a test set without labels and I am tryin to test my classification model on it. In particular, I have also to submit this prediction as a csv. I have the following test set without labels, which is the output of pd.read_json(), so it is the output from the test dataset:

enter image description here

and the point of the classification problem is to predict from the instruction the type of the compiller. The classification problem is already developed, I just need to submit it.

So I have to predict these instructions from the test set, but if I try to do :

test = pd.read_json('test_dataset_blind.jsonl',lines = True)
test

X_new = test['instructions']
new_pred_class = clf.predict(X_new)

where clf is my model, in this case I am using random forests.

I get the following error message:

ValueError: setting an array element with a sequence.

Can anyone please help me? Thank's in advance.

[EDIT] The full error trace is the following:

    ValueError                                Traceback (most recent call last)
<ipython-input-21-ba881bb9e0fe> in <module>
----> 1 new_pred_class = clf.predict(X_new)

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
    543             The predicted classes.
    544         """
--> 545         proba = self.predict_proba(X)
    546 
    547         if self.n_outputs_ == 1:

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, 
X)
    586         check_is_fitted(self, 'estimators_')
    587         # Check data
--> 588         X = self._validate_X_predict(X)
    589 
    590         # Assign chunk of trees to jobs

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in 
_validate_X_predict(self, X)
    357                                  "call `fit` before exploiting the 
model.")
    358 
--> 359         return self.estimators_[0]._validate_X_predict(X, 
check_input=True)
    360 
    361     @property

~\Anaconda3\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, 
X, check_input)
    389         """Validate X whenever one tries to predict, apply, 
predict_proba"""
    390         if check_input:
--> 391             X = check_array(X, dtype=DTYPE, accept_sparse="csr")
    392             if issparse(X) and (X.indices.dtype != np.intc or
    393                                 X.indptr.dtype != np.intc):

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, 
accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, 
ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, 
estimator)
    494             try:
    495                 warnings.simplefilter('error', ComplexWarning)
--> 496                 array = np.asarray(array, dtype=dtype, order=order)
    497             except ComplexWarning:
    498                 raise ValueError("Complex data not supported\n"

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

~\Anaconda3\lib\site-packages\pandas\core\series.py in __array__(self, dtype)
    946             warnings.warn(msg, FutureWarning, stacklevel=3)
    947             dtype = "M8[ns]"
--> 948         return np.asarray(self.array, dtype)
    949 
    950     # ------------------------------------------------------------------ 

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

~\Anaconda3\lib\site-packages\pandas\core\arrays\numpy_.py in __array__(self, 
dtype)
    164 
    165     def __array__(self, dtype=None):
--> 166         return np.asarray(self._ndarray, dtype=dtype)
    167 
    168     _HANDLED_TYPES = (np.ndarray, numbers.Number)

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

ValueError: setting an array element with a sequence.

[EDIT 2] The dataset with labels is the following:

enter image description here

and what I did is the following:

I considered just the operators push,mov,.. and then I created the following dataset with pandas:

enter image description here

after doing this I considered only the values and I used a tf ì-idf vectorizer. Then I slitted the data as;

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
    test_size=0.2, random_state=15)

and I used support vector machine as model.

[EDIT 3] I have achieved eliminated the error and as output I have:

array(['icc', 'gcc', 'gcc', ..., 'clang', 'clang', 'clang'], dtype=object)

now I do the following:

pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class})

and I get the error message:

ValueError                                Traceback (most recent call last)
<ipython-input-41-da853bce8ce2> in <module>
----> 1 pd.DataFrame({'instructions': 
test['instructions'],'compiler':new_pred_class})

~\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, 
index, columns, dtype, copy)
    409             )
    410         elif isinstance(data, dict):
--> 411             mgr = init_dict(data, index, columns, dtype=dtype)
    412         elif isinstance(data, ma.MaskedArray):
    413             import numpy.ma.mrecords as mrecords

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
init_dict(data, index, columns, dtype)
255             arr if not is_datetime64tz_dtype(arr) else arr.copy() for  
arr in arrays
    256         ]
--> 257     return arrays_to_mgr(arrays, data_names, index, columns, 
dtype=dtype)
    258 
    259 

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
arrays_to_mgr(arrays, arr_names, index, columns, dtype)
 75     # figure out the index, if necessary
    76     if index is None:
---> 77         index = extract_index(arrays)
    78     else:
    79         index = ensure_index(index)

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
extract_index(data)
379                         "length {idx_len}".format(length=lengths[0], 
idx_len=len(index))
    380                     )
--> 381                     raise ValueError(msg)
    382             else:
    383                 index = ibase.default_index(lengths[0])

ValueError: array length 30000 does not match index length 3000

apparently I have that new_pred_class has 30000 elements, while the test dataset is 3000 rows. What should I do in this case? Thank's in advance.

$\endgroup$
  • 1
    $\begingroup$ Full error trace, please. And is your image the output from test? $\endgroup$ – Ben Reiniger Nov 5 at 23:23
  • $\begingroup$ Thank's for your answer, and yes the image is the ouput from test. I have edited my question with the full error trace. $\endgroup$ – J.D. Nov 6 at 10:53
  • $\begingroup$ How did you preprocess the data for training? $\endgroup$ – Ben Reiniger Nov 6 at 14:37
  • $\begingroup$ thank you again. I have edited my question summarizing what I did $\endgroup$ – J.D. Nov 6 at 16:34
0
$\begingroup$

The data that you're loading into the model is malformed.

Usually this error arises when not all of the elements in an array share the same size. You need to make sure that each row/array contains the same number of elements (otherwise you can't form a 2D array out of the data). Maybe there's an issue with reading the data?

$\endgroup$
  • $\begingroup$ thank you for the answer. What do you mean with issue reading the data? In the column I have lists which are of different sizes, but in this case what should I do? Should I modify the test data? $\endgroup$ – J.D. Nov 6 at 16:37
  • $\begingroup$ You need to transform the test data in the same way as you did the train data, subsetting the lists and applying tf-idf. $\endgroup$ – Ben Reiniger Nov 6 at 17:25
  • $\begingroup$ thank's for answering. I did as you suggested, don't kwow if correctly, but I get an error. I edited my qiestion explaining what I did. $\endgroup$ – J.D. Nov 7 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.