# problem submitting classification problem

I am trying to make a submission, so I have a test set without labels and I am tryin to test my classification model on it. In particular, I have also to submit this prediction as a csv. I have the following test set without labels, which is the output of pd.read_json(), so it is the output from the test dataset:

and the point of the classification problem is to predict from the instruction the type of the compiller. The classification problem is already developed, I just need to submit it.

So I have to predict these instructions from the test set, but if I try to do :

test = pd.read_json('test_dataset_blind.jsonl',lines = True)
test

X_new = test['instructions']
new_pred_class = clf.predict(X_new)

where clf is my model, in this case I am using random forests.

I get the following error message:

ValueError: setting an array element with a sequence.

Can anyone please help me? Thank's in advance.

[EDIT] The full error trace is the following:

ValueError                                Traceback (most recent call last)
<ipython-input-21-ba881bb9e0fe> in <module>
----> 1 new_pred_class = clf.predict(X_new)

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
543             The predicted classes.
544         """
--> 545         proba = self.predict_proba(X)
546
547         if self.n_outputs_ == 1:

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self,
X)
586         check_is_fitted(self, 'estimators_')
587         # Check data
--> 588         X = self._validate_X_predict(X)
589
590         # Assign chunk of trees to jobs

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in
_validate_X_predict(self, X)
357                                  "call fit before exploiting the
model.")
358
--> 359         return self.estimators_[0]._validate_X_predict(X,
check_input=True)
360
361     @property

~\Anaconda3\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self,
X, check_input)
389         """Validate X whenever one tries to predict, apply,
predict_proba"""
390         if check_input:
--> 391             X = check_array(X, dtype=DTYPE, accept_sparse="csr")
392             if issparse(X) and (X.indices.dtype != np.intc or
393                                 X.indptr.dtype != np.intc):

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array,
accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite,
ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype,
estimator)
494             try:
495                 warnings.simplefilter('error', ComplexWarning)
--> 496                 array = np.asarray(array, dtype=dtype, order=order)
497             except ComplexWarning:
498                 raise ValueError("Complex data not supported\n"

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
536
537     """
--> 538     return array(a, dtype, copy=False, order=order)
539
540

~\Anaconda3\lib\site-packages\pandas\core\series.py in __array__(self, dtype)
946             warnings.warn(msg, FutureWarning, stacklevel=3)
947             dtype = "M8[ns]"
--> 948         return np.asarray(self.array, dtype)
949
950     # ------------------------------------------------------------------

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
536
537     """
--> 538     return array(a, dtype, copy=False, order=order)
539
540

~\Anaconda3\lib\site-packages\pandas\core\arrays\numpy_.py in __array__(self,
dtype)
164
165     def __array__(self, dtype=None):
--> 166         return np.asarray(self._ndarray, dtype=dtype)
167
168     _HANDLED_TYPES = (np.ndarray, numbers.Number)

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
536
537     """
--> 538     return array(a, dtype, copy=False, order=order)
539
540

ValueError: setting an array element with a sequence.

[EDIT 2] The dataset with labels is the following:

and what I did is the following:

I considered just the operators push,mov,.. and then I created the following dataset with pandas:

after doing this I considered only the values and I used a tf ì-idf vectorizer. Then I slitted the data as;

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
test_size=0.2, random_state=15)

and I used support vector machine as model.

[EDIT 3] I have achieved eliminated the error and as output I have:

array(['icc', 'gcc', 'gcc', ..., 'clang', 'clang', 'clang'], dtype=object)

now I do the following:

pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class})

and I get the error message:

ValueError                                Traceback (most recent call last)
<ipython-input-41-da853bce8ce2> in <module>
----> 1 pd.DataFrame({'instructions':
test['instructions'],'compiler':new_pred_class})

~\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data,
index, columns, dtype, copy)
409             )
410         elif isinstance(data, dict):
--> 411             mgr = init_dict(data, index, columns, dtype=dtype)
412         elif isinstance(data, ma.MaskedArray):
413             import numpy.ma.mrecords as mrecords

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in
init_dict(data, index, columns, dtype)
255             arr if not is_datetime64tz_dtype(arr) else arr.copy() for
arr in arrays
256         ]
--> 257     return arrays_to_mgr(arrays, data_names, index, columns,
dtype=dtype)
258
259

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in
arrays_to_mgr(arrays, arr_names, index, columns, dtype)
75     # figure out the index, if necessary
76     if index is None:
---> 77         index = extract_index(arrays)
78     else:
79         index = ensure_index(index)

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in
extract_index(data)
379                         "length {idx_len}".format(length=lengths[0],
idx_len=len(index))
380                     )
--> 381                     raise ValueError(msg)
382             else:
383                 index = ibase.default_index(lengths[0])

ValueError: array length 30000 does not match index length 3000

apparently I have that new_pred_class has 30000 elements, while the test dataset is 3000 rows. What should I do in this case? Thank's in advance.

• Full error trace, please. And is your image the output from test? – Ben Reiniger Nov 5 '19 at 23:23
• Thank's for your answer, and yes the image is the ouput from test. I have edited my question with the full error trace. – J.D. Nov 6 '19 at 10:53
• How did you preprocess the data for training? – Ben Reiniger Nov 6 '19 at 14:37
• thank you again. I have edited my question summarizing what I did – J.D. Nov 6 '19 at 16:34

## 1 Answer

The data that you're loading into the model is malformed.

Usually this error arises when not all of the elements in an array share the same size. You need to make sure that each row/array contains the same number of elements (otherwise you can't form a 2D array out of the data). Maybe there's an issue with reading the data?

• thank you for the answer. What do you mean with issue reading the data? In the column I have lists which are of different sizes, but in this case what should I do? Should I modify the test data? – J.D. Nov 6 '19 at 16:37
• You need to transform the test data in the same way as you did the train data, subsetting the lists and applying tf-idf. – Ben Reiniger Nov 6 '19 at 17:25
• thank's for answering. I did as you suggested, don't kwow if correctly, but I get an error. I edited my qiestion explaining what I did. – J.D. Nov 7 '19 at 7:04