I am trying to make a submission, so I have a test set without labels and I am tryin to test my classification model on it. In particular, I have also to submit this prediction as a csv. I have the following test set without labels, which is the output of pd.read_json(), so it is the output from the test dataset:

enter image description here

and the point of the classification problem is to predict from the instruction the type of the compiller. The classification problem is already developed, I just need to submit it.

So I have to predict these instructions from the test set, but if I try to do :

test = pd.read_json('test_dataset_blind.jsonl',lines = True)

X_new = test['instructions']
new_pred_class = clf.predict(X_new)

where clf is my model, in this case I am using random forests.

I get the following error message:

ValueError: setting an array element with a sequence.

Can anyone please help me? Thank's in advance.

[EDIT] The full error trace is the following:

    ValueError                                Traceback (most recent call last)
<ipython-input-21-ba881bb9e0fe> in <module>
----> 1 new_pred_class = clf.predict(X_new)

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
    543             The predicted classes.
    544         """
--> 545         proba = self.predict_proba(X)
    547         if self.n_outputs_ == 1:

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, 
    586         check_is_fitted(self, 'estimators_')
    587         # Check data
--> 588         X = self._validate_X_predict(X)
    590         # Assign chunk of trees to jobs

~\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py in 
_validate_X_predict(self, X)
    357                                  "call `fit` before exploiting the 
--> 359         return self.estimators_[0]._validate_X_predict(X, 
    361     @property

~\Anaconda3\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, 
X, check_input)
    389         """Validate X whenever one tries to predict, apply, 
    390         if check_input:
--> 391             X = check_array(X, dtype=DTYPE, accept_sparse="csr")
    392             if issparse(X) and (X.indices.dtype != np.intc or
    393                                 X.indptr.dtype != np.intc):

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, 
accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, 
ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, 
    494             try:
    495                 warnings.simplefilter('error', ComplexWarning)
--> 496                 array = np.asarray(array, dtype=dtype, order=order)
    497             except ComplexWarning:
    498                 raise ValueError("Complex data not supported\n"

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    537     """
--> 538     return array(a, dtype, copy=False, order=order)

~\Anaconda3\lib\site-packages\pandas\core\series.py in __array__(self, dtype)
    946             warnings.warn(msg, FutureWarning, stacklevel=3)
    947             dtype = "M8[ns]"
--> 948         return np.asarray(self.array, dtype)
    950     # ------------------------------------------------------------------ 

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    537     """
--> 538     return array(a, dtype, copy=False, order=order)

~\Anaconda3\lib\site-packages\pandas\core\arrays\numpy_.py in __array__(self, 
    165     def __array__(self, dtype=None):
--> 166         return np.asarray(self._ndarray, dtype=dtype)
    168     _HANDLED_TYPES = (np.ndarray, numbers.Number)

~\Anaconda3\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order)
    537     """
--> 538     return array(a, dtype, copy=False, order=order)

ValueError: setting an array element with a sequence.

[EDIT 2] The dataset with labels is the following:

enter image description here

and what I did is the following:

I considered just the operators push,mov,.. and then I created the following dataset with pandas:

enter image description here

after doing this I considered only the values and I used a tf ì-idf vectorizer. Then I slitted the data as;

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
    test_size=0.2, random_state=15)

and I used support vector machine as model.

[EDIT 3] I have achieved eliminated the error and as output I have:

array(['icc', 'gcc', 'gcc', ..., 'clang', 'clang', 'clang'], dtype=object)

now I do the following:

pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class})

and I get the error message:

ValueError                                Traceback (most recent call last)
<ipython-input-41-da853bce8ce2> in <module>
----> 1 pd.DataFrame({'instructions': 

~\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, 
index, columns, dtype, copy)
    409             )
    410         elif isinstance(data, dict):
--> 411             mgr = init_dict(data, index, columns, dtype=dtype)
    412         elif isinstance(data, ma.MaskedArray):
    413             import numpy.ma.mrecords as mrecords

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
init_dict(data, index, columns, dtype)
255             arr if not is_datetime64tz_dtype(arr) else arr.copy() for  
arr in arrays
    256         ]
--> 257     return arrays_to_mgr(arrays, data_names, index, columns, 

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
arrays_to_mgr(arrays, arr_names, index, columns, dtype)
 75     # figure out the index, if necessary
    76     if index is None:
---> 77         index = extract_index(arrays)
    78     else:
    79         index = ensure_index(index)

~\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in 
379                         "length {idx_len}".format(length=lengths[0], 
    380                     )
--> 381                     raise ValueError(msg)
    382             else:
    383                 index = ibase.default_index(lengths[0])

ValueError: array length 30000 does not match index length 3000

apparently I have that new_pred_class has 30000 elements, while the test dataset is 3000 rows. What should I do in this case? Thank's in advance.

  • 1
    $\begingroup$ Full error trace, please. And is your image the output from test? $\endgroup$
    – Ben Reiniger
    Nov 5, 2019 at 23:23
  • $\begingroup$ Thank's for your answer, and yes the image is the ouput from test. I have edited my question with the full error trace. $\endgroup$
    – J.D.
    Nov 6, 2019 at 10:53
  • $\begingroup$ How did you preprocess the data for training? $\endgroup$
    – Ben Reiniger
    Nov 6, 2019 at 14:37
  • $\begingroup$ thank you again. I have edited my question summarizing what I did $\endgroup$
    – J.D.
    Nov 6, 2019 at 16:34

1 Answer 1


The data that you're loading into the model is malformed.

Usually this error arises when not all of the elements in an array share the same size. You need to make sure that each row/array contains the same number of elements (otherwise you can't form a 2D array out of the data). Maybe there's an issue with reading the data?

  • $\begingroup$ thank you for the answer. What do you mean with issue reading the data? In the column I have lists which are of different sizes, but in this case what should I do? Should I modify the test data? $\endgroup$
    – J.D.
    Nov 6, 2019 at 16:37
  • $\begingroup$ You need to transform the test data in the same way as you did the train data, subsetting the lists and applying tf-idf. $\endgroup$
    – Ben Reiniger
    Nov 6, 2019 at 17:25
  • $\begingroup$ thank's for answering. I did as you suggested, don't kwow if correctly, but I get an error. I edited my qiestion explaining what I did. $\endgroup$
    – J.D.
    Nov 7, 2019 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.