0
$\begingroup$

Any binary one-hot encoding is aware of only values seen in training, so features not encountered during fitting will be silently ignored. For real time, where you have millions of records in a second, and features have very high cardinality, you need to keep your hasher/mapper updated with the data.

How can we do an incremental update to the hasher (rather calculating the entire fit() every time we incounter a new feature-value pair)? What is the suggested approach here the tackle this?

$\endgroup$
1
$\begingroup$

If you're doing anything at the millions of records per second level, you're going to need something more high-performance than Python. Spark/PySpark is ideal for this since it interfaces with your existing Python knowledge and supports HDFS/distributed technologies that are made to take on the millions per second sized tasks.

As for sklearn: you might be out of luck here. The fit_transform() or fit() methods necessarily have to consider every record that you pass to them to keep the hash up to date, so if you have millions of data points coming in you're going to have to do millions of computations to keep moving, whether you do them one at a time or in a batch. The source code for the transform() method should convince you that doing either is equivalent.

One caveat is that if you have predictable features, e.g. maybe you're receiving counts of foo's bar's and baz's, like in the first link:

D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]

Then you don't need to update the hash every time, all you have to do is transform D into the vectorized format and add it to your collection. As long as you don't need to fit new features this can be done relatively fast. However, if you're doing something less structured like NLP this won't give you hardly any useful information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.