I am trying to embed a text data which is in the form of list, since its a huge data I wanted to embed it using the multiprocessing Pool map() function.
The embedding technique I'm using is google's universal sentence encoder - 'Transformer Architecture' which outputs the text into 512 dimensions.
When I tried encoding with a straight forward method, that is without the multiprocessing Pool, it works perfectly and gives the output is 7.96sec.
input is a list of sentences , for eg:
test = [hey how you doing?, did you eat? , how was your day?, the car looks so new, the bike is mine,......] tf.logging.set_verbosity(tf.logging.ERROR) with tf.Session() as session: session.run([tf.global_variables_initializer(), tf.tables_initializer()]) message_embeddings = session.run(embed(test))
But When I try to use the Pool map() function it keeps on running for a very long time. the code is given below.
import multiprocessing as mp pool = mp.Pool(mp.cpu_count()) def embed(row): tf.logging.set_verbosity(tf.logging.ERROR) with tf.Session() as session: session.run([tf.global_variables_initializer(), tf.tables_initializer()]) message_embeddings = session.run(embed(row)) return(message_embeddings) result = pool.map(embed, [row for row in test])
The expected output is a 2D matrix of shape (10, 512), where 10 is the number of sentences in the list sent as the input and 512 dimensions is the dimension in which the sentence encoder converts it into.
for eg: array([[ 0.03187958, -0.0350634 , -0.04103173, ..., -0.05661938, -0.04859127, 0.05627369], [ 0.01406917, 0.06345133, -0.06354789, ..., -0.0324694 , 0.0584636 , -0.00601132], [ 0.02827099, -0.01823348, -0.02626591, ..., -0.06193804, 0.04049303, 0.08589993], ...,
I want to know if my code is correct or not, because all the solution I find online tells me that the syntax is correct, except for the fact that all the examples online are not related to sentence encoding process but I don't think that has something to do with the Multiprocessing.
Any help is hugely appreciated.