1
$\begingroup$
from keras.models import Model
from keras.layers import Input, Dense, Dropout, Embedding, Conv1D, MaxPooling1D, Flatten, Bidirectional, GRU, Concatenate, Lambda, Multiply, Permute, RepeatVector,dot



text_input = tf.keras.layers.Input(shape=(), dtype=tf.string)
encoder_inputs = preprocessor(text_input)
outputs = encoder(encoder_inputs)
#pooled_output = outputs["pooled_output"]      # [batch_size, 768].

sequence_output = outputs["sequence_output"]
dropout_layer = Dropout(0.3)(sequence_output)  
# add BiGRU layer with attention mechanism
bigru_output= Bidirectional(GRU(units=64,activation='tanh',return_sequences=True))(dropout_layer)


# Add a CNN layer
conv_layer1 = Conv1D(filters=128, kernel_size=2, activation='relu',padding="same")(bigru_output)
conv_layer2 = Conv1D(filters=128, kernel_size=3, activation='relu',padding="same")(bigru_output)
conv_layer3 = Conv1D(filters=128, kernel_size=4, activation='relu',padding="same")(bigru_output)
# max_pool_layer = MaxPooling1D(pool_size=2)(conv_layer)
conv_layer= tf.keras.layers.Concatenate()([conv_layer1,conv_layer2,conv_layer3])
# Add a dropout layer after the CNN layer
conv_layer = Dropout(0.3)(conv_layer)
# Map each cnn output vector to a unique context vector using a Dense layer
context_vectors = Dense(128, activation='tanh')(conv_layer)

# Define a function to compute attention scores
def compute_attention_score(context_vector, query_vector):
    """
    Computes the attention score between a context vector and a query vector.
    """
    score = dot([context_vector, query_vector], axes=[1, 1])
    score = Activation('softmax')(score)
    return score

# Compute attention scores for each context vector using a lambda function
attention_scores = Lambda(lambda x: compute_attention_score(x[0], x[1]))([context_vectors, bigru_output])

# Compute the weighted sum of the context vectors using the attention scores
weighted_context_vectors = Lambda(lambda x: dot([x[0], x[1]], axes=[1, 1]))([attention_scores, context_vectors])

# Concatenate the weighted context vectors with the BiGRU output vector
attention_output = Lambda(lambda x: tf.concat([x[0], x[1]], axis=-1))([bigru_output, weighted_context_vectors])


# Add max pooling layer
max_pool_layer = MaxPooling1D(pool_size=2)(attention_output)

# Flatten and add dense layer for final output
flatten_layer = Flatten()(attention_output)
output_layer = Dense(units=1, activation='sigmoid')(flatten_layer)


# define the model
model = Model(name="BBRCA",inputs=text_input, outputs=output_layer)
$\endgroup$
3
  • 2
    $\begingroup$ Welcome to DataScienceSE. Please add more detail: what is the task, data? size of the training data? How many classes? Distribution of the classes? Accuracy of 0.77 can be very good (if there are many classes and the task is hard) or very bad (if it's binary and easy, like spam classification), there's no way to know without more info. Btw accuracy is rarely a good way to measure performance. $\endgroup$
    – Erwan
    Apr 14, 2023 at 10:24
  • $\begingroup$ Thank you for the help @Erwan ! I am having problem to classify amazon questions into pre-purchase and post-purchase category and training data is of 160,000 and validation data is of 80000 records. Also the above model is giving validation accuracy of 99% on spam-ham dataset. $\endgroup$
    – rutvi
    Apr 14, 2023 at 11:55
  • $\begingroup$ the posted code would fail with indentation errors $\endgroup$ Apr 15, 2023 at 16:50

1 Answer 1

1
$\begingroup$

Your question can be given a general answer along the lines of the comment from @Erwan . Scoring highly on a chosen metric (and as he mentioned accuracy is only one type and possibly not the best one) is affected by a myriad of considerations. Trying to compare the results of a metric on completely distinct types of problems does not have any meaning

  • The datasets are different
  • The objectives to be measured are different

There should be no expectation of results on the one hand having any carry-over meaning to the other. Even if you had 99% precisely on both datasets it would not indicate that the ham-spam results were indicative of likely statistical strength on the second problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.