Timeline for How do the linear layers in the attention mechanism work?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 23, 2021 at 21:26 | comment | added | noe | Yes, that is correct. I'm glad the answer was useful. | |
| Jan 23, 2021 at 18:28 | vote | accept | T Piper | ||
| Jan 23, 2021 at 18:27 | comment | added | T Piper |
thank you for your detailed response! So when I have 1 head and input of shape (5 x 128), the W_k/q/v matrices are simple shaped (128 x 128) so that when I do matmul(input, W_k/q/v) I get the output matrix of shape (5 x 128) right? I was just always thinking about linear layers that the input "must/should" be a 1d vector but your answer made it clear thanks :)
|
|
| Jan 22, 2021 at 16:34 | history | edited | noe | CC BY-SA 4.0 |
added 20 characters in body
|
| Jan 22, 2021 at 16:27 | history | edited | noe | CC BY-SA 4.0 |
added 20 characters in body
|
| Jan 22, 2021 at 16:22 | history | edited | noe | CC BY-SA 4.0 |
added 20 characters in body
|
| Jan 22, 2021 at 13:36 | history | edited | noe | CC BY-SA 4.0 |
added 38 characters in body
|
| Jan 22, 2021 at 13:31 | history | edited | noe | CC BY-SA 4.0 |
added 47 characters in body
|
| Jan 22, 2021 at 12:31 | history | edited | noe | CC BY-SA 4.0 |
added 5 characters in body
|
| Jan 22, 2021 at 12:06 | history | answered | noe | CC BY-SA 4.0 |