I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 being meaningless.

I've used a RELU activation function for my output node as this seems to, at first glance, make most sense. However, this does (occasionally) lead to values higher than 27. I don't find this particularly problematic, because I can just post-hoc change all values >27 to 27. However, it has been suggested that I scale the data to be bound between 0 and 1 so I could use a sigmoid function as an activation function. However, I'm skeptical that this makes sense given the data. My data is linear, while the sigmoid function exponentially approaches both 0 and 1 with increasing difficulty, and is generally only used when fitting Boolean data. However, this is only an intuition, and I can't really support my hypothesis mathematically. Is there any reason a sigmoid function is or isn't a good fit for this data?

Bonus: anyone every implemented with Keras (R)? I've had issues even attempting to train a sigmoid function with non-Boolean data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.