# How to chose the right activation function for CNN output depending on the output value ranges?

I'm working on training a CNN model that takes an eye image as input and outputs the 5 coordinates of the ellipse representing the pupil (center x, center y, major axis, minor axis, angle). I started by training a small CNN with a linear activation function on the output layer. This worked, but the model was producing negative values, which while valid coordinates, ideally I'd prefer to have all positive outputs.

So I changed the output activation function to ReLU, which ensures all outputs are non-negative. This seems to be working well, but I've read that using ReLU on the output layer is not always recommended.

If I use the sigmoid function it will have an upper-bound, but to do so I have to normalize my labels to the 0-1 range. My images are not squares, I was thinking about normalizing x and y coordinates by respectively width and height but the ellipse major/minor axes are not necessarily aligned with the x/y axes. Could I normalize all coordinates and lengths by the diagonal of my image for example?

And if I use the sigmoid function I have the feeling it will less likely output values close to 0 and 1, while having an angle of 0 (0°) or 1 (180°) is as likely as an angle of 0.5 (90°). Should I use a hard sigmoid function instead?

I'm open to any other suggestions you may have as well. I am pretty new to deep learning and I would like to know and understand the best practice.

If I use the sigmoid function it will have an upper-bound, but to do so I have to normalize my labels to the 0-1 range. My images are not squares, I was thinking about normalizing x and y coordinates by respectively width and height [...]

That sounds right to me. The net will then predict x/y location as a proportion of the image's width/height (relative to the corner), rather than making predictions that will depend on the image's absolute pixel dimensions.

[...] but the ellipse major/minor axes are not necessarily aligned with the x/y axes. Could I normalize all coordinates and lengths by the diagonal of my image for example?

For the axes, apart from their origin, I presume you have: major axis length, minor axis length, major axis angle. Scaling to the diagonal sounds sensible so that the targets are bounded up to 1.0. When a net predicts the major/minor axis lengths, neither of them would need to exceed 1, so you could safely use a sigmoid. The net will thus be predicting 'major/minor axis length as a proportion of the diagonal'.

And if I use the sigmoid function I have the feeling it will less likely output values close to 0 and 1, [...]

I concur with that intuition, but I don't think it necessarily prevents the net from converging on the right values. I'd opt for sigmoid, and if you find that pupils closer to the edges are being systematically under-estimated, then you could explore ways of dealing with it like a hard sigmoid. My feeling is that whilst a hard sigmoid might help with edge behaviour, it also has a flat part that would inhibit gradient updates if a neuron landed there. So I'd go for sigmoid initially, and iterate on that if necessary once you've got a baseline model going. I think sigmoid has been used for coordinates in YOLO and DETR.

[...] while having an angle of 0 (0°) or 1 (180°) is as likely as an angle of 0.5 (90°).

I think angle could be measured as the counter-clockwise rotation about (x, y) between the x-axis and the long axis. It wouldn't be affected by the length of the long axis, because the idea is you'll extend the long axis out either side of the origin and find the angle between that line and x (thus limiting it between 0°-180°).

I can't speak to what the standard practice is in the field of image processing, but above is how I'd approach things based on my general experience with nets.