Apparently it's impossible to bind to a port under 2000 with a non privilaged user. Not sure how I got this far into my career without knowing that.
Anyway the solution is to run this as a root user to give the binary permission.
sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/binary
This can be problematic as every-time the binary changes you have to do it again.