"listeners" manipulation when bind-address set breaks ability to use SSL

[codenamelxl]

Since commit ed0ff0732990c2fc4a7f3ef192b15a34ca2f12b3
The listeners config is forced to only PLAINTEXT when bind-address is defined which effectively disable the ability to use more than one listeners.