There's no way, from the lobby, to select which server you join.
At the moment, that means that if you want to play on a specific server, you have to join the server, and then call votes to get the settings you want.
So, in the console,
callvote map blabla
callvote changedifficulty blabla
etc.
There's also callvote restartround or something, which I find useful, as the game starts as soon as one person joins the server. Once everyone is in, you can restart the game.

[EDIT] Disclaimer: The only callvote arugment I can actually remember is kick, so the above may be wrong, but you get a little 'auto complete' box that should help you out [/EDIT]