Its best to do this when the server is open to the public and at 48-50/50 but heres the best way I have found.
Check the server population and take note of it... then go into the change servers and filter by minimum players to just below the number you saw on the server. Sort it, then basically scroll through until you see it and favorite it from there.
Its best to do when its a near full server because that really limits the list and makes it a quicker find. For whatever reason it seems that you can only favorite servers from the main list, not the remote list.
Hope that helps!