Padang Bai main beach is used by the fishermen, so although you can still use it, it will not be empty by any imagination. There are two other beaches reached from Padang Bai; Blue Lagoon, and Bias Tugel.
Blue Lagoon is smaller, has more facilities (small restaurants, toilets, sun loungers)...