You're best bet is probably Dive Otago - HOME | Dive Otago. They will be able to rent you gear and tell you where to go.
The only diving I've done around Dunedin was at Aramoana Mole, a short walk if you don't have the key to the gate - but not arduous. Good diving, some OK wrecks, but cold...