Cayman Brac is terrific. That having been said, the single best dive I did when I stayed at Brac Reef Beach Resort was on Bloody Bay Wall on Little Cayman (the two islands are only 7 miles apart, so they go over there for 2 dives once during the week if conditions allow and if passengers are willing to pay a fuel surcharge).
You really can't go wrong with either, but based on my experience, I'd say Little Cayman is probably slightly better. We certainly saw more big stuff over there. Brac might have more different types of sites - there's a wall, shallow coral gardens, or a terrific wreck (the Captain Keith Tibbetts) depending on where you go.
James