Hmm... I seem to have offended some folks and I apologize. Let me see if I can clarify.
First off, Aruba certainly has some nice diving. It's warm and the water is clear. I come from Boston so just ANYTHING would seem nice to me. However, over the 10 year period between visits to Aruba, a LOT has changed and the diving is not exempt. The reefs and the wrecks have suffered from storms and over use. I even had an unfortunate incident regarding payment with one of the more popular dive ops, though I will not mention them by name.
Crime: Please note that my caution is aimed at Aruba's widely known "no crime" PR campaign. For anyone that's been to Aruba in the last 10+ years, you know how heavily they push the "no crime" message and it's simply untrue. On my most recent visit, people in our group had anything and everything offered to them (drugs - lots, prostitution, you name it). Does it make it "unsafe"? Hardly. I can walk out my office door and find all of those things at lunch time. But it hardly lives up to the squeaky clean image that they try to portray and the uninformed may be lulled into a false sense of security and be very disappointed to learn their lesson the hard way.
That said, the OP inquired about staying outside of the tourist zone. If you've seen the rest of Aruba (jeep tours don't count), you would agree that there is no place outside of the hotel area that you could knowingly recommend to anyone. The fact is that it's a man made island who's sole purpose is tourism. Look under the thin veneer a bit and you'll see what I mean.
So, now I REALLY feel like I've rained on the OP's vacation plans. Please accept my apologies. These negatives aside, you can certainly have some fun in Aruba and with only 3 days, there's no shortage of diving. Any of the places mentioned in this thread are worth a boat ride and a tank of air. Throw in a sunset cruise and a nice dinner and you'll have a great time!