I'd say use common sense like anyplace. There are lots of urban areas I wouldn't hang out at at night. But we typically drive downtown for dinner almost every night and have never felt unsafe in any of the areas by hotels and restaurants, or other areas we've had occasion to go to at night.
We did have one little incident last year where some guy claimed to have been watching our car for us so it was safe while we gone, he was basically trying to shake us down for a tip, we pretty much ignored him, got in the car and drove away. (It probably does not hurt that my husband is a big guy.) We were parked on the road along the water near the cruiseship facility in the channel.