Personally, I love to swim through and around wrecks just to see. I do have friends and acquaintances who love to bring things back. I've even seen one guy using a baby sledge attempting to dislodge a brass prop from a torpedo! As previously stated, different areas have different rules. Most are unrestricted, while others are heavily to the contrary. The USS Monitor comes to mind, which was a protected NOAA site. Illegal to dive except by special permit. Expensive special permit, I should say. Some are in the "should not go inside" range, like sunken u-boats left behind by patrolling german crews off the atlantic coast.
In my part of the world, the carolina coast is very popular as the gulf stream comes close enough to shore, the continental shelf is relatively shallow, and there are plenty of wrecks there. Anything 120fsw and shallower is well picked over. Below 140 or so, not so much. I remember visiting the Naeco one fall (she lies at about 150fsw) and while everything was in shades of blue, lots of stuff went "clink" when struck together.
I know of one person in particular who used to go inside a german u-boat with a gear bag. He would shovel muck into the bag, then feel his way out and let the currents sift the muck back out of the bag. He found interesting things lie board-game pieces, and I think a cup or two.
So, the right thing to do is consider where you are diving, and on what. Make sure of local laws and such. Then find what moves you. My personal favorite is to take pictures at most, but as I said, usually I just love to drift through it.
Just my two cents.