I like to eat nice food. When I have a chance, I will kill a fish, whether by pole and line or spear. Do I do it just for the sake of killing it? No, I do it so I can eat it. Can I go to fish market and buy it? Yes, but I do not know how fresh it is, and besides, that kind of fishing is probably more detrimental to the environment because of bycatch.
On the other hand, when I want to eat roasted lamb or pig, I butcher it myself. First, I can do it as efficiently as possible, to prevent it to suffer unnecessary, and I can use money for other purposes, like going for a dive.
You don't like it and you will look down on me? Newsflash, I do not give a **** if you look down on me. Live and let live.
To get back to the OP, I take empty shells all the time. While it is technically illegal to do so, I can see kids bringing dead shells out of water all the time and LEO not giving a single look. I will not take a live organism, unless I am going to eat it. I will also not take a souvenir from a wreck. So, just use some common sense and you should be OK.