I haven't read all the posts on this thread so forgive any repetition. As has been mentioned, diving site and conditions have a profound effect on a new diver's first impressions. Even if you were experienced, it doesn't sound like your dive would have been fun at all.
However, I would also like to mention that pre-dive prep is important. Sometimes being rushed as you kit up, or struggle with your exposure suit can raise your stress level. Being in a crowded area with everyone buzzing around getting their equipment together can be chaotic... especially on a day boat. Try to plan ahead. Take your time to set up your gear at a relaxed pace, try to be ready to go before the mad rush to jump in the water. When putting on your wet suit, slow down and don't struggle as it can get tiring and also raise your heart rate. Go over your equipment methodically and slowly and do the same during your buddy check. Take some time to chill before the dive and get into the right state of mind. The more prepared and relaxed you are going into the water, the better off you will be as the dive progresses. Believe me, it makes a difference... especially when you're just starting out and trying to get comfortable in the water and your equipment.
Lastly, being properly weighted is very important and it may take a few dives to fine tune it, but it can make a big difference in how streamlined you are moving through the water. If you're overweighted you will try to compensate by over-filling your BC which creates excessive drag and unnecessary effort to move in the water... that can get tiring and you will be using up more gas at a faster rate... this faster rate of breathing can also be a cause of stress. When you're weighting is optimal, you can move with ease and breathe at a slower rate... the buoyancy control will come from your lungs. So again, the more relaxed you are, the calmer your breathing will be... and that creates a better mindset as you progress through the dive - hopefully letting you enjoy the scenery.
Good luck and take it easy.