@Kevin W. Blaylock
As
@TMHeimer said. You don't mention the visibility or the amount of light.
As has been said, cold, or overly tight hood or suit can present a problem.
The other consideration is visibility and light. Even if the visibility was good, but If it was dark, that can raise the stress.
One of the UK tech instructors told a story of when he first started diving. On his first wreck dive. He got to the bottom of the shot with his buddy, it was dark as anything. His breathing rate was through the roof, he was just about to signal up and leave, and his buddy turned on this huge torch. They where in a hold, and suddenly it was as if they where in room with flood lights. He had a great dive, when he got home, he went out and bought the biggest torch he could find.
You did the right thing by aborting the dive. If you are uncomfortable again.
STOP.
Let your buddy know you have a problem.
Breath in for a count of four, breath out for a count of four. - do this a few times.
If you are still unhappy, then abort the dive or ascend a bit until you feel comfortable.
Cold, poor visibility, darkness, physiological stress (concern over being deep), working hard, breathing restrictions (tight suit), all are potential contributors. A tight hood freaks a lot of people out.
Continue your shallow dives, and make small progressive increases in depth. Ensuring you are comfortable at the new depth before the next increment. Make use of the topography, a gentle slope gives you more control than a wall.
Also, keep the work load down, swim slowly rather than sprinting, keeping your breathing relaxed.
It is always better to stop and regain control, than press on. If you are out of breath, if you have a kit problem, if you are apprehensive, stop, regain control. If things can't be resolved, ascend.