No, don't give up!
Diving is completely different from swimming, except that at the start and end of a dive you find yourself on the surface and that's where a lot of incidents can occur -- if you are in 40' of water and not confident in deep water (BCD or not), panic
can set in under the right circumstances. That's where what they call
watermanship makes all the difference.
Only you can say for sure if your skills could really use some improvement. We can obviously only go on what you tell us here, but it sounds to me like it could be the case. If your heart beats quicker as you swim from the shallow end to the deep end of the pool, that's a pretty accurate sign.

If so, maybe consider some coaching (swim coaches aren't that hard to find if you are already at a pool). Or just practice, laps are a confidence builder and will build your endurance in addition to all-around fitness. Another confidence builder that I know Walter recommends highly is skin diving skills, which you can also practice in your local pool.
As a self-taught swimmer I could meet the requirements of six untimed laps in a pool, but it took
forever on my back and any attempt at freestyle left me completely winded (lack of technique).
At that time the swim test for me was a means to an end, I just wanted to dive. I'd bet that sounds familiar.

Eventually, a couple of situations on the surface and a timed swim test for a class I wanted to take got me to take the advice I'm suggesting to you. I have never regretted it.
FWIW, I now swim a mile +/- (1600m = 3 sets of 10 laps each, plus a couple, in a 25m pool) two to four times a week. Not only am I in better shape than I have been in a long time, my diving is more fun, I'm more confident in dicey surface conditions, and my air consumption has improved.
YMMV, but probably not. :14: HTH!