I have the same problem with boat dives in unfamiliar places!
You can do a lot of navigation using natural features. For one thing, if you are diving off an anchored boat, make careful note of the depth where the anchor is sitting, and what the terrain around it looks like. (For example, is it sitting on top of a knob, or on a flat, sloping bottom, or in a pile of boulders?) Where the bottom has a distinct slope, you can pay attention to whether you are swimming with the shallow side on your right or on your left (getting home is the reverse). Most of our Red Sea dives were either on or near walls, or on sloping bottoms (or on wrecks, which are their own navigational landmarks). A compass is helpful if the visibility is too poor to orient on surrounding landscape, or if the bottom has very little slope or is extremely irregular.
Time can also be useful. If you know you swam 20 minutes in one direction, then the anchor line isn't likely to be more than 20 minutes back (unless you were swimming with, and then against the current).
Noting unusual features that you pass can help reassure you, too. I've even been sure I'd found the spot for the exit because we swam over the same furious nest-guarding fish we passed on the way out! My husband did some navigation in his IE by nudibranch

But anything that's likely to stay put and be recognizable is a way to reinforce your orientation.
I guess what it comes down to is that good navigation requires pretty good situational awareness, which means noticing more of what's around you than the particular fish or coral or sponge that's caught your eye. As a guided diver, you don't have to do this much, but as an independent diver, it's key.