Don't really want to sink too much money into it since photography won't be my main diving niche.
In that case you're better off with a gopro (or a knockoff) or a phone housing. As pointed out by
@tbone1004, D80 is ancient by now, with its housings long out of production, and while you might be able to get a used one, it's going to be
old and components like o-rings and springs wear out with time, so it will likely need servicing and refurbishing. A housing is also just the bare start - you will need lenses (ultrawide, fisheye, macro - types of lenses that are not very commonly used in land photography), ports, strobes, tray, arms, clamps, maybe a viewfinder - all of these are expensive. Finally, a DSLR rig is going to be
big unless you restrict your photography to a fisheye in a compact port with natural light. I dive with a crop mirrorless (Sony A6300), which is a lot more compact than a typical DSLR, and it still pretty much dominates every dive I take it on. With an action camera, a phone, or a small compact, you can dangle it off your wrist, or clip it to a D-ring, or stuff it into a pocket, but a large camera rig always demands attention. If you dive specifically to take photographs, then yes, a large rig makes sense (though housing a fifteen year old camera doesn't), but if you just want to take an occasional snapshot - get something small and easy to handle.