I try to spread out the prep and clean up over several days. Keeping the gear always ready to go is a big time saver as well. Loading the vehicle the night before the drive helps too.
Driving to the diving site after work lessons the stress on diving day for me. I live and work several hours' drive away.
I normally spend the night before and the night after a dive day in a local motel. Most gear can be rinsed there, at the motel, especially if they cater to scuba divers and have a rinse area. Then you can also let the gear dry out overnight, and go have dinner locally and enjoy whatever the diving area has to offer in vistas or entertainment.
Driving back is usually the morning or afternoon of the last weekend day or holiday.
Re-rinsing, drying, and staging for the next dive can then happen at home, several days later.
Trying to do all this in 1 or 2 days can be exhausting, yes. Especially if you also must move gear from your vehicle to a tarp, and back, as well.
When I am diving at a resort, I am usually one of the few, if not the only person, who brings my gear back to the hotel room to rinse it myself. I normally do this after every dive day, before going down to get dinner. I don't like leaving my gear on the boat, ever.