My wife and I recently completed the course. I'd say that the swimming is somewhat challenging but mostly because you want to achieve good times to treat the rescues as the real thing. We had to bring unresponsive divers to shore from 25-50 yeards (can't recall, seemed like a long distance, though!) while stripping their gear and giving rescue breaths. The rescues are tiring because you're giving rescue breaths along the way (and you have to fin up to get part of your body out of water to truly represent an actual rescue breath), getting out of your gear (buckles and so forth) and taking the victim's gear off. There were time limits, but all of us came in under the ceiling.
There's also a lot of panicked diver drills where they try to drown you, take your mask off, etc. It's all about learning how to react and remain calm. In fact, some rescues just call for swimming back to shore while a panicked diver tries to grab at you, until they realize there back at shore. But, then again, our instructors kept us guessing.
The lost diver drills are mostly about quick reaction and organization. But again, every drill is about time (oxygen deprivation, etc.), so you tire yourself out trying to do your best.
It's a great course and a lot of fun. I would try to stay in shape so you can help yourself and others when something goes wrong, but none of us were olympic swimmers.