Well I would have been glad that the boat didn't cancel, given the conditions you state. It's something you have to decide yourself if the charter won't cancel. Even if it means no refund it's better to call the dive than do something you feel is unsafe.
I haven't been out in a boat that I have felt has pushed it a bit with the weather but I have a high tolerance for bad weather. I like the fact that there are charters that have different tolerances for weather. Some go out in all kinds of conditions and that suits me but if I am diving with someone new or who gets seasick I'll pick a charter that is more likely to cancel if the weather is iffy.
Before each dive I look up all the weather info to see what to expect. Most charters will give a refund if you cancel more than 24 hours out and locally to me you can get a pretty good idea of the weather a day before.