Size: Most manufacturers specify a 2 - 012 0-ring. Some specify 2 - 011. Either will work, for different reasons. A 011 will expand slightly as tank pressure is applied and jam into the corner between the reg body and the hose. For most regulators, a 012 is slightly oversized and will scrunch into place and seal just as well. This is the preferred mechanism, but you don't want the O-ring to wrinkle microscopically, hence a little lube.
EDIT: See more info in post #10 below!
Lube: This o-ring is a static o-ring. There is no movement once it's snugged up. Therefore there's no need for lubrication since the O-ring or the part doesn't slide. But you don't want it to catch microscopically on the fittings when it's first installed. A little lube helps there but doesn't help the seal. For nitrox you don't need an 02-compatible lubricant, or for that matter, any lube at all, if it seals.
Material: Viton has the highest Ignition temperature and is commonly used in oxygen environments. Nitrox doesn't count here. So what else is important? Viton is the poorest at taking a set. It will indent permanently, sooner than the other two. But that's more a theoretical concern than a real one if you use a 012 o-ring. If you use a 011, then it might have slightly more significance. In other words, I wouldn't use 2-011 viton here.
Between EPDM and Nitrile, the only significance is shelf life. If you grab a hardware store nitrile o-ring, it might have come from China with high carbon black content and already be ready to crack. From a reputable manufacturer, it's a great and inexpensive choice.
EPDM lasts longer and has great flexibility. Many manufacturers use EPDM in their service kits.
Durometer: As a high pressure seal, the standard would be to use a duro 90 for this hose. That said, I've seen many manufacturer service kits where the high pressure hose O-ring is definitely softer than that. A Duro 70 will seal better, but extrude easily if your hose unscrews even a hair.
So after all that, what's best? In my mind, a duro 90 EPDM 2 - 012 o-ring with a wisp of lube on it. If that means a special trip or special order, then just drop by the hardware store with wet lips.
And don't snug that hose too tightly. Remember, the reg body is just brass.
The definitive answer lies in
@Akimbo 's
1: O-rings for Divers Question: I'm a recreational diver. Why should I care about O-rings? Answer: For starters, an O-ring failure could empty your Scuba tank in about a minute. (Table Reprinted by permission) This table appeared in Life Ending Seconds, 3000 to Zero in 72 Seconds, Advanced...
scubaboard.com