For something like an outboard motor, if I knew anything about it, I'd just go with the dry weight. Naturally, that'll be an overestimate of the lift required, but if it's just a big hunk of metal, it's pretty close. If I didn't know anything about it, I'd basically just guess. I've got 50-, 100-, and 200-pound lift bags. I'd guess that it'd be too big for the 50 (unless it was really small), so I'd probably start at 100 and go from there.
The nice thing is that a full open-bottom lift bag will not increase in lift during ascent. As the air in it expands, it'll just spill out the bottom. So, if you've got, say, two 100-pound bags on something that's only 150 pounds, you could fill one bag completely and control it with the other alone. Of course, if it's much larger than an anchor or a small outboard, I'm going to use lines. If it's big and unwieldy, trying to control it mid-water can be a bad idea.
Of course, there's also the "stepladder" style of lifting large things. You have a line with loops every 10 feet (or whatever) from the item all the way to the surface. You attach to the first submerged loop and lift that to the surface. (No need to worry about runaway, as you're just lifting 10 feet.) You leave that bag in place while you attach to the next loop (which will now be 10 feet from the surface), and you lift *that*. Then you detach the first bag and lift the *next* loop, and so on, until the item is all the way up. By doing the lift a segment at a time, you minimize the "rocket to the surface, plummet back to the bottom" potential.