I've not done this from a boat and only saw a couple boats stop here, but I'm talking about the Ukumehame Beach dive, just north of Coral Gardens and the last tunnel around the pali - I don't know if this is exactly what is referred to as Manta Point. It doesn't seem like a 'Point' to me...
If you go there most afternoons, I bet you'd see dive boats that would give you an
idea of the distance. It's a fair swim from shore, I'd estimate 300 yds minimum, but the real issue is that the first 200 yds are very shallow over the reef. It slopes gradually to maybe only 5 ft deep out at 200 yds, meaning that most of that distance is not swimmable over the reef in anything except the absolute flattest of swell conditions. To add to that, there are some inshore patches of higher reef top that you wouldn't want to pass over EVER!
There is however a sand channel you can swim OUT pretty easily. Back IN is a different matter. We encountered rip current in the channel - flowing away from the shore - that was all but impossible to swim against on one of the dives, and I would say the groundswell was only a couple of feet that day. Anything higher and I would seriously question whether you would be able to make your way back in. As it was, we swam off to the side of the channel, outside the rip, pulling ourselves along the bottom, until shallow enough that we could remove our fins and drive ourselves forward with our feet, up the sand channel. Given that you have to make 200 yds of headway across the shallows to hit shore, the line between manageable and IMPOSSIBLE at this spot is liable to be fine, and hard to assess precisely. It would be a long, long swim to anywhere with an easier exit, so you DON'T want to get stuck offshore by doing this on a chancy day - also don't forget the rip current strength probably varies with tidal state, which can change noticeably over the course of a 90 minute dive.
To find the entry, go north towards Lahaina from the Ukumehame parking lot a short distance, looking for some small sand dunes right along the edge of the road. The dunes are a little more than car-top height, and there are a couple of cuts through them that are about a car's width. The second to last cut, if I recall, is right about where the sand channel is. The channel bottom is clearly visible from the beach, and is about the width of a narrow two-lane road. There is more than one channel along that stretch but the one we used appeared to be the widest (and was described to us by a local), and heads more or less straight off shore. Once you get out to 5-6 ft depth the channel becomes less distinct. At that point we headed slightly leftwards, not quite at a 45 degree angle, until we got to 25 ft depth. We encountered the mantas over some nice patches of finger coral, but I have no idea how localized the cleaning stations are. My recollection is that the boats stopped well outside this area, but also a little bit farther south. Their divers swam through our area.
Good luck if you try it, just please pick your day carefully, and note a beach marker to help you find the channel back in!