syn--More folks probably create lens coating damage trying to "help" their lenses (believing that
anything on the surface of the lens will be in their photo) than will ever see a problem in their images. The simple fact is the image is not formed on the lens surface, so modest dust, spots etc. don't really need cleaning. Big blobs of "stuff" or seriously spotted optics or ports, well ok. In the wrong conditions with serious back-lighting you could easily get flare.
Coated, and especially multi-coated, lenses (and many ports) should get different treatment though than just plain glass. Be leery of any cleaner, they typically are not needed and some that may be ok for your reading glasses can contain solvents that will damage many types of lens coatings. At least you put the cleaner on the cloth. There are folks who drip it directly on the lens only to watch it race around the edges and leer at them from in between the lens elements.
More times than not a simple breath fogging of the lens and gentle, circular wiping with a proper cloth will do fine. For some reason a "proper" cloth is not what some manufacturers send out with their lenses. Back in ancient times (film days) Kodak sold a lens chamois that was great. Now days some of the artificial micro-fiber cloths made for glass cleaning might be acceptable but me, I still have a few lens chamois' stuck in film cans (another artifact of the past!) that I use.
Whatever, just do NOT soak these in anything, keep them dry and DO put them in a sealed container so as not to collect any grit, which would really wreck your day if scrubbed around on a pricey lens surface. As my old friend Les Izzmore used to say (alas, Les is no more): "Dust on my lenses? Yeah, shows I use 'em!". Of course he was the sort that would pull his shirt tail out and shove it around on his lenses. Somewhere in-between there lies a happy medium, eh?
The smears and swirls you mention won't bother your shots but they will bother you, especially if you keep breathing on your lens and looking for them! The right sort of micro-fiber cloth should remove them. For the OP's port concerns, if it is plain glass the previous postings could work. If you hold the port sideways to catch light and see any colors on either surface indicating a non-glare lens coating though you might want to be a bit more careful. Maybe test your cleaning method out near an edge first? Oh and if you take the potato route skip the butter, salt and pepper!
// ww