I would say the housing can have an effect on image quality, but the skill of the photographer is much more important.
The lens port will have various aberrations associated with it that will degrade the image slightly. The same is true of any lens in front of the camera, but housing ports are not made to the same exacting standards as the lenses inside the camera. If they were, the cost would be much higher. The material of the lens port (often acrylic) will cause some chromatic aberrations that can cause colors (mostly near the edges of the frame) to blur some. The shape and position of the port can contribute to edges being blurred as well.
I guess it could go without saying, but scratches on the port will definitely reduce image quality.
As mentioned above, proper technique and a creative photographer is much more important than these small effects, At least for those of us who aren't pro underwater photographers.
