If you're moving at about the same speed as your flag, it's not so bad
And now my nerdy engineering background kicks in. Velocity profile is slowest at bottom (friction with bottom), next slowest at surface (air friction), and fastest at mid-depth. This can be significant. It takes no effort at all to pull under a pencil float with dealing with the flow. You also get significantly dragged from tube floats due to the friction component of it against the water surface.
River drift diving is a "unique" experience.....