We all would like the world to be a matter of black and white, yes or no, 1 or 0, but there is a spectrum between healthy and "immuno-compromised" and there are all kinds of environments. I see people at the bus stop who look like they have health issues and are laboring just to get themselves to the supermarket and back home. If younger and slightly more healthy, they may be commuting to a job. There are a whole lot of semi-unhealthy people who are susceptible to serious damage from Covid, and these same people can't afford to stay home. Until ventilation on public transport and in stores is improved, masking in these places "on the off chance" it may help susceptible people makes sense to me. Nobody is asking us to wear masks when walking outdoors or in large, well-ventilated places anymore. It's not a matter of masking or not masking but rather looking at where we are and who we are likely to come into close contact with.
In your post you mentioned you would be willing to wear a mask if you knew you were visiting an immuno-compromised person. So, why not extend that, based on your own judgment on the spot? In some situations, you can be pretty sure there are no such persons around, pretty sure the ventilation is adequate, etc., but in other situations you might know it's likely you will come into close contact with people with health issues. These days, I keep a mask in my pocket and use it when my judgment tells me it might help.