It's not quite as simple as that.
Early cell phones had user replaceable batteries. It was often not even worth it as the battery pack was proprietary, so expensive, and only available for a short while. On top of that, there were conflicting demands placed on cell phone design. Presumably from what consumers were asking for. We wanted long battery life, big screens, fast processors, all sorts of radio communications (5G, WiFi, BT, etc.) and some level of water/contaminant protection. All while keeping the size and weight manageable. User replaceable battery packs eat up a lot of real estate, and on top of that you also need a means to keep water out. Years ago, I had a Motorola Defy, which was one of only a handful of IP67 rated phones. It actually had a removable battery than was under a gasket cover. It had a latch and the cover had a bit of a curve, so the latch would compress the gasket. Even so, I wouldn't trust that very deep. Battery pack probably accounted for 40% of the overall size. That's really not doable anymore today. So many cameras, and radios, memory, processors etc. the battery needs to be squeezed into whatever space is available.
And that's just for cell phones. Watches are much smaller, so constraints even greater.