I think the contention of this comes from something else. Reading you write this is not troubling, because I know you're acting in good faith. The problem is that when the likes of a Matt Walsh or a Ben Shapiro say "Trans women aren't women", they're not trying to be definitionally accurate, or biologically accurate, or semantically accurate... they're trying to find a way to discriminate.
Are they biologically different from women? Yes. But the only people who should really give two hoots about that are doctors and biologists. When people outside those groups appear to be getting very het up about defining them as different, it's usually for a nefarious reason. Neil DeGrasse Tyson pinned Shapiro down on this, and I think he sums up the reason it's contentious to hear people arguing about whether trans women are women pretty well here. The whole thing is worth watching, but I've linked a timestamp for the most salient bit.