Part of it I think it’s that you’re describing what the general population would call swinging or hotwifing. Cuckolding has an implied humiliation/non-consensual connotation in the public mind so that’s why it’s seen as derogatory. A cuck isn’t having fun, or might not even know it’s happening.
Husband is on-board and good communication? Not a cuck
Husband is seen as weak/humiliated, wife unilaterally controls it? Cuck
Whether these definitions are “correct” or not, that’s the public perception