The problem is the people doing this maliciously are already breaking several laws and scamming people, and certainly don’t care whether it’s legal or not. They’re also outside of the US.
For instance an ex-girlfriend answered a phone one day that was from a bank number, Chase, which was saved in her Contacts, so she assumed it was really them. They told her some stupid story about a fradulent Zelle payment not going through and she swears she signed in to her bank and saw a $5000 payment there declined that then, she says, disappeared from the list. Then they told her she had to send money to a special service number so they could ‘backtrace’ the fraudulent payment. So she did (she was also credulous/dense/gullible I guess). Obviously, this was bullshit. it was especially annoying because I walked over a couple of times and asked her “wtf are you doing? Who is that?” and she just waved me away and ignored me. She ended up losing $600. So anyway, the main thing that gave them credibility to her was calling from the Chase number.