The anxiety I was thinking of is a feeling, while physiological responses are stress that, in adverse circumstances, easily turns into anxiety.
Past struggles don't make people "stronger" or tougher, but even more sensitive and traumatized. So if one has, for any reason, experienced eg. a lot of anxiety in childhood (like just about all autists have), we are likely be more sensitive.
The big question to me is, is it unavoidable for an autistic person to experience excessive amount of stress from early on? Today, in practice it probably is. But I can quite easily imagine a different world. So how much can this broken world of ours be fixed?
Well, IMO, we should not predict it but instead try find out by doing our best to promote change.