Exactly. Historically, there have also been a couple of cases of Senators succeeding to the post when the incumbent PM died (for example, John Abbott ).
I’m not sure there’s even any formal legal barrier to the Governor General picking someone off the street at random; it’s just never done.