I think you’re overthinking this. The presence of water is required to make clay minerals, and water has covered just about the entire surface of the planet. I’m also pretty sure the people making clay over 10,000 years ago didn’t do a fine analysis of the raw dirt they refined their clay from. This video also showed 3 ways to refine clay, depending on the clay content. And given that the profile of soil generally includes some amount of clay (loam consists of less than 40%, deserts are often over 90% sand and still have clay) pretty much any natural soil that grows plants probably has some amount of clay.
So yes, you need specific minerals to make clay, they just happen to be almost everywhere plants, and thus people, are found. And no, this won’t give you pure clay minerals, but if the clay isn’t white, that’s already the case.