The downside is the leaves act as storage for mobile nutrients, cutting them early lowers reserves. But that’s only really an issue if something unexpected happens later in the grow. Admittedly a much larger risk when doing hydroponics than soil because the soil also acts as a buffer and nutrient store.
Secondarily, I’ve read that the larger fan leaves produce more sugars for the plant and should only be removed sparingly, and generally during the vegetation phase. I’ve had great luck following that methodology.
To be clear, I took defoliation to mean removing live, healthy leaves, in pursuit of a specific growth objective, not pruning detritus. My bad on that front.