It’s not about “all life” but about “all sentient life”. Only beings that are able have pleasant and unpleasant experience should be considered. If something (living or not) cannot experience suffering then you can’t harm it, by definition.
Sentience is studied scientifically. It cannot be stated with absolute certainty but scientists have good sets of criteria and experiences that helps identify it. With the current knowledge it’s almost certain that all mammals are sentient, like us. Fishes and birds are also very likely to be sentient. Some species of insects are probably sentient while others may not be. And plants are likely not sentient.
But even if all living things are sentient, it doesn’t change very much. Speciesism means treating beings differently only because they belong to some specific species. There are good reasons to treat different beings differently but they should be based on the beings’ interests, not their species (and studying sentience helps identifying these interests). It’s very likely that we do less harm by growing plants than by breeding animals. And even if it was the same amount of suffering we would still do less harm by avoiding eating animals because breeding them to eat them actually requires more plants than just eating plants. We should seek to minimise suffering and avoiding eating animal is a good way to do that.