Except some of the earliest uses of the word “literally” that didn’t pertain to letters and glyps we in the form of hyperbole.
Literal as factual and literal as exaggeration both about the same age and precedent, and have been used long enough that it’s just part of the English language at this point.
May as well complain about how “discreet” and “indiscreet” are opposites, but “flammable” and “inflammable” are the same.