Vanilla just means unmodified, exactly what you think.
However, some people might still refer to mods that don’t significantly change gameplay as vanilla. For instance, the Vanilla Tweaks resource pack is filled with small changes to the appearance of things in the game. Some might also consider the Optifine or Sodium mods to be Vanilla since they don’t change gameplay.