This used to be my field before I quit academia. There are two answers both indicating towards - yes, babies remember:
every time we (scientists) devise a new way to ask even younger kinds (infants) whether they understand one thing or another, or whether they remember, we find out that they do. The problem is how to communicate with a nonverbal infant, let alone a newborn.
A lot of brain development happening in the first 6 years or so is killing a lot of neural connections and strengthening others.
The leading theory (10 years ago, when I left the field, science can move fast) proposed that this may be why adults rarely remember things before age 3 - but young children have LOADS memories of before age 3 with accounts (anecdotes) of young children having memories of the prenatal phase (“when I was in mama’s belly …”) - I call these anecdotes because I know of of many parents with these anecdotes, but no study that actually looked into validity of these stories.
The theory then simply argues that as the brain matures and kids learn all the new things they need to learn, they retire these very early memories - they simply forget. But they did have them.