What you’re describing is a perfect thermal insulator.
What does it feel like when you’re wrapped in blankets? You feel warm, despite the blankets generating no heat of their own. They do this by insulating you from the outside temperature.
It would feel like being wrapped in a warm blanket, and if you were fully submerged, you’d end up overheating as we’re constantly generating heat that we need to get rid of.