They indeed need to make it more conversational. I think this is a big thing Jobs would harp on if still alive. It should feel like always having a friend/assistant in the room who knows everything.
It sucks for privacy but if you trust apple enough it’d be nice to have an always-on microphone for Siri so you could be like “hey siri that tour we talked about at breakfast- can you bring up directions to that?” Stuff like that