I don't see what's different about a phone call on hands free to a conversation with someone next to u. I definitely don't think being on the phone while driving is a good idea(not hands free)
....The difference is that you can much more easily stop talking to a person sitting next to you. In fact, whenever someone sits in my passenger seat for the first time and they seem to be a talkative type (usually females), I politely warn them that I may suddenly stop talking mid sentence or stop listening to them because my attention and focus is needed driving. It's also a good way of shutting them up!
When someone has phoned you, it is because a particular subject needs thought and discussion - It's not usually just idle chat. Therefore you politely pay attention and hence may be much more easily distracted. You are not together and talking within the same environment, so all communication is different.