I'd keep the deposit and not feel any guilt whatsoever.
All you can do is tell him you are not prepared to return it, explaining your lost money and time, and wait for his reply.
It's 50/50 really if this guy wants/knows how to fight for it back.
He could have just asked for it back hoping to appeal to your better nature. Maybe he is not expecting to see it again.
Like you say you could offer him 1/2 the cash back. If you did he should consider himself very lucky.
I, and I'm sure 99% of people on here know the meaning of a deposit. It's a bit of security for the seller against things like this occurring.
Stick to your guns mate.