Why though? That won't make him want to spend time with you when he comes back. It might make him want to spend MORE time away. Which isn't fair to anybody.

Instead of being mad, you should come over and we'll try to be happy together. Maybe we could play a board game or something? We haven't done that in a while.