Why does he do this? I don't understand...?

He is clearly uncomfortable with hanging out with your family, for whatever reason. You should have a gentle talk with him to try to find out why. This will always be a pressing issue until you two talk and resolve it. Dont' try to rush into conclusions; perhaps he is just nervous. Just talk with him, be supportive and find out why. Communication is always key.