Grumpy Toad,

It sounds like both of you do not know how to communicate.  You are afraid of conflict and he/you get defensive, walks away.  Not a good formula for a long lasting, intimate/close relationship.

Find a counselor to teach both of you how to communicate.  You say it is too early.  I say that if either of you did not know how to deal with concerns and differences before the marriage then it is not too early.  Non-Violent Communication is a great tool (you can Google it) for communication.

Plus you are not setting boundaries on what you do and do not want, e.g. having him share personal information with his office girlfriend (and yes I am using that term deliberately).  Whether it is a sexual affair or not, it really sounds like an emotional affair.

Mark