You have a right to be angry, no need to apologize or feel guilty for your feelings . If you want to let go the key for you is to cut off communication with him would be a start he was only in your life a short period, don’t let it be a lifetime of misery he will cause you. He is set in his ways and you will be in his control as you are now by letting him linger in your life. Go out and do things to make you happy, focus on a new life as if he is not in it , because he is not. If you have not already consult a lawyer and proceed with a divorce, you deserve to be happy , the key to letting go in your situation, is to let go and move on with your life and not hang on to someone who cannot provide the love you want .