Hi Sylvia,
After reading your story, all I could think of is that his past has not been too great and he really need somebody who could understand him and deal with his mood swings. If you really want to be with him than you would really need to keep a a lot of patience and definitely not judge him for anything that he does to understand him better .
But as you said you are losing self respect for him and are not able to love him than you need to discuss everything you feel with him and take a decision whatever you both think is right for you guys.
You must read this what happens when we love a person and what happens when we start hating that same person?
http://www.magicalvibe.com/love-is-the-beauty/
Mishika 🙂