Airene is right. There is a reason the term is SELF esteem. It’s very hard for someone to change how you feel about you on a fundamental level. The only way they can tear you down is if you already have insecurities for them to prey upon.

The conversation you need to have with this man is simple.

“I am miserable with you. You prey on my insecurities and make me feel terrible. I cannot be with you anymore.”

The more difficult part here is going to be staying away from him after. It sounds like you have some codependency issues that you should really talk to a counselor about. Otherwise you WILL just continue to relive this cycle over and over. And the end of the cycle, if left to its own devices I fear will be very bad for you.