It sounds like he is trying to manipulate you. You have spent a lot of time trying to make it work, and you are certainly not guilty of quitting on him nor do you have anything else to feel guilty about, do not let him make you feel that way. I am struggling with a similar situation right now with my boyfriend, and it is really hard. Every time I think I know what decision is right, I end up second guessing myself. I have been second guessing myself for years, and guess what? Seven years later nothing has changed. I am still unhappy and dissatisfied. The only difference is that now, I am seven years older and I have wasted most of my twenties. I advise you to really try and imagine what your future looks like, with him in the picture, and with him out of the picture. In which scenario are you happy? I advise you to really follow your heart and have no regrets. I know how hard it is from personal experience, but try not to second guess yourself. Three years turns into five, five turns into ten, and before you know it you have lost a huge chunk of your life and have nothing to show for it. Life is too short as it is, do not let fear hold you back from following your intuition. It will hurt temporarily, but it will be well worth it a few years down the road when you have found the love of your life and you are happy and at peace.