Even though rationally I recognise his reasons for not committing further is all about him – my self-esteem is suffering from the fact he doesn’t seem to want more than friendship despite the fact we were more before.

I think this is totally understandable, but it’s possible that he didn’t realize that he wasn’t ready for a relationship or had things that he really should deal with first until he was in one and then he felt he needed to back off.  Has he said specifically what is going on in his personal life that he needs to deal with?

It really sounds like he has some issues he needs to deal with because something is stopping him from moving forward, and I really doubt it has anything to do with you, even though you’d been more in the past. I hope he’s able to figure things out, but, in the meantime, I really think it’d be smart to back off, at least for a while. If he seems afraid and craving to be single yet is seeking constant companionship, he’s likely trying to distract himself from his problems rather than having to actually be alone and face them, which isn’t going to help him resolve them.  So continuing to be so close with him might actually enable him to keep spinning his wheels, if that makes any sense. Plus, it seems to be affecting you negatively emotionally, right? So it might be a good idea to just take a step back for a while, and you can always check in later and see how he’s doing.  I don’t think you have to cut him off cold turkey if you’re worried about that damaging your friendship. I’d just probably stop being so available to hang out and only catch up once in a while. You’re just really busy, you know?