I think you should let your common friend know that you are not a type of person who likes hanging out with others and is happy in your own little world. You should not do something which you don’t like and the way you have mentioned it , it feels like you are just being pushed to go out with him , It’s better if you oppose him and ask your common friend to intervene and talk things with this guy .