should I just man up and be miserable at the parties or do I have a right to say no without feeling guilty

You of course have a right to say no.

Notice though you are projecting into the future that you will be miserable and guilty, which you probably will be if you look at attending the party of having to “man up”.  We create what we fear. That problem isn’t about your relationship but your insecurities, which are impacting your relationship.

That said its important that you hear what your boyfriend is saying to you when he says he doesn’t feel he can invite you to things and participate in his life. How might he be feeling? Could you be ashamed of him? Do you want to be a part of his life? You’ve explained your behavior as not wanting to ruin his night and yet going or not going your creating a negative experience for both of you. Which negative experience has the most weight?

One of the purposes of relationship is to confront our fears and deal with them. Nothing like a relationship to reveal our shadow.

As an introvert I get it. I always feel uncomfortable in large gatherings, yet when I make the effort I’m almost always glad that I had.  But it is a effort, and also my problem. As you posted in a Buddhist site I suspect you already know that you need to let go of your concerns of how you imagine others might be thinking of you. In this case you don’t have to imagine you can ask how your boyfriend feels when the two of you go out to visit friends. He may even enjoy staying beside you and helping you engage.