I am unsure if I am transferring stuff onto them and should therefore withdraw
or if I feel connected (and hence its safe to spend time with them).
In either of the cases, see if you are in touch with yourself during these interactions.
Below is a quote from Eckhart when read and understood deeply are interconnected and has several meanings within itself and an answer to both sides of the doubts.
“When you lose touch with inner stillness, you lose touch with yourself. When you lose touch with yourself, you lose yourself in the world.” ~Eckhart Tolle