I think you need to talk to him about the situation at hand. It seems as you could be ruining his relationship. I doubt you would want that to happen to your own relationship.
Lets say you have a bf and your bf was being taken by another girl would you enjoy that?
I think you need to talk to him and end it or try to swing with him
I highly doubt they want to swing, but you never know. But I find it wrong to ruin someone's relationship.
Your just going to have to decide whats right for you and what you can life with.