I can sympathize with you. I had a friend with benefits who really enjoyed anal too. As far as the safety issue, as long as you're always using a good lube and he's being gentle there shouldn't be any safety issues. But the issue is pleasure. Sex should be pleasurable for both (or all) partners. This is not to say that the occasional "taking one for the team" or in this case for your partner isn't off limits. Sometimes doing something you're not crazy about is fine if it pleases your partner but not all the time. As Scarbowl said, you can make a demand of him as he is doing to you. That's the old giving him a taste of his own medicine. However, I feel the best way to approach this initially is to tell him you don't mind doing anal from time to time as you know he enjoys it, but that it is not pleasurable for you. Let him know what you do like that he does, don't be all negative. I'm sure there are parts that you enjoy, and be sure to stress those, and tell him how much you like him inside you (vaginally) and how good it feels.
Hope this helps.