I don't think one or two cups a day are going to destroy you. I think that if you were knock off ten or twenty cups a day then it's another question.
Again it really is about the amount rather than what it is. The caffiene content in a few cups don't amount to a hill of beans (to paraphase Bogart).
My wife insists on having skim milk in her cappuccino. The amount of difference between regular and skim is very small. It seems hardly worth the effort to be so petty.
Milk is good for you (hot water for a cuppa what's that about?). A glass full of milk is very good for you. Osteoporosis is something that should be avoided at all costs.