I'm assuming that your baby is formula fed? If he/she is breastfed then they should be getting all their water from you, so if you just want to give a few sips of bottled water I'm sure it's ok.
If it's for formula, can you bring your own kettle, one of those electric ones? Or buy one?
I don't know about the bottled water, but I agree it would be better than the unsafe tap water. I wonder if some bottled water is better than others, maybe there is a kind that is of better quality.