As far as I know there is no direct harm in it - the former reason is the applicable one, because growing children need the fat. Remember that if your baby is below 12 months they shouldn't be having cow's milk as a drink, though it's fine in cooking and to mix cereals etc.
HTH!