If she settles fairly quickly for nap and bedtime, then the routine looks ok for now. Does she go straight to sleep as long as you or your husband are there? If not she may need a routine adjustment too, but I'll assume things are ok on that front for now.
I think gradual withdrawal is probably going to be the best technique for you and your husband to get out of the room -
Walk In/Walk Out vs. The Gradual Withdrawal Method (HOW TO CHOOSE) (ignore the information on WIWO). I would also change up her bedtime routine if she has the cup of milk directly before sleep - not least because she needs to brush her teeth afterwards. She can still have milk as part of the routine, but maybe give it to her downstairs before heading up for bath/stories/whatever else you do. Then at least you know she isn't hungry if you end up with a tantrum.
With the overnight milk, I would spend time in the day talking to her about how there will be no more milk at night. For now, it's ok for your husband to go in and comfort her when she wakes, but you need to be consistent about no milk. If you do give in it will make it ten times harder the next time. Once she will settle with your husband going in, but no milk, start working on gradual withdrawal so that she gets used to going back off to sleep herself.
How does that sound?