It varies for me. I gave up caffeine entirely for a month a while ago and didn't see a big difference EXCEPT sometimes the afternoon sleeps, when she'd just gotten the milk that would have had the caffeine in it.
Now I do drink SOME caffeine, and there are days that she sleeps dreadfully, and some of those days I've not even had any. :? Go figure. In any case, I try not to drink any after about 9-9:30AM, and I'll drink lots of water and other stuff to try to dilute it if I do.
That said, Josie was terribly sensitive to caffeine as a baby and I was off it entirely most of her BFing life.