For my own personal sanity, I would leave mine there just because they'd be happier comfort sucking than not. And yes, your supply should settle down within a few days from what other ladies have said.
2 out of my 3 were super cranky at that time of day too until about the 3/4 month mark and I never quite figured out what I could do to make them happier aside from lots of nursing! Hope it all sorts out soon!