Hi Kirsty
If you're not ready to drop the 11am bottle yet how about compromising and giving him half of his bottle at 11.30 followed by his lunch and then offering the other half of his bottle straight after - if he wants it. You could keep doing this until he's hardly taking any of the milk and then finally drop it.
Kelly x