Hi there! Yes, some people drop the DF earlier (some people don't use it at all!). If he's sleeping well through the night, then you could give it a try.
I'd be more cautious about the solids, amount, actually. Remember that most guidelines don't suggest solids much before 6 months, and even after that, they're more for practice than anything. You don't want him filling up on rice cereal which has almost no nutritional value, when he still needs the maximum of breast milk. As long as he's getting full feeds for those 4 bf's, and you make sure not to give too much solid, then dropping the DF should be worth a try.
HTH!