I say listen to the HV, look at your baby, then decide what you want to do. I think you know it's not the right time to wean off the BF or else you wouldn't have posted the question - right?!
My HV told me to keep giving my DD unlimited amounts of milk - sometimes she would down 11oz - and hold off on weaning until hitting 6 months as this was the 'gospel'. I decided to wean a month earlier - she's a big baby and clearly milk wasn't enough for her so it was right to go against her advice. I've never regretted it and have to say, was proud that i finally followed my instinct and had the self belief to do it.
HVs can be great but they don't know your baby like you do. It's not like you're feeding on the boob every five mins. if I were you I'd probably ignore the advice. If your baby wants more food, then give it to him but you will know when that's the right time.
Be careful not to feed too much in one go to start with as their tummies need time to get used to digesting the food and all that poo-ing!