I can recommend two great books which have lots of activities for letter and number games but which also focus on the importance of gross motor, core development and play.
Marshmallow Maths
The Write Start
They both cover a variety of ages/abilities and interests, worth a read.
I used to 'disguise' various activities for DS letting him think they were all about letters/numbers/shapes/colours (he really loved all that) when they were really more focused on gross motor, fine motor etc. So for instance creating a large floor maze or road map with chalks or masking tape on the kitchen floor and getting him to drive a toy car from one letter to the next along the maze or road route. Floor play is such a vital aspect of child development. Another was marking out big shapes on the kitchen floor and writing a number in each one then asking him "can you hop to 3?" "can you crawl to 7?" "can you frog jump to 9?" or I'd set up a group of objects in the living room and he'd have to walk with a little basket to fetch "4 red cars" to bring back to put in the 4 circle, "3 blue cars" to bring back and so on, physically selecting, lifting, carrying, placing the objects rather than just rote counting numbers.
Also rather than continuing to count higher and higher you can establish a greater understanding of how numbers work with those numbers from 1-10 or 1-20 by using mathematical vocabulary. Counting starting at a random number rather than 0/1 (so you lead him in "3,4,5...what's next? ...keep going to 10" if he can do this then don't lead in just give the starting number "count from 6 to 11". Counting backwards, one more than, one less than, weight and size vocab, small, big, biggest, heavy/light etc. These sort of play can really extend number play without going forever higher and higher on the numbers.
Perhaps he'd like some pattern making activities? Rather than building with lego you can set out a simple pattern (red blue red blue) and ask him to copy it with more bricks. Adapt and increase the difficulty as needed.
Even pretend play can be encouraged by including his favourite subjects (letters and numbers), say using little toy people/characters counting how many are at home, how many are out, if one comes home how many are there at home now? And whilst you do it show him how to make up stories with the people, put on voices or make up adventures etc. Role play shopping is an easy one for a number lover, set him up as shop keeper and keep asking for various numbers of items, use play money etc.