Brain shrinkage is de-evolution. Humans aren't particularly big creatures to begin with, so the likelihood they would shrink to the size of, say a rat, is practically impossible in an evolutionary sense. Same goes for becoming much larger; that would involve not just building a bigger structure but building a significantly larger brain.
It's not bragging to suggest that humans are unique animals; we rely more heavily on our tool creating abilities than any other species by far. It's a feature that allows us to have an outsized ability to manipulate and adapt to our environment without altering our physiological traits. Humans are the only animals that can adapt to every single environment on the planet in some form exclusively because we have an ability to create assistant devices to enable that adaption. At the same time, in a physiological sense, animals of any species of any order evolve specifically in order to survive either in a home environment that becomes inhospitable to their old trait set or in an expanded range because of population density. Our tool making ability makes the need to physically evolve immaterial: either we build an apparatus to adapt ourselves to a new and inhospitable environment or we go extinct.
You can facepalm all you want, but your premise is fundamentally preposterous.