The biggest thing about switching software is that in order to keep my server secure, I have to keep updates to things like SQL, PHP and Apache going.
SMF is developed so poorly and slowly that it doesn't work with newer versions of that server software - so every time I end up updating the server, I have to hack more of SMF to get it to work, and more shit breaks. Eventually we're going to get to the point where SMF just doesn't work anymore.
As for the server - I originally was hosting this on a $25/month VPS. That's when we had all the slowdown and crashing issues. The fact of the matter is that this site can see anywhere from 10 - 120 people actively clicking around and using it at any given time and that takes enough processing power that it really isn't suitable for a virtual server. So I used a server that I own, and the couple hundred a month I pay is to have it co-located in a data center instead of running out of my home internet connection.