Not a scary game, but the bonelords and greater bonewalkers from morrowind scared the hell out of me.
Its not that they kill you, its that they destroy your strength with every hit. Now you've killed it and you can't move because you aren't strong enough to lift a foot. Have fun being imprisoned in a dungeon forever.
Also, that alien game. Again, not super scary, but impossible for me to keep going after i die continuously in the exact same spot because that motherfucker is inescapable.