In the end, you always need to do what's best for yourself. No company will be there for you when you're sick or in trouble.
You're guilt is making this bigger in your head, I think. I've managed a long time, and never focus on this stuff, nor have I worked with other managers who hold on to shit like this.... it was handled poorly on their behalf, and sounds like they scrambled or should have pushed back themselves.
You good, homie.