That's akin to saying "My legs are what gives me locomotion. I don't want to create stress and tear their muscle tissue by exercising. It will just mess with them."
Please note that I'm not taking one side or another, but caffeine has been shown to provide more benefits than harms in moderation.
I also gave up caffeine in university. I found myself drinking bottles of Mountain Dew during lectures and still feeling drowsy. So I gave it up.
After two weeks of fatigue and lethargy, I was fine. I slept better, and was more alert.
But most of all, I gained the ability to actually abuse caffeine. If I need to work through the night and still function the next day, all it takes is a single can of Doctor Pepper or a cup of tea. It's like a super power.
Please note that I'm not taking one side or another, but caffeine has been shown to provide more benefits than harms in moderation.