Having a water bottle dilemma

I moved my accessories from my old bike to my new bike and now I'm unable to get the water bottle out of the holder on the seat tube (To insert the water bottle in that holder I have to take off the top of the bottle also). Would side loading water bottle cages solve my problem? Any recommendation for one that'll hold a CamelBak Podium Big Chill 25oz Insulated Water Bottle? Also, would I be able to keep my pump on my bike? Thanks!