But if we travel back in time and change a fixed event, wouldn't it create a hole in time and space as we know it ?
No. Then time would simply progress forward as if nothing happened.
If period A = present, period B = past and period C = future of altered past, the 'infinite loop' argument only applies if you try to travel back to period A. But even that can be refuted, to be honest. Suppose you edit a picture in Photoshop, then decide to move back to the photo before the edit. Does the universe end? I suspect the same logic would apply to time travel.