Under fire for the late night crackdown on Baba Ramdev and his followers at Ramlila Maidan in New Delhi, Union Home Minister P Chidambaram today defended the government's decision, saying action was taken only after the yoga icon went back on his word on calling off the fast.