Amarinder Singh, who stepped down as Chief Minister on Saturday following an escalation in the political crisis in the state stemming from a bitter tussle with rival Navjot Singh Sidhu, ruled out accepting his party colleague as his successor and left the door open on leaving the Congress.