Chhota Rajan had shuttled between Australia, Malaysia and Cambodia since 2000, when Dawood Ibrahim's men tracked him to a hotel in Bangkok and tried to kill him.
Chhota Rajan, the gangster wanted in India for 17 murders, drug trafficking and arms smuggling, was
caught soon after he arrived at the resort island of Bali in Indonesia at around 10 am on Sunday.
He had flown a Garuda Indonesia from Sydney, where he had reportedly stayed for a month.
Rajan, a former aide-turned-rival of terrorist Dawood Ibrahim, had evaded the police in several countries for 20 years.
The police in Bali was tipped off by the Australian police about a murderer travelling on an Indian passport as "Mohan Kumar".
It soon emerged that Mohan Kumar was an alias used by the 55-year-old crime boss wanted for charges that include contract killing, arms smuggling and extortion.
Sources say Rajan's role is suspected in the assassination of Nepal lawmaker Mirza Dilshad Beg, who was shot dead in Kathmandu in 1988.
Rajan had shuttled between Australia, Malaysia and Cambodia since 2000, when Dawood Ibrahim's men tracked him to a hotel in Bangkok and tried to kill him. The gangster was shot at and escaped through the roof of the hotel.
Born Rajendra Sadashiv Nikalje in Mumbai, Rajan is known to have three passports.
His gang was seen by Indian officials as the second most notorious and powerful after Dawood's.
The CBI tracked his movements and reportedly informed the police in Indonesia and Australia about him.
"There was obviously some coordination between India, Australia and Indonesia," Kiren Rijiju, Minister of State for Home Affairs, said.
The police believe that his questioning may solve many cases that have piled up over the years, and also reinforce India's case against Dawood Ibrahim.
There is no official confirmation, but Rajan's hand in eliminating anti- India and pro-Pakistani elements in the 1990s is well-established folklore within the security circuit.
There were reports that the security establishment leaned on Rajan's network to plan an assassination attempt on Dawood Ibrahim in 2005 when his daughter was getting married to the son of legendary Pakistani Cricketer Javed Miandad. The plan failed.
Over the years, however, Rajan's network and ability are believed to have depleted.
Questions have been raised about whether Rajan, after playing the cat-and-mouse game with the police and rival gangs for years, wanted the safe sanctuary of Indian jails.