Kapil Sibal, said the observers were appointed in 2003 and have dealt with the issue since then.
The Supreme Court on Monday directed the Chief Justice of the Allahabad High Court to nominate two additional district judges within 10 days as observers to deal with the upkeep and maintenance of the disputed Ram Janmbhoomi-Babri Masjid site in Uttar Pradesh's Ayodhya.
A bench headed by Chief Justice Dipak Misra passed the order after senior advocate Rakesh Dwivedi, appearing for the Allahabad High Court registry, informed it that one of the observers has retired and the second one elevated as a judge of the high court.
Mr Dwivedi also handed over to the top court a list of additional district judges (ADJs) and special judges who could be considered for appointment as observers.
"As the list is long, we think it appropriate that chief justice of the Allahabad High Court shall nominate two persons from the cadre of additional district judges or special judges keeping in view the nature and tenor of the earlier orders passed in this case," the judges said.
The top court asked its registry to forward the order to the high court registrar and said, "the chief justice is requested to nominate two names within 10 days hence".
During the brief hearing, senior advocate Kapil Sibal, appearing for one of the parties in the matter, said the two observers, T M Khan and S K Singh, were appointed in 2003 and they have been dealing with the issue since then.
"Why should the court change them when they are here for 14 years? This is a very sensitive matter," he told the bench, adding, "please ask them (Khan and Singh), will they continue".
The bench, however, said, "one of them is not holding a post and he cannot continue now".
"We will ask the chief justice of the high court to decide this," the top court said, adding that the "tenor of this court's order was that they must be a part of the system. One of them is no more a part of the system".
"One of them has been elevated as a high court judge. It is not proper that a high court judge is asked to go there and observe all the things. We can't ask a high court judge to do this," the bench said.
The top court had on August 11 said it would commence the final hearing in the long-standing Ram Janmabhoomi-Babri Masjid title dispute from December 5, a day before the 25th anniversary of the demolition of the medieval-era structure.
The court had reached a consensus on commencing the hearing on a total of 13 appeals filed against a 2010 judgement of the Allahabad High Court in four civil suits.
The high court had ruled a three-way division of the disputed 2.77 acre area at Ayodhya among the parties -- the Sunni Waqf Board, the Nirmohi Akhara and Lord Ram Lalla (deity).
Another sect of Muslims under the banner of Shia Central Waqf Board of Uttar Pradesh had recently moved the top court offering a solution that a mosque could be built in a Muslim- dominated area at a "reasonable distance" from the disputed Ayodhya site.