Police fired shots in the air but were unable to stop the angry crowd from surrounding the building, a witness was quoted as saying by news agency AFP.
The Harbour Master confirmed that two Sri Lankan Navy ships SLNS Sindurala and SLNS Gajabahu left from the Colombo Port amid protests in crisis-hit Sri Lanka.
Sri Lanka is facing a severe foreign exchange shortage that has limited supplies of fuel, food and medicine, leading the island nation into its worst financial crisis in seven decades.