About 600 people gathered outside Parliament in Hamilton, the island's capital, chanting "Brown must go" and waving banners as they marched to the Cabinet office.But Mr Brown was defiant. "As some of you might know, I grew up in the protest era," he shouted at the booing crowd. "This is nothing new to me. I have seen them larger and longer," he said.
Mr Brown negotiated a secret deal with the US last week members of China's Muslim Uighur minority to be sent to the British. The Foreign Office was not informed and the Obama administration also bypassed the British.
The US, which is desperate to unload the prisoners to fulfil a promise made by president Barack Obama to close the jail by next January, said it could not send the Uighurs to China because they faced persecution there. Congress, however, blocked efforts to release them in the US, even though they had long ago been deemed no threat.
David Miliband, the Foreign Secretary, protested to Hillary Clinton, the US Secretary of State, about the secret deal and questioned whether Mr Brown had authority to admit the Uighurs. Britain is responsible for the foreign policy and security of Bermuda.
Janice Battersbee, who described herself as a lifelong supporter of Mr Brown's Progressive Labour Party, stepped up to the microphone when Brown invited the protesters to send a representative to speak. "The leadership of this country seems to be on a course heading toward dictatorship that the majority of Bermudians are no longer willing to tolerate," she said. "This latest action is the final straw. We are fed up, disgusted, disrespected and angry."
Mr Brown said he had been summoned to meet with Sir Richard Gozney, Bermuda's governor, but did not elaborate on what was discussed. When Mrs Battersbee finished speaking, Mr Brown was directed away by police and left in an official car.