At least 55 parliamentarians signed the letter to the Metropolitan police commissioner, Cressida Dick, after the Speaker of the House of Commons urged police to do more to protect MPs on Monday afternoon.

Scotland Yard later confirmed it had opened an investigation into the incident involving Soubry, in which the protesters repeatedly shouted that she was a “Nazi” while she spoke on live TV.

The letter, signed by at least 55 MPs, reads: “After months of peaceful and calm protests by groups representing a range of political views on Brexit, an ugly element of individuals with strong far-right and extreme-right connections, which your officers are well aware of, have increasingly engaged in intimidatory and potentially criminal acts targeting members of parliament, journalists, activists and members of the public.

“We understand there are ongoing investigations but there appears to be an ongoing lack of coordination in the response from the police and appropriate authorities including with Westminster borough policing, and despite clear assurances this would be dealt with following incidents before Christmas, there have been a number of further serious and well publicised incidents today.”

In the letter, the MPs said they wanted to ensure people retained the right to peacefully protest outside parliament. “It is, however, utterly unacceptable for members of parliament, journalists, activists and members of the public to be subject to abuse, intimidation and threatening behaviour and indeed potentially serious offences while they go about their work.”

Soubry criticised the lack of police response after chants of “Soubry is a Nazi” could clearly be heard while she was being interviewed by BBC News on Abingdon Green, a grassed area outside parliament used by broadcasters.

It is the second time in recent weeks the pro-remain MP has been targeted by a small group of pro-Brexit protesters wearing yellow vests, some of whom have links to the far right. On the earlier occasion, she was surrounded by the shouting men, who called her a traitor.

British MP Anna Soubry accosted by pro-Brexit demonstrators in December. Guardian video

After the latest incident, the Conservative MP Nick Boles raised the issue with John Bercow, the Commons Speaker, asking what could be done to end the harassment.

Bercow said safety off the parliamentary estate was not part of his remit, but he took the issue very seriously and had been in touch with police, “who have been made very well aware of our concerns”.

The Speaker told MPs: “Peaceful protest is a vital democratic freedom, but so is the right of elected members to go about their business without being threatened or abused, and that includes access to and from the media stands in Abingdon Green. I am concerned at this stage about what seem to be a pattern of protest targeted in particular – I don’t say exclusively – at women.”

He was backed by other MPs, among them Labour’s Mary Creagh, who said such “vile, misogynist thuggery, abuse and harassment” raised particular worries following the murder of her colleague Jo Cox in 2016 by a far-right terrorist.

The small but highly visible and noisy band of yellow-vested activists harassing MPs and journalists outside parliament are ostensibly protesting in favour of Brexit, but have links to wider far-right activism of the sort carried out by Tommy Robinson.

Robinson, real name Stephen Yaxley-Lennon, has built both a huge social media following and a handsome living from anti-Muslim protests. He has recently tried to latch on to pro-Brexit activism, co-organising a so-called Brexit Betrayal march in London in December.

The group outside parliament, almost all of whom are men, are not formally linked to Robinson, or his new political partners in Ukip, but have similar roots. They have modelled their outfits on the French gilets jaunes protests.

James Goddard, a self-styled “political activist” who filmed Tory MP Anna Soubry being abused and harassed in December, is the most prominent of the group, and has appeared at many of their small-scale protests, including blockades of Westminster bridge.

He gave a speech at a rally in the summer in support of the then-jailed Yaxley-Lennon, and has posted anti-Islam comments on Gab, a niche Twitter-mimicking social media site popular with far-right users.

Goddard has also been involved in protests connected to an anti-Muslim conspiracy theory which claims that claims an incident in which three teenage boys were killed by a speeding drunk driver as they walked to a party in west London was in fact a terror attack.

Like Robinson, Goddard has been seeking to make such protests his full-time job by soliciting donations from supporters online.

Bercow said he was aware of incidents “involving aggressive and threatening behaviour towards members and others by assorted protesters who have donned the yellow vests used in France”.

He said it was a matter for the Metropolitan police rather than parliamentary authorities as it had happened in the street, but added: “Female members, and in a number of cases I’m advised, female journalists have been subjected to aggressive protest and what many would regard as harassment. I can assure the House that I am keeping a close eye on events and I will speak to those who advise me about these matters.”

During the BBC interview, Soubry broke off from answering questions to tell the presenter Simon McCoy: “I do object to being called a Nazi actually. I’m sorry but I just think this is astonishing.”

The abuse continued as Soubry was interviewed by the Sky News presenter Kay Burley, with chants of “liar, liar” heard throughout the live broadcast.

Soubry told Burley: “I don’t have a problem with people demonstrating and making their views heard. I have a real problem with people who call me a traitor, or ‘Soubry, you Nazi’. That is a criminal offence and I’m a criminal barrister.

“I’m told that we should get used to it, but we shouldn’t have to. Apparently it’s democracy in action and the CPS [Crown Prosecution Service] believe that no offences are being committed.”

In a later tweet, Soubry said: “Apparently MPs and politicians are meant to accept it as part of the democratic process. I fail to see why journalists and technicians should be subjected to the same abuse & intimidation as the police stand by and do nothing. They tried to stop me getting into parliament.”

The political commentator Owen Jones was accosted by a group outside parliament earlier on Monday, with some wearing union flags. He shared a video of his encounter on Twitter, where they could be heard calling him a “traitor” a “horrible little man” and accusing him of writing “fake news”.

Many of the incidents have been carried out by the same small group of protesters wearing yellow hi-vis jackets, modelled on the French gilets jaunes movements, who like to livestream their actions on Facebook. They have also blocked Westminster Bridge and harassed other politicians and journalists, including shouting racist and sexist abuse at a Sky News team.

A key member of the group is James Goddard, who identified himself as the person filming the first abuse of Soubry. Goddard is a supporter of the far-right activist Tommy Robinson and has posted anti-Muslim messages on the social media site Gab, which is popular with far-right users.

Caroline Davies, Peter Walker and Kevin Rawlinson