The aftermath of the violence at a neo-Nazi and white nationalist rally in Charlottesville, Virginia, represents the latest break between Trump, who sold himself as a businessman president, and leaders of corporate America.
Frazier, who as a lawyer helped free a black death row inmate falsely accused of murder, and who now leads one of the country's most powerful drug companies, was first to quit the manufacturing council.
After his remarks, the fifth member of his manufacturing panel resigned, AFL-CIO president Richard Trumka, who said: "We can not sit on a council for a president who tolerates bigotry and domestic terrorism".
Later in the day, Trump went back at Frazier, saying that Merck was "a leader in higher & higher drug prices while at the same time taking jobs out of the U.S. Bring jobs back & LOWER PRICES".
Trump has come under fire for his comments Saturday that "many sides" are to blame for the violence.
Merck CEO Kenneth Frazier is leaving President Trump's American Manufacturing Council, saying, "I feel a responsibility to take a stand against intolerance and extremism".
In announcing his resignation, Plank said Under Armour "engages in innovation and sports, not politics".
But he plans to stay on a separate Trump advisory panel and said the president's follow-up remarks on Monday that named white supremacists were a step in the right direction.
The demonstration in Charlottesville by hundreds of white nationalists took a deadly turn on Saturday when a auto plowed into a group of counter protesters and killed one person.
"I resigned because I want to make progress, while many in Washington seem more concerned with attacking anyone who disagrees with them", he noted.
"Americans deserve to know that white nationalists, white supremacists, and neo-Nazis are not in a position to influence US policy", says the letter dated Monday.
Frazier and Merck will likely be applauded in Austin for the CEO's response.
Richard Trumka, president of the largest federation of US labor unions, the AFL-CIO, questioned the council's effectiveness and said the group was mulling leaving. The president on Monday later said condemned groups tied to racism. Later, Kevin Plank, of Under Armour, and Brian Krzanich, of Intel, also said they would leave the president's committee, known as the American Manufacturing Council.
Later Monday, Trump made a statement calling out specific hate groups.
"Racism is evil - and those who cause violence in its name are criminals and thugs, including KKK, neo-Nazis, white supremacists, and other hate groups are repugnant to everything we hold dear as Americans", Trump said.
SpaceX founder Elon Musk and Disney CEO Robert Iger bowed out of their respective positions when Trump said he would pull the USA out of the Paris climate agreement.
Former Uber Technologies Inc [UBER.UL] CEO Travis Kalanick quit the business advisory council in February amid pressure from activists and employees who opposed the administration's immigration policies.
Trump on Saturday initially denounced what he called "this egregious display of hatred, bigotry and violence on many sides".