WASHINGTON — (CNN) Wells Fargo CEO John Stumpf apologized to customers for more than 2 million fake accounts opened in their names, but denied any orchestrated fraud by bank management.

In testimony to the Senate Banking Committee Tuesday, Stumpf also said the bank plans to expand the internal review of accounts and refund process by two years, starting in 2009 now.

He addressed the outrage over Carrie Tolstedt, the head of the division where the fake accounts were created who is set to walk away with $124 million in stocks and options, when she retires later this year.

Stump indicated Tolstedt was pushed out after the bank determined she “did not do enough” to fix issues in the retail bank.

Stumpf also suggested that the Wells Fargo board, in an independent process, will consider possible clawbacks of her compensation.

“The board has the tools to hold senior management accountable, including me and Carrie Tolstedt,” Stumpf said.

The Wells Fargo CEO said the scandal pains him more than any problem faced by the bank in his 35 years there, and that he accepts responsibility.

“I am deeply sorry that we failed to fulfill our responsibility to our customers, to our team members, and to the American public,” he said in his prepared remarks.

“I do want to make very clear that there was no orchestrated effort, or scheme as some have called it, by the company,” he said.

Wells Fargo employees have said they were pressured to meet unrealistic sales goals and that they opened the bogus accounts so they wouldn’t lose their jobs.

Stumpf said the fake accounts cost the bank both money and satisfied customers.

He said the scandal goes against “our values, ethics and culture and runs counter to our business strategy.”

Wells Fargo also said it will now alert every single deposit customer to review their accounts and shut down ones they don’t want or recognize.

Previously the bank had only agreed to do that for its California customers, as part a settlement with Los Angeles.

Further, Wells Fargo said it has begun contacting hundreds of thousands of customers with open credit cards to confirm they need or want their card.

Stumpf has refused to step down, despite calls for him to do so.

Senator Elizabeth Warren, a member of the committee, has called the scandal a “staggering fraud.”

She has said that if Stumpf and other top management really did not know the scope of the scandal, “that tells me this is simply a bank that is too big to manage.”

Stumpf told the committee that the bank fired 5,300 employees involved in the fake accounts starting in 2011, not in response to the Consumer Financial Protection Bureau, which revealed the scandal earlier this month.

Lawmakers pressed Wells Fargo to explain who it fired besides low-level employees among the 5,300 workers that were let go as part of the scandal.

Stumpf reiterated that “managers and managers of managers” were fired, and for the first time said at least one “area president” was also terminated.

The Wells Fargo hearing inspired a rare burst of bipartisanship from Republicans and Democrats who teamed up to hold the hearing.

“You have done something I’ve never seen in 10 years: You have united this committee — and not in a good way,” Sen. Jon Tester told Stumpf.

Jaret Seiberg, an analyst at Cowen & Company, said Stumpf’s testimony is unlikely to satisfy either Democrats or Republicans.

He said Wells Fargo has admitted it was aware of the problem since at least 2011, but was evidently unable to stop it despite increasingly aggressive steps to do so.

That suggests “employees believed the aggressive cross-selling was required to stay employed,” Seiberg wrote in a report.

Analysts believe lawmakers may take the hearing as an opportunity to pressure the Justice Department to prosecute senior Wells Fargo executives for fraud.

The Justice Department has issued subpoenas to Wells Fargo over the scandal, a U.S. official told CNN last week.

“There is tremendous pressure on Justice to bring criminal charges against bankers. That is why the stakes are so high here for Wells Fargo,” Seiberg wrote.

Also due to testify at the hearing are two regulators: Richard Cordray, director of the consumer bureau, which fined Wells $185 million, and Tom Curry, the U.S. comptroller of the currency, which oversees the bank.

Also testifying will be James Clark, the deputy Los Angeles city attorney who brought the lawsuit that helped bring the scandal to light.