(CNN Money) — Bernie Madoff’s victims are getting more of their money back.

The U.S. government on Thursday announced that it would begin paying out $772.5 million to more than 24,000 victims of the Ponzi scheme.

The money comes out of the Madoff Victim Fund, an account the government put together four years ago to help people who were scammed by the notorious fraudster.

All told, federal authorities say they’ll pay out $4 billion from the fund. The money was collected through seized assets and settlement agreements with people and companies connected to the case.

“We have recovered billions of dollars from third parties — not Mr. Madoff — and are now returning that money to tens of thousands of victims,” said Rod Rosenstein, the deputy attorney general. “This is the largest restoration of forfeited property in history.”

Madoff stole an estimated $20 billion from investors who thought he was running a legitimate Wall Street firm. He pleaded guilty in 2009 and was sentenced to 150 years in prison.

The government began recovering money as it pursued legal action against Madoff, his conspirators and others associated with the case.

But there’s another effort to repay Madoff’s victims.

When investors filed a class action lawsuit against Madoff, a court appointed the attorney Irving Picard to help them get their money back.

He has distributed around $10 billion to victims so far — money amassed largely through civil settlements.

The U.S. government took that tack, too. About $1.7 billion in its own victims fund came from JPMorgan Chase, which agreed three years ago to pay to settle allegations that it ignored red flags about Madoff’s crimeswhile it was his banker.

The bank insisted that its employees did not knowingly assist in the crimes.

But federal authorities also funded their account through the forfeiture of civil assets. The agency seized around $2.2 billion from the estate of Madoff investor Jeffry Picower, who died in 2009.

Picower is said to have benefited the most from Madoff’s scheme, though his wife has denied his involvement.

The U.S. collected additional funds through forfeiture actions against Madoff, his co-conspirators and certain Madoff investors, according to officials.

The government’s asset forfeiture program has been the subject of controversy over the years. It allows the federal government to work with state and local officials to seize cash and other assets from suspects before they have been convicted of a crime.

Rosenstein, the deputy attorney, defended the use of the program in the Madoff case in a Wall Street Journal op-ed on Thursday.

“Not everyone who possesses illegal proceeds can or should be criminally prosecuted,” Rosenstein wrote. “Many criminals transfer ill-gotten gains to relatives or friends, and others use couriers to transport cash. In such cases, civil forfeiture enables the government to recover property when prosecuting the person caught holding it may not be appropriate or feasible.”

The-CNN-Wire™ & © 2017 Cable News Network, Inc., a Time Warner Company. All rights reserved. (Photo: CNN/US Department of Justice)