WASHINGTON — (CNN) A federal judge on Wednesday had a stern message for a government lawyer defending the legality of the National Security Agency phone record collection program.

“I am not going to allow any impression that the government is running out the clock,” U.S. District Judge Richard J. Leon said.

Leon held in 2013 that the program was likely illegal but he put his ruling on hold pending review by an appeals court. That court did not rule until late last week when it threw out Leon’s original opinion and sent the case back down to him to resolve threshold issues.

The three-judge panel of the U.S. Court of Appeals for the D.C. Circuit vacated Leon’s opinion, but in doing so, it did not rule one the merits of the program. Instead, it questioned whether there was clear enough evidence that the plaintiffs had been harmed.

During the hearing on Wednesday, Leon showed unveiled frustration and repeatedly expressed interest in resolving the concerns of the appeals court.

“This court believes that there are millions of Americans whose constitutional rights have been violated,” Leon said.

He said that because the program in its current form lapses at the end of November, there is a very small window for him to be able to reach the procedural issues as directed by the appeals court and issue another opinion on the merits.

At times, Leon seemed close to offering Larry Klayman, the lawyer who is challenging the program, advice on how to proceed. He asked whether Klayman planned to amend his original complaint and to ask the appeals court to expedite the mandate necessary for Leon’s court to once again take up the case.

“Judge Leon clearly wants this case to go forward, even though the Court of Appeals only left the door open a crack,” said Stephen I. Vladeck, a professor of law at American University.

Klayman faces an uphill challenge. Since he filed his original claim, Congress changed the law in passing the USA Freedom Act.

At one point the judged referenced the appeals court ruling, saying, “It remains to be seen if anyone agrees with me.”

Klayman responded with praise.

“You are the last line of defense for the American people,” he said.

In June, when Congress passed the Freedom Act, it changed the structure of the program. It stipulated a transitional period to allow the current program to expire on Nov. 29.

Rodney Patton, a lawyer appearing for the government, seemed to caution moving too quickly and urged Leon to wait to act until the official mandate is received from the higher court, sometime near the end of October.

But Leon warned that the lawyers should not go on any vacation and discussed holding another hearing as early as next week.

There are two similar cases pending in other appeals courts.

The-CNN-Wire ™ & © 2015 Cable News Network, Inc., a Time Warner Company. All rights reserved. (Photo: CNN)