The lawsuit alleging the rape says that the assault occurred between November 2016 and March 2017, when the men were staying in the same cell in Metropolitan Correctional Center in Lower Manhattan.
The inmate has not been identified because he is an alleged sexual assault victim, The New York Times reported.
The alleged rape victim and Zarrab became friends in the jail and Zarrab offered to pay for him to retain a private lawyer, as he was being represented by a public defender, the paper said.
The lawsuit also alleges that the man "felt helpless and unable to fight off the younger and stronger man," and the alleged victim was "too scared and embarrassed at that time to complain or to seek help."
According to the lawsuit, the inmate again was attacked by Zarrab on March 8 and started yelling.
"Being in jail is a bad enough experience for someone, and this just made his experience so much worse," the lawyer for Zarrab's ex-cellmate, Alexei Schacht said.
Robert J. Anello, a lawyer for Zarrab denied the allegations, saying "The allegations are outrageous and false from a source that is not remotely credible."
In a testimony Tuesday, Zarrab testified that he bribed federal prison officials to the tune of $45,000 in exchange for alcohol, synthetic drugs and the use of a cellphone.
Zarrab last month accepted all the U.S. charges against him, including the violation of now-lifted U.S. sanctions against Iran, money laundering and other charges and agreed to cooperate as a witness against Mehmet Hakan Atilla, the former deputy CEO of Turkey's state-run lender Halkbank.
Zarrab, 34, was arrested by U.S. authorities in March 2016 on suspicion of taking part in the alleged sanctions-busting scam but is now cooperating with the authorities in an apparent plea bargain.
Since Zarrab became a witness, deputy chief executive of Turkish lender Halkbank Mehmet Hakan Atilla has become the lone defendant in the dock accused of violating sanctions, bribery and money laundering.
On Wednesday, a letter of the defense cited a September 2016 jailhouse call in which Zarrab said he needed to lie in "'in order to get out or to get a reduced sentence." Defense attorney Victor Rocco has raised questions over Zarrab's credibility, suggesting he made a deal to get out of jail free.