Devidas Paliutis, 27, originally from Lithuania, appeared at Newry Magistrates Court, via video link from the prison, on Wednesday.
The Prison Service was advised that the holding charges were withdrawn from the Magistrates' Court.
"However, they failed to act on the notification that other charges had been substituted, for which he was returned for trial to the Crown Court, and erroneously released Mr Paliutis", the Prison Service said in a statement.
"Steps are being taken to safeguard the interests of the victim of the alleged crime."
The error was discovered on Thursday, but prison authorities delayed publicising the mistake for 24 hours in the hope he could be picked up in parts of Co Armagh and Co Monaghan, in the Republic of Ireland, which he had frequented.
The Prison Service website lists Paliutis as having been charged with rape.
He is being described as having brown eyes, brown hair, of medium build, with a left ear piercing, a vaccination mark upper left arm and scars on both knees and the forehead.
'Embarrassing'
Justice Minister David Ford has ordered an immediate and full inquiry into the circumstances which led to this release.
He said he will invite the Chief Inspector of Criminal Justice to review the findings.
Mr Ford has described the mistake as "embarrassing".
"It is clearly an embarrassing incident; it needs to be set against the fact that several thousand similar cases are heard either by video link or in court every year but any one case is clearly worrying", he told UTV.
"That is why we took such serious action."
The PSNI, An Garda Siochána and Interpol are co-ordinating efforts to have Mr Paliutis returned to custody, the prison service confirmed.
Ulster Unionist Party leader Tom Elliott says he is "deeply concerned" by the mistake.
"This is a highly unusual and disturbing circumstance, and I hope that the authorities will locate the man in question as quickly as possible", Mr Elliott said.
"I would question how on earth something like this could happen.
"The fact that the prisoner was charged with serious sexual offences and unlawful and injurious imprisonment adds to my concern, and of course my thoughts are with the victim at this time."
Last month, Mr Ford launched a review into how other sex cases are dealt with in Northern Ireland, in the wake of the controversial handling of the Donagh case.
He also ordered an audit on other court rulings against sexual offenders after it emerged that there were clerical errors in the judge's orders issued to the paedophile McDermott brothers, who were allowed to return to live in the community where they committed the abuse.
"All departments make administrative errors, but the Department of Justice deals with some of the most serious issues that arise in our society and has care of its most dangerous members", SDLP Justice spokesperson Alban Maginness said.
"Coming so soon after the administrative errors in the Donagh case, this must send a strong signal to Justice Minister David Ford that standards in his department must match the seriousness of the issues that it deals with."
Sinn Féin Foyle MLA and deputy Chair of the Assembly's Justice Committee, Raymond McCartney, says he will be raising the issue through the Justice Committee.
"That a man accused of such serious offences could be released through 'error' does little to reassure those many victims of crime who already lack significant confidence in the justice system," he said.
