The Howard League for Penal Reform says it will spend two years looking at all elements of the issue, from consensual to coercive sex in jails.
The investigation in England and Wales will also look at what can be done to improve the sex education of adolescent and teenage inmates.
The UK has a ban on conjugal visits for prisoners - although more than half of European countries allow them.
There is virtually no reliable information about the extent of consensual and coercive sex in jails, although ministers recently told Parliament there were almost 140 sexual assaults in 2010, up on the two previous years.
The Howard League said it wanted its independent commission to shed light on the issue, which had never been properly discussed.
The study has partly come about after the charity got involved in the cases of three teenagers who said they had been raped by other inmates.
The research team will try to assess the scale of sexual offending in prisons - but will also look at whether the ban on consensual sex should go.
Sex is unlawful in prisons because cells are deemed to be public places - but other countries take a completely different view.
'Behind the times'
Frances Crook, chief executive of the Howard League, said: "These are three very big issues that have huge implications for the justice system. Nobody has looked at these at all in this country."
She said that attitudes differed from country to country.
Stay informed. Subscribe to our newsletter
More than half of European countries allow some form of private visits involving partners, although the rules differ widely.
Canada has one of the most liberal systems and it allows private relationship visits for up to three days at a time for qualifying inmates.
Canada's prisons service says the scheme was established to "develop and maintain family and community ties in preparation for their return to the community". Other countries take a similar view that private family time while in jail improves the chances of an offender resettling after release.
The Ministry of Justice, which runs prisons in England and Wales, has no plans to change the law.
"The issues have never been thought through [in the UK]," said Ms Crook. "We are behind the times by not talking about them."
A key part of the investigation will look at the experiences of teenagers jailed for a year or more and how their sexual development and education compares to those on the outside.
"If you have a 15-year-old who is jailed for two years, they are missing out on what their friends are getting which is the development of their relationships, girlfriends and so on," Ms Crook said.
"They come out of prison with none of these experiences and we expect them, as a miracle, to catch up."
-BBC