Hannibal Gaddafi and his wife, along with the Libyan state, filed a civil lawsuit against the Geneva authorities in a Geneva court on Wednesday.
They are claiming damages for the "disproportionate" way in which they were treated by the Geneva police who detained them in their hotel in July, after receiving complaints that they had seriously mistreated two of their servants.
The couple were released on bail after two days, and charges against them were dropped in September after the servants withdrew their complaint, having reached a financial arrangement with Hannibal.
"Police did not hesitate to use physical force and act with brutality despite a lack of resistance" and along with the justice authorities "deliberately chose to inflict on Mr Gaddafi the most humiliating treatment possible", Swiss television TSR quoted the complaint as saying.
The claimants also say that the Geneva authorities had not complied with the Vienna Convention on diplomatic standards and had set an "astronomical" bail of SFr500,000 ($435,000).
They further claim that police had refused to act on a complaint of theft made by the Gaddafi couple.
The Swiss lawyer for the Gaddafi couple said they were claiming SFr50,000 ($43,000) in moral compensation and SFr470,00 in material damages.
Lars Knuchel, spokesman for the Swiss foreign ministry, said: "The case has been filed with the Geneva courts and it is a good thing." The authorities in Geneva have consistently backed the actions of their police.
Relations between the two countries have been strained ever since the arrest, with Libya taking a number of retaliatory measures against Switzerland.
Libya demanded an apology for the arrest and imposed economic measures against Switzerland, including closing Swiss companies and stopping flights.
Two Swiss nationals working in Libya were also arrested and later released but were refused permission to leave the North African country. The foreign ministry says one of them is unwell and has appealed to the Libyan government to allow the citizens to leave the country on humanitarian reasons.
Bern has ruled out an apology over the affair and stressed the Geneva authorities acted within the law. In a briefing on Tuesday, the foreign ministry noted it could take "a long time" before the crisis with Libya was resolved.
Foreign Minister Micheline Calmy-Rey added that dialogue was continuing but the matter was now between the Geneva authorities and Libya.
swissinfo with agencies