A UN nuclear agency team returned Friday from Iran without a deal on additional inspections, as diplomats briefed afterwards voiced scepticism that more talks next month could bridge the differences.

Dampening hopes of progress in parallel talks with world powers that might take place in late January, the International Atomic Energy Agency told member states in a statement that "important differences" remain.

"We could not finalise the structured approach to resolve outstanding issues regarding possible military dimensions of Iran's nuclear programme," chief inspector Herman Nackaerts told reporters at Vienna airport, without elaborating.

One Western diplomat told AFP that Iran had put on the table at talks on Wednesday and Thursday "unacceptable conditions" and that he would be "very surprised" if a deal were reached in Tehran at another round set for February 12.

A second Vienna diplomat agreed, saying that IAEA director general Yukiya Amano's comment last week that he was "not necessarily optimistic" is "still probably accurate".

"I think that on balance I would be surprised if there was an agreement in February, although I wouldn't write it off," said a third.

Iran's representative in the talks and its ambassador to the IAEA, Ali Asghar Soltanieh, said: "Some of the differences were solved but it is a very complex issue."

The IAEA conducts regular inspections of Iran's declared nuclear facilities but it also wants access to what it believes are sites where undeclared activities aimed at developing nuclear weapons took place until 2003, and possibly since.

Nackaerts had said in December after an earlier visit to Tehran that he had expected, after a string of fruitless meetings this past year, to at last sign a deal this week.

On Friday he made no such prediction about the next round.

Nackaerts said that during the talks "also on this occasion no access was granted to Parchin", one of the sites the agency would like to visit.

Iran denies ever having worked on nuclear weapons and says that the IAEA's evidence is based on faulty intelligence that it has not even been allowed to see.

It says that because no nuclear activities took place at Parchin, the IAEA has no business conducting inspections there and that it already went there twice in 2005.

The talks this week were originally due to last one day but were extended.

It was unclear what went wrong this time but in the past Iran has insisted that the agreement include clauses that could infringe on the IAEA's ability to conduct proper inspections.

"It's disappointing, but not all that surprising, that Iran has yet again failed to follow through on earlier indications of flexibility," said Mark Fitzpatrick from the International Institute for Strategic Studies.

The talks with the IAEA came ahead of a new meeting between Iran and the P5+1 powers -- the United States, China, Russia, Britain, France and Germany.

This parallel effort is focused more on Iran's current activities, in particular uranium enrichment, a process that can be used for peaceful purposes but also for creating the core of a nuclear bomb.

At their last meeting in Moscow in June, Tehran rejected P5+1 calls for it to scale back its nuclear enrichment activities, while asking for relief from sanctions that in 2012 began to bite.

The six are since reported to have reworked the proposal, albeit not substantially.

Watching both the IAEA and the P5+1 meetings is Israel, the Middle East's sole if undeclared nuclear-armed state, which has threatened to bomb Iran's nuclear programme sites.

"I believe some of the actors in Tehran really do want to strike a deal with the IAEA as a prelude to the more sensitive talks with the P5+1," Fitzpatrick told AFP.

"But getting the political forces in Tehran in alignment on what might look like a compromise is proving to be too difficult. Uncompromising hardliners remain in command."