North Korea on Thursday fired shells into waters near a South Korean warship on patrol south of the disputed Yellow Sea border, prompting an evacuation of residents on a nearby island, officials said.

Two shells fell near the South Korean ship, which was sailing near the frontline island of Yeonpyeong, the South's defence ministry said.

Residents on the island were advised to move to civilian shelters and nearby fishing boats were told to return to port.

North Korea shelled Yeonpyeong in November 2010, killing four South Koreans, briefly triggering concerns of a full-scale conflict.

"North Korea fired shells which fell near our ship, but it did not cause any damage to our ship," a ministry spokesman told AFP.

The North's move prompted a response from the South Korean vessel which fired several rounds into waters near a North Korean vessel, he said.

The exchange of fire sparked a tense confrontation between warships from the two sides, but there was no additional provocation from North Korea, the spokesman said.

"The situation is now stable but we are closely watching the movement of North Korean troops," another ministry official said.

There were no reports of casualties or damage on the island from authorities, the Yonhap news agency said.

The incident came ahead of a Friday visit to Seoul by the European Union's foreign affairs chief Catherine Ashton, and the EU confirmed that talks would include "security cooperation, non-proliferation and recent developments in our respective neighbourhoods including the DPRK (North Korea)."

Ashton will meet President Park Geun-hye and Foreign Minister Yun Byung-se in the South Korean capital.

The North's military had threatened Wednesday to attack South Korean warships "without any warning" if there was even a "trifle" violation of the maritime border, after a South Korean naval ship fired warning shots to stop an incursion by three North Korean patrol boats across the sea.

The South's navy urged the North to stop "absurd threats" and warned: "We will mercilessly punish any provocative actions by North Korea."

The North does not recognise the Yellow Sea border, the scene of brief but bloody naval clashes in 1999, 2002 and 2009.

In March the North fired hundreds of shells in a live exercise near the sea boundary. About 100 shells dropped into South Korean territorial waters, and the South responded with volleys of shells into North Korean waters.

Cross-border tension has been high for months, amid signs that the North may be preparing to conduct a fourth nuclear test.

This month the two Koreas have upped the ante in their verbal exchanges over crashed surveillance drones recovered on the South Korean side of the border.

Seoul said a joint investigation with US analysts had provided "smoking gun" evidence that the drones came from the North. Pyongyang flatly denied any involvement.