Jerusalem (CNN) -- The convoy of United Nations Secretary-General Ban Ki-moon was pelted with shoes by Palestinians in Gaza Thursday, according to witnesses and a U.N. official.

Upon entering Gaza from Israel, Ban's car was met by a protest of several dozen people who threw objects, including shoes, at it, said U.N. spokesman Eduardo del Buey.

Witnesses said the demonstration was made up of relatives of Palestinian prisoners in Israeli jails, protesting what they see as a lack of U.N. attention to the plight of imprisoned Palestinians.

The convoy did not stop, del Buey said, and the protest did not interrupt Ban's planned meetings with U.N. personnel. "All is fine," the spokesman said.

But the U.N. later said that representatives of civil society had canceled a scheduled lunch with Ban to protest "against the situation of Palestinians in Israeli jails."

In an open letter to Ban released to the media, the representatives accused the secretary-general of not upholding international law as it applies to Gaza and treating Palestinians in "a manner which can only be described as subservient to the will of powerful states."

The letter also criticized Ban for not holding Israel accountable for crimes during the course of its three-week military campaign in Gaza beginning in 2008 and for not meeting with family members of Palestinian prisoners, when he held such meetings with the family of captured Israeli soldier Corp. Gilad Shalit.

When asked about the shoe-throwing incident, Ban said he understood the protesters' frustrations.

"I met many people who were waiting for me at the entrance," he said at a short press conference at a U.N. event in Khan Younis. "I fully share their concerns and frustrations, and for that reason I am here for the third time. There is a very dire economic and social and humanitarian problem."

Ban met Wednesday night with Issa Karake, Palestinian minister of prisoners affairs, "and received a letter outlining specific concerns," the U.N. said in a statement. "The United Nations continues to call on Israel to abide by its obligations under international law."

There are more than 5,000 Palestinians currently being held in Israeli prisons, according to human rights groups, and the issue continues to be a sensitive one in the Israeli-Palestinian conflict.

Ban is on a tour of the Middle East including stops in Jordan, Israel and the West Bank.