The White House is "frustrated" by remarks from Afghan President Hamid Karzai, who on Sunday promised tribal leaders he would hold back a NATO military offensive in violence-plagued Kandahar province until he had their backing, a spokesman said Monday.

"We will not conduct the operations in Kandahar until you say we can," Karzai told about a thousand tribal leaders at a shura, or conference, at the governor's compound in the southern province.

"The remarks are genuinely troubling," White House spokesman Robert Gibbs told reporters in an off-camera session.

Gibbs on Friday characterized as "troubling" a speech made by Karzai last week, days after the Afghan leader hosted President Barack Obama on a surprise trip to Afghanistan. Karzai accused the United States and other Western governments of wanting a "puppet government" in Afghanistan and orchestrating fraud in the recent Afghan elections.

"It was disturbing Friday," Gibbs said Monday. "Obviously, it didn't get better. ... On behalf of the American people, we're frustrated by the remarks."

Asked whether the administration has confidence in Karzai, Gibbs said, "President Karzai is the elected leader of Afghanistan." The White House has outlined several things Karzai and others must do to make sure military gains are solidified.

A planned May 12 visit by Karzai to the White House is still on "as of now," Gibbs said.

Karzai was accompanied by Gen. Stanley McChrystal, the U.S. commander in Afghanistan, as he made his comments Sunday.

U.S. and NATO troops are planning a massive military offensive to wrest control of the Kandahar region from Taliban militants. Most of the gathered leaders were apprehensive about the operation. They were worried that the military push was planned during harvest time.

Referring to last week's speech, Karzai said Sunday, "in today's speech, I am pointing the finger at myself," implying that his government needs to hear from the public.

Asked by CNN about Karzai's recent remarks, McChrystal said, "I spend most of my time thinking not about what people say but what they do, and my partnership with President Karzai has been something I've been pleased with and something I've relied on as we've moved forward in operations, and that's reflected in his people and his government that I've worked with."

- CNN's Atia Abawi contributed to this report.