The prince appeared angry over his fall, but later revealed he was upset over losing a bet. He stood to win $B100,000 ($US48,000) for his charity, Sentebale, if he had managed to stay in the saddle for the entire match.
The generous donor readily accepted afterwards that Prince Harry had not fallen off - his horse had slipped over - and agreed to pay the full amount to the charity.
The prince was in the Caribbean island to play in the inaugural Sentebale Polo Cup.
It will become an annual fundraising match for the charity, which he set up with Lesotho's Prince Seeiso to improve the lives of impoverished children in the African kingdom.
The event may have raised as much as $US800,000.
Prince Harry said afterwards that he was a bit sore after the fall, in which the horse slipped from under him.
He said he had tried to throw himself clear from the falling horse, which may have made it look more dramatic than it actually was.
The prince captained his side to a win over a South African team at the Apes Hill Club.
Before the game, Prince Harry made a speech on behalf of Sentebale.
He expressed his sympathy for the victims of the Haitian earthquake and spoke of how when "natural disaster strikes" eventually a sense of hope would follow.
The Prince said: "It seems to be one of life's cruelest ironies that when natural disaster strikes - whether it be earthquake in Haiti, or years of drought and ravaging by HIV/Aids in Lesotho, it is always the most vulnerable the defenceless, normally the children, who are left exposed and needing more help.
"We seem so powerless to prevent these things from happening, but out of such tragedy, comes the purest of human reactions: love, charity, and, in time, a sense of renewed hope. This is really what Sentebale is about."