Mercury affects the behavior of white ibises by "turning them homosexual", with higher doses resulting in males being more likely to pair with males.
Scientists in Florida and Sri Lanka studied the effect of mercury in the birds' diet. Their aim was to find out why it reduced the ibises' breeding.
Mercury pollution can come from burning coal and waste, and run-off from mines.
The report, in Proceedings of the Royal Society B, shows that wetland birds are particularly badly affected by it.
Although the researchers already knew that eating mercury-contaminated food could affect an animal's development, they were surprised by the "strange" results of this experiment.
"We knew mercury could depress their testosterone (male sex hormone) levels," explained Dr Peter Frederick from the University of Florida, who led the study. "But we didn't expect this."
The team fed white ibises on food pellets that contained concentrations of mercury equivalent to those measured in the shrimp and crayfish that make up the birds' wetland diet.
The higher the dose of mercury in their food pellets, the more likely a male bird was to pair with another male.
Dr Frederick and his colleagues say the study shows that mercury could dramatically reduce the breeding rates of birds and possibly of other wildlife.
The exact mechanism that causes this change in behaviour is not yet fully understood.
But mercury is known to disrupt hormonal signalling, so it could have a direct impact on the sexual behavior that is mediated by those hormones.
Importantly, the males with the higher mercury doses performed far fewer courtship displays, so they were more likely to be "ignored" by females.
Males fed higher mercury doses performed fewer courtship displays
Wetland habitats, like the Florida Everglades that are home to these birds, are particularly vulnerable to mercury contamination.
Bacteria that live in the thick, oxygen-free sludge chemically alter the mercury, turning it into its most toxic form - methylated mercury.
And this chemical can act as a sort of biological impostor, mimicking hormones that act as the body's natural chemical signals.
Some of these signals are involved in reproductive behavior - they may stimulate an animal to carry out a courtship display or motivate it to mate.
"We're seeing very large reproductive effects at very low concentrations [of mercury]," said Dr Frederick. "So we really need to be paying more attention to this."