How did 5% of the population of the South manage to keep a system in place that was detrimental to the overwhelming majority of southerners, both...

The system you are referring to is the institution of slavery (before 1865). At the height of slavery, only about 5% of white southern families owned slaves. To perpetrate this system, the southern plantation owners co-opted poorer whites into supporting the system of slavery.

This cooperation goes back to Bacon's Rebellion of 1676, in which poor whites and former indentured servants revolted against the white elite. As a response, according to historians such as Edmund...

The system you are referring to is the institution of slavery (before 1865). At the height of slavery, only about 5% of white southern families owned slaves. To perpetrate this system, the southern plantation owners co-opted poorer whites into supporting the system of slavery.

This cooperation goes back to Bacon's Rebellion of 1676, in which poor whites and former indentured servants revolted against the white elite. As a response, according to historians such as Edmund S. Morgan (author of American Slavery, American Freedom), the white elite instituted an increasingly harsh and inflexible color line that kept African-Americans enslaved and that gave poor whites a superior position to slaves in the social hierarchy. The elite also supported the movement of white settlers onto Native American lands; the restriction of white westward movement had been one of the causes of Bacon's Rebellion, and the southern states (and the federal government) supported encroachment on Native American lands into the 19th century with the policy of Indian Removal. If poorer whites were allowed to expand, they were pacified and less likely to revolt against the elite. Therefore, the color line was a tool that the white southern elite used to hold power before the Civil War.

Fear was another tool the southern white elite used to maintain their power and a reason why 95% of the population did not revolt against the system. Escaped slaves were generally shot, and slaves were beaten and treated cruelly to prevent their solidarity and to break their will. Families were routinely broken up, and most slaves, as documented in Frederick Douglass's autobiography, were prevented from learning skills, such as reading, that would allow them the means and the will to escape.

In the north, power brokers often used a religious or ethnic line to co-opt other whites into cooperating with them. For example, northern factory owners increasingly turned toward employing Catholic people from Ireland to work in their factories. The northern elite often stoked feelings of nativism, the belief that American-born Protestants from Northern Europe were superior to foreign-born Catholics and later (in the late 1800s) to Jews, Slavs, Italians, and other people from Southern and Eastern Europe. This system co-opted poorer whites into supporting the status quo.