I Am a Descendant of James Madison and His Slave

My whole life, my mother told me, ‘Always remember — you’re a Madison. You come from African slaves and a president.’

Illustration: Sophia Zarders

PPresident Madison did not have children with his wife, Dolley. Leading scholars believe he was impotent, infertile, or both. But the stories I have heard since my childhood say that James Madison, a Founding Father of our nation, was also a founding father of my African American family.

In my childhood, whenever I whined or squirmed or got into trouble, my mother repeated the refrain: “Always remember — you’re a Madison. You come from African slaves and a president.” This is my family’s credo, the statement that has guided us for 200 years.

Though many in our family have heard we descend from President Madison and his slaves, only the griots — the one-in-a-generation oral historians in the family — know the full account of our ancestors, White and Black, in America. Gramps had told me many stories, but the detailed family history was Mom’s responsibility to convey to me when I became the next griotte.

The night my mother passed those stories on to me, I understood for the first time why some of the details of our family history were passed only from the griot of one generation to that of the next. Not only were some of the stories intimate, but this tradition safeguarded their accuracy, truth, and longevity. I sank into the sofa with my mother and listened with a new awareness of the significance of her words and what they meant to me. She began.

“When your uncle John, your uncle Mack, and I were children, Daddy would call us to his library. He would spin the big globe that sat on his desk, then stop it to point to different continents and countries and teach us about ancestors who had lived there. When Daddy spun the globe and asked us, ‘Who was the first African in our family to come to America?’ John would stop the spinning globe, touch the outline of Africa, and shout, ‘Mandy! She was kidnapped in West Africa and put on a slave ship.’

“John Maddison — that’s spelled with two d’s,” Mom said, beginning the saga of the Other Madisons, “was our first white ancestor in America. His son, John Jr., kept that spelling, but his grandsons, John the third, Henry, and Ambrose, spelled it with one d.

“Mandy was our first black ancestor stolen from Africa. That happened sometime in the middle of the 18th century,” Mom said. “She was the mother of our family.

“Mandy and her master, President Madison’s father, had a daughter, Coreen,” Mom went on. “She was the first African American Madison and the second of our family’s griots.

“James Madison Jr., the future president, saw Coreen walking back and forth between the kitchen and the mansion. And he wanted her,” Mom stated simply. “As soon as she became pregnant with his child, she began to worry she would not be allowed to keep the baby for more than a few years. Coreen gave birth to a boy. She named him Jim. Raising him, Coreen lived in constant dread that he would be taken from her. She had heard of family members, even mothers and infants, being sold and separated by hundreds of miles, never to see each other again.

“In Jim’s teenage years, Coreen’s fear became reality. He had been born around 1792. A few weeks after his birth, Dolley’s sister-in-law died, leaving two daughters — Susan, a toddler, and Victoria, an infant. Dolley agreed to take care of them. When the children arrived at Montpelier, she assigned Coreen to be Victoria’s wet nurse. Coreen nursed Victoria on one breast and Jim on the other. The two children became inseparable.

“Many plantation owners believed that black people lacked the ability to read, write, or ‘figger.’ The slaves knew this was not true. As Jim grew up, he hid behind the door and listened in on Victoria and Susan’s lessons. His father saw him hiding there but did nothing. Allowing Jim to learn,” Mom speculated, “was Madison’s way of showing love for his son.

“When Victoria was 12, Dolley told her she could no longer be around boys, especially slave boys, like Jim.

“In 1809, when Madison became president, he brought Coreen and Jim to Washington. Dolley directed the other house slaves to make sure her niece was not around when Jim was working. But Victoria was hardheaded; she hid in the armoires in the Madison family bedrooms, where they shared their deepest thoughts and feelings. It didn’t take long for Jim and Victoria to fall in love.

“One of the maids found out and suggested they stay away from each other. Jim was worried the maid would report them to Dolley, so he went to his mother. She burst into tears when he disclosed how he felt about Victoria. Coreen knew her boy could be sold or killed. She persuaded the steward to let Jim work in the kitchen, where it would be easier to keep the young lovers apart. But Victoria followed him there. One of the chefs, a slave, warned her that if she didn’t stop sneaking into the kitchen, he would have to tell the mistress.

“In 1812,” Mom continued, “the United States declared war on Great Britain, and on August 24, 1814, British soldiers and slaves who had been freed and recruited to fight with them advanced on Washington. Dolley told Jim to save the American flag. He folded it, secured it under his shirt, then ran to hide in the woods with the other slaves.

“Years after the war,” Mom said, “Jim told his children how worried he was that lightning and flames from the burning city would reveal his hiding place.

“In December 1814, two months after Britain and the United States negotiated a peace treaty, Dolley gave a party to celebrate. She ordered the male slaves, including Jim, to stand along the walls holding rushlights.

“A decade later, he told his children how sore his arms became, but he would never forget the music, the dancing, and the ladies and gentlemen in their finery. Standing there like a statue, he must have looked like the hero he truly was.

“A few days after the party, Victoria sneaked into the kitchen again. The chef promptly carried out his threat. Dolley was furious. She had assumed that Victoria and Jim knew better than to fall in love. Dolley arranged to sell Jim immediately, and the president made only weak objections.

“Just before Jim stepped onto the wagon that would take him away, Coreen held him tight and wept bitter tears. Her only hope was that the Madison name might serve as a tool to help them find each other someday.”

I knew what was next. I could hear Coreen whisper to her son: “Always remember — you’re a Madison.” This is how the credo was started. Over the generations, as America changed, words were added and we were reminded that we weren’t just descendants of a president, we were descendants of slaves, too.

“Jim was sold twice,” Mom went on. “The first time, he was sold to a nearby plantation, but Victoria drove herself there on a wagon to see him. The plantation owner sent her back and informed Dolley, who begged the new owner to sell Jim to someone far away.

“He ended up in Tennessee and never saw Victoria or Coreen again. But he remembered he was a Madison.”

Jim, I would learn when I tried to find him, was one of the countless slaves who was not valued by those who created America’s written record. Much of his story is lost to history.

TThere were more stories for my mother to tell me, but the hour had grown late, and Mom and I were tired. I accompanied her upstairs, kissed her good night, then returned to the living room.

I did not question Madison’s greatness, but I did question his goodness.

I have always been very private and reserved by nature, but I knew the source of my reluctance to accept this new role was about something more. For my mother, being the griotte meant being proud of descending from a U.S. president. She was not ashamed of having slaves in her family tree, yet, although most black Americans had enslaved ancestors in their genealogies, few had presidents. And she believed that President Madison was a great and good man.

I did not question Madison’s greatness, but I did question his goodness. James Jr. inherited more than fortune and power. He also inherited the Southern way of thinking and behaving. Madison was a Founding Father who was reputed to have been kind to the human beings listed among his possessions, but I knew that he, like his father and many other plantation owners, sexually assaulted or coerced the women he owned. In my eyes, the griots before me had glossed over the less than admirable behavior that had given James Madison Jr. a place of honor in my family tree.

The next day, my mother said, “I have something to tell you before I go back to Oakland tomorrow. I’ve never told you before. Your father is the only person I’ve ever mentioned it to.” She then told me about how her mother had been hard on her growing up — more than hard, really. “She loved us, I’m sure, but sometimes it seemed she was trying to break us by enforcing a list of rules that was as long as her arm. She was hardest on me, claiming I was the most disobedient.

“Mother would push me facedown on the floor, sit on my back, and beat my behind with a belt. I cannot justify her abusive actions to anyone, especially myself. I can only suppose Mother loved me in her way and was trying her best to direct me. I was a girl, so maybe she singled me out for punishment because she was afraid some man might try to hurt me if I didn’t follow her rules carefully. Maybe Daddy understood her reasons and accepted her methods. A little misstep on my part could have cost everything they worked for and put me in danger. They were only one generation removed from being slaves themselves.

“What happened to Mandy was much worse than what happened to me. I tried to be strong like her. And Daddy had told me I was President Madison’s great-great-great-granddaughter. I was proud of that. Even Mother couldn’t beat that pride out of me. Mandy and the president got me through those beatings.”

I wanted to say something that would take away that long-lasting hurt, but nothing seemed adequate. Mom blinked away tears. The beatings and Mom’s anger and pain would now be part of the family saga.

Over the next two years, I went through the box again and again. I studied maps of West Africa and researched the history of slavery in America. I read biographies of James and Dolley Madison and historical studies on the relationship between Thomas Jefferson and his slave Sally Hemings. And I tried to figure out the message the directive should impart to the current and future generations of African American Madisons.

My mother, like the ancient storytellers of West Africa, retold the family history that had been passed from generation to generation faithfully and accurately. She added her own stories and messages but never challenged any part of the saga. However, unlike the ancients, who relied strictly on oral tradition, my mother embraced a new tradition started by my great-grandfather Mack. To make the stories tangible and support them, he had gathered up letters, documents, and photographs. He wanted his children and grandchildren to see their ancestors, read their words, and hold in their hands evidence of what they had accomplished once they were no longer in bondage.

I, the newest griotte, would be the first to write it all down, and I began to realize I would be the first to explore the discomforting parts of our story. I had many questions. Who were these slaves and slave owners I had heard so much about? How had they influenced who I was? I wanted my ancestors to become real to me. I needed to visualize all of them and understand their sorrows, joys, and passions.

I needed to know how Mandy had survived the Middle Passage and life in bondage and how generations of her descendants had endured unrelenting, sometimes life-threatening, racism.

And I had to try to understand how James Madison could own some one hundred human beings while knowing that the widespread institution of slavery spat on the moral principles underlying the nation he had helped create.

To know my ancestors and find myself in their stories, I had to walk in their footsteps. Virginia, I decided, would be the best place to begin.

Excerpted from THE OTHER MADISONS: The Lost History of a President’s Black Family by Bettye Kearse. Copyright © 2020 by Bettye Kearse. Used by permission of Houghton Mifflin Harcourt. All rights reserved.

Bettye Kearse is a retired pediatrician and an oral historian who explores her unique family legacy. She descends from President James Madison.

Get the Medium app