Banneker born in Baltimore County, Maryland, was one of several children born to Robert, a freed slave from Guinea, and Mary Banneker. Mary's mother, Molly Welsh, came to the American colonies as an indentured servant from England and later married one of her slaves, an African of royal descent named Bannaka or Banneky. Banneker and his sisters were born free and grew up on a self-sufficient tobacco farm of 40 hectares (100 acres). Banneker received the equivalent of an eighth-grade education at a local integrated school and was also tutored by his grandmother. Growing up, he spent much of his free time devising and solving mathematical puzzles. He took over the farm after his father's death in 1759.

In the 18th century, clocks and watches were rare devices constructed in metal by skilled artisans. At the age of 22, Banneker created a working clock from wood after studying the watch of a friend. Having no metal at his disposal, he meticulously carved each component from wood with a pocketknife. It took him two years to finish the clock. Banneker's wooden clock kept accurate time in hours, minutes, and seconds throughout his lifetime.

After his retirement from farming at the age of 59, Banneker began to study astronomy, becoming a man of science and mathematics through unassisted experimentation and close observation of natural phenomena. He became interested in astronomy through a local surveyor named George Ellicott, who loaned him astronomy books. Banneker employed his knowledge of astronomy and mathematics to help plan the city of Washington, D.C. In February 1791 United States president George Washington commissioned Ellicott and French engineer Pierre L'Enfant to help plan the construction of the nation's capital on an area of land 25 sq km (10 sq mi) in Virginia and Maryland. Ellicott invited Banneker to be his assistant. A dispute between some Americans and Frenchmen on the project led L'Enfant to abandon it and take the drafted plans with him. Over the course of two days, Banneker reproduced the intricate plans from memory, preventing a major delay. For this reason, some historians refer to Banneker as "the man who saved Washington, D.C."

Shortly after returning to his farm in April 1791, Banneker issued his first of some ten annual almanacs, which were published by several printers and sold widely in both England and the United States. Banneker charted the movement of heavenly bodies and successfully predicted several solar eclipses. Farmers and navigators relied on this important information. In addition, Banneker reproduced road maps, conversion charts, and literature in his almanacs.

On August 19, 1791, Banneker sent a copy of his first almanac to Thomas Jefferson, Secretary of State under President George Washington, in an effort to dispute Jefferson's belief that blacks were intellectually inferior to whites and in order to protest slavery in the United States. Jefferson congratulated Banneker on his publication and expressed his wish for more proof "that nature has given to our [black] brethren talents equal to that of other colors of men." Furthermore, Jefferson forwarded a copy of Banneker's almanac to the Academy of Sciences in Paris, France, one of the leading scientific societies in the world during the 18th century.