Among Knoxville's earliest settlers, the Williams family left its name on nearby Williams Creek. Col. John Williams II (1818-1881) built the home in 1842 with the help of slaves who reportedly made the bricks on site. The Williamses were Unionist Democrats, and their guests included William’s Greeneville friend, Andrew Johnson -- who would later become president. The house was the childhood home of Williams' sons, both of them attorneys prominent in local politics: J.C.J. Williams and Thomas Lanier Williams II. The latter was the grandfather of Thomas Lanier Williams III, better known as Tennessee Williams. The famous playwright never lived in Knoxville, but his father and several other ancestors were born and raised in the city. The Riverside house originally faced in the opposite direction, toward Dandridge Avenue, where other Williams family homes were located. But in 1899, a remodeling by the Richards family reoriented the house south, toward the river and Riverside Drive.
An older house, up the hill on Dandridge Avenue at Williams Creek Golf Course, belonged to an earlier John Williams (1778-1837), a U.S. Senator and diplomat who sometimes opposed to the policies of President Andrew Jackson. It's the oldest house in this part of town, dating to 1826.