Kennesaw is the best battlefield to walk.
Downtown, there is Cyclorama with Civil War Museum.
Other historically significant things to do include the Carter Presidential Library, the Margaret Mitchell House, and The Atlanta Historical Society (exhibits and grounds).
Stone Mountain Park has lots of family oriented attractions.
If you want some exercise, there is a 1.3 mile walking trail up the back side of Stone Mountain, and you can ride the cable car down.
Other touristy options include the World of Coca-Cola and the Georgia Aquarium downtown.
I would endorse a visit to the Varsity, but would also recommend Mary Mac's Tea Room on Ponce near the Fox Theatre and Peachtree. From there, its also not too far to Little Five Points for food, drink and Atlanta's version of boho culture. Buckhead used to be the place to go, but its a little overdeveloped for me these days.