Savannah is a pretty town in a low-lying marshy area. Although its topography is flat, it makes up for this with its architecture and squares. What a delight! Walk around the town and every few blocks you come to a square with statues, a fountain, and incredible old oak trees hung with moss (these squares were used as military defenses in the past). Sit on a bench in the shade and watch the tourists walk by. Sometimes there is a street musician playing a tune. The buildings all around you are lusciously decorated with ironwork fences, balconies, and stair railings.

You can visit Chippewa Square, which is where Forrest Gump sat on a bench in the movie waiting for a bus.
If you walk south you will come to Forsyth Park where a very grand fountain greets you. Here, homeless people are pushy and this takes away from the experience. There is a fantastic gated playground for kids.
Warning- do not come in summer! It is beyond hot, and gnats swarm around you sometimes getting in your ears or nose! It is just very unpleasant.
Besides the playground in Forsyth Park, there's not much to do for young kids in Savannah, so come on a romantic weekend without them, eat leisurely meals, and enjoy some tours of the historic homes. 
A lovely place to eat breakfast or lunch is Le Cafe Gourmet, where you can have a baguette sandwich or crepe like in Paris! This cafe works with kids too since it is counter service.
An impressive place to go with kids is Jepson Center, which is part of Telfair Museums. It is in an incredible glass building with fun activity stations for kids, plus modern art for parents to check out. Two other museums come with your admission. We checked out the yellow Telfair Museum, just a block away from Jepson Center, but it is musty and kind of rundown. Though it has some American artists from the late 19th century, which we usually love, we weren't very impressed. The yellow Telfair Museum contains the Bird Girl statue.
Some places I have not tried yet that you could check out:
If you walk all the way north you will come to the river at River Street. Tall former cotton warehouses line the street here providing shade at certain times of day. Here you can check out the stores, or walk along the river admiring the Georgia Queen riverboat.
If you visit Savannah with kids, eat at the Soda Pop Shoppe, for a fun and tasty meal.
Savannah Children's Museum is fun for kids to explore- it's mostly outdoors so come in good weather.
Kids will also like Old Fort Jackson (11 minutes drive east) where they have live cannon firings at 11am and 2pm daily (again, don't go when it's hot).
Kids can also play in the sand and sea at Tybee Island Beach (24 minute drive east). The sand is hard-packed and brown, and the water is cloudy, but it's always nice to be at the ocean. Parking is $2 per hour at the beach and there are no free times.

Directions

Chippewa Square is located at 222 Bull St, Savannah GA 31401. There is metered parking along the road.
Forsyth Park is located at Drayton St and East Park Ave, Savannah GA 31401.
Le Cafe Gourmet, 53 Montgomery Street, Savannah GA. Open breakfast and lunch.
Soda Pop Shoppe, 114 Bull St, Savannah GA 31401.
Savannah Children's Museum, 655 Louisville Rd, Savannah GA 31401, call (912) 651-4292. Open Wed-Sat 10-4, Sun 11-4. Summer hours are Mon-Sat 9-2. Admission is $7.50 for all people aged 18 months and over.
Old Fort Jackson, 1 Fort Jackson Rd, Savannah GA 31404. Open daily 9-5. Closed Jan 1, Thanksgiving, Dec 25. Admission is $8 for adults, $4 for kids aged 2-12.
Tybee Island Beach, 1613 Strand Ave, Tybee Island GA 31328. Parking is $2 per hour and enforced at all times.

Click on map for interactive view