After walking miles and miles in Washington DC, there's not that much point walking further to get to the Smithsonian Castle. But if you happen to pass by it anyway, check out its beautiful rust-colored exterior of Seneca red sandstone from Maryland. The castle was built in 1855.

If you walk through the castle to the back (you must get your bag checked first and then checked again when you re-enter, making this not worth it) there's the Enid A. Haupt Garden that smells very nice, but is sun-exposed (impossible to spend time there in summer). It is a long lawn with criss-cross borders of flowers, and some blossoming trees on the perimeter with benches underneath. If you want to see just the garden, you can enter it from the south without having to go through the castle. 
Inside the castle, it's musty, and there's not much except some chairs and tables, an information stand, a cafe with food that does not look good, and a small gift shop. 
There is a carousel near the front/north/National Mall entrance to the Smithsonian Castle that kids might enjoy. Adjacent to the garden, south of the Castle, is the: Arts and Industries Building (beautiful architecture and many murals inside), the Smithsonian National Museum of African Art (headpieces), and the Sackler Gallery (Asian art and items).
An eight minute walk west of the Smithsonian Castle brings you to the US Holocaust Memorial Museum. A six minute walk east of the Smithsonian Castle brings you to the National Air and Space Museum.

Directions

Smithsonian Castle is located in the center of the National Mall, on the southern end.

Click on map for interactive view