The Zimmerli building is pretty interesting actually. It's built in the burnt out remains of Ballantine Gym, which burned down about 100 years ago. You can actually find hints of Ballantine's brickwork if you walk around the outside of the museum. Not only that, but Zimmerli's art storage room is the old pool of Ballantine, complete with shallow and deep end, pool tiles, and grates.
The museum connects directly into Voorhees Hall, which was once it's own, stand alone library. (Many of its windows have been filled in, but it was the University's main library after Kirkpatrick.) Voorhees Hall of course, is now physically connected to the art library.