About 40% of U.S. energy is used in buildings, mostly for lighting, heating, and cooling. The Department of Energy, and many others, have set ambitious goals for reducing building energy use. Proposed energy reduction methods range from the mundane (insulation) to the exotic (using waste heat to generate electricity). We investigate the challenges from two very different directions: (1) very simple physical models that elucidate the basic physical and engineering relationships among energy flows, and (2) complicated simulation models that predict the behavior of individual building systems (such as cooling system components) and that are used to attempt to optimize building design and operation. We compare predictions from these approaches to data from real buildings, and discuss how buildings use energy, how they can use it better, and why current goals will be hard to meet.