Stephen Colbert shares the worst part of leaving the “Late Show” set: 'Breaks my heart'

"It's the best broadcast space in New York City," the host said of filming within the historic Ed Sullivan Theater.