He gets it; you don't.
Just because the football program may be the most visible part of the university, that doesn't make it the most important part.
We're all football fans on this board, so we recognize that the football program is a way that the university can connect with alumni and non-alumni fans. But if that connection never runs deeper than the football program, then it is a superficial connection that is of little significance to the university.
The university is much more than just football.
There are a handful of universities where the culture and identity of the place revolves around football. But those universities are the exception. And Rutgers is not one of those exceptions.
What you leave out, of course, is that if there is no football program, there is no invitation to the Big 10 (and all the academic benefits that come with it). There's an argument to be made (a strong argument) that, yes, the football program is the most important part of the university ... it is our single best available source of advertising (to potential students, employers, faculty, etc) and without it, we don't have the academic position and associations that we do. It's also the best available method of generating goodwill with alumni and local businesses and, more importantly, the donations that come with that goodwill.
Simply put and generally speaking, nobody cares about the English Department. People do care about the football program. And as long as that is true, it is more important.