Because for decades they did everything they could to be anti-intellectual in the name of having the "elites" instead. Yake boasted they had the most gentlemen and least scholars, Princeton said they didn't want scholars but that "all-round man". They also had the worst test scores in the country. Harvard's Lawrence Lowell consistently fought for racism, anti-semitism, etc. Harvard was the best of the three academically, but still.
The ivies (especially the big 3) managed to turn their reputation for regular elitism into academic elitism in the post-WW2 era, but the way they acted before that leaves a bad taste in my mouth.
That said, Douglass had a nazi leading its German department, so it's not like a lot of schools didn't change.