I'm in the same boat as you. Slogged it through RU, paid my own way, and graduated with $6K in debt that I paid off within 6 months because debt has always scared me.
If I had to choose, and I don't have a choice, I would rather these decisions be based on economic disadvantages than on race/color alone. But will the top schools then be excluding the best and the brightest in favor of lowering standards to less smart and/or less qualified applicants? We are seeing this in a range of professions such as airline pilots. Quite frankly, I want the most skilled pilot, and I don't care whether the cockpit is full of white men, black women, or an all LGQBTQIA crew (we have several family members in the LGBTQUIA crowd), as long as they are all highly qualified and up to the job.
Anecdotally, however, knowing someone who has a family member that is a teaching at an Ivy medical school, some students are being "waved through" despite their medical school performance. In high risk fields such as operation of aircraft, military, police, firefighting, etc, it is a fair question to ask whether society should care about diversity over skill level and aptitude for the job.
Back to your kid/kids. I truly believe that instilling the values of hard work, empathy, community service and being reliable and stepping up to do the hard tasks are more important than where they go to school. Sure, an Ivy or a Johns Hopkins will open doors, but as you may have seen, some (maybe many) who went to the "finer" schools do not have the same grit as a people such as you or your wife, who made the "hard way" earning it. We were concerned that our kids might feel privileged, and not be hard workers, and would coast. Kid 1 is out and doing well. Kid 2 is well on the way (half way through Physician Assistant program) and absolutely crushing school, a public health fellowship in South America this summer, and an active volunteer. YMMV, but the traits instilled by parents can mean more than an expensive degree from a fancy school.