I think it boils down to desirability/demand. UC receives a far greater than proportionate number of applications (relative to number of available seats/capacity) compared to Rutgers. They both only have so many seats across their campuses, and this comes into play with how many applications they each have to accept (and from which, how many will enroll), in order to fill their incoming classes at each campus..., i.e. the yield.
Unless things have changed recently, like the UC system and probably several other universities, Rutgers also has a single application from which you can apply to various undergrad schools at NB, at Newark, and/or at Camden. Also there may be a limit to the number of schools you can apply to for an initial application fee, with probably a higher tiered fee structure to be able to apply to more schools.
I'm not sure CA and NJ would fare much differently relative to their populations, in fact it might suggest the reverse of your claim since NJ likely has a relative shortage of supply of seats to meet its in-state demand compared to CA. We hear every year about significant brain drain/net export of students from NJ due to this issue...perhaps more than any other state it "needs" the Shop Rite moms to send their Joey and Sally to the OOS publics and to privates, nevermind that some of them desire to attend those schools anyway.
Rutgers-NB isn't able to have as high of a selectivity as the top UCs partly because it doesn't attract as many applications and not as many out of state applications either. Just need to continue trying to market the school better.
Probably true. Same with Rutgers with a NJ kid aiming for Newark and/or Camden and also applying to New Brunswick (potential reach).
When it comes to available seats versus demand for admission at certain undergrad schools, however, there may be something to consider. There is wider availability of programs/majors across different UC campuses compared to the Rutgers campuses. Intra-school transfer within Rutgers notwithstanding, one can only study in certain programs at New Brunswick (engineering, pharmacy, ag, et.al.), or transfer in to NB to finish the upper division of study. Whereas in the UC system, for example, one can complete a reputable degree in engineering at more than just Berkeley or UCLA (in fact it is possible to do at more than half, if not all, of their campuses - probably not UC Merced). That's one difference in the broader university system setup that UC has versus the smaller setup that Rutgers has (only one larger flagship campus, two smaller regional campuses). Rutgers is generally more similar to U of Michigan in that context.