Agree to a point, and we did see Mathis' minutes go down as Pike used him less in the latter part of the year.
But as you say, it's team defense - so looking at how different lineups perform overall gives you a sense of which combinations of players works better than others offensively and defensively. If you take a lineup with 4 players, and then swap other players into the 5th spot, you can at least gain a little insight into their impact on the overall group.
Because being a "good defender" is a lot more than just 1x1 on-ball defense.. it's switching appropriately, overall positioning on the floor when your man doesn't have the ball, it's staying with your man on cuts, fighting over screens, closing out on shooters, forcing the ball away from or to certain players, defending without fouling, getting defensive boards, getting back in transition, etc.
That really can't be measured easily in a single metric - which is why you try to look at overall points allowed while a player is on the floor, or even more specifically when they are on the floor with certain lineups vs. other players on the floor with the same lineup. It's all imperfect to do it in aggregate - but what you see with your eyes is frequently backed up by what you see in a comprehensive box score.