Rights to people are the sum of the user's rights via People, People by type and People by group, with rights determined dynamically based on the types and groups of the person in question.
If a user is given rights to people in a certain group, and that group has "children" in the group hierarchy, the user gets those same rights to all people in the group's children. For example, a user given "Edit" rights to the group "West Campus", which has groups beneath it "Facility 1" and "Facility 2", the user automatically gets "Edit" rights to both of these sub-groups. Changing the group hierarchy automatically updates these rights as these are implied rights and not explicitly stored.
Based on the current user's rights, if he does not have permission to at least view a person, in the person lookup tool certain aspects of that person will not appear on the list of people. See Person lookup in the System preferences and feature enabling tool.