In principle, it would be best to confer all the benefits of adulthood (voting, marriage, driving, etc) to anyone who passes an exam testing things like ability to identify self-interest, ability to reason and think critically, understanding of cause and effect.
This will never happen, though, because a) examiners are corruptible and the passage of time is not. b) millions of people who are currently legal adults - in fact even plenty of people who have gotten to retirement age - would never pass it, and therefore never support it.
As we can't have a good system, then, we're stuck with a standard age for certain rights. All of these will always look completely arbitrary if you look at them in enough detail (because they are), so quibbling about the precise age is somewhat meaningless.