The Constitution does not state that. The Supreme Court stated that, and assumed the responsibility. Most scholars grant it (that they have the power, not that it is their primary duty). But some do not. Also, your point does not refute Baron's point.
Actually, your point is an extension of what the Left is doing with speech, and wants to do with guns. You want to extend "normal limits" (for the amendments) and "interpretation" (for SCOTUS), both of which are generally accepted by both sides, to substantive limits on the Bill of RIghts and activist creation of law through radical "interpretation" which is actually legislation.
You will sit here and say they are just interpreting, or using reasonable limits. But, you are actually trying to legislate and establish substantive limits.