In a letter to Abigail Adams in 1804, Thomas Jefferson wrote of his concern that the courts could potentially take away too much power from the other two branches of government. Remember that our three branches of government were supposed to be equal and serve as checks/balances on each other. Jefferson feared the courts were in a position to have the last word on issues. He said:
"The Constitution... meant that its coordinate branches should be checks on each other. But the opinion which gives to the judges the right to decide what laws are constitutional and what not, not only for themselves in their own sphere of action but for the Legislature and Executive also in their spheres, would make the Judiciary a despotic branch."
The Constitution does provide a fuse for this. Congress can specify areas where the court does not have jurisdiction. Certainly there is potential for abuse here, too, but I am not aware of Congress having used that power very often.
More of Jefferson's thoughts can be found on the University of Virginia site