Justice Richard Storey was appointed to the Supreme Court by James Madison. In 1833, Storey wrote "Commentaries on the Constitution." In it he said,
"It yet remains a problem to be solved in human affairs, whether any free government can be permanent, where the public worship of God, and the support of religion, constitute no part of the policy or duty of the state in any assignable shape. The future experience of Christendom, and chiefly of the American states, must settle this problem, as yet new in the history of the world, abundant, as it has been, in experiments in the theory of government."
See the excerpt on Google Books