After George Washington was sworn in as President, he lead the entire House, Senate, and cabinet to St. Paul's Episcopal church for a two-hour service, lead by the Chaplains of the House and Senate.
The Founders included the First Amendment to the U.S. Constitution partly to protect religious rights. It was intended to keep the Federal government from interfering in religious practices and to keep it from establishing a national religion. Today, the 1st Amendment is turned on its head due to lack of historic perspective. For more insights, check the reading list: The First Amendment and The U.S. Consitution