The first presidential inauguration was George Washington's, on April 30, 1789. The US government was headquartered in New York City at the time, so the ceremony was held at Federal Hall in lower Manhattan.
While Washington's inauguration was a large public event, it did not begin a tradition. The United States government moved to Philadelphia in 1790, and Washington's second inauguration took place within the US Senate chamber on March 4, 1793.
John Adams also took the oath of office in Philadelphia, in the chamber of the House of Representatives. The government moved to Washington, DC, and Thomas Jefferson was the first president to take the oath of office at the US Capitol. Jefferson was inaugurated in the Senate chamber of the new Capitol building on March 4, 1801, and again on March 4, 1805.
The tradition of holding the inaugurations inside the Capitol building, in either the House or Senate chamber, continued for decades. One exception was the inauguration of James Monroe on March 4, 1817. The Capitol and White House had been burned by the British in 1814 and were still being rebuilt, so Monroe's inauguration took place in front of a Washington hotel.
Andrew Jackson's inauguration on March 4, 1829 was held on the east portico of the US Capitol so the public could attend. The tradition of large public inaugurations continues to this day, though the ceremonies were moved to the west side of the Capitol beginning with Ronald Reagan's first inaugural in 1981.
And, of course, presidents who ascended to the office due to a president's death or resignation have taken the oath of office in various places, ranging from hotel rooms to Air Force One.