In search of a warmer climate due to his ailing health, he sought the position of British Ambassador in Constantinople, to which he was finally appointed in 1799. Shortly prior to his departure, he married Mary Nisbet, an heiress from a wealthy family. This proved to be a timely move, as Elgin had already accumulated considerable debts borrowing large sums of money to pull down his father's house in Fife and building in its place a luxurious mansion, which he named Broomhall.
It was around this time that his architect, Thomas Harrison, gave him the idea of acquiring plaster replicas of ancient Greek architecture, to decorate his mansion. In his attempt to recruit the best artists, he turned to the talented painter JWM Turner but they could not reach an agreement over payment, especially as Elgin wanted exclusive rights to every drawing and painting of Turner.
Along with two secretaries, the Reverend Phillip Hunt, his wife and staff he departed for Constantinople. Once in Sicily he managed to recruit, at a better rate, the necessary workers and appointed the Italian painter Giovanni Battista Lusieri to head the mission.