There are many legends surrounding Hermitage. One is that an early owner of the first castle on the site, Sir William de Soulis, was in league with the devil. He had an arrangement with the devil that he could not be killed by iron or rope. The borderers were nothing if not ingenious though – the legend is that they boiled him alive!
Another, in this case almost certainly true story, relates to another owner, Sir William Douglas. Sir William committed a particularly evil act in the castle. Jealous of Sir Alexander Ramsay, who had been appointed by the King as Sheriff of Teviotdale, he kidnapped Sir Alexander, imprisoned him in Hermitage and starved him to death. Sir William was not a man to fall out with. When Sir Willaim died, the castle passed to his son James Douglas, the hero of the battle of Otterburn, and then to George Douglas, the illegitimate son of Sir William. George became the first Earl of Angus and founder of the Red Douglas line, so called because of his red hair.
However by the 1490s Archibald Douglas, the then owner of Hermitage was getting too close to the English for the liking of the Scottish king, James IV. Hermitage was near the border and protecting an important invasion route from England. Therefore James IV required Archibald to exchange Hermitage for Bothwell Castle, then controlled by the Earl of Bothwell. Bothwell Castle was less strategic as it was in Lanarkshire, much further from the border.
The Bothwell Earl’s control of Hermitage resulted in one of Hermitage’s most famous incidents. In 1566 James Hepburn, the fourth Earl, was injured in a skirmish with Little Jock Elliot of Park, a reiver, and was taken to his castle of Hermitage. Mary Queen of Scots, who had been linked romantically with Boswell, was in Jedburgh about 25 miles away, on a royal tour of the Borders. When Mary heard of Boswell’s injury, she immediately made the 25 mile journey with a small party to Hermitage to see him. After two hours with Bothwell, she rode back the 25 miles to Jedburgh. She may have had pressing business in Jedburgh, or considered it inappropriate to spend the night in the castle. These would have been very difficult and exposed journeys across bleak moorland in October. On the return journey her horse threw her at one point. When back in Jedburgh she contracted a fever that nearly killed her. Much later, whilst the long term prisoner of her cousin, Elizabeth 1 of England, Mary is reported to have wished she had died of the fever at Jedburgh.