"Gh" is a strange way for an English word to start. There are only a handful of commonly used words that begin with this spelling. Beyond the spirit cluster of ghost, ghastly and ghoul, we have borrowed words like ghetto, gherkin, and ghee, some place names like Ghana and Ghent, and that's about it.
"Ghoul" was borrowed into English in the 1700s from the Arabic ghul, but at first without the "h," as "goul" or "goule." It was later lured over to the "gh" group by its semantic similarity to "ghost."
But how did ghost get its "gh"? Compared to the other "gh" words, ghost is both a lot more common and a lot older, going all the way back to Old English gást. Until the 1500s, over a few centuries of language change, it was spelled gast, gæst, gost, goste, goost, and goist. "Ghastly," from the related, Middle English gastliche, also came in "h"-free spellings until the 1500s.
We can trace the introduction of the "h" in ghost and ghastly back to William Caxton, the man who brought the printing press to England. He had established his first press in Bruges, and he brought some Flemish typesetters back with him when he returned to set up business in Westminster. David Crystal writes, in his history of English spelling, that "in Bruges they would have been used to reading manuscripts in Flemish spelling. So if a word reminded them of its Flemish counterpart, why not spell it that way? The boss wouldn't mind, as long as the words were intelligible. He had more to worry about than spelling."
The typesetters also used "gh" in their spellings of goose, goat, and girl, but those spellings never caught on. For some reason, only ghost and ghastly kept the "h." Maybe because the words looked spookier that way. Indeed, a story about the ghost of a "ghoos ghoot gherle" sounds downright terrifying. Thank ghoodness those spellings have ghone.