English spelling is a crazy mess, but it's a mess that makes sense if you look at the history of how it got that way. Writing changes much more slowly than speech, so often our spelling reflects older pronunciations. Also, we've borrowed so much from other languages that their spelling habits have gotten mixed in with our own. Nobody meant for English spelling to become so complicated; it just turned out that way.
Except in some cases, where people did choose to make it harder than it needed to be. Rhyme came to English from French where it is spelled "rime." And that's how it was spelled in English at first too. But in the 16th and 17th centuries, when English spelling conventions were getting standardized by printers, fancy-pants writers started to spell "rime" as "rhythm" or "rythme" to show off that they knew "rime" was ultimately derived from Greek rhythmos through Latin rythmus. Other show-off spellings started around this time, including receipt (instead of receyt), indict (instead of indite), and many others.
At first, "rhythm" (pronounced "rime") meant rhythm or rime. The meaning of rhythmos in Ancient Greek was broader — it referred to the use of cadence, stress, pause, length, and repetition for melodious effect. In English, two forms started developing into two different meanings after 1600 when rhythm took on a different meaning from rhyme.
In an ironic twist, rime is now the fancier term. It's a technical term in linguistics for the part of a syllable after the onset. (The rime of "Greek" is "eek.") Rhyme and rhythm are simply Greekified English that we've all gotten used to. Thank (or curse) our meddling, snooty linguistic ancestors for that.