Sure, giants exist. Haven't you heard about Andre the Giant? We're not just being clever, here. Per Health Direct, rare conditions like gigantism and acromegaly (which also affects their body's proportions) can cause people to grow well past the norm, to a point that an average person might well consider them to be giants. This is often caused by a tumor in the brain's growth hormone-producing pituitary gland, or a rare genetic condition.
Gigantism can be treated with drugs, surgery and radiotherapy, and when it goes unchecked, you pretty much grow until you stop, on account of dying before your time. Case in point: According to Guinness World Records, the tallest man in the world was a fellow called Robert Wadlow. His condition went untreated, and as a result, he was a whopping 8'11.1" tall before his untimely death at 22.
So, yeah. Life with gigantism might not be particularly nice, and presumably, it wasn't any nicer back in the days when there was no medical understanding of the condition. Imagine what a man like Wadlow would have looked like to a random peasant back in the 14th century. Imagine a 9th-century monk coming across Andre the Giant in the woods. Wouldn't they spin a tall tale or two out of such an encounter?