Well, that depends on how you define hard. This article at 3 Quarks Daily talks about the fact that struggling with maths doesn't mean you're bad at it -- it means you're learning. I wholeheartedly agree with that sentiment and I wish more of my friends and classmates did too. For that matter, I wish more of my lecturers did!
The article goes on to say that anyone can appreciate the beauty of mathematics once they're past the initial hurdles. I agree more reservedly there. Perhaps everyone can appreciate maths, but that doesn't mean they will. I'm pretty sure I was the only kid in my class grinning like a fool when we proved the fundamental theorem of Calculus. Or when we started axiomatic vector analysis. Or when we did half a dozen other things that are pretty awesome, but under-appreciated.
Now I'll buy that some people can appreciate mathematics without grinning like fools; it's just not one of my particular talents. On the other hand, the fact that most of my friends gently but firmly point out that I must be crazy when I walk out of a maths lecture saying "That was awesome!"* is harder to overcome. There are a large number of people who pass courses called things like 'Advanced Calculus' quite well, but don't think that Calculus is awesome. And I don't just mean on the days when we have to integrate by parts three times, make an obscure substitution and remember the derivative of arctan(x) from eighteen months ago. Nobody I know or have heard of really likes Calculus on those days -- but hardly anybody seems to like it even on the days when we do epic proofs.
I could claim that I'm some kind of super genius wonder child, but did you notice how many times I used 'fool' up there? (There are more objective things that suggest otherwise too, but they'd be boring to go into.) I think it's far more likely to have to do with believing that maths is impossibly difficult -- or with something I haven't thought of†. And so we come back to what you mean by 'hard'.
If hard means that you have to put something in to get something out, then yes, maths is hard (but so is nearly everything worthwhile, with a few notable exceptions).
If hard means that there will be struggling and use of intellectual stamina, then yes, maths is hard (but so is reading a good book).
If hard means that most people haven't managed to understand it, then yes, maths is hard (but so is, say, giving directions to my house, which most people couldn't do, although I don't find it desperately taxing).
If hard means that you'll only succeed by luck and that actually enjoying that success comes at the expense of being at all 'normal', then saying that maths is hard is nonsense.
You cannot use the first statement to prove that maths is hard, and then proceed to use the last statement to say that therefore you simply can't do it. Well, apparently you can, because people do it all the time, but it oughtn't to be possible. Maths is hard, but so are myriad other things that we learn to do and subsequently enjoy very much. Maths is fun.
Yeah, maths is fun. That seems like a good place to end. I'm not sure if this is coherent; there are so many mathematical ideas fighting for precedence in my mind that I'm not sure any of them have escaped, but at least I've tried!
Maths is fun.
*Yes, yes, I really do walk out of maths lectures saying things like that. Maths is not as horrible as people seem to think. It's kind of like art, but with less emphasis on being able to draw.
†See above re:foolishness and extrapolate to propensity for not thinking of important things as appropriate.