Better to say that editing is the "soul," rather than the "curse," of video. Editing is what makes all the difference between what people will watch or not. Only certain highly sequential, staged, or scheduled events will attract viewers with little or no editing. That might include a performance, ceremony, or public event of some sort. However, even those events need some editing to fit into a <60 second "spot" consistent with a program or viewer attention span.
Any humble photo will draw a polite 10 second glance and compliments, provided the viewer knows the people in the picture or knows the place where it was shot. Viewers will positively love the photo if it serves as a cue to talk for an hour about what they themselves know of the people or place portrayed. Meanwhile, the photo can stay quietly on the shelf or wall.
Video, on the other hand, demands viewer attention over a longer stretch of time. Anything longer than 30 seconds becomes a challenge to the eye, or become an outright sedative, unless:
1) the subject is very compelling (one's own kids, etc),
2) the (usually borrowed) music is popular and entertaining,
3) or the editing job is excellent.
A few camera junkies may stare at a screen simply to savor the video quality or editing skills. However, the vastly dominant quotient of any audience won't know or care about the camera performance or technique, but only whether they like the subject, action, or story. If you have the energy or art to edit something people will watch, perhaps you'll get some compliments or lots of YT "hits." Otherwise, video editing is a formidable chore, where it takes geometrically greater amounts of time to attain a slighly longer attention tolerance, but still leaves the result of the standards people take for granted, without any concept of the work that requires.
"Dad, can't we see the rest of this, ugh, later? I mean, how about we turn off this stuff so now watch some real movie or show?" [At least one viewer is candid. The rest are dozing or have gone off to get another beer.]
It is an irony that people will sit alert, an hour or longer, through a scoreless ball game, celebrity gossip show, or vapid sit-com or soap. However, those are things they can later chatter about with peers, and "stay in the know" about stuff considered newsworthy among friends. That factor, alas, sets any amateur, or even the most clever Indie, at a disadvantage. Wedding videographers may be the only exception, since at least the bride and her mom will watch the entire 1-hour video (edited from 5 hours of raw video, requiring 30+ hours to edit).
That said, I'll still say that editing is the soul or core of video. If you can love it, perhaps the way fly fishers can love tying or sailors can love boat maintenance, then it's worthwhile. Pro's who eck out a living may also make the claim, the same way a restaurant owner knows that cleanup and service (the editing) matters as much, or more, than the cuisine (the camera work).
But here are some real curses of video:
1) Camera shake: hard to avoid if you can't set up a tripod and orchestrate actors on a set. Most amateur video must be hand held, which requires an awful lot of attention to keeping the shots as steady as one can, or the result will appear downright clutsy. Even if you can afford a Merlin, you're not likely to have it on hand or set up when you need it.
2) Bad light or backlighting. John Ford used lamps to illuminate actors when shooting outdoor westerns. You probably don't have that option. The occasion will be a dark room with floor or table lamps that cause blowouts, but leave faces in the dark. Your "actors" aren't about to congregate where the light is best, nor will they stay natural if you shine an LED against their shy eyes or reflective glasses.
3) Native sound. The ambient audio may include sounds or voice you want, but also feature distortions or racket the human ear ignores but mics do not. It is also hard to edit the video and retain continuity of the audio. Essentially, you must recreate the audio in the fashion of old-times Italian flicks. Even contemporary movies, with all the set control in the world, also rely on elaborate sound mixing and additives.
4) Camera phobia. Most human subjects will halt and pose for some still shots. It is a completely different story, when it comes to get people to face a camera or behave naturally for an extended period. Polished conversationalists or extemporizers have a way of turning into stutterers or mutes if they see a videocam pointed at them. This is why a small, inocuous P&S can sometimes capture better social or public video than a big pro device, unless people have a major publicity or financial reason to stand in front of that big camera and "perform."
These are problems that the still photographer can avoid or deftly work around, one way or another. For the videographer, on the other hand, they are simply facts of life, and simply add to the pleasure / anxiety of the venture.
"Yeah, Dad. Real nice. But, like I said, isn't it time to shut that stuff off, so we can watch a real movie? There's a playoff game coming up soon, too. Don't want to miss that. The team has been losing, but we gotta see it the coach puts the injured and suspended so-and-so back on the field."