I know I'll probably get a lot of responses along the lines of, "it'll be better if you start from scratch",
Only because that's usually what the original poster ends up concluding.
The thing I'm worried about is the filters; I dont know how to get around them.
You don't, not without a schematic and good knowledge of electronics. Since you're asking if the filter is on the amp, I'd use a different amp.
The passive HP filter (speaker-level) is not what you'd worry about, anyway, it's all the eq/filtering built in to get a desired response curve from, and to protect, that specific driver with that specific amp/box. Merely replacing the driver with a "better" one (based on specs) would most likely result in worse performance.
All mass-loading really does is roll off the high end...better off using eq to boost.