Making an actual driver absolutely from scratch that produces sound can be done. It won't sound good. It won't sound much like music. More like buzzing. It isn't just about winding a good coil and finding the right magnet (though that's important). The shape of the flux in relation to the coil, shorting rings etc, shape and stiffness of the suspension, surround, cone, dustcap, even details like how the leads are woven (or not) are important. Some companies invest significant cash into R&D over years just to make crappy sounding products (Pyle). Even crappy sounding drivers are difficult to design.
You could assemble drivers from some combination of existing motors and soft parts. A frankendriver of sorts. This has a much higher liklihood of at least producing crappy sounding music, not just crappy sounding indeterminate buzzing.
If you want to do either of those as a hobby then by all means, let us know how it turns out.
If you want to make something likely to be useful for parties or concert type applications, then let your design center around choosing the right combination of drivers and designing appropriate crossovers and cabinets. Many hobbyists spend a lifetime getting good at just that that part, and that alone is a surprisingly deep and complicated undertaking. But rewarding.