Why drywall at all?

Code issues notwithstanding - the drywall is central to soundproofing. Without it, you'll be listening to the rest of the house and neighborhood instead of your system. That's enough of a reason for me.

Further, I imagine that you'd end up with better ambiance and envelopment by having a consistent barrier of fixed distance and quality.

Fred

Resale value of the house is one good reason. Don't leave yourself a list of things that you think you will get around to changing when the time comes. when the time comes you never have the time.

Lack of sound containment would be the 2nd reason not to skip the drywall. Your family's safety would be the first. Drywall is a necessary component of your wall assembly in order to achieve a specific fire rating. 1/2" drywall has a 30 minute rating and is a critical piece in a) keeping fire from spreading from one side of the wall to the other and b) reducing the amount of O2 in your wall cavity should a fire start within your wall. Fabric alone won't do either of those things.

You could however do essentially just what you proposed with the addition of a second drywalled structure several feet or more beyond your fabric walls. This is precisely what is used in some professional control room or mastering type environments. The inner shell deals with speculation energy and reflections, the outer shell deals with modal energy and behavior.