I've seen it happen before where someone has accidentally connected the cable to the pre-input terminal (circled in red) rather than the pre-output terminal (circled in green). So, just to confirm, you do have the cable connected to the pre-output terminal?
If 'yes', have you tried another cable just to make sure there's not a problem with the one you're using?
It does have RCA to RCA connection but has "digital" written on the connectors
That's meaningless advertising piffle. Wire is wire. Insert the cable into the sub amp, turn it on, put your finger on the center conductor at the other end of the cable. The sub should hum. If it doesn't the cable may be bad. If it does the cable is good and the problem lies either in the AVR or in how you have it hooked up.
It does have RCA to RCA connection but has "digital" written on the connectors, which made me think it was a coaxial digital cable.
It is, in the sense that it's not an optical digital cable. But coaxial digital is a cooler-sounding way of saying "wire with RCA connectors on both ends".
Might order another RCA to RCA to see if that might work?
Save some time and money by using that cable for a left or right analogue connection from a source (DVD or BD player) to the AVR. If you get audio through the channel that cable is on, then the cable's fine.