As a guy who has designed a lot of circuitry, waded around in waveguides, magetrons and other esoteric hardware for a long time, I think I've got the subject down pretty well. It has been my experience that vswr is rarely related to "bad transmitter audio", could be, of course. But it would be a rare instance.
Allow me to comment as designer of a number of aviation band radios and other RF products.
AM transmitters can be severely affected by bad VSWR. Reflected energy can be significant and interfere directly with the modulation circuit using a number of different pathways depending on phase relationships and modulation schemes. The most common effect however is creating a audio feedback path via grounds which can include your audio wiring. This can lead to severe distortion, howling or in mild cases something that sounds echoey like "bathroom sound". Bad VSWR can, via grounds, cause trouble completely outside of the radio itself - via the audio paths.
Bad VSWR can actually destroy your transmitter - that's how bad it can get. Some radios have built in protection for this and can reduce or disable the transmitter if this is detected (this is not a 100% solution due to lag but much better than nothing).
A more likely cause, and one I ran into often, was off frequency receivers and/or transmitters. Channels are very tight now-a-days. A particularly vexing combination to troubleshoot is a tx leaning one way and the rx the other way. Both stations work ok except the combination, and often only in one direction because one unit is more bandwidth tolerant. This result is consistent with the OP's description.
To the OP: If this get to be more than that single instance, I would check the transmitter frequency. All later aircraft radios use synthesizers. So, freq tolerance checks are simpler. Checking one freq checks them all.
This is highly unlikely in my own experience - older radios with 25Khz channel spacing tend to be around +/-20ppm accurate while newer radios with 8.33Khz spacing have to be better than +/-5ppm. In reality - most will be around +/-1ppm. It is no longer much of an issue to produce a radio that will perform to what once was expensive laboratory standards. Temperature compensated active crystal oscillators are now the norm and low cost - thanks in no small part to mobile phones.
I have never had a (older) radio on the bench that deviated more than 8ppm or so after decades of use - and that's pretty decent going.
AM receivers are also quite tolerant of deviation (both side bands contain all of the audio information) - you have about +/-12Khz on each side of center frequency available with 25Khz spacing where the receiver will work just fine. So the combined, worst case tolerance to make it work is about 50ppm !!! (100ppm per side, assuming the other is spot on).
If your TX audio is bad, there are four major causes: The first, as mentioned is VSWR or simply bad audio wiring or antenna much too close to headsets etc.
The second affects radios that use "drain" or "collector" modulation using a relatively simple modulator connected directly to the power supply. This type of modulator transposes whatever rubbish you may have on your radios power supply directly onto the transmitted signal - and usually wide band as well (so that splatters into adjoining channels). While a fail in FCC terms, this is not revealed by FCC test methods as a clean power supply is always used. Many older radios do it this way.
The third cause is a damaged transmitter - usually the final output stage transistor - power check tells the story here.
The fourth cause is low supply voltage during TX - usually caused by a bad contact somewhere (corrosion), unsuitable wiring, tired battery etc. In these cases RX is usually just fine (very low power draw) but the moment you press that PTT a few amps are needed to keep the transmitter happy. If the supply cannot provide that the voltage drops, possibly below minimum the transmitter needs to operate properly.
No matter what radio you may have - if your transmit is iffy, your first step is to measure TX Power, VSWR (or SWR) or use a dedicated antenna tester.
Rainier
CEO MGL Avionics