Wow. Talk about apples and oranges. This thread is about whether an approved ADS-B out system can be removed from an aircraft to become non-compliant, yet still airworthy and flyable outside of rule airspace. That is a completely different topic than the exemptions allowed for non engine-driven electrical aircraft.
So, for the question at hand about removing ADS-B. Yes you can. Of course then you cannot legally fly in Rule airspace without a temporary one-time ADAPT approval from Washington and carrying a print out of that approval onboard the airplane (online application and approval). Allegedly the ADAPT process removes ATC from making the call to allow you to deviate although many controllers will still accommodate.
So, for the issue about non engine-driven electrical system which the OP did not ask about but someone else mixed into the fray: As long as an airframe was originally certified with a non engine-driven electrical system, and has never subsequently been modified to add a engine-driven electrical system during it's lifetime, then the ADS-B exemption (Mode C veil) applies to these airframes. The Mode C veil is effectively history due to the ADS-B out rules incorporating a parallel exemption. So if you have a 1946 Luscombe originally delivered with a Continental A65-8 and no engine-driven electrical system and it has maintained the non engine-driven electrical system status uninterrupted for it's life it is exempt from the equipage requirement under Charlie and the Veil and hence under Bravo. However, if that same aircraft was modified in the early 1960's with a Continental C85-12 with an generator and then a subsequent owner in the 1980's decided to a complete restoration re-installing an A65-8 without engine-driven electrical system so the airframe is exactly as delivered by the factory in 1946 the airplane is still no longer exempt due to it's history of having flown with an engine-driven electrical system some time in it's previous life. So this might be where some here are confusing the non-electrical airframe exemption being a one way street might somehow bleed over to an equivalent interpretation of rule-compliant ADS-B equipment. It does not.
So the non engine-driven ADS-B exemption still won't allow a non-electrical aircraft to operate inside or out of a Mode C airport.
But then again why would it even come up with respect to our Van's aircraft? I might guess there may be one or two RVs out there that were certified without having an alternator or generator mounted on the engine and and maintained that status since new. But I doubt it.