Someone has typed in your N-number (accidentally or intentionally) when configuring their ADS-B out device, and that’s what FlightAware is showing. You dont have to contact ATC for flight following or be talking to anyone at all for FLightAware to show your ADS-B track (note: it doesn’t work all the time...some flights never get shown...real reliable, huh?).
What can you do about it? I have no idea, but maybe contact your FSDO and see if they have an idea, just to be on the record as having notified them that someone else is using your ID. They may have no clue what you’re talking about however - some of them are still trying to figure out transponders, much less ADS-B....