I'm gonna take a stab at this, because I had similar concerns when I first started on the tail of my RV-8.
I tried riveting the skin on my horizontal stab and wasn't happy at all with the way the flush skin rivets sat. None of them were perfectly flush.
I found out that I was under dimpling. I read something that referred to the dimple dies as "spring-back" dimple dies. This got me to thinking about bottoming out the die set in the squeezer and then giving that extra little push. And, sure enough, there was a bit of spring back when you let off. It leaves a perfectly symmetrical little halo on the dimpled skin, around the rivet hole when done properly. Its especially noticeable when using the DRDT-2 that is bolted down to a sturdy surface.
Try this and see if it doesn't cure your problem.
Mount Vernon, WA
2018 VAF Supporter