Any tips for driving out a blind rivet shaft?

So I'm in the process of deconstructing my horizontal stab after a botched SB attempt. Tonight I was working on removing the rear spar. Got the skin rivets and solid skeleton rivets drilled out and removed. For the four BSPQ-5-4 rivets attaching the rear spar to the main ribs, I drove out the mandrels and drilled far enough to take off the rivet heads. However, the spar still isn't coming off, thanks to those rivet shaft apparently having a good solid grip on the spar bars.

Clearly the shafts need to be driven out to release the spar here, but I'm trying to figure out the best way to do this without potentially damaging the hole. Since the rivet shaft is just a tube, the only option here seems to be putting a small punch against the edge of the hole in the spar bar and punching the shaft out that way - but this seems like a really good way to bugger up the hole.

Is there some magic trick I can use here, or is it just a "be careful" type of situation?

Dont drive the mandrel all the way out when starting, just a bit to get the drill going.

After the head is off, you can usually drive out the rest of the rivet tail using the mandrel.

Q rivets grip the end of the mandrel pretty good.

Turns out there was a stupid simple solution to this problem. I didn't have a #21 size punch on hand, but I did have... a drill bit. And that worked perfectly well for driving the things out. Rear spar is now removed.