Got it done with your advice and
-flycutter in drill press
-workpiece securely attached to a large plank
-placed a 3/16" steel plate between them with a 1/" hole to guide the drill.
-the whole works then attached to the drillpress platform
NO shaking, drifting of part.
-Took the unfluted part of a dull/scrap 1/4" drill bit and used it where the drill bit was in the flycutter.
-predrilled the center of each hole then lined it up with holes in the steel plate and wood plank.
-freshened flycutter cutting bit (as depicted in previous post)
-minimized the amount that the cutting tool stuck out so as to reduce chance of it shifting in the flycutter
-really cranked on the flycutter allen screws and the drill press chuck
-lots of WD40 as I cut.
-patience, don't rush the cut
-started on one side, flipped and finished on the other for one but not the other holes.
-the piece was a bit unlevel, it was cutting on one side mostly so I shimmed it with thin cardboard scraps.