the reason see and avoid works as well as it does, judging by how few mid air collisions we see, is that there are at least two sets of eyeballs looking. the thing I always try to keep in mind is even if the other guy is technically not (following the right pattern, right of way, or other rule or suggestion) it is still my responsibility not to hit him just as much as its his responsibility not to hit me.
In the OPs case, you saw him the entire way around the pattern, you knew where he was and you avoided him; the system worked. Was he wrong, technically yes as you had right of way as aircraft on final. However, he thought he was doing everything correctly too; possibly even looking down final and not seeing you. Things I personally do differently, that may have prevented this "close call":
1) I respond to every CTAF call that isn't obviously a response to my standard call, in other words I would have quickly responded when he said he was backtaxing and/or departing "RV half mile final Runway XX at Somewheresville". I do this even if I have been making calls around the pattern, as this shows it's not a given the other guy is on frequency unless he responds directly to you.
2) If there is an aircraft waiting at the hold short/run up area, I always land past the numbers where he would pull out. If he does decide to pull out when I'm flaring I should land safely past him and hopefully he can see the airplane that is on the runway in front of him.
FLYING!!! as of 7/22/10