Brother, from the jump this was just a set up.
I can't say I wouldn't have done the same as you but, I agree with your wife.
Think about this from a point of view watching from the porch. Given the history, looking back, do you think the guy was ever gonna say "yeah my bad, I'll put those back"? No. He wants to prove a point and isn't going to back down Eben if he's wrong. Playing chess with a pigeon man.
I think your best option is to have a law enforcement officer handle any situation going forward. You don't need to get yourself in a bind over an asshole neighbor. You get sent to the clink and all this arguing about property lines means dick cause then, he'll have free reign.
Alternative, drive railroad spikes through 2*4's and bury them a Foot inside your property line. Won't be no more galavanting around on atvs lol