Get arfff moy lannnd!

Finally I have met someone on my site that I do not like. I lurk on a few facebook groups and I've read some sorry tales of allotment neighbours; rudeness, theft and vandalism, snitchy jobsworthiness and honestly, mine isn't that bad. Those stories make my plot neighbour seem fine, actually. But he isn't fine. He's a...boor.
Imagine the scene if you will. On return trip 4 of 13 to the water tap because you haven't sorted out your water butt and hose, and there is a man standing on your plot. He makes you jump actually, because you do not expect, I think, a man to be standing on your plot when one was not there two minutes ago. He introduces himself as one of the previous tenants (3 years ago) and explains that he had to give the plot up due to having another child, but that he still keeps the plot opposite. You nod politely and explain you've only come to water your plants very quickly because you are busy. 
He takes a phone call on your only direct path off the plot to the water tap. You have to say 'excuse me' to get past on YOUR OWN PLOT.
A few more trips to the tap and he's still talking on the phone whilst standing on your plot. Eventually he finishes. This is how the conversation goes;
'So this used to be our plot,'
'Yes, I think you might have said...'
'I left a bit of stuff on your plot.'
I do not say, 'You left it three years,'
'Oh?' I say, 'Well...'
'Yeah, two tonnes of topsoil,'
'Yes, it came in very handy, thank you,'
'And loads of really good compost,'
'Um?'
'You know, in the bins,'
I begin to water my runner beans at the back of my plot. He follows me.
'There wasn't really, I mean...'
'I was hoping to get some of my stuff back before someone took on the plot,'
Three years!
'I was hoping to have the bins back, but I guess you're using them,'
I do not say 'You also left a lot of shit for me to clear up and dispose of, but I don't expect you want that back, do you?'
Idiotically, I point to one of the bins at the end.. 'well, I'm not using...'
'Do you see that bed at the end, I triple dug that,'
I do not say 'triple dug, what the fuck are you talking about?'
'Oh yes,' I say, 'great. Thank you,' (Thank you!)
'Yeah, the soil is really good, I triple dug it...'
'Right, well, it was covered with couch grass, but yeah..'
'I was planning to plant sweetcorn in it.'
'Ok, well, I'm putting sweetcorn in a different bed, I thought I'd try tomatoes there,'
AND HE SAYS *chuckle* 'Good luck with that!'
He strides then, uninvited, to the back of my plot, picks up the compost bin and leaves. I search 'electric fencing' on my phone.

I can't decide if he was being ignorant of my personal space, or passive aggressive. It doesn't matter, the effect was the same. The only trouble is that if he does it again, uninvited, I will have to tell him I don't want him to come onto my plot. Which puts me in the role of being pissy, unfriendly tenant (I am, granted) and him in the role of being friendly neighbour, victim of shrew. Hm.