you'll be really lucky if he stays away----Hmmm...it could go either way. He could know of your DH, like the way I knew of my new neighbor's ex husband because his father used to be my other neighbor's plumber (seriously).
I wouldn't know him to stop him on the street and have a conversation but I would "know" him enough to pick his house instead of the house beside him if my truck broke down in the area.
I hope he pays you back. If not, he may have really really needed the money. Your DH is a good guy.
P.S. If I was home alone, I wouldn't have opened the door to him, either, although I would have offered to call someone for him.