Lots has been written since I read this thread last. So here is my input.
I agree with MyBuddy that I would practice the: no talk, not touch, no eye contact with him. Just ignore him and let him get a sense of people. Don't force the interaction. He sounds like an insecure dog that's all and if you just let him come to you he will be fine. I would stop the barking and growling and maybe make him physically back off a little but that is all. Then just ignore until he is ready to come over in a relaxed and less unsure manner. If he shies away when someone wants to pet him he is not ready.
Also don't baby him when he is unsettled when you wife is gone. I would tell him to lay down, stop the unwanted behavior but don't pet him and tell him it's alright as what you are telling him with that is that it is alright for him to be unsettled.
When you work on the recall, do you give him a treat when he comes to you? I would recommend you do that so it's a good experience and he wants to come every time you call. Make sure it's something really yummy.
I agree that he is too skinny. I like my dogs lean but he is quite underweight. I would up his feed right away, and you can up it a good bit. Even if they go through a growth spurt they should not look like that. If they do grow more and burn more calories you obviously up their feed, so the previous owners excuse was baloney.
Also I was thinking about the energy you put into trying to get him to like you and make everything great. Don't try too hard. Again ignore him a bit and let him settle in his own time. If you put too much energy into it you may be pushing him away instead of making him interested. It's kind of like when you play with a puppy. You want to run away from the to make them follow not run after them. I would live my life the same as normal, correct him when needed and just pretend he has been part of the family always.
It does sound like you are making great progress but the above may help you along a bit.

Great job.