Sounds like you have a scaredy cat chinchilla.
They are all so very different, and they are creatures of habit.
If he was always like that then it could be his personality.
Or yes something in his past made him so skittish.
You've had him long enough now that I would be handling some every day. Start slow, and try not to spook him.
Make him come to hand for his treat. Even break it up so small or use something like oatmeal, so he has to eat it out of your hand.
Or don't give him a treat unless he's in your arms.
Sprinkle dust on your hand and see if he'll warm up to it that way.
Hold him some every day and make it rewarding for him to let you.
Other than that get more chins so you see the different personalities, if you have the time money & space.