Long text, but please read. Need advice on cat after being at a "cat hotel".

I went on vacation between Thursday-Monday, because of that I dropped my cat off at a boarding facility from Wed-Mon. It was a really nice one, a "cat hotel"with tons of cat trees and toys, both inside and in cattios they could access outside.

Problem is, my cat is not used with other people, except for me and my partner, and because of that he's REAL shy. As well, he's an only pet (he's turning 1 year old in half a month) so he isn't used with other animals either. He's quite timid in nature.
I was really nervous placing him there because of that, but we told the lady about it and she said it should be fine, and well, when we picked him up we found out he got along well with other cats but he spent 98% of the time just being terrified. He just hid most of the time.
He already feels thinner, even though she said he ate and drank well. Within 40 minutes of being home he ate a little food, and drank almost 1.5 liter of water, but since then (20ish hrs later) he's done nothing. He doesn't want to drink or eat, neither kibbles or wet food, nor snacks.
He doesn't use the litterbox (I got him to pee once like 6 hrs ago, but it was very little), and he doesn't want to play. Just sleep and cuddle. I've read it can take cats some time to get back to their old routine, but I am just wondering at what point I should... well, start worrying for real? When should I call the vet? Should I give him 2-3 days and see if he's gotten better or should I just call right now? I don't want him to get dehydrated or starve or anything.
Sorry for long text, I am just worried about him, since I think this was way worse for him than I could have imagined. :( Thanks for all the advice.