Logistics of leaving primary care job?

Im thinking of leaving my current job where I’ve been for two years fresh out of residency (Ill also be taking a mat leave in a bit)- I’m just super anxious about the patients I’ll be leaving behind and the organization- I’m scared of being “black listed” (what if I want to come back) and worried about leaving all these patients who I inherited from two physicians who also left.

How common is this? Are my anxieties unfounded?