Anyone else having issues with their ghoul builds?

My ghoul builds SPECIALs keep changing and several of my PERK cards unequip randomly. It's usually right after I fast travel or loot something. Is it just me?