iCloud and number switching on others phones

I have an iPhone 15 Pro, iOS 18.1.1. I’m going to do my best to explain the problem. I was texting from an iCloud account for awhile and then got a SIM card which gave me a number. However for some of my contacts it will seperate the two. Some people get texts from both my number and iCloud. In group chats there’s always two of my for everyone else one iCloud and one number. It doesn’t affect anything on my end but it confuses people sometimes and is quite annoying when there are group chats cause even removing one means that I can’t type from the other. I’ve gone into settings and tried changing the send and receive to only my number however then I won’t get texts from certain people. There’s no way of knowing who has my iCloud and who has my number. Is there anyway to fix this and combine both of them to make things easier without having to individually text everyone to tell them to combine the two. I don’t know if any of this makes sense but I’m lost and really need help.