Should I let my 11 yr old quit baseball?

My 11 yo has played baseball a few times but he’s not on the same skill level as his other teammates. Over 50% of his team is also on travel ball so they’re just better because they have more experience/practice. Also my son is on the smaller side so majority of the kids have a good foot and 50 pounds on him. I quite literally am scared he will get hurt. I’m deeply disappointed little league is no longer what it used to be about (having fun and learning) it’s all about being the best now. I just feel like this is way out of his league and I’m not sure what to do. I’ve always told him once he starts a sport he can’t quit mid season but I’m highly considering it now. Thoughts/advice?