I was in a book club for years. Like you I often felt a bit out of place for similar reasons. When the pandemic hit, I made a break from them. I think it’s totally okay to leave and find people who are your people.