I've noticed that deciding whether to close a question on this site is a bit different from some other sites. I've become persuaded that it makes sense to be less stringent here than on some other sites, but I still feel hesitant to cast a vote unless it's incredibly obvious. Therefore, I think it would be helpful if the moderators shared their magic sauce for how they sift out the untenable questions. In addition, other experienced reviewers could share how they think about it. Someone might have ended up with a similar approach to how the moderators decide these things, but with perhaps a different way of thinking about it or explaining it.
The system, for better or worse, requires five users to agree to close a question unless they are either a diamond mod or have the gold tag badge for one of the tags attached to the question. So as long as the moderators hold off (nobody on this site has a gold tag badge) five people with different personal views have to agree. My conclusion from this is that an individual user should not worry too much about being correct but should just give an honest opinion. After all if it is a truly bizarre opinion nobody else will vote to close so it will have no effect.
The moderators have no "magic sauce". We might make "mistakes" like any other users (and I quote "mistakes" because there are some very blatant off-topic questions and others that fall in a grey area, and it is up to the community to decide how to hanlde certain situations. You may even need a Meta-discussion to decide as a community what to do with some questions that might stretch the limits of the on-topicness).
To add to the problem, mods' votes are binding, so we can close/reopen a question just with our vote (as opposed to community consensus). This doesn't make mods infallible, nevertheless. For that reason we mods have been trying to stay away from the close/reopen queues and maybe cast our vote when there are already 3 votes or if the need for the binding vote is quite blatant.
What I can advise you is not to be so afraid of casting the wrong vote (See mdewey's excellent answer:
an individual user should not worry too much about being correct but should just give an honest opinion). If the community does not agree with your vote they won't provide the other votes needed to close or reopen. Also, any closed questions can be reopened if needed, no big deal (except for the time needed to get the 5 votes needed, which should not be the end of the world).
If you think that a question falls in a "grey area" feel free to discuss on meta if it should be closed or reopened or if the definitions of on-topic/off-topic should be revisited.