In "The Cask of Amontillado," what is the significance of Fortunato belonging to the order of Masons? How does Fortunato determine that Montresor...

At this point in the story, Montresor has lured Fortunato into the catacombs below his family residence, and intends to do him some type of harm (as yet undisclosed) in retaliation for Fortunato's "many insults." Montresor is also attempting to keep Fortunato drunk in order to dull his wits. 

Fortunato gives an indecipherable sign with his hands, a form of signaling that would be recognized had Montresor been a Mason, and taught how to recognize...

At this point in the story, Montresor has lured Fortunato into the catacombs below his family residence, and intends to do him some type of harm (as yet undisclosed) in retaliation for Fortunato's "many insults." Montresor is also attempting to keep Fortunato drunk in order to dull his wits. 

Fortunato gives an indecipherable sign with his hands, a form of signaling that would be recognized had Montresor been a Mason, and taught how to recognize it. Montresor's lack of response, or of recognition, is how Fortunato determines that Montresor is not a member. 

The Masons are a fraternal order, something like a club or social group, with a series of traditions and practices that distinguish them, such as giving signals and handshakes which would only be recognized by other Masons. One of these traditions is the requirement that Masons may not lie to each other. If one is versed in Masonic culture, they may begin to understand how endangered Fortunato is at this point.

We might also question why Fortunato gave the sign in the first place - perhaps he was suspicious of Montresor's intentions, and was subtly asking "are you lying to me?" - however, we would presume that Fortunato would address this directly when Montresor fails the test. Instead, it may have simply been a mark of Fortunato's ever-present hubris, and he was simply making a show of his status.