What is the difference between Hamlet and Macbeth? Why is it bad luck to say "Hamlet" in the theatre?

First, there is no history of saying "Hamlet" in the theatre -- you are thinking of the traditional taboo about saying the name of the play Macbeth backstage in the theatre -- one should say "the Scottish play" instead of saying "Macbeth."  That taboo came from some Victorian theatre event where some sort of disaster befell a cast member in a Macbeth production.

Now to your real question:  While both these plays can...

First, there is no history of saying "Hamlet" in the theatre -- you are thinking of the traditional taboo about saying the name of the play Macbeth backstage in the theatre -- one should say "the Scottish play" instead of saying "Macbeth."  That taboo came from some Victorian theatre event where some sort of disaster befell a cast member in a Macbeth production.

Now to your real question:  While both these plays can be seen as Elizabethan tragedies, they differ in three important respects:  First, Hamlet is Shakespeare's greatest play (some say King Lear is a rival), while Macbeth is good but has not received the same level of critical examination as Hamlet.  Secondly, the characters differ in their intelligence, complexity, nobility, and linguistic grace.  Finally the two plays treat other-worldliness in completely different ways.  Hamlet examines the value of "ocular proof," of the validity of Christian beliefs in the after-life, ghosts, Lucifer, etc.  While Macbeth begins with pagan folk-rituals and witch figures that predate Christianity's appearance in England and Europe in general.  It is the difference between believing in angels and saints, or in the Tooth Fairy, Santa Claus, or the Easter Bunny.