The dipole moment depends on both the electronegativity difference and the geometry. Nitrogen is the most electronegative element in its group, and the bond angle in $NH_3$ is also favorable, resulting in ammonia having the highest dipole moment among these hydrides.