When is the appeal to authority a fallacy?

Appeal to authority is a fallacy when those who use it do not provide any justification to support their argument. Instead they cite someone famous who agrees with their viewpoint, but is not qualified to make reliable claims on the subject.