Least of all has 2 different meanings across 1 category:
Adverb
especially not
"nobody, least of all Joe, agreed with me"
to a lesser extent than anything else