It's more afraid of you than you are of it

They always used to tell me this when I was a kid. But they aren't more afraid of me than I am of them. Because they come into my house! I wouldn't go into their house!