Former Olympic champion Joshua had to square up to Miller during his meet with Povetkin after the American invaded the stage.
Miller has wanted to face Joshua – who fights Povetkin in his seventh world title defence at Wembley Stadium on September 22.
Miller is ranked at number three with the IBF and WBA and remains a viable opponent for Joshua.
Joshua told Sky Sports: “This is where I’m at right now, I can’t sit down for too long because there’s always people clucking around me feet. I have got to kick them back down.
“People talk about me and see me face-to-face and they still want to talk.
“I defend my titles in the ring time and time again. I just won’t tolerate it out of the ring either.
“Anyone is welcome to come and get the smoke as well. Firstly focus on Povetkin and you never know what the future holds.”
Joshua holds three of the four major world titles – the IBF, WBA and WBO – with Deontay Wilder’s WBC belt between him being undisputed heavyweight champion.
He fights Povetkin as a mandatory bout but Wilder is the opponent he wants.
He said: “It’s one thing me saying it and it’s another me doing it. I’ve secured the date, location, my side has been signed. All we need to do is wait for the ink to dry and the fight is locked in.
“He’s got up until now and January to sign the contract. Unfortunately there was a time scale and it didn’t happen so I have a mandatory fight. I need to secure my destiny of becoming undisputed heavyweight champion of the world.
“Povetkin is good, a solid fighter, an Olympic champion and I respect him. I respect every fighter. He is a banana skin.
“He has fought people like me all his life, he knows how to slip punches. It’s not the height difference which is the issue, it’s how determined he is and how willing he is.”