The Bundesliga champion had Jerôme Boateng sent off nine minutes into the clash at Commerzbank-Arena and was made to pay as it suffered its biggest league defeat since 2008-09, AS reported.
In-form striker Robert Lewandowski was on target for Bayern after Filip Kostic and Djibril Sow had put Frankfurt two goals ahead, but David Abraham, Martin Hinteregger and Gonçalo Paciência struck in the second half to pile the pressure on Kovac.
Bayern has now dropped points in half of its 10 league matches this term and is fourth in the table, four points adrift of surprise leader Borussia Monchengladbach.
However, Kovac is standing firm amid speculation that Bayern chiefs are preparing to pull the trigger ahead of next weekend's Klassiker showdown with Borussia Dortmund.
"I did not give up last season and I will not give up now," he said at his post-match news conference.
"When it comes to such situations, you must not get restless.
"I leave here sad and disappointed. I'm not starry-eyed – I know the business. But I will not give up."
Further pressed on his future at the Allianz Arena, Kovac said, "How should I know? You probably know better. I cannot say anything about that.
"My feelings are not important. Those who decide must be asked."
Bayern has conceded 16 goals in its opening 10 league matches, the most it has let in since 2008-09, and Kovac admitted his side struggled after Boateng's early dismissal.
The experienced center-back was shown the seventh red card of his Bundesliga career for a last-man tackle on Paciência inside the opening nine minutes.
"We got a red card and if you play with ten players for 80 minutes, it will be difficult in Frankfurt," he said.
"That did not make it easy for us. But you cannot lose even with a man less. It is disappointing and annoying.
"It cannot be used as an excuse. We still did not play like we should in the first half."
Victory for Frankfurt was its first over Bayern in 17 attempts and lifted it to within a point of their opponents.
"This was not to be expected," Frankfurt boss Adi Hutter told reporters. "I didn't have a good record against them, but we caught them at a good time.
"We have a great collective and our players took their chances. We are very happy today."