Sean Rigg put the U’s in front with his first goal for the club, but a Michael O’Connor brace and a late strike from Jason Taylor gave Rotherham United a 3-1 win at the New York Stadium.
The six-game losing streak is now the club’s worst in a single league season for 25 years and increased the pressure on the under-fire Wilder and his players.
But Brown, who was handed his first start of the season on Saturday, stressed the squad remain loyal as they attempt to regroup for tomorrow’s home game against AFC Wimbledon.
He said: “Everybody’s playing for Chris, he’s hurting more than anybody.
“It becomes really difficult for ourselves now we’ve got to play on Tuesday.
“We’ve got to go out there and do it right, it’s massive and is a must-win game.”
He added: “It’s got to change, otherwise you don’t know what’s going to happen.
“I know there are people outside the club who are trying to put a divide between us, but we’re one unit.
“It’s not for the want of trying. The boys are working so hard and have been different class in training, but it’s just not happening at the moment.”
Brown was surprisingly included in the starting line-up as Ryan Clarke was dropped to the bench for the first time this season.
It was a gamble, with the 35-year-old goalkeeper’s last appearance of any kind coming in pre-season.
Brown was unhappy with his own performance, but admitted it was not easy to come straight into the side.
He said: “I was a bit edgy, it probably showed in the performance actually.
“I’m not particularly happy with it, I felt like I let the team down as well.
“It is tough, but you get on with it and try to be professional. I tried to use a bit of experience, but sometimes you need a bit of game time to help you out.
“I made a couple of saves, my kicking was awful and there were lots of things I should have done better.”
The defeat leaves Oxford 17th in the table, two points clear of tomorrow’s opponents Wimbledon, who lie in 22nd place.