Brown never complained. Instead, he just kept working, knowing his chance would come.
"He is the ultimate team guy," NFL sacks leader Robert Mathis said. "He doesn't like to talk about himself because he's not his favorite subject. He just goes to work and he just lets his play do all the talking."
Brown prefers it that way.
But this was not the plan Indy had when it took the UConn star with the 27th overall pick in 2009. Back then, the Colts envisioned Brown becoming the workhorse back and the eventual replacement for Joseph Addai.
Though he never quite lived up to the billing, he refused to give up and became one of the most respected guys in the locker room.
"He's a great pro. He's reliable. He's accountable," Pagano said.
It's been tough.
Brown started only nine times in his first two seasons, running for just 778 combined yards and five touchdowns. Even in 2011, with Peyton Manning out and the Colts' offense sputtering, Brown was one of the lone bright spots averaging 4.8 yards on 134 carries. Yet he still found himself playing behind Addai and making only two starts.
It looked like Brown might get another chance when Indy cleaned house last season and kept Brown around.
Instead, after taking Luck's first NFL pass for a long touchdown in the 2012 preseason opener, Brown lost the starting job to Ballard when he got hurt and never won it back. Brown finished the season with 57 carries for 324 yards and three TDs.
Critics have long complained Brown can't pass block, pointing to a blown pickup on fourth down late in a loss to Miami in Week 2 as one illustration, and that he didn't hit holes hard enough. Today, those same people champion Brown's patience to set up blocks, something they don't believe Richardson has done well enough. Richardson is averaging 2.8 yards per carry since coming over from Cleveland in a September trade, while Brown has thrived in mostly limited action.