For the third consecutive season, Kenny Miller struck a double in the first Old Firm fixture of the campaign to win the match for the Light Blues.
Gary Hooper had put the home team in front seconds before the half time whistle but a Glenn Loovens own goal made it 1-1 just three minutes after the restart.
Miller then took over with two excellent finishes – including a clinching penalty – to give Walter Smith’s men their best start to a national league campaign since 1921.
The victory was remarkable in that it was the first time a team had gone behind first then won in the Glasgow derby in 27 meetings.
It was also only the fourth time in 52 fixtures this century it has happened – and each time the victors have been
The Ibrox team is now riding solo at the top of the SPL table, three points clear of the Parkhead side, after nine successive wins.
Manager Smith made two changes to the side which played so well in the 1-1 Champions League draw with Valencia at Ibrox.
As he shuffled his pack to accommodate a 4-5-1 formation, both Richard Foster and Vladimir Weiss dropped to the bench.
In came the fit again Lee McCulloch and Kyle Lafferty, with the Ulsterman handed a role on the left side of midfield.
This was Smith’s 50th Old Firm game in charge of Rangers and he knew getting a win would take him beyond the tally of victories recorded in the fixture by Scot Symon.
Although he thought he had seen it all by now, even he won’t have imagined the 90 minutes would pan out the way they did.
Smith’s biggest concern in the opening stages was an injury suffered by Sasa Papac after less than a minute when he was hurt in a challenge by Anthony Stokes.
The home striker was extremely late as he slid in on the Bosnian down the wing and after the free kick was given, the striker was the first man yellow carded.
When it seemed Papac might have to be replaced, he was able to come back on and take up his place on the left side of defence following treatment.
In a lively start, Steven Whittaker smacked a post as Gers got men forward in numbers with a shot from 16 yards.
Then with only three minutes on the clock, McCulloch was next in the book for sliding in when it seemed as though he had connected with the ball.
The visitors certainly made the better start and although Celtic tried to come into the game, their momentum was nipped when they lost their skipper Shaun Maloney to injury.
Efran Juarez replaced the winger and he gave the hosts a more solid look without making a great impression initially.
Indeed, the game itself lacked rhythm but it was Rangers who appeared more composed and measured in their approach.
That said, a good opportunity was wasted when Steven Naismith was fouled and Papac tried to swing the free kick into the box but only succeeded in sending it out of play.
The SPL champions began to settle and looked the more likely side to do anything when they came forward, with Steven Davis becoming more involved.
Juarez tried his luck with a half volley midway through the first half and although it went well wide, it counted as Celtic’ best chance yet in a match of few openings.
At the other end, Lafferty tried to pick out Naismith with a deep cross but he was crowded out by defenders and his header was off target.
After such a frenetic first few moments, the game was starting to turn into one of the more tame Glasgow derbies.
The home side was starting to find a little more room up front but a fantastic sliding tackle by David Weir on Juarez ensured it remained goalless.
Back came Gers, with Davis turning and firing a fierce shot goalwards after Whittaker’s hard cross only to see Fraser Forster gather comfortably.
It was then Celtic’s turn to move forward again but like everyone else to that point, Emilio Izaguirre couldn’t find his range from 35 yards.
Rangers’ advances upfield were more structured, but Naismith’s lay off only resulted in Madjid Bougherra putting the ball over from the edge of the area.
And Ki Sung Yeung saw his free kick after a McCulloch foul on Georgios Samaras smothered by Allan McGregor.
With half-time nearing, it seemed a lack of clear-cut chances would mean it was goalless at the break.
But then, with the goal gaping, McCulloch should have done so much better than turn over after Whittaker’s mazy run in from the right.
Celtic ended the first half with a series of corners and with each one, they got closer to breaking the deadlock.
But just when it seemed Rangers had got to the break unharmed, they fell behind a minute into injury time.
Ki sent the set piece in and when the ball found its way beyond a ruck of bodies, Hooper was there at the back post to poke in.
Falling behind was nothing new to Gers as they had clawed back deficits in each of their last three SPL games to come out on top.
But with almost six years since a team had fallen behind first then won in an Old Firm fixture, they knew how hard it would be to restore parity then forge ahead.
Boss Smith made one change ahead of the restart, with Kirk Broadfoot coming on for the crocked Papac.
And within three minutes the Light Blues were level, with Lafferty taking the credit for the equaliser after Davis’ free kick was scrambled in.
Replays showed the ball came off Loovens’ chest but either way, the big Northern Ireland hitman will undoubtedly claim it as his first derby counter.
Just six minutes later, the match was turned on its head as Gers went ahead in dramatic circumstances.
Forster’s kick out was poor and only went as far as Naismith, whose first-time touch played Maurice Edu in.
Although he couldn’t get his shot away, the attempt ricocheted off the American and into the previously quiet Miller’s path.
Then, with as clean a connection as you’ll see this season, the striker blasted in his 12th club goal of the campaign with a crisp low volley into the bottom corner from 15 yards.
In control of the match for the first time, Rangers were actually pretty comfortable as seeing their lead evaporate so quickly had visibly sickened the home team.
And their hurt turned to pure agony on 66 minutes as Broadfoot was brought down in the area by Daniel Majstorovic and referee Willie Collum pointed to the spot.
Shaking their heads in disbelief at what they were seeing, home fans began streaming for the exits as Miller slammed his second of the game high into the net.
By contrast, there was sheer bedlam amongst the away supporters and they bounced up and down in delight as they sensed a first victory at Parkhead in more than two years.
Neil Lennon’s team was slow to muster any sort of response but when they did, it took some spectacular goalkeeping to deny them.
Indeed, McGregor was at his very best as he threw himself across his goal to palm a decent Juarez free kick around his post.
From there, it was a very muted Celtic who tried in vain to play their way back into the tie but with some resolute defending, there was no way past Rangers.
The visitors’ one touch football became more confident and if anything, they only looked like they would add to their advantage.
Time was ticking down and with every passing minute, the massed ranks in the away end got louder.
After two minutes of injury time, the toothless hosts were put out of their misery as Collum blew for the end of a brilliant victory for the 53-times Scottish champions.