Rogne signed for Celtic on 20 January 2010, and joined on a three and a half-year contract. Ex-Celtic player Vidar Riseth described him as “the best young talent to come out for Norway in the past ten years”. Rogne was given the number 25 shirt.
On 10 February 2010, Rogne made his Celtic debut coming on as a substitute for Glenn Loovens in a 2–0 home win against Hearts in the SPL. He made his first start for Celtic at home against Dundee United in a 1–0 victory on 20 February 2010, he played very well and impressed the coach Tony Mowbray. Rogne impressed against Rangers on 28 February 2010 but was taken off due to a hamstring injury. He returned two months later in an away game against Dundee United but once again had to leave the field early through injury.
Rogne made his first competitive start of the 2010–11 season for Celtic against St Johnstone in a 3–0 win. In the following game he was sent off for a professional foul against Aberdeen. Despite this, Celtic went on to win the game by a record-breaking 9–0 score. He scored his first goal for Celtic in a 1–1 draw against Kilmarnock in December 2010. Rogne won many plaudits for his performance as Celtic beat Rangers 2–0 at Ibrox in the 2011 New Year derby. He scored his second goal for Celtic against Aberdeen in their 4–1 League Cup semi-final win on 29 January 2011. Rogne took part in his first major Cup Final on 20 March 2011 when Celtic played Rangers in the Scottish League Cup Final, but they lost 2–1. An injury in May sidelined Rogne for the rest of the season.
Injury kept Rogne out of the Celtic side for the first few months of the following season. He returned to the Celtic first team in October 2011 in their League Cup quarter-final tie against Hibernian. Rogne continued to struggle with a series of minor injuries, but nevertheless played well in the majority of his sporadic appearances, having kept a clean sheet in over half the games he played. On 25 March 2012, he scored in Celtic’s 3–2 defeat to Rangers at Ibrox.