Angela Merkel is set for a fourth term as German Chancellor, according to exit polls.
However, the poll suggests the anti-immigration Alternative für Deutschland (AfD) party will enter the national parliament for the first time.
Mrs Merkel’s CDU/CSU alliance took 33% of the vote, well ahead of the second placed Social Democratic Party with 20.5%, which would be its worst result since World War Two.
But in a major shock, the AfD took 12.6% support in Sunday’s vote, making it the country’s third biggest political force.
It is now heading for the opposition benches of the Bundestag lower house – the first time a hard-right openly anti-immigration party with so many seats has entered parliament since World War Two.
The four-year-old party has links to the far-right French National Front and Britain’s UKIP.
At least 500 protesters took to the streets outside the AfD’s election party in Berlin after the results were announced, with some shouting “all Berlin hates the AfD”, “Nazi pigs” and other slogans.
There were other protests in Cologne, Hamburg and Frankfurt.
But AfD supporters cheered as the exit poll results were announced and co-leader Alexander Gauland vowed they would “change” the country.
AfD co-leader Alice Weidel told them: “Millions of voters have entrusted us with the task of constructive opposition work in parliament”.
French far-right leader Marine Le Pen congratulated the AfD on entering the German parliament.
Ms Le Pen, who lost France’s presidential election to Emmanuel Macron earlier this year, wrote on Twitter: “Bravo to our allies from AfD for this historic score! It’s a new symbol of the awakening of the peoples of Europe.”
In her victory speech, Mrs Merkel vowed to win back voters from the AfD and admitted the party’s entry into parliament posed “a big challenge”.
Mrs Merkel failed to secure a ruling majority and now the process of coalition building will begin and could take several months.
She told supporters in Berlin: “Of course we had hoped for a slightly better result. But we mustn’t forget that we have just completed an extraordinarily challenging legislative period, so I am happy that we reached the strategic goals of our election campaign.
“We are the strongest party, we have the mandate to build the next government – and there cannot be a coalition government built against us.”
Another three parties cleared the 5% hurdle to representations, including the liberal Free Democrats who secured 10.7%, anti-capitalist Left and ecologist Greens who both achieved about 9%.
SPD deputy leader Manuela Schwesig said her party would now go into opposition, which rules out a re-run of Mrs Merkel’s existing alliance with the party.
An alternative coalition for Mrs Merkel would be a three-way tie-up with the liberal Free Democrats (FDP) and the Greens.
Mrs Merkel, 63, faces four years of government in a fragmented parliament after the return of the FDP – unrepresented at national level for the last four years – and the arrival of the AfD.
Founded in 2013 by an anti-euro group of academics, the AfD has developed into an anti-immigration party that exploited Mrs Merkel’s 2015 decision to leave German borders open to more than one million migrants.