Abiy had just wrapped up his speech in the heart of Addis Ababa when the explosion went off, sending panicked crowds towards the stage as the prime minister left hurriedly, an AFP correspondent said.
Hundreds of thousands and perhaps around a million people converged on the capital city's Meskel Square starting at 9 am local time.
Mr Ahmed became prime minister after his predecessor Hailemariam Desalegn unexpectedly resigned in February.
The prime minister's chief of staff said 83 people were wounded, with six in critical condition.
The attack was "cheap and unacceptable", the Prime Minister said, and added- "Love always wins". "An individual tried to hurl the grenade toward a stage where the prime minister was sitting but was held back by the crowd". Another witness told Reuters the assailant who carried the grenade had been wrestled to the ground by police before it exploded. "Then we heard the explosion".
AP video from the scene showed bloodstained ground and abandoned shoes while people chanting the prime minister's name fled, some clutching their heads in shock and despair. He called the blast a "well-orchestrated attack" but one that failed.
In recent years, Ethiopia has been riven by protests by members of the country's two largest ethnic groups - the Oromo and the Amhara.
Abiy, 42, took office in April and quickly surprised Africa's second most populous country by announcing a wave of political and economic reforms.
Earlier, the government said it had unblocked 264 websites and TV channels. Many Ethiopians said they could hardly keep up with the pace of change.
"For the past 100 years hate has done a great deal of damage to us", he said, stressing the need for more reforms.
The Eritrean ambassador to Japan, Estifanos Afwerki, said his country "strongly condemns the attempt to incite violence" at what he described the first ever demonstration for peace in the history of Ethiopia.
Mr Abiy stunned Ethiopians this month by saying he was prepared to fully implement a peace deal with Eritrea signed in 2000 and meant to end a two-year war between the country and its neighbour that devolved into a stalemate resulting in huge military build-up by both countries. "I saw some five people injured following the blast".
One organizer of the rally, Seyoum Teshome, says on Facebook that "Prime Minister Abiy and other guests who came in attendance are all fine".
He is the country's first leader from the ethnic Oromo group, which has been at the centre of almost three years of anti-government protests that left hundreds of people dead.