After increasing for eight consecutive weeks, last week sales slipped by 16% to 1.21m (£5.4m), according to industry publication The Bookseller.
However the British novelist still topped bestseller lists by a huge margin, with the first book Fifty Shades of Grey shifting 534,088 copies.
The trilogy's closest rival Flesh and Bones by Kathy Reichs sold just 23,871.
"At some point all books are going to reach their peak," explained The Bookseller's Charts Editor, Philip Stone.
"I think Fifty Shades reached its peak the week before last when it sold 1.4 million. The media were talking about it all the time, it was on The One Show and had had a lot of coverage in the press the weekend before."
The Fifty Shades stories, dubbed "mummy porn" by the press, feature a steamy relationship between billionaire businessman Christian Grey and "unworldly, innocent" literature student, Anastasia Steele.
Sales of the original book Fifty Shades of Grey fell 21% last week with book two, Fifty Shades Darker, and book three, Fifty Shades Freed, suffering 11% and 13% declines to sales of 362,480 and 321,628 respectively.
However despite the media attention having "quietened down", Stone says the trilogy will "still comfortably be at number one for a while yet".
Summer reads Print sales of book one alone now stand at 2,833,988, putting it in 11th place in a list of the bestselling books since records began in 1998.
It is currently behind Dan Brown's Angels and Demons, but has now overtaken huge bestsellers such as Stieg Larsson's The Girl with the Dragon Tattoo, Stephenie Meyer's Twilight and Eric Carle's The Very Hungry Caterpillar.
Stone said this year's summer reads will find it hard to make an impact on the book charts.
"Paperbacks of John Grisham's The Litigator and Lee Child's The Affair are on the horizon. They could normally hope to get to number one, but this year I think that's going to be very difficult," said Stone.
Other big forthcoming titles this summer include James Patterson's Private Games, Stuart MacBride's Birthdays for the Dead and The Haunting by Alan Titchmarsh.
"All of these might have hoped to top the charts in a normal year," said Stone. "But fourth place in the bestseller lists is the new number one!"
For the last four weeks, sales of print books have been larger than during the same weeks last year - with sales of E L James' books accounting for a quarter of all book sales during that time.