And the Saints prop insists he always enjoyed a good relationship with the head coach, who resigned on Wednesday.
Lancaster saw his team fail to progress from the pool stages of the World Cup on home soil.
England became the first host nation to miss out on a quarter-final place and a review into the below-par performance is currently taking place.
But Lancaster appears to have walked before he is pushed, and Brookes is disappointed to see him go.
Kettering Commonwealth Games medalist Kieran receives a hero's return
Chair role to be shared at Diamonds for this season
Glover’s plans for Poppies will take everything into account
NCL: Brigstock stun Peterborough as Finedon pull clear
Poppies boss Glover still on the hunt for fresh faces
“It’s been a tough World Cup for everyone and he’s taken the brunt of it,” said the Saints star, who made three appearances for England in the World Cup.
“It wasn’t my decision at the end of the day and it’s tough on him.
“I’ve always had a good relationship with Stuart and I wish him well for the future.”
Saints forwards coach Dorian West also felt it was a shame that Lancaster decided to leave his post.
West said: “It’s really disappointing for Stuart.
“He’s a really good bloke who has worked hard for England’s cause and it’s always disappointing when you hear that sort of news.”