But the 76-year-old said he had no plans to retire, as long as the public still wanted him.
The veteran all-round entertainer, who is working on his 36th album, said he missed the excitement of one-man concerts.
He also revealed he is in talks about a role in a major West End stage production, but kept tight-lipped about further details, only saying: "It's something I've been keen to do."
O'Connor said he had also been writing children's stories and poems, as his son Adam, who turns four later this year, asked his father to tell him tales.
He said that stories he had written included one about a dancing duck and another about a donkey with a wooden leg.
O'Connor said of his time on Countdown: "It's been wonderful hosting a programme which is part of TV folklore.
"I have really enjoyed my two years with the show and I would like to thank all concerned at Channel 4.
"I've made a lot of new friends. Carol Vorderman has been a joy... it's been a giggle since day one.
"But I miss the thrill of stepping out on a live stage for the one-man concerts and the challenge of live TV entertainment-based shows."
O'Connor said his album, with the working title Setting New Standards, was being recorded with producer and songwriter Alan Hawkshaw, who has written for Barbra Streisand - as well as writing music for Countdown.
O'Connor said he had had a "fantastic" time in Leeds, where the show is filmed and he would be able to see more of his family, who are based in Buckinghamshire, after leaving.
He described Adam as "such a joy".
O'Connor, who also runs his own production company, said: "There are just so many things being offered to me. It just seemed to be the right time."
O'Connor was recently awarded the CBE for his services to entertainment and broadcasting.
Since landing his first television series in the UK in 1963 he has starred in his own mainstream television show every year.
He said he will have done more than 500 shows by the time he leaves in November and he already has some concerts lined up for this autumn.
Asked if he would ever retire he said: "Only if I think they don't want me any more."
Speaking about the loyalty of Countdown viewers, he said the show received a letter from a woman who had only missed one episode - because she was under anaesthetic at the time.
"I hope they will understand," he said. "I'm very grateful to Carol and everyone at Channel 4."
He said he was leaving with a mixture of "sadness and excitement".
Vorderman said she was upset that O'Connor was leaving as he is a "true gentleman".
She said: "Audiences love him and we have had so many laughs in the studio and only last week we recorded some of our best shows.
"Des and myself have spoken with regard to him leaving and he knows how I am about it.
"But I know that everyone on Countdown will want to wish him well and Des and I hope we can work again together in the future.
"Long live Dick-A-Dum-Dum."
O'Connor joined the words and numbers quiz show in January 2007 and helped the series celebrate its 25th birthday in November last year.
He replaced Des Lynam as the anchor, who stepped in following the death of Richard Whiteley.
Helen Warner, Channel 4's Head of Daytime, said: "We are very sorry to see Des leave.
"He has done a fantastic job hosting Countdown, bringing warmth and humour to the role, as well as the benefit of his many years in the business."
Channel 4 said it would be announcing a replacement for O'Connor in due course.