Otley Civic Centre closed to the public in 2010 despite campaigners fighting to keep it open.
Now proposals have been unveiled by Leeds City Council to sell the treasured building to a private company.
In an email sent to ward councillors asking them for their views, the local authority said it is considering selling the council-owned civic centre to Mandale Homes, to replace it with a 21-apartment development.
"We got an email basically saying that the council have made the decision that they are going to sell it [Otley Civic Centre] to a company, to effectively convert into flats," Coun Colin Campbell (Lib Dem, Otley and Yeadon) told the YEP.
"Obviously it's a community building - it's well-loved by the people who live here and we are a bit disgruntled.
"We thought at least they [the council] would do sort of a consultation with us about it."
The building was closed in 2010 because it didn't meet fire regulations, Coun Campbell said.
It was used for civic events, weddings, dances and society meetings and also housed an office for Otley Town Council.
Coun Campbell said he wants the cash raised from any sale of the site to be reinvested into Otley.
"I think our [the ward councillors'] view is that number one: we would like the public to tell us exactly what they think because it is, in effect, their building," he said.
"And if the council are going to sell it, because they own it, then we think that the proceeds from any sale should come back to Otley."
The Give Otley Its Town Hall campaign group in 2016 called on the council to spend Â£1.5m to refurbish the building to bring it back into use, as an alternative to selling it off.
Meanwhile, Alex Sobel, Leeds North West MP whose constituency includes Otley, said he understood "hard decisions" had to be made amid funding cuts to local authorities from central government.
He said: "It would have been preferable to see the building continue to be used for public amenity. However, the building is rapidly deteriorating and I do not want to see it fall further into disrepair.
"I understand the decision has been taken in the context of a 60% funding cut to the council’s budget from central government and therefore hard decisions must be made.
"Although Mandale Homes have a good track record of restoring heritage buildings, I am concerned about the parking and traffic implications that may follow from residential use."
He echoed Coun Campbell's call for the council to reinvest revenue from the sale back into Otley, and said he would take the issue up with the local authority.
A Leeds City Council spokesman said: “These plans would bring this important historical listed building back into use, offering a strong and sustainable future. They would also provide much-needed new accommodation and especially affordable housing in the area, so we look forward to hearing what people think.
“The future of this building has been subject to considerable amounts of consultation and engagement over a period of years, and these plans present an opportunity to get good value for the people of Leeds and Otley, providing resources which will benefit them for decades to come.”