The chimney, at Stonebridge Mills in Farnley, is thought to be more than 150 years old and is viewed as a significant local landmark.
The Grade II-listed old mill buildings are being redeveloped into homes as part of a £25m scheme.
Despite originally promising to keep the 30m chimney in place, developers are now set to reduce its height by 6m after concluding the top portion is not structurally safe.
The top 10m will be removed, before a 4m extension is rebuilt.
Leeds Civic Trust was among the objectors to the removal, though a planning meeting on Thursday was told they could not comment on the chimney’s structural stability as that was not in their remit.
Also objecting to its removal was local Green Party councillor David Blackburn, who argued it could be retained and made safe.
He told the meeting: “We’re talking about a chimney, but it’s a chimney of significant importance.
“I support this development – I went along to see it recently and it’s coming on well. But it’s a sore point is this. There have been a lot of compromises and this is one too far.
“As a Farnley and Wortley resident for most of my life, I don’t want to see it (lost).”
However, representing the company that completed a structural assessment of the chimney, Jeremy Tetley said the height needed to be reduced “on the grounds of public safety”.
He said it was severely damaged and cracked on all four sides.
He told councillors: “It may have stood for 150 years but it doesn’t now sit in accordance with current building standards.
“The only reason we want to reduce the height of the chimney by 6m is to retain the chimney and maintain what we can of it.”
The applicants said they’d initially pledged to leave the chimney untouched because they’d previously been unaware of how structurally vulnerable it was.
Abestos and surrounding buildings had made it difficult to get to its base, the meeting was told.
Conservative councillor Trish Smith however, voiced support for the objectors.
She said: “It’s a heritage structure. Like it or lump it, we can’t erase history. I just think we need to explore every opportunity before we make this a ‘yes’ or a ‘no’ answer.
“The site does have architectural merit. I do think this chimney is worth saving.”
But Labour councillor Neil Walshaw was one of six members of the plans panel to decisively vote in favour of the application.
He said: “Multiple different sources of engineering expertise have come to the conclusion that the most reasonable course of action, to maintain the heritage, is the approach that’s been set out.
“A considerable number of people are going to be living round this tower and I don’t think they want the chimney accelerating to meet them at 10 metres per second. I think they want a safe chimney.”