Emergency services were called to Piper Crescent, in Southey, at about 10.15pm, following reports that the young man had suffered stab wounds to his chest.
He was taken to hospital but sadly died shortly after.
A 39-year-old man was arrested on suspicion of murder on Monday night and remains in custody.
In a statement, South Yorkshire Police said: "Police were called to Piper Crescent at around 10.15pm on Sunday night, following reports that a man had suffered stab wounds.
"The 21-year-old was taken to hospital, where he sadly died a short time later.
"Cordons remain in place in the Southey area this morning."
Police are still appealing for witnesses and urge anyone with information to contact them on 101, or call Crimestoppers anonymously on 0800 555111