Passive cooldowns work different than other cooldowns, as I understand.
If you take a champion like Sinesha, and use her to put Duchess' skills on cooldown, it only puts active skills on cooldown I believe, which means Passive skills are just affected differently.