Distressed-debt manager Invictus lost control of its flagship fund after investors ousted it for business tactics they deemed too aggressive, including during a chapter 11 bankruptcy.