"I know that," he answers, though rather than make it another needle in a sensitive spot he points out, "You've an entire ship full of people to summon when you need them. Teleportation and all. You don't need my tricks, o captain of the Void."

So that's not what he's afraid of. More...

"I still don't want to drag you into my affairs. Little daedra are one thing, but the greater ones, the truly powerful ones... dealing with them can get messy."