"Even aside from the dremora, I don't know what else might have moved in since I left. It... might be more than some... leftover zombies."

As if even that would be the worst thing about going back down there. As if that's what he's most afraid to face. He's not looking at Jim now and it's... it's not exactly terror or stubbornness haunting his expression. It's guilt. Shame.