When I played the campaign, I felt that the level that puts you up against the Barbarian, Lord Slayer, is the hardest, because the Barbarians start sending powerful heroes to attack your castles almost immediately, and with the ‘no movement penalty on rough terrain’ they can cross the desert and be at your door step within 2-3 weeks, making it tough to survive the initial onslaughts. But if you do, it gets easier.
You can avoid this level entirely if you choose the Barbarians to play the campaign to begin with. 🙂