I think it's more that Handler doesn't really have any competition - She's aiming for the female audience awake at night, and her show is more like an R-rated version of The View or a chat show than a late night show. Conan probably splits his audience with Stewart, Colbert, Letterman and Leno.
For what it's worth, I think Conan is doing OK. Someone mentioned a couple months back that he was suffering from a lack of good guests, and while I didn't think it was a problem at that time, I do think he's having that issue now. (Except for Hanks last night, the best guest this week was Elijah Woods or Liv Tyler.) I also think that Ferguson and Fallon at 12:30, and the Comedy Central guys, are putting on the most original and consistently funny bits.