Given how openly goddamn racist the elves are toward humans, there's no way in hell that wasn't a major factor in the downfall of the Riders - or indeed the root cause of it. You can bet your butt they would never have treated poor Galbatorix so cruelly if he'd been an elf.