David Cameron's political rivals have accused him of running scared after saying he will only take part in one televised debate ahead of the election.