Five pictures said to have been painted by Nazi leader Adolf Hitler have failed to sell at auction in Germany.