Warren Buffett thinks the American people thought there would be a speedier recovery for the economy after the credit crisis.