Data-driven decision-making consequential to individuals raises important
questions of accountability and justice. Indeed, European law provides
individuals limited rights to 'meaningful information about the logic' behind
significant, autonomous decisions such as loan approvals, insurance quotes, and
CV filtering. We undertake three experimental studies examining people's
perceptions of justice in algorithmic decision-making under different scenarios
and explanation styles. Dimensions of justice previously observed in response
to human decision-making appear similarly engaged in response to algorithmic
decisions. Qualitative analysis identified several concerns and heuristics
involved in justice perceptions including arbitrariness, generalisation, and
(in)dignity. Quantitative analysis indicates that explanation styles primarily
matter to justice perceptions only when subjects are exposed to multiple
different styles---under repeated exposure of one style, scenario effects
obscure any explanation effects. Our results suggests there may be no 'best'
approach to explaining algorithmic decisions, and that reflection on their
automated nature both implicates and mitigates justice dimensions.