Fired by an algorithm? Exploration of conformism with biased intelligent decision support systems in the context of workplace discipline

Abstract

Purpose

The importance of artificial intelligence in human resource management has grown substantially. Previous literature discusses the advantages of AI implementation at a workplace and its various consequences, often hostile, for employees. However, there is little empirical research on the topic. The authors address this gap by studying if individuals oppose biased algorithm recommendations regarding disciplinary actions in an organisation.

Design/methodology/approach

The authors conducted an exploratory experiment in which the authors evaluated 76 subjects over a set of 5 scenarios in which a biased algorithm gave strict recommendations regarding disciplinary actions at a workplace.

Findings

The authors’ results suggest that biased suggestions from intelligent agents can influence individuals who make disciplinary decisions.

Social implications

The authors’ results contribute to the ongoing debate on applying AI solutions to HR problems. The authors demonstrate that biased algorithms may substantially change how employees are treated and show that human conformity towards intelligent decision support systems is broader than expected.

Originality/value

The authors’ paper is among the first to show that people may accept recommendations that provoke moral dilemmas, bring adverse outcomes, or harm employees. The authors introduce the problem of “algorithmic conformism” and discuss its consequences for HRM.

Citation

Bartosiak, M.L. and Modlinski, A. (2022), "Fired by an algorithm? Exploration of conformism with biased intelligent decision support systems in the context of workplace discipline", Career Development International, Vol. 27 No. 6/7, pp. 601-615. https://doi.org/10.1108/CDI-06-2022-0170

1. Introduction

The idea of building connections between machines and people became the foundation of the fourth industrial revolution, taking place since the second decade of the 21st century. One of the solutions that made it possible is artificial intelligence (AI), defined as “manifold tools and technologies that can be combined in diverse ways to sense, cognise and perform with the ability to learn from experience and adapt over time” (Akerkar, 2018, p. 3). Thus, whenever we refer to an artificial or autonomous agent, we refer to “systems that are capable of sensing, information processing, decision-making and learning to act upon their environment and to interact with humans and other machines to achieve a shared task goal with more or less autonomy” (Seeber et al., 2020, p. 2).

The use of technologies based on AI has grown substantially. AI affords digital innovations in organisations and, thus, transforms the society in general (Trocin et al., 2021). Companies use it to perform tasks like monitoring errors (Davenport and Kirby, 2016), saving costs (Ahmad et al., 2021), improving services (Albrecht et al., 2021), collecting and processing data making companies more data-driven (Al Mansoori et al., 2021), co-creating business value (Grundner and Neuhofer, 2021), or training employees (Novichkov et al., 2021).

A subject that has received much attention from mass media and conceptual scholars is the impact of AI on Human Resource Management (HRM) and career development (Meijerink et al., 2021). HR is familiar with digital tools facilitating daily work (Allal-Chérif et al., 2021) but recently it is AI that transforms multiple activities of HRM (Vrontis et al., 2022). For example, scholars discuss an increase of productivity when employees cooperate with AI (Upchurch, 2018), control and evaluation of employees by algorithms (Kellogg et al., 2020), or consequences of AI implementation for various classes of workers (Frey and Osborne, 2017). Scholars use mostly conceptual theorising. Thus, they have recently called for more empirical research in the area, as experimental research in this field is still scarce (Jia et al., 2018; Pan et al., 2022). One of the gaps regards decisions made from biased algorithms' (Seeber et al., 2020). Such biases may lead to various negative consequences to organisations and individuals (Galaz et al., 2021). However, the scale of the biases and human predispositions to oppose such biases are still scarcely examined.

Therefore, we refer to this gap in our research. Building on the recent debate on algorithmic HRM (Evans and Kitchin, 2018; Meijerink et al., 2021), we simulate a biased algorithm giving too strict recommendations regarding disciplinary actions for employees violating the work rules. We intend to answer the research question: will humans oppose biased algorithm recommendations regarding disciplinary actions in an organisation? We evaluate 76 subjects over a set of 5 scenarios. Our results suggest that humans are harsher in evaluating colleagues when an algorithm suggests stricter disciplinary actions. Thus, our paper offers three contributions. First, we show that biased algorithms may substantially change how employees are treated. Our results suggest that individuals show limited readiness to oppose recommendations made by intelligent decision support systems. Second, we stress that people may accept recommendations that provoke moral dilemmas, bring adverse outcomes, or harm employees. Ultimately, our research visibly accentuates the need for further research that identifies a profile of artificial systems' supervisors that are (1) willing to go against intelligent decision support systems and (2) sensitive to the injustice of information systems.

The remainder of this article is built as follows. First, we present the conceptual background. We refer to artificial intelligent systems and show their current functionalities. We also pay attention to intelligent decision support systems (IDSS) showing their features, strengths and vulnerabilities. On this footing, we contextualise the persuasiveness of the IDSS and demonstrate the human conformity to recommendations made by algorithms. Next, we present the methodology of our experiment, followed by the results of the study. Finally, we discuss the potential consequences of the persuasiveness of biased algorithms by referring to the results of our study and previous literature.

2. Conceptual background

2.1 AI in organisations

Intelligent technologies are involved in the decision-making processes that influence organisations' and teams' performance (Schaefer et al., 2016). Autonomous agents have various degrees and scope of autonomy in performing the tasks. They may retrieve information, give advice or take independent actions (Nissen and Sengupta, 2006). In general, AI-based decisions are often perceived as high quality (Keding and Meissner, 2021). On the other hand, people may not understand how an autonomous agent works, leading to confusion and loss of control over the organisation (Kirchmer, 2017). Yet, the latest report shows that the number of companies that allocate formal budgets for AI development has grown by 80% from 2020 to 2021 (Appen, 2021). Part of the research on autonomous agents to date has been devoted to their adaptation to human teams (Hancock, 2017), especially to the process of mutualising goals to serve the organisation's purpose (Klien et al., 2004). Therefore, it is still in human hands to (1) design autonomous agents to correlate with these goals and (2) learn to collaborate with autonomous agents to keep team goals at the centre of the human-machine relationship.

2.2 AI in decision support systems

Systems based on AI support humans in the decision-making process. Intelligent decision support systems (IDSS) are defined as computation program-based mechanisms that improve the decision-making processes in organisations (Mora et al., 2005). IDSS uses the concept of automation to make faster decisions based on specific standards present in companies (Möhlmann et al., 2021). Earlier research shows that these systems help to make more accurate decisions and to optimise resources (Quintero et al., 2005). Moreover, they reduce uncertainty and increase process transparency (Orlikowski and Scott, 2014). Psychology identifies two major approaches to decision making: clinical and statistical (or actuarial) (Dawes et al., 1989). The clinical judgement is based on informal processes like intuition or experience. The statistical judgement, on the other hand, reduces these informal inferences and builds on the empirical evidence of the relationship between the data and predicted condition (Dawes et al., 1989). The literature highlights higher accuracy of the latter (Ægisdóttir et al., 2006), important particularly in cases that are not unambiguous (Meehl, 1957) or judgements bearing high responsibility, affecting individuals and society (Oleson et al., 2011). Furthermore, scholars have recently theorised about the potential long term negative consequences of replacing human decision-makers with autonomous algorithms (Balasubramanian et al., 2022). It is supposed that relying too much on the algorithm's output may increase the risk of overlooking distant and extreme outcomes – an ability learned with experience by humans (Balasubramanian et al., 2022).

Thus, the role of IDSS should be to offer human workers such an insight into data that humans alone would be either not able to process or need much more time to process when making decisions, but not to limit the human factor. The result is a form of hybrid intelligence, in which the system supports the human decision maker (Ostheimer et al., 2021). Decision support systems based on AI are no longer limited to well-defined problems, but also with complex and unstructured problems (Orriols-Puig et al., 2013), this raises the question about the dependability and trustworthiness of human employees when accepting or rejecting decisions recommended by artificial systems. An ideal IDSS provides status reports (showing actual actions and performance), forecast (signalling expected outcomes of actions), recommendations (indicating the most beneficial actions) and explanation (providing why the system made a particular recommendation) (Phillips-Wren et al., 2009). However, many decision support systems are far from ideal, especially when it comes to decision explanations.

2.3 Conformist behaviour towards disciplinary decisions made by AI in HRM

Artificial intelligence is commonly used in several areas of HRM: HR planning (Mazari Abdessameud et al., 2021), talent management (Black and van Esch, 2021), recruitment processes (Allal-Chérif et al., 2021), candidates' selection and maintenance (Malik et al., 2022). These systems help companies cut the cost linked to the multilevel recruitment process and prepare the company for turnover by suggesting a potential replacement for a vacancy before it occurs. They are also applied to monitor human performance and engagement (Song and Wu, 2021). Since the intelligent systems can support people in making decisions, the question arises whether people supervising the work of IDSS will oppose the recommendation of a system that would expose employees to disproportionate disciplinary consequences. Conformity is defined as a change in personal behaviour due to the real or imagined influence (Asch, 1955). The vast majority of studies in social influence address human-human interactions in both natural and virtual environments (Rosander and Eriksson, 2012). Relatively less attention is paid to the human being influenced by intelligent technology (Salomons et al., 2021), especially in the HRM context. Conformist behaviour among employees is correlated with the ambiguity and perceived difficulty of the task in such a way that the more ambiguous and complex the task is, the higher the probability that humans will manifest conformity (Bond and Smith, 1996). It is worth mentioning this involves both subjective and objective assessment of a task's difficulty (Rosander and Eriksson, 2012). Conformity may potentially become a serious challenge in HR-related decision-making with AI as IDSS creates a black-box for a human employee who loses the context and understanding why a specific recommendation appears. Such a black box creates a distance between human and organisational issues and may lead to confusion or ethical concerns (Zarsky, 2016). When the black box appears, conformist behaviour is more likely to occur. Earlier experiments show that non-human agents (robots and computers) can conform to humans when performing social and analytical tasks using persuasion (Hertz and Wiese, 2018). However, these experiments are not related to professional duties or tasks that may potentially cause moral dilemmas. One case depicted in the literature presents an algorithm favouring white males when assessing employees' performance (Tambe et al., 2019) but it is not known how human supervisors react to it. Surpassingly, there is a scarcity of reliable empirical research studying whether people follow or object to biased IDSS recommendations, which could expose employees to unfair treatment. Our research, therefore, aims to verify this issue through an exploratory experiment on the simulation of a biased intelligent system that we created for this study. Considering that humans use similar heuristics involving humans and technology (Nass and Moon, 2000), we deduce that this may also occur when people begin working with non-human agents, such as IDSS. Therefore, we have formulated an exploratory hypothesis:

He1.

HR system's supervisors display conformist behaviour towards recommendations related to disciplinary decisions made by IDSS.

3. Methodology

3.1 Design

Due to the exploratory nature of the study, our interest was to build a baseline for the theoretical understanding of the effect of an intelligent agent on disciplinary consequences rather than testing existing HR solutions. Thus, we designed our experiment in the well-established psychological tradition (Shadish et al., 2002). We conducted a posttest-only randomised experiment, with participants assigned to a treatment group or control group.

RX1ORO

We used the Wizard-of-Oz-like approach where the treatment group participants were informed they would be interacting with an artificial intelligent agent, while the experimental environment was entirely designed and controlled by the researchers (Kelley, 1983). For this reason, we designed two versions of a fictitious decision-support app–one per group. Each group had access to only one version of the app and could not see the other version. In the control group, the participants were asked to assess five situations without the suggestion of the algorithm, while in the treatment group, the participants were informed that the suggestions they were provided were coming from an algorithm trained on over 1,000 real disciplinary cases coming from various firms. Participants were instructed that they simultaneously teach and test the autonomous system that would be implemented in the HR department of the real company. The ethics of the design, apparatus and survey were evaluated and approved for human-subject study by the institutional board at the university where the data was collected.

3.2 Participants

We employed convenient, non-probabilistic sampling at a European university and recruited 106 participants between 18 and 30 years old. All the subjects were students of the last year of the HR Management specialisation (after six months of apprenticeship). Furthermore, 91% of the sample had previous work experience. Thus, they were familiar with the corporate HR practices in the country of data collection. The study was a voluntary one, thus the subjects could resign at any moment. Of the contacted subjects, 76 records were useable (see section 4.1). All participants remained utterly naïve about the aims and purpose of the study during the treatment – subjects were told that they would assess a new HR-support software and that we would collect their impressions about it. Subjects were debriefed after the experiment.

3.3 Variables

We measured subjects' decisions on disciplinary consequences on an ordinal 0–4 scale – (no consequence, verbal counselling, written warning, suspension and improvement plan, contract termination). For the treatment group, we compared the disciplinary decision with the suggestion of the intelligent agent. This enabled the measurement of participants' conformity with the intelligent agent's suggestion. Furthermore, we measured the time in seconds (T) spent on the tasks. Finally, we measured control variables: perceived trust in the “tested” technology and history of reprimands at work or school.

3.4 Apparatus

First, we prepared a set of scenarios presenting a fictitious employee breaking a workplace rule, varying by severity of the situation and the profile of the presented employee. As a benchmark for the severe AI's recommendations, we invited 6 HR expert professionals to evaluate the scenarios and provide the disciplinary consequences for each situation. We selected five situations where consensus in assessments among the experts was highest (Table 1).

Once the scenarios were ready, we designed two versions of a mock online app, which showed the scenarios, one by one, to the participants. The treatment version presented the study in a first-person narrative as if the program had a conversation with the user. On the other hand, the control group app presented the study in third person (as if the researcher presented them) but presented the same information. In addition to the disciplinary action committed by the employee, the app presented a profile of the fictitious employee (name, surname, gender, age, position and years of experience in the company). However, given our sole focus on the effect of a decision-support intelligent agent, these items were standardised across treatment and control conditions (Figures 1 and 2). Furthermore, all other app elements remained standardised across the two versions and the design stayed neutral not to distract the participants.

Both versions of the app tracked participants' behaviour on screen, including the duration of the visit, time spent reading and evaluating the scenarios and the decision about the disciplinary action. Finally, after the experimental task, the app included a brief survey with statistical control questions. Furthermore, we included a hypothesis guess question and an attention control question in the survey. Survey instruments were administered online at the end of the study.

3.5 Procedure

The experiment took place in an online environment. First, we broadly introduced the study to the participants during an online meeting, without revealing the real purpose of the study. After the introduction, all participants received a link, which directed them to the experimental app. First, participants were asked to read and accept the informed consent form. After this step, each participant was randomly assigned to one of two experimental groups and sent to another page presenting the instructions. In the instruction, subjects were told that they would test a new HR software and were asked to familiarise themselves with the five disciplinary consequences available in the app.

Participants were then redirected to the fictitious app and were presented with the five scenarios, one after another. Subjects were free to use all the information provided on their screen. The subjects of the treatment group had to decide whether to accept or reject the suggestion of the software. In the case of rejection, they were asked to provide their evaluation. The subjects of the control group were asked to provide their own evaluations. We did not set any time limits on the task so that participants would behave in a manner that came naturally to them. After the participants evaluated all the scenarios, the experimental app asked them to complete a survey, also capturing the control measures. After the survey, each subject was debriefed and the real purpose of the experiment was revealed.

4. Results

Before analysing the data, we ensured that the manipulation occurred correctly and that participants followed the experimental procedures. We controlled the IP addresses so that no duplicate answers were allowed. Furthermore, we removed the records of subjects who did not pass the attention control questions. The experimental system ensured that participants were exposed to the same content and read all five scenarios. All the participants who spent less than 40 s on the task were excluded from the analysis. This interval was deemed as not enough time to read and evaluate the scenarios. Furthermore, we excluded four records of the subjects who spent more than 5 min on the task, as their times were classified as outliers (ranging from 5 min to 41 s to 13 min and 26 s). From the original 106 participants, there was a useable sample of 76 records. Each participant evaluated five scenarios. Thus, we had 380 evaluations. Random treatment assignment was unbalanced – 31 subjects in the control group and 45 in the treatment group. However, the treatment group does not pass the threshold of 75% of the total sample. Thus, the efficiency of the comparison does not decline (Pocock, 1979).

4.1 Sample description

In the overall sample, 30 subjects were males and 46 were females. As control questions, we asked the subjects if they had ever received any reprimand at work or school. 61% of the subjects had never received any warning, 26% had received a warning and 13% had never worked. Regarding school reprimands, 41% of participants declared not to have received any and 59% declared to have received at least one. Furthermore, using a 9-item 1–7 Likert-like scale, we measured participants' perceived trust (TR) in the technology they “tested” in the task (Lankton et al., 2015). On average, the participants expressed a moderately positive trust towards the software (mean 4.36, min. 3, max 5.22, SD 0.46). In the control group, 14 subjects were males, 17 were females, while 16 subjects were males and 29 were females in the treatment group. No significant differences were found between the treatment and the control groups regarding perceived trust in technology (p = 0.46).

4.2 Groups comparison by the time of decision-making

First, we used the control question to compare the time of decision between the subjects who had experienced reprimands at school or work vs. those who had not. There were no significant differences between the subjects (t (66) = −0.3, p = 0.76 and t (31) = −1.8135, p = 0.079, respectively). Therefore, we compared the average time spent evaluating the scenarios between the treatment and control groups. The 45 participants who received the experimental treatment (M = 104.12 s, SD = 46.6) compared to the 31 participants in the control group (M = 133.23 s, SD = 57.1) demonstrated significantly faster decision making, t (54) = 2.298, p = 0.025. The effect size of AI assistants on decision-making time is moderate (d = 0.568). Thus, the results suggest that individuals make disciplinary decisions faster when supported by an intelligent agent's suggestions.

4.3 Groups comparison by disciplinary decision

Table 2 shows the mean evaluation by experts, algorithm's suggestion in the treatment group, mean in the control group and mean in the treatment group.

Because we used a Likert-like scale to measure participants' decisions and the consequent non-normality of the sample distribution, we conducted a nonparametric Mann–Whitney U test to investigate the effect of Intelligent assistants on the disciplinary decision. First, we controlled for differences between subjects who had experienced reprimands at school or work vs. those who had not. As we did not find significant differences in any of the five scenarios (Table 3), we continued with the formal analysis. The results suggest that the treatment group participants in each scenario were significantly more severe than the participants in the control group (Table 4). The effect size (η2) for each scenario varies from moderate to high.

The difference in mean ranks is evident in scenarios where the algorithm suggested the highest possible punishment (scenarios 3, 4 and 5). Furthermore, even if the median evaluation of the treatment group is lower than the algorithm's suggestion (Scenarios 2), the mean rank evaluation is still significantly higher than in the control group. Considering the above results, He1 is supported.

5. Discussion

Despite the speed at which AI-based solutions are coming to market and their rising importance to organisations, there is still a paucity of empirical research that rigorously evaluates the AI-based assistants on HR-related decisions (Pan et al., 2022). Our results suggest that individuals who make disciplinary decisions about employees can be affected by the suggestions of intelligent agents. Thus, we contribute to the literature and the debate on the potential use of AI in HRM solutions. Our results show that subjects supported by AI make decisions faster. In line with previous research, we could extrapolate our conclusions to confirm that new technology increases employees’ satisfaction with their decision (Makridis and Han, 2021). However, we should pay attention to the tradeoff between employee satisfaction and the potential negative consequences of biased decisions. Contrasting somewhat with previous research, which claims that individuals tend to oppose moral decisions made by AI (Bigman and Gray, 2018), our subjects did not oppose a biased algorithm. Thus, the results should stimulate a search to understand better why AI affects individuals' disciplinary decisions and the potential consequences of such influence. A concept that can shed light on the phenomenon is persuasive technology (Fogg, 2003). We pay particular attention to currently one of the most intriguing methods of persuasion through technology which is data-centred persuasion (DCP). DCP helps modify human behaviour due to collecting, processing and applying a vast number of behavioural data (Shin and Kim, 2018). Objects such as beacons, home assistants, autonomous cars, or the Internet of things may potentially collect inputs from and interact with individuals' environments so that they actively shape individuals' experiences and behaviours in these environments (Dourish, 2004). In other words, the information system analyses data to find patterns and recommend some action that corresponds to a particular goal set by the user. The higher the perceived credibility of the source and argument's quality, the higher the probability that the user accepts the recommendation (Li, 2013). This strategy is often used in software designed to promote a more healthy and sustainable lifestyle (Böckle et al., 2020) or recommend products to customers (Huang and Hsu Liu, 2014). However, it can be helpful in a HRM context too. For instance, when persuading employees to collaborate with each other (Peng et al., 2019). Despite its potential, DCP may unintentionally be embedded into technology and increase the employees' informational conformist behaviour. It may take place when IDSS recommends specific actions referring to collected and processed data reflecting human behaviour. In other words, a cue sent by a non-human agent as, e.g. “this suggestion is made according to 1,000 similar decisions made earlier by people” may be perceived by the decision-maker as social proof and may increase their obedience towards IDSS, even if such a cue was not purposefully persuasive but informative by nature. If such conformity appears, its consequences may be harmful for the organisation individuals, primarily when IDSS is based on a biased algorithm.

Thus, we see another important avenue for future research: the interpretability and ethics of persuasive AI solutions in the workplace. With the rise of techniques like exponential coding, machine learning and artificial intelligence, creators of such solutions are often unable to fully explain how the algorithms created by them arrive at certain decisions (Ananny and Crawford, 2018). The issue of algorithmic transparency is critical when the outcome of an algorithm affects individuals' decisions and behaviours or brings consequences to individuals' lives. We have witnessed many instances of an algorithm unintentionally programmed to produce biased, unexplainable outcomes that negatively affected individuals' lives (Cossins, 2018). Such an algorithmic bias can be harmful in situations when the suggestions from the algorithm affect individuals' careers. Furthermore, pervasive intelligent HRM tools and potential consequences of biased algorithms call for further debate on the future of employment in the age of AI. Experts predict a high probability of human replacement in the majority of professions in the next decade (Gruetzemacher et al., 2020). However, what is missing in the current debate is the discussion on how biased AI affects employment. Algorithms already decide who to hire by supporting (and, thus, influencing) HRM decisions. Furthermore, current business case studies suggest they have started to decide who to fire too and people may accept the potentially biased recommendations of IDSS in punishing individuals. One of these cases is the dismissal of 150 employees of the Russian company Xsolla, based on the suggestion of the performance evaluation system (Obedkov, 2021). A similar experience was faced by Amazon, which was supposed to use a productivity tracking system to designate employees for layoffs (Lecher, 2019). This also happens in other contexts, as shown by the example of COMPAS – an algorithm assessing the likelihood of recidivism by some US state courts (Humerick, 2019) – accused of racial discrimination. In all cases, there have been suggestions that people accept the IDSS recommendation too thoughtlessly. We do not know the potential future consequences of biased algorithms affecting these activities. Individuals not hired or fired by a biased algorithm may feel treated unfairly or they may start questioning their own value. Such a situation may cause long-term societal consequences. From the organisational perspective, a lack of employees who can oppose a biased algorithm can cause financial and reputational losses. In line with recent theorising, such a reliance on algorithms, rather than human learning, may cause the inability to ignore long-term negative consequences or predict extreme failures (Balasubramanian et al., 2022). In the context of HRM it may mean hiring poorly qualified candidates or firing good employees. However, such biased algorithms could be implemented in other contexts like budgeting or R&D etc. causing tangible losses to organisations' liquidity. Furthermore, we need a separate discussion on the ethical consequences of AI-based solutions on HRM and employment. In line with recent calls, we suggest that any adoption of such solutions should safeguard the well-being of (potential) employees (Nazareno and Schiff, 2021).

Another important implication of our study is the need to specify the employee's profile that will oppose the recommendation of the IDSS and alert the company if s/he identifies irregularities in how the system works. Our research does not show who is prone to accept the recommendations of the IDSS, but such knowledge is key to recruiting reliable “IDSS's supervisors.” On the one hand our results show that HR professionals using clinical judgements may not always be accurate. On the other, we show that an IDSS algorithm used in statistical judgement, when biased, may also bring negative consequences. It is crucial for HR professionals to try to avoid such influences (of their own misjudgement or biased algorithms). Thus, we support the appeal of other researchers to define the critical competencies of employees of the industry 4.0 (Saniuk et al., 2021) and supplement the previous postulates with such attitudes, which we call “algorithmic nonconformism,” i.e. the tendency to oppose IDSS suggestions when they raise ethical doubts of the human supervisor. Therefore, to extend the currently proposed competencies for industry 4.0 (Shet and Pereira, 2021), we shed light on the critical challenge for HRM—discovering who fits? The profile of an algorithmic nonconformist and understanding whether it is possible to teach people such an attitude.

5.1 Limitations

Our research is not free from limitations and, thus, the results should be interpreted accordingly. First, the sample was relatively small and the distribution to experimental groups was unequal. While we controlled for the potential bias, further studies could replicate the experiment on a bigger scale. Furthermore, our subjects were homogenous in terms of their background and experience level. Future studies should examine the effects discovered in this study among professionals of various experience levels and various cultures. As individuals of various cultures perceive technology differently, we may expect possible variations of the results. Finally, due to Covid-19 sanitary restrictions, we had to adjust our study to online settings. Future research should investigate a similar problem in fully controlled lab conditions and, as an additional step, in a live experiment, considering the ethical considerations we mentioned earlier in the discussion.

6. Conclusion

In this paper, we investigate individuals opposing biased algorithm recommendations regarding disciplinary actions in an organisation. We conducted an experiment in which we tested 76 subjects over a set of 5 scenarios in which a biased algorithm gave strict recommendations regarding disciplinary actions for employees violating the disciplinary work code. Thus, we showed that employees follow IDSS recommendations, even if these may be harmful to their colleagues. Our results suggest that humans are harsher in evaluating colleagues when an algorithm suggests more strict disciplinary actions, making their decisions faster. Our results contribute to the HR management and career development literature and the ongoing debate on applying AI solutions to HR problems.

Figures

Figure 1

Control group app

Figure 2

Treatment group app

Table 1

Scenarios applied in the experiment

Scenario 1	Name and surname: Marc Thomson
	Gender: Male
	Age: 23
	Position: Assistant
	Experience in the company: 2 years
	Disciplinary action: being late 15 min (first time)
Scenario 2	Name and surname: Eva Kate Gross
	Gender: Female
	Age: 32
	Position: Senior Executive
	Experience in the company: 5 years
	Disciplinary action: being late 20 min (second time in last 3 months)
Scenario 3	Name and surname: Maria Michels
	Gender: Female
	Age: 28
	Position: Junior Assistant
	Experience in the company: 2 years
	Disciplinary action: stealing (150$ from the team account, confirmed)
Scenario 4	Name and surname: Thomas Redworth
	Gender: Male
	Age: 28
	Position: Service Executive
	Experience in the company: 3 years
	Disciplinary action: discrimination/racism (accused by a teammate, confirmed by antidiscriminatory commission)
Scenario 5	Name and surname: Steven White
	Gender: Male
	Age: 30
	Position: Service Executive
	Experience in the company: 5 years
	Disciplinary action: sexual harassment (accused, not confirmed, case being proceeded)

Table 2

Summary of evaluations in control and treatment groups

	Experts	Algorithm suggestion	Control group (M)	Treatment group (M)	Average difference from experts (control)	Average difference from experts (treatment)
Scenario 1	0	1	0.47	0.84	0.47	0.84
Scenario 2	1	3	0.98	1.55	−0.02	0.55
Scenario 3	4	4	3.29	3.90	−0.71	0.10
Scenario 4	3	4	3.02	3.74	0.02	0.74
Scenario 5	3	4	0.33	3.03	−2.67	0.03

Table 3

Control checks

	Reprimands at school		Reprimands at work
	U	Sig. (p-value)	U	Sig. (p-value)
Scenario 1	612.5	0.46	647.5	0.20
Scenario 2	568	0.17	602.5	0.46
Scenario 3	653.5	0.81	445	0.18
Scenario 4	631.5	0.62	453.5	0.22
Scenario 5	661.5	0.88	604.5	0.49

Table 4

Results

	Treatment group sample size	Control group sample size	Treatment group median	Control group median	Treatment group mean rank	Control group mean rank	U	z-score	Sig. (p-value)	η2
Scenario 1	45	31	1	0	44.18	34.59	521.50	−2.064	0.039	0.046
Scenario 2			1	1	44.21	34.57	520.50	−2.220	0.026	0.046
Scenario 3			4	4	47.61	32.22	415.00	−3.615	0.000	0.117
Scenario 4			4	3	49.26	31.09	364.00	−3.895	0.000	0.163
Scenario 5			4	0	57.92	25.12	95.50	−6.747	0.000	0.533