Does on-the-job training help graduates find a job? Evidence from an Italian region

Abstract

Purpose

The purpose of this paper is to evaluate a recent training programme for graduates, implemented in Italy and entitled Work Experience Laureati and Laureate, i.e. Work Experience for Graduates. The aim of the programme was to increase the career prospects of unemployed graduates in the region of Umbria.

Design/methodology/approach

The authors rely on administrative data and matching methods to evaluate the effectiveness of the intervention in terms of employability of participants.

Findings

The results show that participants are more likely to be employed and to sign an apprenticeship contract within the region boundaries. The authors also find substantial differences in employability and type of contract by gender, with men having a higher probability of finding a job (permanent contract and apprenticeship). The authors show that this may be explained by the different choices in terms of field of study, with males being more prone to enrol in scientific areas and females in the humanities.

Research limitations/implications

It is an intervention implemented in one Italian region.

Originality/value

This is one of the few studies that analyses the effectiveness of active labour market policies targeting unemployed graduates, especially in the Italian context. The authors rely on different administrative data sources that allow them to evaluate the effectiveness of the programme.

Citation

Ghirelli, C., Havari, E., Santangelo, G. and Scettri, M. (2019), "Does on-the-job training help graduates find a job? Evidence from an Italian region", International Journal of Manpower, Vol. 40 No. 3, pp. 500-524. https://doi.org/10.1108/IJM-02-2018-0062

1. Introduction

The effectiveness of labour market policies has been extensively debated in the recent literature. However, no consensus has been reached regarding the impact of some types of policies, such as those supporting the implementation of training programmes, since the effectiveness of these programmes strongly depends on the time horizon over which the employment effects are measured. As summarised by Card et al. (2010, 2017) in their meta-analysis, training programmes show larger average effects in the medium and long terms. This also applies to private sector incentive programmes, and the reason for this lies behind the so-called lock-in effects. As discussed by Ham and Lalonde (1996) among others, participants in training and private sector employment programmes often reduce or suspend their normal job search efforts while participating in these programmes, drastically reducing their employment opportunities in the short term.

Moreover, training programmes have been for long conceived for low educated youth. Hence, most of the existing empirical literature works refer to the impact of training programmes on this specific sub-population. In contrast, evidence on the impact of training measures on highly educated individuals is scarce.

This paper analyses the effectiveness of an intervention targeting graduates that was implemented in 2013 in the Italian region of Umbria. The programme, entitled “Work Experience for Graduates” (Work Experience Laureati e Laureate (WELL)), subsidised on-the-job training for unemployed graduates. It received financial support from the European Social Fund (ESF) and had two main goals: increasing the employability of unemployed graduates residing in Umbria and enhancing the innovative capacity and productivity of participating firms.

This paper aims to contribute to the existing literature in two ways: first, it provides additional evidence on the effectiveness of training programmes in Italy, for which evidence is scarce. Second, it contributes to the literature that analyses the transition from tertiary education to the labour market.

We combine different sources of administrative data, namely, data on the programme participants provided by the Office of Statistics and Evaluation of the Umbria Region, as well as data from the Compulsory Communication Database (CCD), which was collected from local labour offices by the Italian Ministry of Labour and Social Policies.

The effectiveness of WELL in terms of the employability of participants is evaluated using a matching approach where unemployed graduates participating in the intervention are compared to the whole population of unemployed graduates that reside in the region of Umbria but do not take part in the intervention. The outcome variables of interest were measured in December 2015 and referred to: the probability of being employed in the region of Umbria, of being registered as unemployed in the region of Umbria, as well as the type of contract received (if employed).

Our analysis shows that at the end of 2015 (up to two years after the programme), WELL participants were more likely to be employed in Umbria than non-participants. Participating in WELL increases the probability of being employed by about 11 percentage points (pp). We do not find any meaningful effect on the probability of obtaining a permanent contract or a temporary contract, but we find a positive effect (4 pp) on the probability of obtaining an apprenticeship contract. Hence, the latter seems to be the most important route towards employment for WELL participants. Interestingly, when we split the sample by gender, we find that participating in WELL is more beneficial for men, both in terms of employability and the type of contract obtained. We find heterogeneous effects by field of study, with more positive effects for individuals with a degree in science. In contrast, individuals with a degree in the humanities benefit somewhat less from participating in the programme. Such differential impacts may be explained among others by two factors, such as labour demand in the region of Umbria and the relatively lower relevance of job-specific work experience for this type of skills. In light of these results, the aforementioned gender difference in the employment responses to the programme may be explained by the fact that men are more likely than women to complete a degree in science.

Although we refer to a relatively small region and a small scale intervention, the analysis of graduates’ employment prospects is particularly relevant for policymaking, especially given the rising enrolment rates in tertiary education in many countries and the increased emphasis on improving graduates’ employability after the 2008 financial crisis (Pavlin and Svetlik, 2014). In Italy, in particular, the inadequacy of graduates’ skills or work experience is pointed out as a major problem for the labour market, and accordingly, the provision of relevant training and work experience to tertiary students is identified as a key policy measure to facilitate the transition from education to work (European Union, 2015). Analysing the employment prospects of graduates in Italy is also particularly relevant in light of the recent reforms aimed at increasing the flexibility of the Italian labour market.

The remainder of this paper is organised as follows: Section 2 reviews the related literature. Section 3 provides a description of the intervention and the selection procedure. In Sections 4 and 5, we present the data and the descriptive statistics for the sample used in the analysis. Section 6 explains the empirical strategy implemented to quantify the impact of the WELL intervention and presents the main results. Finally, in Sections 7 and 8, we discuss the results and conclusions of the evaluation.

2. Literature review

Empirical evidence on the effectiveness of training programmes across European countries tends to be mixed. Germany is the European country with the most solid experience in terms of the evaluation of training programmes, which can partly be explained by the interest in evaluating the Hartz reforms implemented in the early 2000s to tackle high unemployment rates in the country. Evaluations of various training programmes show both positive and non-significant effects on employability and wages in the medium and long terms.

Kopf (2009) and Rinne et al. (2011) consider both short- and medium-term programmes, finding that public training programmes positively affect employment prospects independently of the skills and age groups of participants, whereas Lechner et al. (2011) find negative effects for all types of training programmes in the short term, but positive effects on employment rates and earnings in the long term (after about two to four years). Finally, Schwerdt et al. (2012) and Górlitz and Tamm (2016) find no significant effects of training programmes on both wages and employability, even in the medium term.

These findings are also confirmed by the evaluations of programmes and similar reforms in France, Belgium, the Netherlands and Denmark. Chéron et al. (2010) estimate the impact of firm-provided training on labour market mobility in France, showing that participation in a training programme reduces the probability of switching jobs or becoming unemployed in the two subsequent years. Cockx and Van Belle (2019) arrive at similar conclusions, finding small positive effects for training and counselling measures devoted to recent graduates in Belgium. Hidalgo et al. (2014) show that in the Netherlands training vouchers on low-skilled workers do not have any significant impact on monthly wages or on job mobility but have a significant impact on future training plans. Danish job training programmes are analysed by Jespersen et al. (2008), who assess the long-term employment and earning effects on participants in both the private and the public sectors. They highlight that the positive post-programme effects on earnings eventually materialise after one to three years. Focusing on youth unemployment and providing a survey of the recent evidence from European countries, Caliendo and Schmidl (2016) confirm mixed effects of labour market training, depending on the type and on the country.

The evidence on the effectiveness of training programmes in Italy is scarce. However, two regional programmes implemented in Italy have been recently analysed. The first study evaluates the “Paid Quality Traineeships” programme within the Giovanisí project that was launched in the Italian region of Tuscany in 2011 (Sciclone et al., 2017). The programme is similar in terms of design to the Umbria programme but has a slightly different target group. In fact, it targets youth aged 18–29 years, who are either graduates or drop-outs in the previous two years, and resident in specific municipalities of Tuscany (about 74 per cent of municipalities have been included). These traineeships have a maximum duration of six months and have an educational scope, as firms that hire the trainees need to assign them a tutor. The authors combine administrative data for both the participant and non-participant groups granted by the Tuscany Region. They use propensity score matching to estimate the impact of the programme on youth employability two years after the completion of the training. Results show the positive effects of participation in the training programme on employment (especially for young people with no working experience), within 18 months from the start of the traineeship. Nevertheless the study has some caveats. One of the main limitations is that the authors are not able to match individuals on the basis of education, which is an important variable to take into account when looking at young people without any working experience. Our analysis overcomes this difficulty and allows us to account for both educational level and field of study.

The second study by Pastore and Pompili (2019) concerns the evaluation of an integrated plan of employment and labour policies (PIPOL), launched by the Italian region of Friuli-Venezia-Giulia in 2014. PIPOL includes a series of programmes financed by the ESF, with the aim of improving the transition process of the youth from the world of school and university to that of work. In particular, the authors evaluate the first phase of PIPOL (2014–2018) and focus on programmes related to extra-curricular training and internships. They employ counterfactual impact evaluation methods (propensity score matching) and also a more qualitative approach to evaluate the impact of PIPOL on job placement for different participants’ characteristics (gender, age, nationality, etc.). The main findings are that the training programme seems to be successful among women, youth younger than 30 years and immigrants. A common result with this study is that we also find that the WELL programme was successful for those younger than 30 years, but different from the study we find men to be more successful in terms of labour market outcomes compared to women.

More generally, the peculiarities of the Italian labour market have recently been investigated to assess the impact of specific measures: the so-called 2015 Good School reform[1], and the 2003 reform of apprenticeship contracts. Picchio and Staffolani (2019) investigate the impact on job opportunities resulting from firm-provided training through apprenticeship contracts. Compared to other types of temporary contracts, apprenticeships are found to be more effective in leading workers to a stable job relationship, especially within the same firm where the apprenticeship was undertaken. Investigating the same topic, Albanese et al. (2017) focus on the impact of the 2003 reform of the Italian apprenticeship contract, showing the positive effect of the new apprenticeship on the transition to permanent employment, mostly in large firms.

Regarding the Good School reform, Pastore (2018b) highlights that the elements that make the transition from school to work problematic stem from the low level of both secondary and tertiary education attainments and the rigidities of the education system (especially tertiary education), which might delay entry into the labour market from university. The Italian institutional setting of education, training and welfare systems was also analysed in a cross-country comparison, to document how it affects the youth disadvantage in the labour market (Pastore, 2015, 2018a, b). In particular, the author discusses that poor performances of youth labour market participation and youth unemployment may depend on the specific school-to-work transition regimes and on poor levels of integration between the education systems and the labour market.

The employability of Italian graduates has been analysed from various perspectives. Ballarino and Bratti (2009) assess if, and to what extent, different fields of study affect graduates’ chances of employment. Taking into account the increasing flexibility of the Italian labour market following the recent reforms, they find that scientific fields, as compared to the humanities, consistently guarantee a higher probability of stable jobs in the university-to-work transition.

Brunello and Cappellari (2008) study the importance of the attended college on the earning and employability prospects of students three years after graduation. They conclude that the choice of college has an impact on employment, especially in the short term.

Finally, Di Pietro and Urwin (2006) assess the effect of over-education on the earnings of Italian graduates. They show that over-educated graduates receive lower wages than peers with a similar level of education who do not experience education mismatch in their jobs. Interestingly, this effect does not depend on the under usage of skills but rather on human capital and job features.

In the same vein, Aina and Pastore (2012) show that delayed graduation increases the chances of over-education, thereby contributing to the wage penalty faced by over-educated workers.

Despite growing interest in the analysis of the employment prospects of tertiary graduates, little evidence is available on the active labour market policies targeting graduates.

3. Description of the WELL programme

The WELL intervention was financed through the ESF and implemented in 2013 as part of the Regional Operational Programme of the Umbria Region[2]. Its main goal was to reduce unemployment among the youth and to strengthen the professional qualifications of graduates by raising the quality of their jobs.

More specifically, it was designed to promote fully subsidised work experience, with the aim of increasing employment among highly educated individuals, who have a higher risk of exclusion from the labour market. The programme also included the provision of wage subsidies to employers who hired WELL participants at the end of the training. The programme also aimed to promote the dissemination of modern and efficient production processes and to increase the innovative capacity and productivity of the participating firms.

The intervention was therefore characterised by a strictly integrated path consisting of two steps: on-the-job training for unemployed graduates; and wage subsidy to firms and organisations that hire the trainee after completion of the training. The training was expected to last six months, with a minimum weekly commitment of 24 h and a gross monthly salary of 800 euros per trainee[3]. The subsidy awarded to firms and organisations varied depending on the type of contract offered. For a trainee hired with a fixed-term contract of at least six months, the subsidy was 2,500 euros, whereas for an apprenticeship contract, it amounted to 4,000 euros. For an open-ended contract, a subsidy of 6,500 euros was offered.

The intervention’s beneficiaries were unemployed people, including first-time jobseekers, who held a bachelor’s or master’s degree and resided in Umbria at the time of programme’s launch (May 2013). Participants’ unemployment status had to be demonstrated through registration in public employment offices. Companies and organisations involved in the intervention, such as associations, foundations, cooperatives and consortia, were required to have at least one production/work unit in Umbria and to have at least two permanent employees[4].

To avoid the possibility that the intervention could produce displacement effects, companies applying for the intervention had to show no dismissals of workers with similar occupational tasks to the ones they were hiring for in the year preceding the traineeship[5].

The WELL project was launched in May 2013 and was completed in September 2014. A shortlist of around 100 eligible firms in Umbria was published in order to encourage participation in the intervention, and participants also used their personal networks to find potential workplaces in which to complete a traineeship. Applications were examined by the regional Department of Labour Policies and eligible applications were ranked according to the following criteria:

1. The commitment of the host company to employ the trainee at the end of the traineeship, depending on the type of job contract:

   • open-ended contract (full-time and part-time) (5 points);

   • fixed-term contract, lasting at least six months (2 points); and

   • other types of contract (1 point).

2. Applicant with a disability under the rules of Italian national law 68/1999 (1 point).

3. Applicant’s age:

   • below 29 year old (2 points);

   • 30–39 year old (3 points); and

   • 40 year old or above (4 points).

4. Innovative activities of the host organisation, defined as participation in regional/national poles or clusters, or ministerial research laboratories (2 points).

In the case of equal scores, the ranking was determined according to the chronological order of submission of the electronic application. For both of the intervention phases, a quota for female applicants was put in place, accounting for 50 per cent of the initial amount of funding for the intervention (€1.2m). The intervention was very successful in terms of participation. In fact, the number of applications received exceeded the expectations of the implementing authority. To meet this unforeseen demand, the budget for the programme was increased to €3.6m and all eligible applications were admitted. Consequently, the quota for women proved to be unnecessary.

The number of applications received for the WELL programme totalled 712, 30 of which were ineligible to participate. Of the 682 eligible applicants, 74 decided not to begin the traineeship and 34 dropped out during training. A total of 574 graduates successfully completed the work experience. The occurrence of drop-outs could be due to the administrative burden related to the framing of the traineeship period. As for the second phase in the intervention, grant subsidies were allocated to 96 companies and host organisations that recruited 98 trainees who had successfully completed the first phase. Of these, 13 workers were employed with a permanent contract, 51 were hired with a temporary contract and 34 were employed with an apprenticeship contract. In the next session, we also provide some descriptive evidence on the characteristics of the firms that hired the WELL participants after the completion of the on-the-job training programme.

4. Data

This analysis combines micro-data on the WELL intervention from the Regional Monitoring System Database and administrative data regularly collected from local labour offices by the Italian Ministry of Labour and Social Policies, i.e., the CCD (CCD) (Comunicazioni Obbligatorie, COB). The CCD collates information about all hirings, as well as any prolongations, transformations and cessations of labour contracts, which private companies and public administrations are required to communicate to the labour offices. In addition, it keeps a record of jobseekers registered at public employment offices, while it does not comprise information on self-employment. This information system has been operated by the Ministry of Labour and Social Policies in cooperation with regional governments, the National Institute of Social Security (INPS), the Italian Government Agency for the Insurance against Work-related Injuries (INAIL) and, since 2015, with the National Agency for Active Labour Policies (ANPAL). Since 2008, all companies operating in both the private sector and public administration have been obliged to communicate hirings, prolongations, transformations and cessations of labour contracts by accessing and entering the data into an online information system referred to as “Compulsory Notifications”[6].

We have access to the CCD for the region of Umbria, which collects information on the employment and registered unemployment periods of people residing in the region. The data used for the analysis consists of information for the period of July 2013–December 2015 for all individuals in the sample. Thus, we have information on individuals in both the treatment and control groups, from before and after the intervention. We observe a number of individual characteristics measured in July 2013, which are pre-determined with respect to the start of the intervention and outcome variables measured in December 2015 referring to the labour market status of the individuals. Specifically, in July 2013, all graduates in the population of interest are unemployed, while in 2015, in addition to individuals’ employment status, we can also observe the type of firm and the sectoral level in which those who are employed are working.

Our sample consists of the target population for the programme, i.e., all graduates residing in Umbria who are unemployed on the day of the application deadline for participating in the WELL intervention (2 July 2013). All individuals are eligible and hence intended recipients of the programme. Some participate (treatment group) and some do not (control group).

The treatment group is composed of 574 participants who completed the training phase of the WELL programme (out of 682 eligible applicants). The control group includes the whole population of registered unemployed graduates residing in Umbria who did not participate in WELL. This group comprises 6,950 individuals.

5. Descriptive statistics of WELL participants and non-participants

This section describes the target population of the programme, which is the sample used for this analysis, and presents the outcome variables used in the evaluation.

5.1 Demographic characteristics

Tables I–IV present the descriptive statistics for participants and non-participants. The population of interest comprises all unemployed graduates (including first-time job seekers), registered as unemployed in the region of Umbria and residing there on the deadline for applications to the WELL intervention. This consists of 6,654 individuals. Of these, 574 participated in the intervention. After applying standard cleaning procedures, we are left with a final sample of 5,816 individuals: 5,266 non-participants and 550 participants[7]. The rest of the analysis is based on this final sample.

Among WELL participants, the number of unemployed people with previous work experience is double that of first-time jobseekers. In the control group of non-participants, the number of unemployed people with previous work experience is four times that of first-time jobseekers. Women are equally represented among WELL participants and non-participants (around 70 per cent).

As for the age distribution of individuals, there is a higher proportion of participants in the younger age group (24–29 years). In contrast, the oldest groups (35–40 and 40+ years) are less represented than the non-participant group. Note that the age of both participants and non-participants is measured at the time in which WELL programme was launched.

As shown in Table IV, there is a large proportion of unemployed people in the humanistic disciplines such as the social sciences, business and law. In contrast, jobseekers with a degree in science tend to be less represented in the WELL group, when compared to non-participants. While we have more detailed information on the specific field of study for WELL participants, this information is not available for non-participants. Therefore, we refer to the definitions of areas of study provided by the Italian Statistical Institute (ISTAT) in order to classify information available in the CCD for non-participants.

In order to ensure the similarity and comparability of the participant and non-participant groups with respect to all observable characteristics such as age, education, gender, etc., we select covariates available for both groups in the two data sources (i.e. intervention data for the treatment group and administrative data for the control group).

Table V reports the observable individual characteristics measured in 2013 (before treatment takes place) for both treatment and control groups.

The first and third columns show the average value of each characteristic for the treatment and control groups, respectively. The last column reports the p-value obtained for the difference between the average values in the first and third columns. In italic, we indicate the characteristics for which the difference between treatment and control groups is statistically significant (at the 95% confidence level). On average, the treatment group is significantly younger than the control group. Conversely, in the treatment group, the proportion of individuals at least 35 years old amounts to only 17 per cent, while in the control group it represents 39 per cent. In addition, a higher proportion of people in the treatment group have a degree in the social sciences, business or law (39 per cent in the treatment group vs 33 per cent in the control group), and have a bachelor’s degree (33 per cent in the treatment group vs 22 per cent in the control group). In contrast, a higher proportion of individuals in the control group have a degree in science (the 3 per cent difference between the two groups is statistically significant at the 95% confidence level) and obtained a master’s degree (78 per cent in the control group and 67 per cent in the treatment group). Lastly, those in the treatment group are significantly more likely to reside in Perugia, the capital of the region.

5.2 Outcomes

To characterise the labour market status of individuals, we consider the following outcome variables:

• Employment status indicator: equal to 1 if the individual is employed according to the CCD data for the region of Umbria, and 0 otherwise. Note that this definition is different from the traditional employment rate. In addition to having a regular job, it requires that the job is to be located in Umbria. Working in a neighbouring region is coded as 0. This definition is not fully satisfactory, as it does not count working outside of Umbria as a success. However, it represents a relevant outcome variable in our context, as the main objective of the programme is to boost employment among graduates residing in the region of Umbria. Since the programme is implemented in this region, it is important to assess the local effects of the programme.

• Unemployment status indicator: equal to 1 if the individual is registered as unemployed on the lists of the unemployment offices of the region of Umbria, and 0 otherwise. Also in this case, our definition of unemployment status does not consider being registered as unemployed in the unemployment offices of other regions.

The following comments are worth noting. First, since our employment rate is based on administrative data, we are not able to observe individuals working in the underground economy. However, to the extent that the objective of the policy is to increase formal employment, this is not a limitation of the study. Second, the administrative data at hand provide us with information on dependent employment. This means that we do not observe self-employed people. However, given the nature of the programme under study, we believe this is a minor issue. We expect a traineeship programme to be successful if the trainees remain employed in the firm once the traineeship ends. Third, since the unemployment rate is based on administrative data, we cannot observe discouraged workers who are out of the labour market (both in Umbria and in neighbouring regions). However, we believe that our measure of the unemployment rate is relevant for the evaluation of the policy, since registered unemployed are the target of active labour market programmes implemented by the local authorities.

In addition to the outcome variables related to one’s labour market status, we also consider the type of contract (permanent, temporary or apprenticeship) registered by December 2015, based on administrative data from the CCD data for the region of Umbria. The outcomes referring to the type of contract are intended to evaluate the effect of the programme on job quality. Accordingly, we study the probability to be hired with permanent, temporary or apprenticeship contract, as proxies of good quality jobs. We ignore precarious contracts, since the aim of the programme is not to enhance the probability to be hired under those types of contracts[8].

All outcome variables are measured in December 2015, i.e., between one and two years after the completion of the on-the-job training programme.

The possibility of extending the time horizon of the post-intervention period beyond the short term turns out to be particularly beneficial in the evaluation of training programmes, since there could be issues related to lock-in effects. Specifically, participants in WELL, as opposed to non-participants, were limited in the time they could dedicate to searching for jobs, while they undertook the training programme. However, the lock-in effect does not rule out the hypothesis that training can increase participants’ employment prospects, since the programme could prove effective if evaluated in the medium or long term, when lock-in effects fade away or are outweighed by the beneficial effects of the programme.

Table VI reports descriptive statistics for the outcome variables, namely, employment status (top panel) and type of contract (bottom panel), for the treatment (WELL participants) and control (non-participants) groups. The first and third columns show the proportion of WELL participants and non-participants, respectively, who are employed and unemployed. The fifth column shows the test statistic for the difference in the averages by treatment status. Column 6 reports the p-values for the t-tests on this difference. WELL participants seem to be more advantaged in terms of labour market outcomes after completion of the training. They are more likely to be employed (the difference in employment rates between the two groups is 16 per cent). As for the type of contract, WELL participants are more likely to get a temporary job or apprenticeship, but no significant differences are found for permanent contracts. Nevertheless, we must be cautious about these comparisons as they may be misleading due to selection bias.

Figure 1 shows the percentage of WELL participants (left panel) and non-participants (right panel) that are employed as measured in 2015, by the municipality of residence. The shading in the map darkens with higher employment rates. Overall, WELL participants are more likely to be employed in 2015. In most centrally located municipalities, the percentage of employed WELL participants ranges between 40 and 80 per cent, whereas the percentage of employed non-participants range between 10 and 40 per cent. In relatively small municipalities, the percentage of employed among WELL participants reached 80–100 per cent (darker areas), whereas the percentage of employed among non-participants was between 10 and 40 per cent.

Besides providing descriptive evidence on the WELL participants, we also show some descriptive statistics on the type of firms that offered the training programme to the 550 participants (demand side). For each firm, we can observe the number of employees (firm size), the sector of activity[9] and the location (municipality). Table VII shows the distribution of WELL participants across firms, separately by firm size (number of employees in the firm). About 86 per cent of WELL participants completed their on-the-job training in very small or small firms having between 1 and 50 employees, whereas 10 per cent of them are working in middle-sized firms and 2 per cent in big firms. Table VIII shows instead the average firm size by the type of industry or sector. Both tables show that the firms hiring WELL participants tend to be small independently from the sector of activity.

Finally, we show the characteristics of the firms hiring WELL participants after the completion of the on-the-job training. As explained above, there are three types of the contract offered: permanent, temporary and apprenticeship. Out of the 550 WELL participants, 98 individuals were hired at the end of the programme, mostly with a temporary or an apprenticeship contract. Interestingly, the very small firms are the only ones offering a permanent contract, whereas the medium-sized firms and large firms did not hire any participant on a permanent contract, preferring mostly temporary contracts.

This piece of evidence is helpful because it highlights the fact that the programme was of small scale (Table IX).

6. Empirical strategy

This analysis aims to evaluate the impact of the WELL intervention on the labour market prospects of the unemployed graduates targeted by this programme. As previously mentioned, the first set of outcome variables that we consider are: the probability of being employed in Umbria (employment status indicator, as observed in the CCD data for the region of Umbria), and the probability of being registered as unemployed in the unemployment offices of the region of Umbria. In addition, we also study the probability of having either a permanent, temporary or apprenticeship contracts.

We estimate the average treatment effect on the treated (ATT), which represents the impact of the WELL intervention for the target population (Angrist and Pischke, 2008). The ATT is calculated as the difference between the average outcome of the treatment group and the average outcome of the treatment group in the counterfactual situation in which the treatment did not take place. The ATT is given by:

(1) ATT = E ( Y 1 | D = 1 ) − E ( Y 0 | D = 1 ) ,

where D is an indicator equal to 1 if the treatment takes place and 0 otherwise, Y¹ is the individual potential outcome given the treatment, and Y⁰ is the individual potential outcome in the absence of the treatment. Note that for the ATT, both potential outcomes refer to the treatment group, since they are conditioned upon D=1. Equation (1) highlights the counterfactual nature of the ATT. The first term of the equation refers to the average employment outcome among the participants in the intervention (WELL participants in our case), whereas the second term refers to the average employment outcome among the WELL participants had they not participated in the programme. This second term is not observable but we can look for a control group that allows us to provide an estimate.

This means that the ATT measures the effect of the programme on intended recipients. As pointed out in Heckman (1997), the ATT this is the relevant estimator for policy, which targets specific populations. A related parameter is the average treatment effect (ATE), which measures the effect of a programme on the entire population – referring to both the intended and non-intended recipients – in case the programme was randomly assigned to individuals in the population (Caliendo and Hujer, 2006)[10]. Since our analysis relies on the target population, conditioning upon D=1 in Equation (1) is irrelevant, as the control group coincides with the universe of the intended recipients of the programme. Thus, in this case, the effect of the WELL programme on a random individual who is exposed to the programme (the definition of ATT) is obtained by estimating the ATE for our sample.

The identification problem for the ATT is that (Y⁰|D=1) the potential outcome of the treatment group in the absence of treatment, cannot be observed. Therefore, the identification strategy boils down to finding a proper control group that mimics the counterfactual situation of the treatment group in the absence of the treatment. Once a suitable control group is available, the identification of the ATT amounts to a simple difference following Equation (1). The ATT amounts to comparing the average of the outcome variable (i.e. the employment status) between the treatment and control groups.

To identify the ATT, we rely on propensity score matching, which ensures that the outcomes of treated units are compared with similar control units. We define the following quantities: Y¹ is the potential outcome given the treatment, Y⁰ is the potential outcome in the absence of the treatment, D is an indicator equal to 1 if the individual receives the treatment and 0 otherwise, and X is a set of observable confounding characteristics that are correlated both with selection into the treatment and with the potential outcomes.

The identification of the ATT relies on the following assumptions:

• conditional independence assumption (CIA): (Y¹, Y⁰)D⊥, X;

• stable-unit-treatment-value assumption (SUTVA); and

• common support assumption: 0<P(D=1 ⊥ X=x)<1.

Where X is a set of observable confounding characteristics that are correlated both with the selection into the treatment and with the potential outcomes.

The first assumption implies that the treatment assignment is independent of the potential outcomes with and without treatment if certain observable covariates are held constant. More specifically, controlling for all observable characteristics X, the decision to participate is uncorrelated with potential outcomes. Hence, if treated and control units with the same values for these observable characteristics show systematic differences in outcomes, these are attributable to the treatment (Imbens, 2004). This assumption requires that all variables that need to be adjusted for are observed; therefore, the extent to which this assumption is reasonable depends on the richness of data available.

The second assumption rules out spill-over effects or general equilibrium effects that may be caused by the treatment. In the context of large-scale labour market policies, this statement may appear quite strong as it prevents crowding-out (displacement) effects in local labour markets: specifically, if treated units are more likely to find a job due to the intervention, this should not decrease the likelihood of control units finding a job. In addition, the intervention should not affect control units via changes in the general equilibrium of wages in the labour market. In our case, SUTVA is a reasonable assumption since the intervention target group is quite small (550 treatment units, compared to 5,266 control units) and thus unlikely to drive general equilibrium or displacement effects.

The last assumption implies that for any given value of the observable characteristics, X, the treatment assignment should not be certain. Therefore, for each value of the confounding variables X, an individual could potentially be observed as treated or not. This assumption ensures that for each treated individual (with given realisations of variables X), we can find a sufficiently similar individual in the control group, i.e., a control unit that is identical to the treated one in terms of variables X.

Basically, the purpose of the matching procedure is to estimate the ATT by comparing treated units with control units that are similar in terms of observable characteristics that affect both treatment participation and outcome variables. Ideally, we would like to compare the outcome value of a treated unit i with the outcome value of a control unit j that is identical to i in terms of a number of characteristics included in X.

Finding an exact match for each individual i becomes more difficult as the number of observable characteristics increases. Adjusting for a set of covariates, X, to eliminate confounding factors leads to the “curse of dimensionality” problem. However, it has been shown that adjusting only for the propensity score is also sufficient to eliminate confounding (Abadie and Imbens, 2016). Formally, the propensity score is the probability of being assigned to the treatment conditional on the observed characteristics (Rosenbaum and Rubin, 1983). It summarises all information contained in X and has to be estimated to assign a given value to each individual in the sample. The propensity score matching procedure involves comparing treated and control units with similar propensity scores. If the propensity score is correctly estimated, individuals with similar scores are also similar in terms of observable characteristics. This also means that one is comparing treated and control units that are similar in terms of potential counterfactual outcomes. This derives from the CIA assumption, replacing X with the propensity score P(X), as shown below: (Y¹, Y⁰) ⊥ D|P(X). The selection process for treatment models the probability of being treated as a function of the aforementioned covariates, as follows:

(2) P ( D = 1 ) = f ( X + ϵ ) .

The propensity score is a function of individual characteristics such as age, gender and all possible observable characteristics. In our context, we also include available educational variables regarding the field of study (e.g. science, education, etc.) and the degree completed (bachelor’s or master’s). These variables are relevant for explaining one’s labour market status after participating in the intervention, as this depends on regional labour demands in terms of educational background. Similarly, individuals with specific educational profiles may be more or less likely to find a firm in which to carry out a traineeship and therefore have higher or lower chances of participating in the programme. Furthermore, we include the municipality of residence and the location of the unemployment office where the individual was registered as unemployed.

This is meant to be a proxy for the local labour market where the individual is looking for a job (or a traineeship through which to participate in the WELL intervention). This is relevant for explaining both the outcomes and the decision to participate.

Finally, we further enrich the set of covariates in the propensity score specification by adding information on the past labour market experiences of individuals.

Gathering information on work patterns prior to participation in the WELL programme is important in order to reduce the bias of self-selection into the programme (e.g. more motivated individuals may be more likely to participate, depending on the quality of the programme), as an individual’s previous work experience can be used as a proxy for that worker’s skills and competences. To this end, we incorporated into the analysis data on the labour market status and the type of contract of WELL participants and non-participants (control group) at fixed dates, namely, on 31 December 2012, 30 June 2012, 31 December 2011 and 30 June 2011. These data were extracted from the CCD.

Following convention, the propensity score is calculated through maximum likelihood estimation (Caliendo and Kopeinig, 2008). The results of the analysis are discussed in Section 7.

7. Results

Table XI reports our baseline results. As previously discussed, the objective of the intervention is to increase the employability of jobseekers in the region of Umbria. Therefore, our outcomes refer to employment status and the type of contract that an individual has if employed in Umbria.

The first row reports the coefficients from a naive linear regression where each outcome is regressed on the covariates on the right-hand side of Equation (2) and the indicator for participation in the intervention D, as in the following equation:

(3) y i = α + β 1 D + β 2 T X i + ϵ i .

The estimated coefficient β₁ amounts to comparing the average for the outcome variable (e.g. employment status) in the treatment and control groups, controlling for a wide set of individual characteristics and past labour market outcomes. The ordinary least squares estimator (OLS) provides an unbiased estimate of the treatment effect (β₁ in Equation (3)) under two assumptions: the CIA, i.e., the treatment indicator is exogenous, controlling for the covariates X in Equation (3); the functional form assumption, i.e., the true conditional expectations of the outcomes are linear so that the linear regression function provides a good approximation of the true conditional expectations (Imbens, 2015). Hence, linear regression provides biased estimates of the treatment effect if the conditional expectations are not linear and if covariate distributions are different in the treatment and control groups.

A problem with this method is that results are very much affected by observations with extreme values in the covariates. Outliers are precisely the units that are not appropriate as counterfactual images of the treated units. While it is difficult to assess if true conditional expectations are linear, so as to justify the use of linear regression, it is quite straightforward to check if the covariate distributions differ by treatment status.

The t-tests for the differences in the means of all covariates between the two groups are reported in Table V and show that the distributions of age, field of study, level of study, past labour market outcomes and residence are different between the treatment and control groups. This suggests that linear regressions provide biased estimations of the treatment effect as the results will be sensitive to outliers (which are not appropriate control units) and to the choice of the (linear) specification. For convenience, we estimate a linear probability model and the coefficients show a positive and statistically significant effect of participating in WELL on the probability of finding a job in December 2015. In particular, participating in WELL increases employment by 18 pp. Furthermore, it increases the probability of having a temporary or apprenticeship contract by 4 pp or 5.8 pp, respectively. Results are more conservative for permanent contracts as they are statistically significant at 10 per cent level.

We now turn to our matching estimates. In general, matching boils down to a number of non-parametric approaches (e.g. exact matching, propensity score matching, sub-classification) that apply the following solution: no functional forms are assumed, but weighting schemes are applied so as to make the covariate distributions in the treatment and control groups as similar as possible.

Figure 2 shows the distribution of the estimated propensity score in the treated and control groups. The estimated propensity score has good balancing properties if both groups are equally distributed along the propensity score. The figure indicates that there is a good overlap in the distributions of the estimated propensity scores in the treatment groups.

Figure 3 shows the overlap of the distribution of the propensity score across treatment and control groups for specific sub-groups in the population: for men, for women, for individuals with bachelor’s degree and for individuals with master’s degree. We believe that the extent of overlap in the distribution of the propensity score is overall satisfactory even when looking at specific categories.

In addition, the propensity score has to satisfy the balancing property. This means that the propensity score should be able to weight units to make the treated and the control group similar and hence comparable. This can be checked by testing that the means for the treated and the control groups in each covariate are not statistically different from each other after applying the matching procedure. If the propensity score effectively enhances the comparability of both groups, then there should be no difference in the means between treated and controls after matching in all covariates. This test allows us to evaluate the quality of the matching procedure.

The t-test statistics for the differences between means are reported in the last two columns of Table X. For each covariate, row “U” and “M” report the test computed before and after the matching, respectively. The column “% bias” reports the standardised percentage bias as computed in Rosenbaum and Rubin (1985)[11]. It is a standard measure to evaluate the quality of the matching procedure. As a rule of thumb, the bias after matching should be less than 5 per cent in each covariate.

The values in rows “U” (i.e. before matching) confirm that before the matching the two groups are unbalanced: for instance, the average age between treated and controls is statistically different. Rows “M” (i.e. after the matching) show instead that the propensity score successfully reduces the covariate imbalances: as for age, the means of treated and controls after the matching are much closer to each other and the per cent bias drops from 59 to 3 per cent. Overall, the matching fixes the balance for all covariates, except for the category related to engineering and manufacturing, the category related to bachelor’s degree and the category related to registering as unemployed in the PES offices of Cittaá di Castello, whose per cent bias remains slightly above 5 per cent (8.2, 5.8 and 8 per cent, respectively). However, it is worth noting that the per cent bias is only marginally above the recommended threshold.

The matching procedure[12] reports also the average per cent absolute bias as the overall measure of covariate imbalance, which according to the rule of thumb should not exceed 25 per cent. In our case, the average per cent absolute bias is 17 per cent. This suggests that our matching has good balancing properties.

The basic procedure consists of the following steps: first, we sort all units according to a propensity score that represents the likelihood of participating in the programme. Second, we compare the average outcomes of treated and control units with similar propensity scores, and finally, we average these differences out over the distribution of propensity scores, so as to estimate the ATT. The second and third rows of Table XI report the results from two different matching procedures. In both cases, we use matching with replacement, which means that each control unit may be used as a match more than once. This improves the comparability between treated and control units, thereby decreasing estimation bias. In the second row of Table XI, each treated unit is matched with the control unit with the closest propensity score, whereas in the third row of Table XI, each treated unit is matched with the five closest control units in terms of the propensity score. The choice of the number of control units to be used in each match (one in row 2 vs more than one in row 3) entails a trade-off between bias and variance. Increasing the number of control units to be assigned for each match tends to increase the bias (since each treated unit is compared with control units that may not be as close in terms of propensity score), but it also increases the precision of the estimate.

Overall, the comparison between the results in the first row and those in the second and third rows suggests that the matching procedure is somehow effective in reducing the bias. When applying nearest-neighbour estimators, the point estimates for the probability of being employed decrease from 0.18 to 0.11. Accounting for as many observable characteristics as can be found in the administrative data helps to reduce the differences between the treatment and control groups. As for the type of contract, we see that participating in the WELL programme has a significant positive effect of 3.8 pp on obtaining an apprenticeship contract, whereas the results are not statistically significant when looking at the probability of obtaining a work contract (temporary or permanent). This result is very much in line with the recent literature on unemployed youth, and especially the unemployed youth in Italy.

The success of the programme is not clear-cut. On the one hand, the employment prospects of participants seem to improve in the region of Umbria. However, this may conceal employment prospects outside of the Umbria region or in the realm of self-employment work arrangement, especially if for some reasons non-participants are more likely to move to other regions or to be self-employed. This is not necessarily a negative outcome, but with the data at hand we are not able to disentangle the effects of the programme outside the labour market of Umbria.

7.1 Heterogeneous effects

We now extend the analysis and provide separate analysis by age group, gender and field of study. As previously, we report the results from the regression analysis in the first row and the results from propensity score matching in the second and third rows. We estimate our model separately by age group considering three categories: below age 30 years, age 31–40 years and 41 years or more. Results are reported in Table XII. Two important results are worth mentioning. First, participants younger than 30 years are 21 pp more likely to be employed compared to the other two groups according to the OLS specification and about 18 pp based on the matching procedure[13]. Second, participants younger than 30 have a higher probability of being employed with a permanent contract (5 pp) or apprenticeship (8 pp) compared to participants aged between 31 and 40 years. A common result for the WELL participants in the age groups 31–40 years and more than 41 years is a higher probability of being unemployed compared to non-participants (respectively, 7 and 21 pp). As expected the programme had a bigger impact on younger participants (Table XIII).

We now turn to our gender-specific regressions (13). From the OLS estimates, we observe substantial differences between males and females in the probability of being employed in Umbria one to two years after the completion of the training. Participating in WELL increases employability by 15 pp for females (top panel) and by 26 pp for males (bottom panel). However, as discussed above, OLS estimates are less reliable compared to matching. In the matching estimates (second and third rows in both panels), this gap in employment rate persists. Participating in WELL increases females’ employability by 4 pp when considering 1 closest neighbour (although the estimate is not statistically different from 0), whereas for males, employability increases by 21 pp, which is statistically different from 0 at the 1 per cent level. We obtain similar results when considering matching with the five closest neighbours. Males participating in WELL also show a higher probability of having a permanent or apprenticeship contract at the end of the training (13 pp or 8 pp, respectively). For females, the magnitude of the coefficients is much lower and the estimates are not statistically significant. One possible explanation for these differences relates to individuals’ field of study. To tackle this issue, Table XIV reports separate estimates by field of study. In order to obtain more robust estimates, we group fields of study into three categories: humanities, which includes “Education”, “Arts and Humanities”, and “Social sciences, Business and Law”; science, which includes “Science” and “Engineering, Manufacturing and Construction”; other, comprised of those graduating in “Agriculture” and “Health”, of which there are very few. We report the results for the “Humanities” (top panel) and “Science”. As expected, we observe large differences in terms of labour market participation. OLS estimates (first row of each panel) show that WELL participants have a higher probability of being employed than non-participants, but the advantage in terms of employability is higher for those graduating in “Science” (25 pp, compared to 14 pp for those graduating in the “Humanities”). This gap becomes even greater when estimating the model through propensity score matching. Graduating in “Science” increases the probability of being employed in the region of Umbria by 18–20 pp (depending on the nearest-neighbour specification), compared to 8 pp for those coming from the “Humanities”. All estimated coefficients are statistically significant at the 1 per cent level. The same pattern is observed when analysing the impact of WELL on the type of contract, although the results are less compelling. Based on OLS results, WELL participants graduating in “Science” have a higher probability of obtaining a permanent job (by 8 pp), but this difference disappears when estimating the model via propensity score matching. However, the difference in terms of obtaining an apprenticeship still remains, benefiting most WELL participants graduating in “Science” (5–7 pp compared to 3–4 pp for those graduating in the “Humanities”). Different mechanisms could be in place. First, our results may be explained by an excess of supply with respect to demand for the type of skills acquired in the “Humanities” field of study. A second possible interpretation could be that for those graduating in humanities work experience might be less important, and they can re-enter the labour market with relatively lower costs due to the specific skills acquired. Unfortunately with the data at hand, we are not able to disentangle between the two mechanisms, but overall our results are in line with the evidence from other OECD countries, which shows that tertiary-educated adults with a degree in science, technology, engineering and mathematics benefit from higher employment rates than their peers[14].

8. Conclusions

In this paper, we analyse the impact of an intervention aimed at increasing the employability of graduates in the small Italian region of Umbria. The intervention subsidised “on-the-job training” for unemployed graduates and had two main aims: increase employment among unemployed college graduates; enhance the capacity and productivity of the participating firms. To evaluate the effectiveness of the intervention, we look at the employment status of participants and similar non-participants in Umbria in 2015 (from one to two years after the intervention). The evaluation exercise is performed using different sources of administrative data. More specifically, information about participants in the WELL intervention was provided by the Regional Monitoring System Database and information on the control group was gathered through the CCD. To estimate the causal effect of the intervention on the labour market career of participants, i.e., the ATT, we employ a propensity score matching technique. Baseline results suggest that WELL participants are more likely to be employed in Umbria, compared to similar non-participants. They also show a higher probability of obtaining an apprenticeship contract after completing the training. Furthermore, we find substantial differences by gender, with males having a higher probability of being employed with a permanent contract or an apprenticeship contract. Heterogeneous effects by field of study suggest more positive employment responses for individuals with a degree in science. In contrast, individuals with a degree in the humanities benefit somewhat less from participating in the programme. A plausible explanation for this differential impact could be due to the excess of supply of individuals with a degree in the humanities compared to demand. Finally, the aforementioned gender difference in the employment responses to the programme may be explained by the fact that men are more likely to obtain a science degree, when compared to women. However, participants and non-participants seem to be equally likely to be registered as unemployed in Umbria, which indicates that the final conclusions apply only to the Umbria region.

Figures

Figure 1

Percentage of employed WELL participants vs non-participants in 2015 by municipality

Figure 2

Distribution of the propensity scores

Figure 3

Distribution of the propensity scores by gender and by level of study

Table I

Labour market status of WELL participants and non-participants in 2013 by gender (per cent)

	WELL			No WELL
LM status 2013	Male	Female	Total	Male	Female	Total
Unemployed	31.4	68.6	100	27.55	72.45	100
Unemployed first entry	29.95	70.05	100	33.46	66.54	100
Total	30.91	69.09	100	28.73	71.27	100
No. of obs.	170	380	550	1,513	3,753	5,266

Table II

Labour market status of WELL participants and non-participants in 2013 by age group (per cent)

	WELL			No WELL
Age group	Unempl.	Unempl. (first entry)	Total	Unempl.	Unempl. (first entry)	Total
0–24	6.89	7.49	7.09	2.33	14.41	4.75
25–29	42.42	51.87	45.64	19.33	49.76	25.43
30–35	31.13	26.74	29.64	33.1	23.03	31.09
35–40	12.12	10.16	11.45	20.47	5.69	17.51
>40	7.44	3.74	6.18	24.77	7.11	21.23
Total	363	187	550	4,211	1,055	5,266
No. of obs.	100	100	100	100	100	100

Table III

Educational level of WELL participants and non-participants in 2013 by gender (per cent)

	WELL			No WELL
Edu. level	Male	Female	Total	Male	Female	Total
Bachelor degree	33.53	32.63	32.91	27.3	19.21	21.53
Master degree	66.47	67.37	67.09	72.7	80.79	78.47
Total	100	100	100	100	100	100
No. of obs.	170	380	550	1,513	3,753	5,266

Table IV

Field of study of WELL participants and non-participants in 2013 by gender (per cent)

	WELL			No WELL
Field of study	Male	Female	Total	Male	Female	Total
Education	3.53	10.26	8.18	2.91	10.84	8.56
Humanities and arts	17.65	26.58	23.82	17.32	30.72	26.87
Social sciences, business and law	35.29	41.32	39.45	33.77	32.59	32.93
Science	8.24	8.42	8.36	12.16	11.06	11.37
Engineering, manufacturing and construction	24.71	7.89	13.09	23.27	7.57	12.08
Agricultural	4.12	2.11	2.73	5.09	2.42	3.19
Health	6.47	3.42	4.36	5.49	4.8	4.99
Total	100	100	100	100	100	100
No. of obs.	170	380	550	1,513	3,753	5,266

Table V

Descriptive statistics of covariates for WELL participants and non-participants

	WELL		No WELL			t-test
Variable	Mean	SD	Mean	SD	Diff.	t	p-val.
Female	0.69	0.46	0.71	0.45	−0.02	−1.07	0.28
Age_group: 0–24	0.07	0.26	0.05	0.21	0.02	2.41	0.02
Age_group: 25–29	0.46	0.50	0.25	0.44	0.20	10.21	0.00
Age_group: 30–35	0.30	0.46	0.31	0.46	−0.01	−0.70	0.48
Age_group: 35–40	0.11	0.32	0.18	0.38	−0.06	−3.61	0.00
Age_group: >40	0.06	0.24	0.21	0.41	−0.15	−8.48	0.00
Field of study: education	0.08	0.27	0.09	0.28	0.00	−0.31	0.76
Field of study: humanities and arts	0.24	0.43	0.27	0.44	−0.03	−1.54	0.12
Field of study: social sciences, business and law	0.39	0.49	0.33	0.47	0.07	3.09	0.00
Field of study: science	0.08	0.28	0.11	0.32	−0.03	−2.14	0.03
Field of study: engineering, manufacturing and construction	0.13	0.34	0.12	0.33	0.01	0.69	0.49
Field of study: agricultural	0.03	0.16	0.03	0.18	0.00	−0.59	0.55
Field of study: health	0.04	0.20	0.05	0.22	−0.01	−0.65	0.52
Level of study: bachelor’s degree	0.33	0.47	0.22	0.41	0.11	6.09	0.00
Level of study: master’s degree	0.67	0.47	0.78	0.41	−0.11	−6.09	0.00
PES: Perugia	0.57	0.50	0.50	0.50	0.07	2.95	0.00
PES: Cittá di Castello	0.10	0.30	0.09	0.29	0.01	0.82	0.41
PES: Foligno	0.15	0.36	0.15	0.36	0.00	0.20	0.84
PES: Terni	0.16	0.37	0.22	0.42	−0.06	−3.36	0.00
PES: Orvieto	0.02	0.13	0.03	0.18	−0.02	−2.24	0.03
Prov: Perugia	0.81	0.39	0.74	0.44	0.07	3.46	0.00
Prov: Terni	0.19	0.39	0.26	0.44	−0.07	−3.46	0.00
Nr of past unemployment spells	1.79	1.64	1.93	1.48	−0.13	−1.96	0.05
Nr of past employment spells	0.75	1.23	1.17	1.43	−0.41	−6.52	0.00
Nr of past spells in training	0.06	0.27	0.04	0.21	0.02	2.50	0.01
Nr of past inactivity spells	0.63	1.23	0.35	0.90	0.28	6.58	0.00
Obs.	550		5,266		5,816

Notes: This table tests for each covariate whether the means for the treated (WELL) and for the controls (No WELL) are statistically different between each other. The column “Diff” shows the difference between the mean for the treated and of the controls. The t-test is computed based on a regression of each covariate on the treatment indicator. Column t reports the t-statistic of the estimated coefficient and the column p-val. reports the corresponding p-value

Table VI

Outcome variables for WELL participants and non-participants in 2015

	WELL		No WELL		t-test
	Mean	SD	Mean	SD	Difference	p-val.
Employment indicator in Umbria	0.52	0.50	0.37	0.48	0.16	0.00
Unemployment indicator in Umbria	0.25	0.43	0.22	0.41	0.04	0.06
Permanent contract	0.16	0.37	0.15	0.36	0.01	0.53
Temporary contract	0.17	0.38	0.13	0.34	0.04	0.02
Apprenticeship contract	0.09	0.29	0.02	0.15	0.07	0.00
Obs.	550		5,266		5,816

Table VII

Distribution of WELL participants, by firm size

Firm size (nr of employees)	Freq.	%	Cum.
1–9	297	54.00	54.00
10–49	181	32.91	86.91
50–249	60	10.91	97.82
250+	12	2.18	100.00
Total	550	100.00

Table VIII

Average firm size by type of sector

Type of industry	Average firm size
Accommodation and food service	18
Activities of households	9
Administrative, support service	102
Agriculture, forestry and fishing	22
Arts and recreation	27
Construction	5
Education	16
Electricity, gas supply	7
Financial and insurance	3
Information and communication	8
Manufacturing	47
Other services	49
Professional activities	21
Public administration	13
Real estate	8
Transporting and storage	48
Health and social work	76
Retail trade; repair of vehicles	81
Missing	29

Table IX

Distribution of WELL participants by type of contract in phase 2 and firm size

Type of contract
Firm size	Permanent	Temporary	Apprenticeship	Total
1–9	8	22	21	51
%	15.69	43.14	41.18	100.00
10–49	5	15	6	26
%	19.23	57.69	23.08	100.00
50–249	0	13	4	17
%	0.00	76.47	23.53	100.00
250+	0	0	2	2
%	0.00	0.00	100.00	100.00
Total	13	50	33	96
	13.54	52.08	34.38	100.00

Table X

Descriptive statistics of covariates for WELL participants and non-participants, after matching

	Unmatched	Mean		t-test
Variable	Matched	Treated	Control	% bias	t	p. val.
Age	U	30.62	34.79	−59.20	−11.53	0.00
	M	30.62	30.83	−3.00	−0.61	0.54
Female	U	0.69	0.71	−4.80	−1.07	0.28
	M	0.69	0.68	2.00	0.32	0.75
Field of study: education	U	0.08	0.09	−1.40	−0.31	0.76
	M	0.08	0.07	3.30	0.56	0.57
Field of study: humanities and arts	U	0.24	0.27	−7.00	−1.54	0.12
	M	0.24	0.22	3.30	0.57	0.57
Field of study: science	U	0.08	0.11	−10.10	−2.14	0.03
	M	0.08	0.08	1.80	0.33	0.74
Field of study: engineering, manufacturing	U	0.13	0.12	3.10	0.69	0.49
	M	0.13	0.16	−8.20	−1.29	0.20
Field of study: agricultural	U	0.03	0.03	−2.70	−0.59	0.55
	M	0.03	0.03	1.10	0.19	0.85
Field of study: health	U	0.04	0.05	−3.00	−0.65	0.52
	M	0.04	0.04	0.90	0.15	0.88
Level of study: bachelor degree	U	0.33	0.22	25.80	6.09	0.00
	M	0.33	0.30	5.80	0.91	0.37
PES: Cittá di Castello	U	0.10	0.09	3.60	0.82	0.41
	M	0.10	0.08	8.00	1.37	0.17
PES: Foligno	U	0.15	0.15	0.90	0.20	0.84
	M	0.15	0.17	−3.50	−0.57	0.57
PES: Terni	U	0.16	0.22	−15.80	−3.36	0.00
	M	0.16	0.14	4.60	0.84	0.40
PES: Orvieto	U	0.02	0.03	−11.40	−2.24	0.03
	M	0.02	0.01	4.60	1.08	0.28
Nr of unemployment spells	U	1.79	1.93	−8.40	−1.96	0.05
	M	1.79	1.87	−4.70	−0.76	0.45
Nr of inactivity spells	U	0.63	0.35	25.60	6.58	0.00
	M	0.63	0.61	1.90	0.27	0.79

Notes: This table reports t-test statistics for the differences between means for each covariate (the last two columns report the t-statistic and the corresponding p-value). The column “% bias” reports the standardised percentage bias as computed in Rosenbaum and Rubin (1985). Rows “U” and “M” report these statistics computed before and after the matching, respectively

Table XI

Effect of WELL on employment status and type of contract

ATT	Employed	Unemployed	Permanent	Temporary	Apprenticeship
Linear regression	0.1754*** (0.0231)	0.0166 (0.0199)	0.0320* (0.0170)	0.0408** (0.0169)	0.0576*** (0.0123)
NN matching (n=1)	0.1069*** (0.0354)	0.0757** (0.0300)	0.0324 (0.0312)	0.0027 (0.0170)	0.0345*** (0.0102)
NN matching (n=5)	0.1150*** (0.0271)	0.0699** (0.0302)	0.0086 (0.0176)	0.0133 (0.0164)	0.0386*** (0.0096)
Observations	5,816	5,816	5,816	5,816	5,816

Notes: This table reports results from linear regression models (first row) and propensity score matching (second and third rows). We use nearest-neighbour matching with replacement; in the second (third) row each treated unit is matched with one (five) control(s). We consider indicators of labour market participation in the region of Umbria in December 2015 (being employed, unemployed, columns 1–2), and the type of contract for employed individuals (columns 3–5). We also account for past labour market experience. Standard errors in parentheses: for matching, robust Abadie–Imbens standard errors, heteroskedastic-robust standard errors otherwise. The specification used for the estimation is as follows: age, gender, field of study categories (social sciences, business and law is omitted as reference), level of study (master’s degree is omitted as reference), dummies for the city in which individuals registered at the Public Employment Office (Perugia is omitted as reference), the number of past unemployment spells and the number of past spells in inactivity. This is the specification which gives the best balancing properties of the estimated propensity score. *0.05<p⩽0.10; **0.01<p⩽0.05; ***p⩽0.01

Table XII

Differences by age group

ATT	Employed	Unemployed	Permanent	Temporary	Apprenticeship
Age<30
Linear regression	0.2119*** (0.0298)	−0.0391 (0.0256)	0.0548*** (0.0212)	0.0324 (0.0210)	0.1014*** (0.0205)
NN matching (n=1)	0.1759*** (0.0356)	−0.0119 (0.0352)	0.0529** (0.0265)	0.0159 (0.0223)	0.0856*** (0.0224)
NN matching (n=5)	0.0002 (0.0861)	0.2113** (0.0882)	−0.0488 (0.0593)	−0.0024 (0.0573)	−0.0033 (0.0022)
Age 31–40
Linear regression	0.1603*** (0.0393)	0.0767** (0.0332)	0.0262 (0.0313)	0.0704** (0.0313)	−0.0088*** (0.0021)
NN matching (n=1)	0.1133** (0.0452)	0.0700* (0.0368)	−0.0148 (0.0348)	0.0408 (0.0387)	−0.0102*** (0.0023)
NN matching (n=5)	0.0124 (0.0729)	0.2673*** (0.0724)	−0.0724*** (0.0084)	−0.0357*** (0.0095)	−0.0053** (0.0022)
Age>40
Linear regression	0.0002 (0.0861)	0.2113** (0.0882)	−0.0488 (0.0593)	−0.0024 (0.0573)	−0.0033 (0.0022)
NN matching (n=1)	−0.0203 (0.0843)	0.2424*** (0.0848)	−0.0576 (0.0577)	0.0064 (0.0582)	−0.0053** (0.0022)
NN matching (n=5)	−0.0393 (0.0371)	0.2117*** (0.0488)	−0.0663*** (0.0127)	−0.0096 (0.0199)	−0.0053** (0.0022)

Notes: The notes under this table are the same as those under Table XI. *p⩽0.10; **p⩽0.05; ***p⩽0.01

Table XIII

Differences by gender

ATT	Employed	Unemployed	Permanent	Temporary	Apprenticeship
Females
Linear regression	0.1454*** (0.0280)	0.0286 (0.0243)	0.0113 (0.0197)	0.0419** (0.0203)	0.0438*** (0.0140)
NN matching (n=1)	0.0498 (0.0354)	0.1409** (0.0598)	−0.0178 (0.0257)	0.0213 (0.0233)	0.0211** (0.0097)
NN matching (n=5)	0.0547 (0.0368)	0.1110*** (0.0366)	−0.0199 (0.0232)	0.0078 (0.0197)	0.0205** (0.0086)
Males
Linear regression	0.2575*** (0.0408)	−0.0078 (0.0351)	0.0925*** (0.0325)	0.0430 (0.0302)	0.0868*** (0.0245)
NN matching (n=1)	0.2136*** (0.0544)	0.0558 (0.0595)	0.1265** (0.0516)	0.0181 (0.0279)	0.0769*** (0.0233)
NN matching (n=5)	0.1935*** (0.0344)	0.0943** (0.0378)	0.0803*** (0.0267)	0.0201 (0.0286)	0.0671*** (0.0200)

Notes: The notes under this table are the same as those under Table XI. *p⩽0.10; **p⩽0.05; ***p⩽0.01

Table XIV

Differences by field of study

ATT	Employed	Unemployed	Permanent	Temporary	Apprenticeship
Humanities
Linear regression	0.1435*** (0.0274)	0.0047 (0.0240)	0.0283 (0.0196)	0.0271 (0.0204)	0.0473*** (0.0136)
NN matching (n=1)	0.0686** (0.0324)	0.0510 (0.0360)	0.0115 (0.0253)	−0.0110 (0.0233)	0.0322*** (0.0112)
NN matching (n=5)	0.0666* (0.0356)	0.0763* (0.0403)	0.0218 (0.0303)	−0.0011 (0.0193)	0.0244*** (0.0086)
Science
Linear regression	0.2502*** (0.0484)	0.0331 (0.0407)	0.0823** (0.0386)	0.0264 (0.0347)	0.0924*** (0.0312)
NN matching (n=1)	0.1755* (0.0937)	0.1076 (0.0898)	0.0223 (0.0394)	0.0436 (0.0405)	0.0682** (0.0271)
NN matching (n=5)	0.1996*** (0.0676)	0.1015 (0.0664)	0.0128 (0.0353)	0.0508 (0.0380)	0.0499*** (0.0187)

Notes: The notes under this table are the same as those under Table XI. *p⩽0.10; **p⩽0.05; ***p⩽0.01

Notes

1. The Good School reform introduced, among other things, compulsory work-related learning in the last two years of higher secondary school, with the aim of facilitating the transition from school to work.

2. It was implemented under the ESF Objective 2 2007–2013 Programming Period and within the Annual Regional Plan for Interventions in Support of Work.

3. In addition, the training experience had to be coherent with the activities and work organisation of the hosting firm, and the list of activities to be undertaken by the trainee had to reflect the knowledge, professional skills, and educational qualification of the trainee.

4. In addition, employers had to be in compliance with workplace security and safety procedures, and to have established specific procedures for employing persons with disabilities (Law No. 68 of 1999).

5. In addition, the companies must not have benefited from the wage guarantee scheme (cassa integrazione guadagni) in the previous year.

6. This information system was introduced on 27 December 2006 in Law No. 296, Art.1, paragraphs 1,180–1,185, which laid down the financial law for 2007. On 29 November 2016, the online ANPAL portal has been activated, following Law 14 September 2015, No. 150.

7. We drop 157 non-participants with a PhD degree. We do this because none of the participants have a PhD degree that means that these control units cannot be exploited in the matching exercise. For the same reason, we drop 24 non-participants whose field of study belongs to the category “Services”, since none of the participants belongs to this field of study. We drop few observations with missing values in the field of study: 561 non-participants and 4 participants. Finally, we drop few individuals who in 2015 are doing an internship: 72 non-participants and 20 participants. In principle, internship would be an interesting outcome. However, this category is too small to be considered as a separated outcome variable. Therefore, we ignore it.

8. Examples of precarious contracts are: contracts for continuative and coordinated services (Co.co.co), intermittent work, domestic work, ad interim employment. Note, the employment rate is one also if one is hired with a precarious job, because these individuals are in fact employed. The employment rate gives a rough measure of the success of the programme, while results by type of contract are intended to give more insight into the quality of the job obtained thanks to the programme.

9. This variable is affected by a large share of missing values, about 48 per cent, so results should be taken with care.

10. As explained in Heckman (1997), the difference between the two parameters (ATT and ATE) relies on the counterfactual group they use: in the ATT, the counterfactual is constructed with intended recipients (i.e. individuals who are eligible to the programme but that have not been treated (yet)). In contrast, the counterfactual group for the ATE comprises all individuals in the population who did not receive the treatment, i.e., both intended recipients and those for whom the programme was never intended.

11. This is the percentage difference of the sample means in the treated and non-treated (full or matched) sub-samples as a percentage of the square root of the average of the sample variances in the treated and non-treated groups (Rosenbaum and Rubin, 1985).

12. The matching procedure was performed using the Stata packages pstest and psmatch2.

13. When we separate the analysis by age, it becomes more difficult to obtain good matches for participants older than 40 years due to a lower sample size, so the results are less robust compared to the other two groups.

14. For more details, see “Education at a Glance 2017: OECD indicators”.