Does onthe-job training help graduates find a job ? Evidence from an Italian region . ∗

This paper provides an evaluation of a training programme for graduates entitled “Work Experience for Graduates” (WELL Work Experience Laureati and Laureate) that was recently implemented in Italy. The aim of the programme was to increase the career prospects of unemployed graduates in the region of Umbria. It consisted of two measures: (i) on-thejob training for unemployed graduates, and (ii) wage subsidies to firms and organisations for hiring the trainees at the end of the programme. We rely on administrative data and matching methods to evaluate the effectiveness of the intervention in terms of the employability of participants. Results indicate that WELL participants are more likely to be employed and to sign an apprenticeship contract within the region. We also find substantial gender differences in employability and the type of contract obtained, with men having a higher probability of finding a job (permanent contract or apprenticeship). We show that this may be explained by different choices in terms of field of study, with males being more prone to enrolling in scientific areas and females in the humanities.


Introduction
The effectiveness of labour market policies has been extensively debated in the recent literature.However, no consensus has been reached regarding the impact of some types of policies, such as those supporting the implementation of training programmes, since the effectiveness of these programmes strongly depends on the time horizon over which the employment effects are measured.As summarised by Card et al. (2010Card et al. ( , 2017) ) in their meta-analysis, training programmes show larger average effects in the medium and long terms.This also applies to private sector incentive programmes, and the reason for this lies behind the so-called lock-in effects.As discussed by Ham and Lalonde (1996) among others, participants in training and private sector employment programmes often reduce or suspend their normal job search efforts while participating in these programmes, drastically reducing their employment opportunities in the short term.
The remainder of this paper is organised as follows: Section 2 reviews the related literature.Section 3 provides a description of the intervention and the selection procedure.In Sections 4 and 5, we present the data and the descriptive statistics for the sample used in the analysis.Section 6 explains the empirical strategy implemented to quantify the impact of the WELL intervention and presents the main results.Finally, in Sections 7 and 8, we discuss the results and conclusions of the evaluation.

Literature review
Empirical evidence on the effectiveness of training programmes across European countries tends to be mixed.Germany is the European country with the most solid experience in terms of the evaluation of training programmes, which can partly be explained by the interest in evaluating the Hartz reforms implemented in the early 2000s to tackle high unemployment rates in the country.Evaluations of various training programmes show both positive and non-significant effects on employability and wages in the medium and long terms.Kopf (2009) and Rinne et al. (2011) consider both short-and medium-term programmes, finding that public training programmes positively affect employment prospects independently of the skills and age groups of participants, whereas Lechner et al. (2011) find negative effects for all types of training programmes in the short term, but positive effects on employment rates and earnings in the long term (after about two to four years).Finally, Schwerdt et al. (2012) and Górlitz and Tamm (2016) find no significant effects of training programmes on both wages and employability, even in the medium term.
These findings are also confirmed by the evaluations of programmes and similar reforms in France, Belgium, the Netherlands and Denmark.Chéron et al. (2010) estimate the impact of firmprovided training on labour market mobility in France, showing that participation in a training programme reduces the probability of switching jobs or becoming unemployed in the two subsequent years.Cockx and Van Belle (2019) arrive at similar conclusions, finding small positive effects for training and counselling measures devoted to recent graduates in Belgium.Hidalgo et al. (2014) show that in the Netherlands training vouchers on low-skilled workers do not have any significant impact on monthly wages or on job mobility but have a significant impact on future training plans.Danish job training programmes are analysed by Jespersen et al. (2008), who assess the long-term employment and earning effects on participants in both the private and the public sectors.They highlight that the positive post-programme effects on earnings eventually materialise after one to three years.Focusing on youth unemployment and providing a survey of the recent evidence from European countries, Caliendo and Schmidl (2016) confirm mixed effects of labour market training, depending on the type and on the country.
The evidence on the effectiveness of training programmes in Italy is scarce.However, two regional programmes implemented in Italy have been recently analysed.The first study evaluates the "Paid Quality Traineeships" programme within the Giovanisí project that was launched in the Italian region of Tuscany in 2011 (Sciclone et al., 2017).The programme is similar in terms of design to the Umbria programme but has a slightly different target group.In fact, it targets youth aged 18-29 years, who are either graduates or drop-outs in the previous two years, and resident in specific municipalities of Tuscany (about 74 per cent of municipalities have been included).These traineeships have a maximum duration of six months and have an educational scope, as firms that hire the trainees need to assign them a tutor.The authors combine administrative data for both the participant and non-participant groups granted by the Tuscany Region.They use propensity score matching to estimate the impact of the programme on youth employability two years after the completion of the training.Results show the positive effects of participation in the training programme on employment (especially for young people with no working experience), within 18 months from the start of the traineeship.Nevertheless the study has some caveats.One of the main limitations is that the authors are not able to match individuals on the basis of education, which is an important variable to take into account when 502 IJM 40,3 looking at young people without any working experience.Our analysis overcomes this difficulty and allows us to account for both educational level and field of study.
The second study by Pastore and Pompili (2019) concerns the evaluation of an integrated plan of employment and labour policies (PIPOL), launched by the Italian region of Friuli-Venezia-Giulia in 2014.PIPOL includes a series of programmes financed by the ESF, with the aim of improving the transition process of the youth from the world of school and university to that of work.In particular, the authors evaluate the first phase of PIPOL (2014)(2015)(2016)(2017)(2018) and focus on programmes related to extra-curricular training and internships.They employ counterfactual impact evaluation methods (propensity score matching) and also a more qualitative approach to evaluate the impact of PIPOL on job placement for different participants' characteristics (gender, age, nationality, etc.).The main findings are that the training programme seems to be successful among women, youth younger than 30 years and immigrants.A common result with this study is that we also find that the WELL programme was successful for those younger than 30 years, but different from the study we find men to be more successful in terms of labour market outcomes compared to women.
More generally, the peculiarities of the Italian labour market have recently been investigated to assess the impact of specific measures: the so-called 2015 Good School reform [1], and the 2003 reform of apprenticeship contracts.Picchio and Staffolani (2019) investigate the impact on job opportunities resulting from firm-provided training through apprenticeship contracts.Compared to other types of temporary contracts, apprenticeships are found to be more effective in leading workers to a stable job relationship, especially within the same firm where the apprenticeship was undertaken.Investigating the same topic, Albanese et al. (2017) focus on the impact of the 2003 reform of the Italian apprenticeship contract, showing the positive effect of the new apprenticeship on the transition to permanent employment, mostly in large firms.
Regarding the Good School reform, Pastore (2018b) highlights that the elements that make the transition from school to work problematic stem from the low level of both secondary and tertiary education attainments and the rigidities of the education system (especially tertiary education), which might delay entry into the labour market from university.The Italian institutional setting of education, training and welfare systems was also analysed in a cross-country comparison, to document how it affects the youth disadvantage in the labour market (Pastore, 2015(Pastore, , 2018a, b), b).In particular, the author discusses that poor performances of youth labour market participation and youth unemployment may depend on the specific school-to-work transition regimes and on poor levels of integration between the education systems and the labour market.
The employability of Italian graduates has been analysed from various perspectives.Ballarino and Bratti (2009) assess if, and to what extent, different fields of study affect graduates' chances of employment.Taking into account the increasing flexibility of the Italian labour market following the recent reforms, they find that scientific fields, as compared to the humanities, consistently guarantee a higher probability of stable jobs in the university-to-work transition.Brunello and Cappellari (2008) study the importance of the attended college on the earning and employability prospects of students three years after graduation.They conclude that the choice of college has an impact on employment, especially in the short term.
Finally, Di Pietro and Urwin (2006) assess the effect of over-education on the earnings of Italian graduates.They show that over-educated graduates receive lower wages than peers with a similar level of education who do not experience education mismatch in their jobs.Interestingly, this effect does not depend on the under usage of skills but rather on human capital and job features.
In the same vein, Aina and Pastore (2012) show that delayed graduation increases the chances of over-education, thereby contributing to the wage penalty faced by over-educated workers.

On-the-job training
Despite growing interest in the analysis of the employment prospects of tertiary graduates, little evidence is available on the active labour market policies targeting graduates.

Description of the WELL programme
The WELL intervention was financed through the ESF and implemented in 2013 as part of the Regional Operational Programme of the Umbria Region [2].Its main goal was to reduce unemployment among the youth and to strengthen the professional qualifications of graduates by raising the quality of their jobs.
More specifically, it was designed to promote fully subsidised work experience, with the aim of increasing employment among highly educated individuals, who have a higher risk of exclusion from the labour market.The programme also included the provision of wage subsidies to employers who hired WELL participants at the end of the training.The programme also aimed to promote the dissemination of modern and efficient production processes and to increase the innovative capacity and productivity of the participating firms.
The intervention was therefore characterised by a strictly integrated path consisting of two steps: on-the-job training for unemployed graduates; and wage subsidy to firms and organisations that hire the trainee after completion of the training.The training was expected to last six months, with a minimum weekly commitment of 24 h and a gross monthly salary of 800 euros per trainee [3].The subsidy awarded to firms and organisations varied depending on the type of contract offered.For a trainee hired with a fixed-term contract of at least six months, the subsidy was 2,500 euros, whereas for an apprenticeship contract, it amounted to 4,000 euros.For an open-ended contract, a subsidy of 6,500 euros was offered.
The intervention's beneficiaries were unemployed people, including first-time jobseekers, who held a bachelor's or master's degree and resided in Umbria at the time of programme's launch (May 2013).Participants' unemployment status had to be demonstrated through registration in public employment offices.Companies and organisations involved in the intervention, such as associations, foundations, cooperatives and consortia, were required to have at least one production/work unit in Umbria and to have at least two permanent employees [4].
To avoid the possibility that the intervention could produce displacement effects, companies applying for the intervention had to show no dismissals of workers with similar occupational tasks to the ones they were hiring for in the year preceding the traineeship [5].
The WELL project was launched in May 2013 and was completed in September 2014.A shortlist of around 100 eligible firms in Umbria was published in order to encourage participation in the intervention, and participants also used their personal networks to find potential workplaces in which to complete a traineeship.Applications were examined by the regional Department of Labour Policies and eligible applications were ranked according to the following criteria: (1) The commitment of the host company to employ the trainee at the end of the traineeship, depending on the type of job contract: • open-ended contract (full-time and part-time) (5 points); • fixed-term contract, lasting at least six months (2 points); and • other types of contract (1 point).
(2) Applicant with a disability under the rules of Italian national law 68/1999 (1 point).
(3) Applicant's age: (4) Innovative activities of the host organisation, defined as participation in regional/national poles or clusters, or ministerial research laboratories (2 points).
In the case of equal scores, the ranking was determined according to the chronological order of submission of the electronic application.For both of the intervention phases, a quota for female applicants was put in place, accounting for 50 per cent of the initial amount of funding for the intervention (€1.2m).The intervention was very successful in terms of participation.In fact, the number of applications received exceeded the expectations of the implementing authority.To meet this unforeseen demand, the budget for the programme was increased to €3.6m and all eligible applications were admitted.Consequently, the quota for women proved to be unnecessary.
The number of applications received for the WELL programme totalled 712, 30 of which were ineligible to participate.Of the 682 eligible applicants, 74 decided not to begin the traineeship and 34 dropped out during training.A total of 574 graduates successfully completed the work experience.The occurrence of drop-outs could be due to the administrative burden related to the framing of the traineeship period.As for the second phase in the intervention, grant subsidies were allocated to 96 companies and host organisations that recruited 98 trainees who had successfully completed the first phase.Of these, 13 workers were employed with a permanent contract, 51 were hired with a temporary contract and 34 were employed with an apprenticeship contract.In the next session, we also provide some descriptive evidence on the characteristics of the firms that hired the WELL participants after the completion of the on-the-job training programme.

Data
This analysis combines micro-data on the WELL intervention from the Regional Monitoring System Database and administrative data regularly collected from local labour offices by the Italian Ministry of Labour and Social Policies, i.e., the CCD (CCD) (Comunicazioni Obbligatorie, COB).The CCD collates information about all hirings, as well as any prolongations, transformations and cessations of labour contracts, which private companies and public administrations are required to communicate to the labour offices.In addition, it keeps a record of jobseekers registered at public employment offices, while it does not comprise information on self-employment.This information system has been operated by the Ministry of Labour and Social Policies in cooperation with regional governments, the National Institute of Social Security (INPS), the Italian Government Agency for the Insurance against Work-related Injuries (INAIL) and, since 2015, with the National Agency for Active Labour Policies (ANPAL).Since 2008, all companies operating in both the private sector and public administration have been obliged to communicate hirings, prolongations, transformations and cessations of labour contracts by accessing and entering the data into an online information system referred to as "Compulsory Notifications" [6].
We have access to the CCD for the region of Umbria, which collects information on the employment and registered unemployment periods of people residing in the region.The data used for the analysis consists of information for the period of July 2013-December 2015 for all individuals in the sample.Thus, we have information on individuals in both the treatment and control groups, from before and after the intervention.We observe a number of individual characteristics measured in July 2013, which are pre-determined with respect to the start of the intervention and outcome variables measured in December 2015 referring to the labour market status of the individuals.Specifically, in July 2013, all graduates in the population of interest are unemployed, while in 2015, in addition to individuals' employment status, we can also observe the type of firm and the sectoral level in which those who are employed are working.
Our sample consists of the target population for the programme, i.e., all graduates residing in Umbria who are unemployed on the day of the application deadline for participating in the WELL intervention (2 July 2013).All individuals are eligible and hence intended recipients of the programme.Some participate (treatment group) and some do not (control group).
The treatment group is composed of 574 participants who completed the training phase of the WELL programme (out of 682 eligible applicants).The control group includes the whole population of registered unemployed graduates residing in Umbria who did not participate in WELL.This group comprises 6,950 individuals.

Descriptive statistics of WELL participants and non-participants
This section describes the target population of the programme, which is the sample used for this analysis, and presents the outcome variables used in the evaluation.

Demographic characteristics
Tables I-IV present the descriptive statistics for participants and non-participants.The population of interest comprises all unemployed graduates (including first-time job seekers), registered as unemployed in the region of Umbria and residing there on the deadline for applications to the WELL intervention.This consists of 6,654 individuals.Of these, 574 participated in the intervention.After applying standard cleaning procedures, we are left with a final sample of 5,816 individuals: 5,266 non-participants and 550 participants [7].The rest of the analysis is based on this final sample.Among WELL participants, the number of unemployed people with previous work experience is double that of first-time jobseekers.In the control group of non-participants, the number of unemployed people with previous work experience is four times that of first-time jobseekers.Women are equally represented among WELL participants and non-participants (around 70 per cent).
As for the age distribution of individuals, there is a higher proportion of participants in the younger age group (24-29 years).In contrast, the oldest groups (35-40 and 40+ years) are less represented than the non-participant group.Note that the age of both participants and non-participants is measured at the time in which WELL programme was launched.
As shown in Table IV, there is a large proportion of unemployed people in the humanistic disciplines such as the social sciences, business and law.In contrast, jobseekers with a degree in science tend to be less represented in the WELL group, when compared to non-participants.While we have more detailed information on the specific field of study for WELL participants, this information is not available for non-participants.Therefore, we refer to the definitions of areas of study provided by the Italian Statistical Institute (ISTAT) in order to classify information available in the CCD for non-participants.
In order to ensure the similarity and comparability of the participant and non-participant groups with respect to all observable characteristics such as age, education, gender, etc., we select covariates available for both groups in the two data sources (i.e.intervention data for the treatment group and administrative data for the control group).
Table V reports the observable individual characteristics measured in 2013 (before treatment takes place) for both treatment and control groups.
The first and third columns show the average value of each characteristic for the treatment and control groups, respectively.The last column reports the p-value obtained for the difference between the average values in the first and third columns.In italic, we indicate the characteristics for which the difference between treatment and control groups is statistically significant (at the 95% confidence level).On average, the treatment group is significantly younger than the control group.Conversely, in the treatment group, the proportion of individuals at least 35 years old amounts to only 17 per cent, while in the control group it represents 39 per cent.In addition, a higher proportion of people in the treatment group have a degree in the social sciences, business or law (39 per cent in the treatment group vs 33 per cent in the control group), and have a bachelor's degree (33 per cent in the treatment group vs 22 per cent in the control group).In contrast, a higher proportion of individuals in the control group have a degree in science (the 3 per cent difference between the two groups is statistically significant at the 95% confidence level)

507
On-the-job training and obtained a master's degree (78 per cent in the control group and 67 per cent in the treatment group).Lastly, those in the treatment group are significantly more likely to reside in Perugia, the capital of the region.

Outcomes
To characterise the labour market status of individuals, we consider the following outcome variables: • Employment status indicator: equal to 1 if the individual is employed according to the CCD data for the region of Umbria, and 0 otherwise.Note that this definition is different from the traditional employment rate.In addition to having a regular job, it requires that the job is to be located in Umbria.Working in a neighbouring region is coded as 0. This definition is not fully satisfactory, as it does not count working outside of Umbria as a success.However, it represents a relevant outcome variable in our context, as the main objective of the programme is to boost employment among graduates residing in the region of Umbria.Since the programme is implemented in this region, it is important to assess the local effects of the programme.• Unemployment status indicator: equal to 1 if the individual is registered as unemployed on the lists of the unemployment offices of the region of Umbria, and 0 otherwise.Also in this case, our definition of unemployment status does not consider being registered as unemployed in the unemployment offices of other regions.
The following comments are worth noting.First, since our employment rate is based on administrative data, we are not able to observe individuals working in the underground economy.However, to the extent that the objective of the policy is to increase formal employment, this is not a limitation of the study.Second, the administrative data at hand provide us with information on dependent employment.This means that we do not observe self-employed people.However, given the nature of the programme under study, we believe this is a minor issue.We expect a traineeship programme to be successful if the trainees remain employed in the firm once the traineeship ends.Third, since the unemployment rate is based on administrative data, we cannot observe discouraged workers who are out of the labour market (both in Umbria and in neighbouring regions).However, we believe that our measure of the unemployment rate is relevant for the evaluation of the policy, since registered unemployed are the target of active labour market programmes implemented by the local authorities.
In addition to the outcome variables related to one's labour market status, we also consider the type of contract (permanent, temporary or apprenticeship) registered by December 2015, based on administrative data from the CCD data for the region of Umbria.The outcomes referring to the type of contract are intended to evaluate the effect of the programme on job quality.Accordingly, we study the probability to be hired with permanent, temporary or apprenticeship contract, as proxies of good quality jobs.We ignore precarious contracts, since the aim of the programme is not to enhance the probability to be hired under those types of contracts [8].
All outcome variables are measured in December 2015, i.e., between one and two years after the completion of the on-the-job training programme.
The possibility of extending the time horizon of the post-intervention period beyond the short term turns out to be particularly beneficial in the evaluation of training programmes, since there could be issues related to lock-in effects.Specifically, participants in WELL, as opposed to non-participants, were limited in the time they could dedicate to searching for jobs, while they undertook the training programme.However, the lock-in effect does not rule out the hypothesis that training can increase participants' employment prospects, since the programme could prove effective if evaluated in the medium or long term, when lock-in effects fade away or are outweighed by the beneficial effects of the programme.

509
On-the-job training by treatment status.Column 6 reports the p-values for the t-tests on this difference.WELL participants seem to be more advantaged in terms of labour market outcomes after completion of the training.They are more likely to be employed (the difference in employment rates between the two groups is 16 per cent).As for the type of contract, WELL participants are more likely to get a temporary job or apprenticeship, but no significant differences are found for permanent contracts.Nevertheless, we must be cautious about these comparisons as they may be misleading due to selection bias.
Figure 1 shows the percentage of WELL participants (left panel) and non-participants (right panel) that are employed as measured in 2015, by the municipality of residence.The shading in the map darkens with higher employment rates.Overall, WELL participants are more likely to be employed in 2015.In most centrally located municipalities, the percentage of employed WELL participants ranges between 40 and 80 per cent, whereas the percentage of employed non-participants range between 10 and 40 per cent.In relatively small municipalities, the percentage of employed among WELL participants reached 80-100 per cent (darker areas), whereas the percentage of employed among non-participants was between 10 and 40 per cent.
Besides providing descriptive evidence on the WELL participants, we also show some descriptive statistics on the type of firms that offered the training programme to the 550 participants (demand side).For each firm, we can observe the number of employees (firm size), the sector of activity [9] and the location (municipality).Table VII shows   the firms hiring WELL participants tend to be small independently from the sector of activity.
Finally, we show the characteristics of the firms hiring WELL participants after the completion of the on-the-job training.As explained above, there are three types of the contract offered: permanent, temporary and apprenticeship.Out of the 550 WELL participants, 98 individuals were hired at the end of the programme, mostly with a temporary or an apprenticeship contract.Interestingly, the very small firms are the only ones offering a permanent contract, whereas the medium-sized firms and large firms did not hire any participant on a permanent contract, preferring mostly temporary contracts.
This piece of evidence is helpful because it highlights the fact that the programme was of small scale (Table IX).

Empirical strategy
This analysis aims to evaluate the impact of the WELL intervention on the labour market prospects of the unemployed graduates targeted by this programme.As previously mentioned, the first set of outcome variables that we consider are: the probability of being employed in Umbria (employment status indicator, as observed in the CCD data for the region of Umbria), and the probability of being registered as unemployed in the unemployment offices of the region of Umbria.In addition, we also study the probability of having either a permanent, temporary or apprenticeship contracts.

On-the-job training
We estimate the average treatment effect on the treated (ATT), which represents the impact of the WELL intervention for the target population (Angrist and Pischke, 2008).The ATT is calculated as the difference between the average outcome of the treatment group and the average outcome of the treatment group in the counterfactual situation in which the treatment did not take place.The ATT is given by: where D is an indicator equal to 1 if the treatment takes place and 0 otherwise, Y 1 is the individual potential outcome given the treatment, and Y 0 is the individual potential outcome in the absence of the treatment.Note that for the ATT, both potential outcomes refer to the treatment group, since they are conditioned upon D ¼ 1. Equation ( 1) highlights the counterfactual nature of the ATT.The first term of the equation refers to the average employment outcome among the participants in the intervention (WELL participants in our case), whereas the second term refers to the average employment outcome among the WELL participants had they not participated in the programme.This second term is not observable but we can look for a control group that allows us to provide an estimate.This means that the ATT measures the effect of the programme on intended recipients.As pointed out in Heckman (1997), the ATT this is the relevant estimator for policy, which targets specific populations.A related parameter is the average treatment effect (ATE), which measures the effect of a programme on the entire populationreferring to both the intended and non-intended recipientsin case the programme was randomly assigned to individuals in the population (Caliendo and Hujer, 2006) [10].Since our analysis relies on the target population, conditioning upon D ¼ 1 in Equation ( 1) is irrelevant, as the control group coincides with the universe of the intended recipients of the programme.Thus, in this case, the effect of the WELL programme on a random individual who is exposed to the programme (the definition of ATT) is obtained by estimating the ATE for our sample.
The identification problem for the ATT is that (Y 0 |D ¼ 1) the potential outcome of the treatment group in the absence of treatment, cannot be observed.Therefore, the identification strategy boils down to finding a proper control group that mimics the counterfactual situation of the treatment group in the absence of the treatment.Once a suitable control group is available, the identification of the ATT amounts to a simple difference following Equation (1).The ATT amounts to comparing the average of the outcome variable (i.e. the employment status) between the treatment and control groups.
To identify the ATT, we rely on propensity score matching, which ensures that the outcomes of treated units are compared with similar control units.We define the following quantities: Y 1 is the potential outcome given the treatment, Y 0 is the potential outcome in the absence of the treatment, D is an indicator equal to 1 if the individual receives the treatment and 0 otherwise, and X is a set of observable confounding characteristics that are correlated both with selection into the treatment and with the potential outcomes.The identification of the ATT relies on the following assumptions: • conditional independence assumption (CIA): (Y 1 , Y 0 )D⊥, X; • stable-unit-treatment-value assumption (SUTVA); and Where X is a set of observable confounding characteristics that are correlated both with the selection into the treatment and with the potential outcomes.
The first assumption implies that the treatment assignment is independent of the potential outcomes with and without treatment if certain observable covariates are held constant.More specifically, controlling for all observable characteristics X, the decision to participate is uncorrelated with potential outcomes.Hence, if treated and control units with the same values for these observable characteristics show systematic differences in outcomes, these are attributable to the treatment (Imbens, 2004).This assumption requires that all variables that need to be adjusted for are observed; therefore, the extent to which this assumption is reasonable depends on the richness of data available.
The second assumption rules out spill-over effects or general equilibrium effects that may be caused by the treatment.In the context of large-scale labour market policies, this statement may appear quite strong as it prevents crowding-out (displacement) effects in local labour markets: specifically, if treated units are more likely to find a job due to the intervention, this should not decrease the likelihood of control units finding a job.In addition, the intervention should not affect control units via changes in the general equilibrium of wages in the labour market.In our case, SUTVA is a reasonable assumption since the intervention target group is quite small (550 treatment units, compared to 5,266 control units) and thus unlikely to drive general equilibrium or displacement effects.
The last assumption implies that for any given value of the observable characteristics, X, the treatment assignment should not be certain.Therefore, for each value of the confounding variables X, an individual could potentially be observed as treated or not.This assumption ensures that for each treated individual (with given realisations of variables X ), we can find a sufficiently similar individual in the control group, i.e., a control unit that is identical to the treated one in terms of variables X.
Basically, the purpose of the matching procedure is to estimate the ATT by comparing treated units with control units that are similar in terms of observable characteristics that affect both treatment participation and outcome variables.Ideally, we would like to compare the outcome value of a treated unit i with the outcome value of a control unit j that is identical to i in terms of a number of characteristics included in X.
Finding an exact match for each individual i becomes more difficult as the number of observable characteristics increases.Adjusting for a set of covariates, X, to eliminate confounding factors leads to the "curse of dimensionality" problem.However, it has been shown that adjusting only for the propensity score is also sufficient to eliminate confounding (Abadie and Imbens, 2016).Formally, the propensity score is the probability of being assigned to the treatment conditional on the observed characteristics (Rosenbaum and Rubin, 1983).It summarises all information contained in X and has to be estimated to assign a given value to each individual in the sample.The propensity score matching procedure involves comparing treated and control units with similar propensity scores.If the propensity score is correctly estimated, individuals with similar scores are also similar in terms of observable characteristics.This also means that one is comparing treated and control units that are similar in terms of potential counterfactual outcomes.This derives from the CIA assumption,

513
On-the-job training replacing X with the propensity score P(X ), as shown below: (Y 1 , Y 0 ) ⊥ D|P(X).The selection process for treatment models the probability of being treated as a function of the aforementioned covariates, as follows: The propensity score is a function of individual characteristics such as age, gender and all possible observable characteristics.In our context, we also include available educational variables regarding the field of study (e.g.science, education, etc.) and the degree completed (bachelor's or master's).These variables are relevant for explaining one's labour market status after participating in the intervention, as this depends on regional labour demands in terms of educational background.Similarly, individuals with specific educational profiles may be more or less likely to find a firm in which to carry out a traineeship and therefore have higher or lower chances of participating in the programme.Furthermore, we include the municipality of residence and the location of the unemployment office where the individual was registered as unemployed.This is meant to be a proxy for the local labour market where the individual is looking for a job (or a traineeship through which to participate in the WELL intervention).This is relevant for explaining both the outcomes and the decision to participate.
Finally, we further enrich the set of covariates in the propensity score specification by adding information on the past labour market experiences of individuals.
Gathering information on work patterns prior to participation in the WELL programme is important in order to reduce the bias of self-selection into the programme (e.g. more motivated individuals may be more likely to participate, depending on the quality of the programme), as an individual's previous work experience can be used as a proxy for that worker's skills and competences.To this end, we incorporated into the analysis data on the labour market status and the type of contract of WELL participants and non-participants (control group) at fixed dates, namely, on 31 December 2012, 30 June 2012, 31 December 2011 and 30 June 2011.These data were extracted from the CCD.
Following convention, the propensity score is calculated through maximum likelihood estimation (Caliendo and Kopeinig, 2008).The results of the analysis are discussed in Section 7.

Results
Table XI reports our baseline results.As previously discussed, the objective of the intervention is to increase the employability of jobseekers in the region of Umbria.Therefore, our outcomes refer to employment status and the type of contract that an individual has if employed in Umbria.
The first row reports the coefficients from a naive linear regression where each outcome is regressed on the covariates on the right-hand side of Equation ( 2) and the indicator for participation in the intervention D, as in the following equation: The estimated coefficient β 1 amounts to comparing the average for the outcome variable (e.g.employment status) in the treatment and control groups, controlling for a wide set of individual characteristics and past labour market outcomes.The ordinary least squares estimator (OLS) provides an unbiased estimate of the treatment effect (β 1 in Equation ( 3)) under two assumptions: the CIA, i.e., the treatment indicator is exogenous, controlling for the covariates X in Equation ( 3); the functional form assumption, i.e., the true conditional expectations of the outcomes are linear so that the linear regression function provides a good approximation of the true conditional expectations (Imbens, 2015).Hence, linear 514 IJM 40,3 regression provides biased estimates of the treatment effect if the conditional expectations are not linear and if covariate distributions are different in the treatment and control groups.
A problem with this method is that results are very much affected by observations with extreme values in the covariates.Outliers are precisely the units that are not appropriate as counterfactual images of the treated units.While it is difficult to assess if true conditional expectations are linear, so as to justify the use of linear regression, it is quite straightforward to check if the covariate distributions differ by treatment status.
The t-tests for the differences in the means of all covariates between the two groups are reported in Table V and show that the distributions of age, field of study, level of study, past labour market outcomes and residence are different between the treatment and control groups.This suggests that linear regressions provide biased estimations of the treatment effect as the results will be sensitive to outliers (which are not appropriate control units) and to the choice of the (linear) specification.For convenience, we estimate a linear probability model and the coefficients show a positive and statistically significant effect of participating in WELL on the probability of finding a job in December 2015.In particular, participating in WELL increases employment by 18 pp.Furthermore, it increases the probability of having a temporary or apprenticeship contract by 4 pp or 5.8 pp, respectively.Results are more conservative for permanent contracts as they are statistically significant at 10 per cent level.
We now turn to our matching estimates.In general, matching boils down to a number of non-parametric approaches (e.g.exact matching, propensity score matching, sub-classification) that apply the following solution: no functional forms are assumed, but weighting schemes are applied so as to make the covariate distributions in the treatment and control groups as similar as possible.
Figure 2 shows the distribution of the estimated propensity score in the treated and control groups.The estimated propensity score has good balancing properties if both groups are equally distributed along the propensity score.The figure indicates that there is a good overlap in the distributions of the estimated propensity scores in the treatment groups.Figure 3 shows the overlap of the distribution of the propensity score across treatment and control groups for specific sub-groups in the population: for men, for women, for individuals with bachelor's degree and for individuals with master's degree.We believe that the extent of overlap in the distribution of the propensity score is overall satisfactory even when looking at specific categories.
In addition, the propensity score has to satisfy the balancing property.This means that the propensity score should be able to weight units to make the treated and the control group similar and hence comparable.This can be checked by testing that the means for the treated and the control groups in each covariate are not statistically different from each other after applying the matching procedure.If the propensity score effectively enhances the comparability of both groups, then there should be no difference in the means between treated and controls after matching in all covariates.This test allows us to evaluate the quality of the matching procedure.
The t-test statistics for the differences between means are reported in the last two columns of Table X.For each covariate, row "U" and "M" report the test computed before and after the matching, respectively.The column "% bias" reports the standardised percentage bias as computed in Rosenbaum and Rubin (1985) [11].It is a standard measure to evaluate the quality of the matching procedure.As a rule of thumb, the bias after matching should be less than 5 per cent in each covariate.Note: This figure shows the distribution of propensity scores for: men and women (top panel), bachelor's degree and master's degree as the highest degree obtained (bottom panel) The values in rows "U" (i.e.before matching) confirm that before the matching the two groups are unbalanced: for instance, the average age between treated and controls is statistically different.Rows "M" (i.e. after the matching) show instead that the propensity score successfully reduces the covariate imbalances: as for age, the means of treated and controls after the matching are much closer to each other and the per cent bias drops from 59 to 3 per cent.Overall, the matching fixes the balance for all covariates, except for the category related to engineering and manufacturing, the category related to bachelor's degree and the category related to registering as unemployed in the PES offices of Cittaá di Castello, whose per cent bias remains slightly above 5 per cent (8.2, 5.8 and 8 per cent, respectively).However, it is worth noting that the per cent bias is only marginally above the recommended threshold.The matching procedure [12] reports also the average per cent absolute bias as the overall measure of covariate imbalance, which according to the rule of thumb should not exceed 25 per cent.In our case, the average per cent absolute bias is 17 per cent.This suggests that our matching has good balancing properties.for the differences between means for each covariate (the last two columns report the t-statistic and the corresponding p-value).The column "% bias" reports the standardised percentage bias as computed in Rosenbaum and Rubin (1985).Rows "U" and "M" report these statistics computed before and after the matching, respectively The basic procedure consists of the following steps: first, we sort all units according to a propensity score that represents the likelihood of participating in the programme.Second, we compare the average outcomes of treated and control units with similar propensity scores, and finally, we average these differences out over the distribution of propensity scores, so as to estimate the ATT.The second and third rows of Table XI report the results from two different matching procedures.In both cases, we use matching with replacement, which means that each control unit may be used as a match more than once.This improves the comparability between treated and control units, thereby decreasing estimation bias.In the second row of Table XI, each treated unit is matched with the control unit with the closest propensity score, whereas in the third row of Table XI, each treated unit is matched with the five closest control units in terms of the propensity score.The choice of the number of control units to be used in each match (one in row 2 vs more than one in row 3) entails a trade-off between bias and variance.Increasing the number of control units to be assigned for each match tends to increase the bias (since each treated unit is compared with control units that may not be as close in terms of propensity score), but it also increases the precision of the estimate.
Overall, the comparison between the results in the first row and those in the second and third rows suggests that the matching procedure is somehow effective in reducing the bias.When applying nearest-neighbour estimators, the point estimates for the probability of being employed decrease from 0.18 to 0.11.Accounting for as many observable characteristics as can be found in the administrative data helps to reduce the differences between the treatment and control groups.As for the type of contract, we see that participating in the WELL programme has a significant positive effect of 3.8 pp on obtaining an apprenticeship contract, whereas the results are not statistically significant when looking at the probability of obtaining a work contract (temporary or permanent).This result is very much in line with the recent literature on unemployed youth, and especially the unemployed youth in Italy.
The success of the programme is not clear-cut.On the one hand, the employment prospects of participants seem to improve in the region of Umbria.However, this may conceal employment prospects outside of the Umbria region or in the realm of self-employment work linear regression models (first row) and propensity score matching (second and third rows).We use nearest-neighbour matching with replacement; in the second (third) row each treated unit is matched with one (five) control(s).We consider indicators of labour market participation in the region of Umbria in December 2015 (being employed, unemployed, columns 1-2), and the type of contract for employed individuals (columns 3-5).We also account for past labour market experience.Standard errors in parentheses: for matching, robust Abadie-Imbens standard errors, heteroskedastic-robust standard errors otherwise.The specification used for the estimation is as follows: age, gender, field of study categories (social sciences, business and law is omitted as reference), level of study (master's degree is omitted as reference), dummies for the city in which individuals registered at the Public Employment Office (Perugia is omitted as reference), the number of past unemployment spells and the number of past spells in inactivity.This is the specification which gives the best balancing properties of the estimated propensity score.*0.05 op ⩽ 0.10; **0.01o p ⩽ 0.05; ***p ⩽ 0.01 arrangement, especially if for some reasons non-participants are more likely to move to other regions or to be self-employed.This is not necessarily a negative outcome, but with the data at hand we are not able to disentangle the effects of the programme outside the labour market of Umbria.

Heterogeneous effects
We now extend the analysis and provide separate analysis by age group, gender and field of study.As previously, we report the results from the regression analysis in the first row and the results from propensity score matching in the second and third rows.We estimate our model separately by age group considering three categories: below age 30 years, age 31-40 years and 41 years or more.Results are reported in Table XII.Two important results are worth mentioning.First, participants younger than 30 years are 21 pp more likely to be employed compared to the other two groups according to the OLS specification and about 18 pp based on the matching procedure [13].Second, participants younger than 30 have a higher probability of being employed with a permanent contract (5 pp) or apprenticeship (8 pp) compared to participants aged between 31 and 40 years.A common result for the WELL participants in the age groups 31-40 years and more than 41 years is a higher probability of being unemployed compared to non-participants (respectively, 7 and 21 pp).As expected the programme had a bigger impact on younger participants (Table XIII).
We now turn to our gender-specific regressions (13).From the OLS estimates, we observe substantial differences between males and females in the probability of being employed in Umbria one to two years after the completion of the training.Participating in WELL increases employability by 15 pp for females (top panel) and by 26 pp for males (bottom panel).However, as discussed above, OLS estimates are less reliable compared to matching.In the matching estimates (second and third rows in both panels), this gap in employment rate persists.Participating in WELL increases females' employability by 4 pp when For females, the magnitude of the coefficients is much lower and the estimates are not statistically significant.One possible explanation for these differences relates to individuals' field of study.To tackle this issue, Table XIV reports separate estimates by field of study.In order to obtain more robust estimates, we group fields of study into three categories: humanities, which includes "Education", "Arts and Humanities", and "Social sciences, Business and Law"; science, which includes "Science" and "Engineering, Manufacturing and Construction"; other, comprised of those graduating in "Agriculture" and "Health", of which there are very few.We report the results for the "Humanities" (top panel) and "Science".As expected, we observe large differences in terms of labour market participation.OLS estimates (first row of each panel) show that WELL participants have a higher probability of being employed than non-participants, but the advantage in terms of employability is higher for those graduating in "Science" (25 pp, compared to 14 pp for those graduating in the "Humanities").This gap becomes even greater when estimating the model through propensity score matching.Graduating in "Science" increases the probability of being employed in the region of Umbria by 18-20 pp (depending on the nearest-neighbour specification), compared to 8 pp for those coming from the "Humanities".All estimated coefficients are statistically significant at the 1 per cent level.The same pattern is observed when analysing the impact of WELL on the type of contract, although the results are less compelling.Based on OLS results, WELL participants graduating in "Science" have a higher probability of obtaining a permanent job (by 8 pp), but this difference disappears when estimating the model via propensity score matching.However, the difference in terms of obtaining an apprenticeship still remains, benefiting most WELL participants graduating in "Science" (5-7 pp compared to 3-4 pp for those graduating in the "Humanities").Different mechanisms could be in place.First, our results may be explained by an excess of supply with respect to demand for the type of skills acquired in the "Humanities" field of study.
A second possible interpretation could be that for those graduating in humanities work experience might be less important, and they can re-enter the labour market with relatively lower costs due to the specific skills acquired.Unfortunately with the data at hand, we are not able to disentangle between the two mechanisms, but overall our results are in line with the evidence from other OECD countries, which shows that tertiary-educated adults with a degree in science, technology, engineering and mathematics benefit from higher employment rates than their peers[14].

Conclusions
In this paper, we analyse the impact of an intervention aimed at increasing the employability of graduates in the small Italian region of Umbria.The intervention subsidised "on-the-job training" for unemployed graduates and had two main aims: increase employment among unemployed college graduates; enhance the capacity and productivity of the participating firms.To evaluate the effectiveness of the intervention, we look at the employment status of participants and similar non-participants in Umbria in 2015 (from one to two years after the intervention).The evaluation exercise is performed using different sources of administrative data.More specifically, information about participants in the WELL intervention was provided by the Regional Monitoring System Database and information on the control group was gathered through the CCD.To estimate the causal effect of the intervention on the labour market career of participants, i.e., the ATT, we employ a propensity score matching technique.Baseline results suggest that WELL participants are more likely to be employed in Umbria, compared to similar non-participants.They also show a higher probability of obtaining an apprenticeship contract after completing the training.Furthermore, we find substantial differences by gender, with males having a higher probability of being employed with a permanent contract or an apprenticeship contract.Heterogeneous effects by field of study suggest more positive employment responses for individuals with a degree in science.In contrast, individuals with a degree in the humanities benefit somewhat less from participating in the programme.A plausible explanation for this differential impact could be due to the excess of supply of individuals with a degree in the humanities compared to demand.Finally, the aforementioned gender difference in the employment responses to the programme may be explained by the fact that men are more likely to obtain a science degree, when compared to women.However, participants and non-participants seem to be equally likely to be registered as unemployed in Umbria, which indicates that the final conclusions apply only to the Umbria region.

521
On-the-job training the distribution of WELL participants across firms, separately by firm size (number of employees in the firm).About 86 per cent of WELL participants completed their on-the-job training in very small or small firms having between 1 and 50 employees, whereas 10 per cent of them are working in middle-sized firms and 2 per cent in big firms.TableVIIIshows instead the average firm size by the type of industry or sector.Both tables show that Figure 1.Percentage of employed WELL participants vs nonparticipants in 2015 by municipality This figure shows the distribution of propensity score

Figure 2 .
Figure 2. Distribution of the propensity scores

Figure 3 .
Figure 3. Distribution of the propensity scores by gender and by level of study This table tests for each covariate whether the means for the treated (WELL) and for the controls (No WELL) are statistically different between each other.The column "Diff" shows the difference between the mean for the treated and of the controls.The t-test is computed based on a regression of each covariate on the treatment indicator.Column t reports the t-statistic of the estimated coefficient and the column p-val.reports the corresponding p-value Table VI reports descriptive statistics for the outcome variables, namely, employment status (top panel) and type of contract (bottom panel), for the treatment (WELL participants) and control (non-participants) groups.The first and third columns show the proportion of WELL participants and non-participants, respectively, who are employed and unemployed.The fifth column shows the test statistic for the difference in the averages This table reports t-test statistics

Table XI .
Effect The notes under this table are the same as those under TableXI.*p ⩽ 0.10; **p ⩽ 0.05; ***p ⩽ 0.01 1 closest neighbour (although the estimate is not statistically different from 0), whereas for males, employability increases by 21 pp, which is statistically different from 0 at the 1 per cent level.We obtain similar results when considering matching with the five closest neighbours.Males participating in WELL also show a higher probability of having a permanent or apprenticeship contract at the end of the training (13 pp or 8 pp, respectively). considering The notes under this table are the same as those under Table XI.*p ⩽ 0.10; **p ⩽ 0.05; ***p ⩽ 0.01 Notes: The notes under this table are the same as those under Table XI.*p ⩽ 0.10; **p ⩽ 0.05; ***p ⩽ 0.01