The Australia Pacific Training Coalition (APTC) is a major Australian government foreign aid initiative, commencing in 2008, that has spent over $350 million, and has turned out over 15,000 graduates with Australian qualifications in skilled occupations from carpentry to plumbing, cooking, and personal care.
APTC is to be commended for running annual tracer surveys of its graduates. Would that other long-running aid initiatives so consistently collect vital monitoring data. Between 2011 and 2019, ATPC surveyed 5,975 students, about half of all their graduates, generally within six to twelve months of graduating. Our new Development Policy Centre Discussion Paper uses that data to analyse APTC graduate employment outcomes.
APTC graduates can be divided into two groups – what we call “job-keepers” and “job-seekers”. “Job-keepers” are those students who have an employer to return to. “Job-seekers” are those who don’t have continuing employment – those who need to look for a job after graduating. The number of job-seekers has been growing over time and now makes up 45% of the total number of graduates.
To look at how successful APTC graduates are at finding a job, we need to focus on the 45% of ATPC graduates who are job-seekers. Job-keepers have jobs to return to, so their success rate at “finding” a job is 100%. That is not informative. Hence our analysis focuses on the job-seekers.
We use three-year averages to reduce volatility. The graph below shows that the proportion of graduates with full-time work at the time of the tracer survey has fallen from 77% in 2011-13 to 55% in 2017-19, and the proportion with no paid work has risen over the same period from 9% to 35%. Of course, some may find work after the tracer survey, but the trends are sharp, and worrying.
Employment outcomes for job-seekers, 2011 to 2019, in three-year averages (%)
We cannot be sure what explains these results. Certainly, they fly in the face of concern about middle-level skills shortages and “brain drain” in the Pacific. The most likely reason, we argue, is that APTC, with its set intakes for long-established courses, has a high risk of oversupplying graduates for the available domestic demand. This is to be expected for Pacific countries which have small wage-based economies. It may even apply to Fiji with its much larger formal economy.
The high levels of unemployment and underemployment for graduates with a range of APTC qualifications suggest that domestic labour markets in the Pacific cannot absorb a continuous flow of post-school technical graduates with the same qualifications.
What is the solution? Clearly, a more demand-led approach to student admission and course selection is needed, as well as a review of the quality of graduates. But the best way to promote demand for APTC graduates is to link them with New Zealand and Australian employers. Although the APTC was established in response to Pacific demands for greater access to Australian labour markets, opportunities for migration by APTC graduates have been very limited.
In summary, what our analysis shows is that APTC needs to focus less on outputs (qualifications) and more on outcomes (jobs). The best way to turn qualifications into jobs is to help APTC’s skilled graduates migrate to Australia and New Zealand under both countries’ skilled migration programs. We have developed a policy brief to explain how this might work, and will be blogging about that next.
Read the full Devpolicy Discussion Paper “Worsening employment outcomes for Pacific technical graduate job-seekers” here.
Disclosure
This research was undertaken with support from the Pacific Research Program, funded by the Department of Foreign Affairs and Trade. The views are those of the authors only.
Critique of Curtain & Howes’ analysis of APTC Graduate Tracer Survey data
Richard Brown is an Honorary Associate Professor in Economics at the University of Queensland. He is also a member of APTC’s Advisory Board. The comments expressed below reflect his personal views only, and not those of the Board.
Preliminary remarks
In this note I argue that while I welcome Curtin & Howes’ (2021) (henceforth C&H) original and, so far, only longitudinal analysis of data on APTC graduate outcomes, I am concerned that a number of limitations in their methodology give rise to an exaggerated measure of the APTC graduates’ ‘poor’ employment performance.
Their analysis splits the sample of graduates into two categories:
• “job-seekers”, which they define as those who did not already have a job to return to on graduation, and their measure of APTC’s performance is then based on the success of this sub-group in finding full-time employment within 6–12 months after graduation; and
• “job-keepers”, which they define as all those who return to work for the same employer after graduation, irrespective of the level at which they are employed.
They find that the employment rate among job-seekers fell from around 80% to 56% between 2011–13 and 2017–19. Although they do not state it, this could be (and is likely to be) interpreted that almost half of APTC’s graduates end up unemployed.
In this note I seek to identify the main shortcomings of their study and to demonstrate the extent to which their findings are affected (exaggerated) by the assumptions and methodology of their analysis.
I conclude that rather than treating C&H’s study and policy recommendations as in any respect conclusive, it should rather be seen as a welcome contribution to, and invitation to concerned stakeholders to participate in, a process of evidence-based, rigorous and transparent analysis of APTC’s performance and policy options for the future.
What’s wrong with C&H’s analysis?
My main bone of contention is with their measure of job-seekers.
APTC’s annual Graduate Tracer Survey (henceforth GTS) allows the total sample of graduates to be disaggregated into the following sub-categories:
(1) those who return to the same job with the same employer (pre-APTC training);
(2) those who secure a higher paid position with the same employer;
(3) those who find a job with a new employer; and
(4) those who are not in paid employment.
In their analysis, C&H treat job-keepers as all those in sub-categories (1) and (2) and job-seekers as all those in sub-categories (3) and (4). Although their analysis includes both full- and part-time employed, in reporting their main conclusions they focus exclusively on those in full-time paid employment (e.g. DP 91, 2021, p.19; this blog [PDF, p.2]).
In the rest of this note I address mainly two questions:
• Why is this problematic as a measure of APTC graduate performance?
• How does this categorisation affect the results of their analysis?
In relation to the first question: by excluding sub-category (2) graduates from the job-seeker category they are ignoring a large number of APTC graduates who seek and are successful in gaining promotion to a higher paid position (with the same employer) after graduation. Had the same graduate moved to another employer (at the same or higher position) he/she would be treated as a job-seeker. By virtue of choosing to stay with the same employer, the successfully promoted graduate’s employment to a new position is ignored. This distinction is arbitrary and unjustifiable, and affects the allocation of APTC graduates between their two main categories (job-seekers vs. job-keepers) used in all the subsequent analysis. It concerns me that the authors do not acknowledge the implications of their assumptions, leaving the reader to accept their categories and the resulting analysis as an ‘act of faith’.
Another questionable assumption is the classification of all those in sub-category (4) – the ‘not in paid employment’ – as unsuccessful job-seekers. This assumes that they were all seeking paid employment (at the time of the survey), and if they were not, this was only because they had opted out of the labour force as ‘disillusioned’ unsuccessful job-seekers. This ignores all those who were not seeking paid employment for reasons other than having become disillusioned, unsuccessful graduates. This would include those who were temporarily not seeking paid employment for reasons, for example, of ill health, parental or other family-related leave, further study, or some informal activity not classified under ‘self-employed’. Again, no matter how small they believe to be the likely number of graduates not actively seeking paid employment for such reasons, ignoring this possibility altogether is unacceptable.
Furthermore, C&H’s understanding of ‘successfully employed’ graduates when summarising their findings focuses mainly on full-time employment. Again, there is no discussion of why those employed part-time should be excluded, and thereby effectively grouped together with those not employed at all. The implicit assumption is that they are all involuntarily underemployed, and should therefore not be treated as successful graduates.
Turning to the second question, next I show how a refined definition of job-seekers affects the analysis and conclusions.
Following from the preceding comments, I would argue that C&H’s measure of job-seekers needs to be adjusted in two respects:
(i) all graduates who are successful in gaining promotion to a higher position, whether working for the same employer or a new employer, should be treated equally as job-seekers; and
(ii) not all graduates not in paid, full-time employment should be treated as unsuccessful job-seekers.[1]
By way of illustration I use the GTS data from 2019 to compare the results using the two sets of definitions and measures, C&H’s and mine. (Later I apply my measures to the 2013 survey to estimate changes over time.)
If we then compare the results it can be seen that using C&H’s definition, in 2019 66% of job-seekers were employed, and, excluding those in part-time employment, only 59% were. In comparison, my adjusted definition shows that at least 75% were employed (full-time and part-time) assuming all those not employed were indeed unsuccessful job-seekers. (If we were to assume, for example, that half the 145 not employed were not actively seeking work for reasons other than being disillusioned job-seekers, the employment rate would increase to 80%.)
Repeating the same recalculation for the 2013 GTS survey, using my refined measures and assuming all not employed were unsuccessful job-seekers, I find that approximately 80% of job-seekers were in paid employment. While this supports C&H’s main contention that the percentage of ‘successful’ APTC graduates has fallen, the extent of this decline is nowhere near as drastic as they assert; “… that in 2011–2013, three-quarters of APTC job-seekers were employed at the time of the tracer surveys, but by 2017–2019, only slightly more than half” (DP 91, 2021, p.9).
In other words, while my analysis shows a decrease in the percentage of job-seekers in employment over this time period, the magnitude is substantially less; viz. in the order of 5 percentage points rather than the 22 percentage points they claim.
Moreover, it needs to be stressed that this decline, in percentage terms, occurred in the context of a rapidly growing annual cohort of APTC graduates. Over the same time period the number of graduating APTC trainees rose from 692 to 1,603; a 132% increase. A decline in employment among job-seekers of 5 percentage points could be considered relatively small in the context of this 132% increase in the number of new graduates over the same time period.
Do other GTS data support C&H’s findings?
C&H make it explicit that their paper focuses exclusively on actual employment data as their preferred approach to gauging APTC graduate success. However, it is interesting to ask why it is that other data, based on responses to questions from the same GTS surveys, do not lend support to C&H’s main contentions. For example, from the 2019 survey, of all respondents:
• 89% reported it ‘Easy’ or ‘Very Easy’ to find work;
• 8% reported it ‘Difficult’ ‘or ‘Very Difficult’; and
• only 3% reported that they could not find work in the same area as their APTC training.
Further:
• 96% reported that their training ‘Improved employment prospects’;
• 91% reported ‘Improved current employment position’;
• 96% reported ‘Increased job ease and satisfaction’; and
• 95% reported ‘Improved sense of job security’.
Finally:
• 43% reported that they had gained promotion after graduation; and
• 49% reported that they were earning more after graduation.
The obvious inconsistency between these assessments and responses about actual employment as reported by C&H is worth at least noting for further investigation, but should not be ignored completely in a paper that uses GTS survey data to assess APTC graduate performance.
Conclusions
Longitudinal analysis of APTC’s GTS data is welcomed, especially if it provides credible, evidence-based analysis and assessment of APTC’s performance as a basis for discussion of APTC’s policies. I argue that such analysis and policy debate should be based on a common understanding among relevant stakeholders of the appropriate criteria and measures of success best suited to assessing APTC’s training objectives, outcomes and policies. Transparency and rigour would obviously also be required.
Unfortunately C&H’s study falls short of these in a number of respects.
1. Their use of a questionable measure of graduate success: especially in relation to the definition of a job-seeker, their restriction of ‘success’ to full-time paid employment, and, their treatment of the ‘not employed’.
2. Their lack of transparency in failing to:
a. adequately justify the definitions used or to even acknowledge the possible implications of alternative assumptions and definitions for their main findings; and
b. provide details of the raw numbers from the surveys on which their categorisations and disaggregated analysis are based. (An appendix with the relevant data allowing the reader to understand and replicate their results would be appropriate.)
3. Lack of appropriate checks of robustness, such as the sensitivity testing of the main results to alternative measures and assumptions.
4. Failure to use or even acknowledge other data from the same GTS surveys that report feedback from graduates on their job search, employment, income, etc. While such data are obviously less objective than actual employment status data, it would have been informative had the authors compared their findings with graduates’ reported experience, especially as these are not supportive of the authors’ findings.
In this note I have sought to demonstrate that applying alternative, and in my opinion more acceptable, criteria and measures of employed job-seekers would not provide the same level of support for the somewhat alarmist conclusions the authors have drawn.
I would favour an approach that encourages further evidence-based discussion of the way forward for APTC, grounded on a clearly articulated understanding of APTC’s main objectives and assessment criteria, and drawing on rigorous and transparent analysis of the available data. C&H’s paper should be treated as a most welcome initiation of such a process, but by no means providing conclusive evidence to inform the direction of APTC and DFAT’s future training policy in the Pacific islands.
_______________________________
[1] Although part-time employed are not treated explicitly as unsuccessful graduates, by limiting their analysis to full-time employed only C&H are effectively relegating part-time employed to the same status as not employed. Their results indicate that in 2019 approximately 7% of their job-seekers were part-time employed, with only 59% ‘successfully’ employed full-time. They only cite this full-time employed figure in the remainder of their analysis and in other presentations including this blog, and ABC Radio Australia’s Pacific Beat interview by Richard Ewart (19 Feb 2021).
Many thanks to Richard Brown for his extensive comments on our paper, and apologies for the long delay in responding.
Before getting to his main point, we want to reply to Richard’s suggestion that we are too negative and ignore important data when it doesn’t suit our case. In particular, Richard says we ignore the 2019 APTC tracer survey result that 89% of respondents said that it was easy or very easy to find work, which, he says, undermines our result that APTC graduates are finding it increasingly difficult to find work.
However, the APTC reports don’t make clear that this question is only asked to those in paid work. Good that most of APTC’s employed graduates said it was easy to find work, but this doesn’t speak to our finding that an increasing share of APTC graduates who need to find a job just can’t find one. Of APTC graduates who did not have a job to return to, only 9% were not in paid work at the time of the 2011-13 tracer surveys but this had grown to 35% by the 2017-19 surveys.
We turn now to Richard’s more general critique. Before we summarise his position, allow us to reiterate that our analysis of APTC graduate employment outcomes is not seeking to evaluate APTC graduates’ performance in terms of their value-added contribution that they are making to their employer. Our sole focus is instead on the employment outcomes of APTC graduates, a measure of employer demand. This means that our focus is only on those graduates who are not employed while students and need to find a job on graduation—that is, job-seekers.
Richard suggests we also include in our analysis those graduates who are employed while students (and at the time of graduation) but who have since gained a higher position with the same employer. In our terminology, these are a subset of what we call job-keepers, i.e., employed students, so are not included.
A focus on job-keepers is not relevant to our purpose. To judge the demand for APTC graduates, we need to look at those not employed at graduation.
A thought experiment might help. Imagine a scenario in which all APTC students were employed while they were studying and at the time of graduation. Then we would say, according to Richard’s method, that APTC graduates were 100% successful in finding jobs. That would be a meaningless finding. And that is why all APTC graduates who continued to be employed by the same employer while studying and after graduating need to be excluded from our analysis of demand for APTC graduates.
The fact that some job-keepers have gained a higher position with the same employer is irrelevant for our purposes. They might well have moved to a higher pay level while still doing the same work. We have no way of knowing whether anyone else has been hired to do their old job. Their having a new position with the same employer cannot be taken as a sign of demand for APTC graduates.
Crucially, Richard never says what question he is seeking to answer by grouping one subset of job-keepers with all the job-seekers. Our question is clear: how many APTC graduates who are not employed at the time of graduation have found paid work by the time of the tracer survey?
Think of APTC as a curative pill. To test the pill, we need to see how it works on diseased (unemployed-at-graduation) patients (graduates), not healthy (employed-at-graduation) ones.
To summarise, we welcome Richard’s comments but they do not alter our findings, or lessen the urgent need for APTC to face the fact that those of its graduates who need to find a job on graduating are finding it increasingly difficult to do so.
Stephen Howes and Richard Curtain
Thanks to Stephen and Richard for pointing out that the tracer survey questions about ease/difficulty in finding a job are asked only of those in employment – a pity that this is not stated in the APTC Tracer Survey Reports. However, this does not affect my main argument that their categorisation and definition of Job Seeker, and who is included/excluded from this category, is arbitrary and inconsistent.
They have still not given a satisfactory explanation of why a graduate who seeks a new job at a higher level is treated as a Job Seeker only if he/she finds a job with a new employer. They say that they exclude those who gain a higher level job with the same employer on the grounds that they were already employed. They seem to be assuming that any graduate who gains a job with a new employer must have been unemployed while studying and on graduation – otherwise they would not be a Job Seeker by their definition. And yet, when one looks at the Tracer Survey Reports (e.g. 2019 Report, Table 2, p.9) it states clearly that those who gained employment with a new employer did so in a ‘different company … to the one I was employed by when I studied at APTC’. Unless Stephen and Richard have other information to the contrary there is no basis for treating these graduates as Job Seekers, nor for differentiating them from graduates who gained new, higher positions with the same employer. Unfortunately, neither their blog nor the more detailed Devpolicy Discussion Paper provide the reader with the details of how they arrived at their numbers. The reader is left having to accept their results as an act of faith.
Perhaps more useful than assuming a particular basis on which to evaluate APTC’s performance, DFAT could be invited indicate what they consider the most important outcomes and criteria on which APTC’s performance should be assessed?
Richard Curtain: Thank you for your invitation. For what it is worth on job seekers. Jobs are scarce and competition will continue to be high and others are doing what the graduates should be doing. So during training some emphasis should be placed on trainees to think about about how they can create their own employment out of the skills they gain through APTC. PNG Government’s policy is five hundred thousand 500,000 SMEs by 2030-50. This presents an opportunity for anyone who is willing to make a positive difference in their own lives and those others less fortunate. Tertiary institutions are being encouraged to offer assistance to make real the government’s aspiration of growing the economy through small and medium size enterprises.
Sustainability expert Jeffrey Sachs, outlined that Australia can limit its carbon footprint by moving from coal mining to building it’s tourism sector. Maybe, the APTC graduates could provide that much needed hopitality workforce the sector desperately needs (specially Western Australia).
Arun,
thanks for your comment. Unfortunately, APTC graduates in hospitality are not eligible for a temporary skills shortage visa. Only APTC graduates can access this visa if their occupation is on the medium term or short-term skill shortage lists.
Oh dear, my apologies, I should have looked at that first. Thanks Stephen.
Thanks for the blog. Was there any sector- or sex-disaggregated data available for analysis?
Hi Juliet, yes it is all in the Discussion Paper linked at the bottom of the blog. The gender dimensions are particularly interesting, and we’re hoping to write a separate blog on this – so watch this space.