In short, it means that the quality and usefulness of any review work undertaken may be undermined where reviewers have conflicts of interest, are not objective and do not report on an open and candid basis.
In other words, it is unrealistic (and contrary to the independence principle) to ask people directly engaged on (or connected with) a project to effectively “mark their own work” or the work of people with whom they have an association.
Similarly, open and candid reporting is required in order to expose any areas that need to be addressed. Senior officials need to know the facts in order that an informed decision on how to proceed can be taken.
Incomplete information
In order to be effective, review teams will be heavily reliant on documents and other information provided to them by interviewees.
If relevant material is omitted then assurance processes may be undermined: reviewers will be working from incomplete information and problem areas may remain undetected. Issues or problems that remain hidden are only like to get worse over time.
It is therefore vital that reviews are set up in a way that promotes full co-operation by interviewees. In my experience people tend to be less co-operative if they are suspicious of the inquiry or do not receive adequate comfort about its purpose and how information they provide will be used.
Problem areas need to be identified and flushed out in order that they can be addressed before it is too late.
A few simple pointers should pay dividends in this regard:
• Explain the context and process to the interviewee (including confidentiality);
• Explain that the process is intended to be positive and constructive;
• Highlight the downside of problem areas remaining undetected (and therefore the importance of being open and honest); and
• Allow adequate time for interviewees to prepare.
Recommendations by assurance reviewers that are ignored by agencies or departments
And what about situations where the review team does its job but senior public sector officials refuse to adopt the recommendations made? This has been a problem area in the UK and was picked up by a (UK) Public Accounts Select Committee last year who found that:
"... the MPA [the UK body with responsibility for assurance work, as described above] only has informal influence over departments. It supports the Treasury in approval and funding decisions but there is no obligation on the Treasury to follow its recommendations. It has no powers if a department decides to proceed with a project against MPA advice.” "It needs to have stronger, more formal mechanisms for driving change, and there should be transparency where ministers or officials have rejected its recommendations.” As a result, the Select Committee recommended that:
• the Major Projects Authority be given more power to drive change; and
• there be greater transparency if ministers or other officials do not follow recommendations from assurance reviewers.
These recommendations appear sensible and may be worth considering here in New Zealand, even at this early stage.
Typical problem areas for reviewers to target may include the following (this is a selection only – not an exhaustive list):
Pre-contract
• Is the department or agency buying too much (test “needs” versus “wants”)?
• Is the project too big? If so, can it be broken down into smaller parts?
• Is there too much complexity in current business processes and, if so, are adequate plans in place to simply these?
• Are users actively engaged in setting requirements?
• Is the project framed as a business endeavour rather than an ICT project?
• Is the scope clear?
• Are the timescales realistic?
• Have any “agreement to agree” scenarios been minimised?
• Is the project adequately resourced (capability and quantity)?
• Are contingency plans in place?
Delivery
• Is staff continuity good?
• Are contractual obligations being performed (by all sides)?
• Have any variations (for example, to scope, timetable and associated commercials) been properly documented?
• Are “spot-checks” required to check for scope creep and/or delay?
• Does the project need to be re-set?
• Is a testing plan in place and being followed (check for any corner cutting)?
• Does internal reporting reflect the true position on project status?
• Is escalation necessary to address any outstanding issues? Have breach notices been issued where required?
The exercise should, in my view, be largely intuitive and based on experience particularly when interviewing project participants. Too much process (for example, long lists of questions or areas to cover) may run the risk of distracting participants from the key issues and root causes involved.
Next: Smoking guns
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.