What is an Independent Review (IR)?
An IR is required by Vermont statute for IT activities with a lifecycle cost of $1,000,000 or more, or at the discretion of the State Chief Information Officer (CIO). An IR provides an independent assessment of a technology project, proposed solution, proposed vendor, and all the associated lifecycle costs (i.e., implementation and on-going operations). The deliverables of the IR are a/an:
- Acquisition Cost Assessment;
- Technology Architecture Review;
- Implementation Plan Assessment (which includes a Risk Assessment);
- Cost Benefit Analysis;
- Impact Analysis on Net Operating Costs for the agency carrying out the activity; and
- Procurement Negotiation Advisory Services (optional).
If the IR is not applicable to your project, go to Step 7 in the procurement process.
Timing & Hiring of the IR Contractor:
The IR start date is agreed upon by the Oversight Project Manager (OPM) and the Business contact. The IR is scheduled once a vendor is selected, but prior to entering into a contract (although contract negotiations can begin while the IR is underway).
The OPM (in conjunction with DII's Procurement Specialist) uses the Retainer procurement process to hire an Independent Review contractor.
IR contractor services are paid by the State entity owning the IT activity/project. The cost for a standard IR cannot exceed $25,000. An IR with an expanded scope cannot exceed $50,000. The OPM (in conjunction with the Business contact) defines the scope. Examples of an expanded scope include an IR that covers multiple projects and/or vendors.
A standard IR takes 5 to 6 weeks to complete. An IR with an expanded scope can take 8 weeks.
How can the Business prepare for an IR?
The IR contractor’s job is to independently validate cost and other project related information provided to them by the Business. It is not the IR contractor’s job to create/develop this information themselves or even to facilitate the Business’ creation of it. In advance of the IR start date, we recommend that the Project Manager (or Business Lead if a PM has not yet been assigned) coordinates/facilitates the preparations. Use the IR Report template as your guide for the information you will need. Reach out to folks on your project team or within your department to assist you on their areas of expertise. Here is a list of some recommended preparations:
- Gather project related documentation to provide the IR contractor: the ABC form, Project Charter (this should be completed prior to an IR), the RFP, the selected vendor’s response to the RFP, and the draft contract including the Service Level Agreement (SLA) if applicable.
- Create a Contact list of all the key players in the project including vendor contact information.
- Compile all your IT activity cost information (acquisition costs, lifecycle cost benefit analysis, and impact to net operating costs). Don’t forget to include the intangible costs and benefits. Some or all of this information may be available in your ABC form or Project Charter depending on how thorough/detailed they were completed &/or you may need to solicit information from your Sponsor and key Stakeholders. What’s a tangible versus an intangible cost/benefit you ask?
- Tangible Costs & Benefits: Something that has a direct impact on implementation or operating costs (an increase = a tangible cost and a decrease = a tangible benefit). The cost of software licenses is an example of a tangible cost. Projected annual operating cost savings is an example of a tangible benefit.
- Intangible Costs & Benefits: Something that has a positive or negative impact but is not cost related. Examples: Customer Service is expected to improve (Intangible Benefit) or Employee Morale is expected to decline (Intangible Cost). The IR contractor may also speak of these intangible costs and benefits as being either quantifiable or unquantifiable. If it’s quantifiable then you have a means of measuring it and if it’s unquantifiable then you don’t. Improved system response time would be an example of a quantifiable intangible benefit. This is something that isn’t cost related, so is intangible, but is measurable so is quantifiable.
- Prepare a cost comparison to show what others have paid for the same system/product/services that you are seeking to procure. Compare costs with another governmental entity (State, federal, county or even possibly a municipality) of similar size or volume who has procured that same system or service. The idea is to show that we are either getting a good deal or at least paying no more than the going rate.
- Document project, vendor and solution risks, as well as how you plan to avoid or mitigate each risk you identified. You may have already done this in the Project Charter. If not, we recommend that you facilitate a brainstorming exercise with your project team and key stakeholders to elicit this information. Note, through the process of performing the IR, the contractor may come up with additional risks.
- Will the solution you are implementing integrate with any other internal or external systems? If so, ask your technical project team participants to create a visual depiction of all the integration points. The DII Enterprise Architect group may also be able to assist you with this task.
- Be prepared and/or have project contacts ready to speak about the implementation plans for this solution (e.g., design, configuration, conversion, testing, training and implementation plans). The IR contractor will be asking about these.
- Reserve a conference room at your office location once you know the dates the IR contractor will be on site. The IR contractor can work and conduct meetings in that space. The IR contractor may also ask for assistance in scheduling meetings with internal and external contacts.
The IR contractor won’t just accept/use all your numbers and artifacts. He/she will analyze them, review relevant documentation, and ask questions to arrive at their own independent assessment and conclusions. However, the above preparations will go a very long way toward making your IR go quickly and smoothly!
The IR Process:
- OPM, Business contact, and IR contractor meet by phone or in person to discuss the IR process and roles.
- The Business contact provides the IR contractor with the project artifacts he/she has gathered in preparation for the IR.
- IR contractor generally facilitates a kick-off meeting with the State employees participating in the IR.
- IR contractor will interview all relevant internal and external resources that are needed to complete the work outlined in the Statement of Work (SOW) agreement. A standard IR report template is provided in the SOW.
- IR contractor provides the OPM and Business contact with a draft IR report for review.
- OPM schedules the IR review meeting once the report is ready for presentation to the CIO.
- OPM provides the IR report to the CIO and the State IR participants approximately one week before the scheduled IR meeting. The purpose of the IR meeting is for the IR contractor to present his/her findings and recommendations to the CIO and answer any questions.
- Based on the IR report and meeting, the CIO:
- Approves the project, requests additional information, requests follow-up action prior to granting approval, or declines the project.
- Approves the IR report as written or requests updates. Once complete, the OPM obtains the CIO’s e-signature on the final report. The OPM provides copies of the final report to all who were in attendance at the IR meeting.