HomeMy Public PortalAboutExhibit MSD 84W3 - Review of the Capacity, Management, Operations and Maintenance Program May, 2013Exhibit MSD 84W3
Metropolitan St. Louis Sewer District
Review of the Capacity, Management, Operations,
and Maintenance (CMOM) Program
Fiscal Year 2013
May, 2013
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program FY 2013 AUDIT PLAN
The St. Louis Metropolitan Sewer District
Review of the Capacity, Management, Operations,
and Maintenance (CMOM) Program
May, 2013
Table of Contents
Introduction and Scope 1
Objectives and Methodology 2
Overall Conclusion and Results 3
Observations and Recommendations 4
Opportunity for Improvement 11
Acknowledgements 12
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program FY 2013 AUDIT PLAN
INTRODUCTION AND SCOPE
The Risk Assessment and Internal Audit Plan for Fiscal 2013 developed by MSD Internal Audit (IA)
provides for the performance of an engagement to review the processes and related internal controls
around the District's Capacity, Management, Operations, and Maintenance (CMOM) program. This
report details and summarizes the execution and results of that engagement.
A Consent Decree between the United States of America, the Missouri Coalition for the
Environment Foundation and Metropolitan St. Louis Sewer District ("MSD") in the matter of The
United States et al. v. The Metropolitan St. Louis Sewer District, No. 4:07-CV-01120 (E.D. Mo.) was
entered with an Effective Date of April 27, 2012. Pursuant to sections V.G.31, 32, and 33 of the
Consent Decree (the Decree), MSD was required to submit a Capacity, Management, Operations,
and Maintenance (CMOM) Program Plan. The formal submission of the CMOM Program Plan was
submitted on October 27, 2012, within six months of the Consent Decree Effective Date of April 27,
2012, as required by the Decree.
The District's CMOM Program Plan is designed to minimize service interruptions such as building
backups and non -capacity sanitary sewer overflows (SSOs) resulting from blocked sewers or lines
with capacity limitations or infiltration/inflow (I&I). Understanding capacity, and appropriately
managing, operating, and maintaining the collection system is a central objective of this program.
The program incorporates preventive and predictive maintenance processes to properly maintain the
wastewater collection system, ensuring compliance with the requirements of the Consent Decree and
assisting in the effort to meet service -level goals established by Management.
The system consists of gravity sewers, pump stations, force mains, etc. MSD is the fourth largest
sewer agency in the country based on sanitary and combined sewer system miles. The system
addressed by CMOM consists of approximately 164,000 manholes, 270 sewer pump stations and
approximately 6,300 miles of piping and 130 miles of force mains.
The current CMOM program was initially developed, documented and implemented during 2012. The
procedures have continued to progress and mature during 2013. The program excludes storm water
assets.
Scope
The engagement was conducted during portions of January through March, 2013.at various District
sites such as the main office at Market as well as locations at Sulphur, Grand Glaize, and Mintert.
The scope of the engagement was comprised of a review of the various components of the District's
CMOM program.
The process areas of focus were as follows:
• Inspection of Gravity Sewers
• Sewer Line Cleaning and Root Control
• Manhole Inspections and Repairs
• Repair, Rehabilitation, and Replacement (RRR)
• Standard Asset Management Procedures
• Pump Station Asset Management Standards and Procedures
▪ Force Main Asset Management Performance Standards and Procedures
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 1 FY 2013 AUDIT PLAN
OBJECTIVES AND METHODOLOGY
Objectives
The overall objectives of this engagement were to ensure:
➢ Each CMOM-related requirement of the Consent Decree is identified and addressed by the
District within the District CMOM Program Plan.
➢ District CMOM procedures and controls are adequately designed:
- Process risks are adequately mitigated.
- No gaps in controls exist.
- Duties are properly segregated.
- Procedures are efficient. (i.e., there is no duplication of effort.)
➢ District CMOM procedures and controls are operating effectively:
- Controls are implemented and operating as intended.
➢ The District is operating in a manner that complies with the CMOM requirements of the
Consent Decree and the requirements of CMOM Program Plan document.
Methodology
To accomplish the above objectives, Internal Audit (IA):
> Obtained a copy of the Consent Decree. IA reviewed and identified the specific requirements
of the CMOM Program Plan, which are contained in sections G.31 through G.33 of the Decree.
> Obtained a copy of the District's CMOM Program Plan, which was submitted to the EPA in
October of 2012. TA thoroughly reviewed the document in an effort to gain an adequate
understanding of the District's program.
➢ Reconciled the District's program plan with the requirements contained in the Consent Decree.
IA identified any Consent Decree CMOM requirements not adequately addressed by the
District's program plan.
> Reviewed the design of each process and identified the risks and controls within each process.
IA reviewed for process and control design deficiencies by:
- reviewing the plan document, which includes process flows that document the
various process steps and the related controls,
- performing interviews of process owners to gain a better understanding of each
process and control design, and
- reviewing the setup, design, and operation of the CMMS (Computerized
Maintenance Management System, a.k.a. Maximo).
➢ Performed tests of controls and substantive testing by:
- making inquiries and performing interviews,
- performing walkthrough procedures,
- inspecting and reviewing supporting documentation, system screens, and
system data, and
analyzing and testing system (CMMS, GIS) data.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not be used
by any other parties without the prior written consent of MSD.
Review of the CMOM Program 2 FY 2013AUDIT PLAN
OVERALL CONCLUSION AND RESULTS
In the opinion of Internal Audit, in all significant respects, the controls and procedures utilized for
the CMOM program complied with established policies and are effectively designed and
implemented and are sufficient to satisfy internal control objectives while minimizing undue
business risk. However, we did note matters that are opportunities for strengthening internal
controls and improving the District's ability to comply with the requirements of the Consent
Decree. Such matters are discussed in detail at the Observations and Recommendations
section as well as the Opportunity for Improvement section.
Initial Inherent Business Process Risk: Moderate #
Overall Assessment of Engagement Results: Satisfactory **
** Engagement results are evaluated as satisfactory, generally satisfactory or unsatisfactory.
• Satisfactory (clean opinion) — No significant engagement findings or material
weaknesses were noted.
• Generally Satisfactory (qualified opinion, i.e. "except for') — Results contain
significant engagement findings. No material weaknesses were noted.
• Unsatisfactory (adverse opinion, immediate Management attention required) —
Significant engagement findings and/or material weaknesses were noted.
A DEFINITIONS
Engagement Finding (#Low Risk): An engagement finding is a condition that could adversely
affect the organization but is less severe than a significant engagement finding or significant
deficiency. Classification includes process or control deficiencies that are not significant
deficiencies as well as includes other low risk or low impact conditions.
Significant Engagement Finding (# Moderate to Hiqh Risk): A significant engagement finding is
a condition that could adversely affect the organization. Definition includes all types of findings,
such as irregularities, waste, ineffectiveness, conflicts of interest, illegal acts, errors, and
significant deficiencies in internal control over financial reporting as well as other significant
internal control weaknesses. A significant deficiency is defined as a deficiency, or a
combination of deficiencies, in internal control over financial reporting that is less severe than a
material weakness, yet important enough to merit attention by those charged with governance.
Material Weakness (# Hiatt Risk): A material weakness is a deficiency, or a combination of
deficiencies, in internal control, such that there is a reasonable possibility that a material
misstatement of the financial statements will not be prevented or detected and corrected in a
timely basis. For internal audit purposes, the definition also includes material and/or severe
irregularities, waste, ineffectiveness, conflicts of interest, illegal acts, errors, and other material
control weaknesses, etc.
(The term "material weakness" should be thought of as a serious category of significant engagement findings and/or
significant deficiencies. However, not all significant engagement findings and significant deficiencies are material
weaknesses.)
a - Definitions are based on guidance from the IIA Standards, GAAS, and the PCAOB.
# - Risk Is assessed at the District (Entity) Level. (Risk to the District as a whole).
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 3 FY 2013 AuDJT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 1 — Repair of Assets
Condition:
Regarding assets for which repairs have been completed and deficiencies have been
remediated, controls in place are not effective in ensuring that the required follow-up inspection
work orders are issued. Tests revealed several instances in which assets were not re-
inspected, resulting in the assets remaining at a rating of '5' in the system, despite the
completion of repairs. Re -inspection procedures are required to allow for the assignment of an
improved system rating, such as a '2', indicating no deficiencies. (Assets are assigned condition
ratings of '1' to '5 . Sewer assets rated at '1' through '3' are in a condition to last beyond the
next inspection. For assets rated at '4', repairs are recommended before the next inspection
cycle. Sewer assets rated at '5' are considered acute and require repair within one year.)
In addition, instances were noted in which acute defect work orders were not issued as required
for items assessed at a rating of '5'.
These conditions could detrimentally affect the District's ability to meet the Consent Decree
requirements for repair or remediation of acute defects within one year of discovery.
Recommendation:
IA recommends that Management take the steps necessary to ensure that follow-up and acute
defect work orders are issued and completed. Such steps may involve automation.
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process. Level: Moderate
Process Owner Response:
Newly implemented Escalation 1.1.10 within our CMMS creates an Acute Defect work order for
the Principal Engineers based upon a sewer work order being updated with an asset condition
rating of 5. Engineers review proposed follow-up work, monitor progress for completion within 1
year to comply with the consent decree, and ensure a CCTV re -inspection work order has been
created before they complete the acute defect work order. This acute defect process will be re-
emphasized with the engineers.
Newly implemented Escalation 1.1.14 creates a CCTV re -inspection work order for any
completed repair sewer work order (regardless of asset condition rating) for which there isn't
already an open CCTV work order. This CCTV work order is automatically populated on the
Planners start center (in a result set) that will assure that re -inspection is scheduled and the
repaired asset is given a new correct rating.
Repairs or rehabilitation to manholes should be re -inspected by the crew performing the repair.
Therefore, no escalation is needed. However, a report will be generated that captures any
manhole or sewer assets with a current condition rating of 4 or 5 that do not have open repair or
re -inspection work orders issued. The report will be monitored and appropriate follow-up will be
performed.
Date of Implementation: Recommendations have been implemented.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 4 FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 2 - Work Orders — Status Changes
Condition:
The ability to "cancel" a work order is not restricted to appropriate personnel. In addition, most
Supervisors and Planners are not aware that the system ability to cancel a work order is so
readily available to personnel, potentially creating an inflated comfort level regarding such
cancelled transactions.
There is no standardized methodology in place to review and/or approve cancellation and work -
stop transactions. For the period of April 27, 2012 through February 28, 2013, there were 7,503
work orders that had been cancelled in the system. In addition, as of February 28, 2013, there
were 634 work orders in the "work -stop" status.
This condition may have a detrimental effect on the District's ability to complete specific work
orders within a timeframe that is in compliance with Consent Decree requirements.
Recommendation:
IA recommends the following:
• Restrict the system ability to cancel work orders to appropriate personnel.
• Develop and implement standardized procedures for properly monitoring and/or
approving cancellation and work -stop transactions.
• Adjust the system to require the inclusion of explanatory notes in the transaction log of
such transactions
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process Level: Moderate
Process Owner Response:
The CMMS system will be adjusted by an administrator to take rights away from collection
system Team Leaders and Crew Workers for cancelling work orders.
A process will be established for Planners to be notified if a PM (preventive maintenance) or CM
(corrective maintenance) work order (priority 0-5) should be cancelled. Supervisors should be
notified for EM (emergency maintenance) work orders (priority 6-11) to be cancelled. Planners
and Supervisors will include a status or log note comment explaining why the work order is
being cancelled. They will also make sure correct procedures are being followed to correct any
issues with the asset if this is the reason for those being cancelled (i. e. any needed map update
work orders have been written or corrected attribute information has been included on the work
order spec tab for transmission to GIS.)
The process for review of work -stop work orders by Planners/Supervisors will be updated and
reviewed. Periodic reports of the number of outstanding work -stop work orders will be
generated and monitored to aid in timely resolution and ensure the process is consistently
followed across divisions.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 5 FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 2 - Work Orders — Status Changes (Cont'd)
A request has been generated and submitted asking that IT determine if mere is any way
without customization to require a log note (or status change memo) for "cancelled" or "work -
stop" work orders. in addition, Rules Manager software for Maximo has been submitted in the
upcoming fiscal year budget. If this is approved and we determine that a note can be required,
we will implement it thru this software.
Date of Implementation: The change in who can cancel work orders has been implemented.
Procedures for work orders in workstop status are being reviewed and should be in place by
6/30/13. If Rules Manager software is approved in the coming year's budget and can require
the note on cancelled work orders, it should be in place within 1 year of software installation.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the MOM Program 6 FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 3 — Planner/Schedulers
Condition:
The following was noted regarding the PlannerlSchedulers:
• There is no standardization regarding procedures or reports to be utilized.
• The role of scheduler/planner does not appear to be clearly defined across the District
sites, as IA did not note uniformity to the approaches taken by the various planners.
• Supervision or monitoring of the planning/scheduling process appears to very limited.
This condition may have a detrimental effect on the District's ability to meet certain requirements
of the Consent Decree.
Recommendation:
IA recommends the following:
• Implement the use of best practices by the Planners/Schedulers.
• Clearly define the tasks or role of a Planner/Scheduler.
• Increase the level of monitoring for the planning/scheduling process.
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process Level: Moderate
Process Owner Response:
As indicated in Issue 2 earlier in the report, the procedure for work -stop and cancelled work
orders will be updated and reviewed with the yards. Improvements in scheduling work are being
pursued with IBM per Issue 4 later in the report. In addition, Planner/Schedulers have recently
been assigned to work for the Principal Engineers in each yard. They are meeting with Planners
on a monthly basis and will discuss any needed procedures and consistency across yards.
Date of Implementation: Planners have now been assigned to cancel PM work orders.
Procedures for work orders in workstop status are being reviewed and should be in place by
6/30/13.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program T FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 4 — GIS (Geographical Information System)
Condition:
IA noted the following regarding GIS:
▪ Regarding abandonments and demolitions, the processes in place do not appear to be
effective in ensuring that the results of such events are communicated to the GIS
department for the purpose of updating GIS for the resulting changes to the collection
system assets.
▪ Discrepancies between the CMMS and the GIS accumulate as a result of the interface
process between the two systems.
Recommendation:
IA recommends the following:
• Take steps to reinforce and/or improve the current process to ensure that the results of
demolition and abandonment activities are communicated to the GIS department in a
timely manner, which will allow personnel the ability to update GIS for changes to
collection system assets and increase the accuracy of the GIS.
• Develop and implement a report/program, to be generated quarterly, that compares
individual assets one by one and identifies any differences in the specifications. Such a
report will enable GIS personnel to be more precise in making corrections in the system,
improving the accuracy of the database.
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process Level: Moderate
Process Owner Response:
Abandonments and Demolitions
A process is in place for removals and abandonments done by the Operations department to be
forwarded to GIS once work orders for this type of work (Remove or Replace Manhole, Remove
Sewer, Map Update) are completed. Reports are automatically run monthly for any work of this
type completed in the previous month. Watershed Engineers have 2 weeks to review these
reports before GIS does the map updates and sends the updated information automatically to
Maximo. The Asset Management Team monitors these reports and process.
The Assistant Director of Engineering for Construction has recently communicated and re-
emphasized the importance of clearly including information about removal, replacement, and
abandonment of any existing assets on project plans and as -built drawings. If information is not
clear to GIS, these drawings will be sent back to the appropriate Engineer for clarification.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 8 FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 4 — GIS (Geographical Information System) (Cont'd)
Process to Reconcile GIS and CMMS
The Asset Management group working with IT and GIS at the end of last year made a
comparison of assets in each system within each of the CMOM reporting areas, corrected
discrepancies, corrected system deficiencies, updated procedures, and followed up with
personnel as needed to keep the 2 systems synced. A request (PMO 20121195) has been
placed for and discussed with IT to develop a more frequent check between GIS and Maximo of
asset or asset specification discrepancies not only for CMOM related data but overall.
Date of Implementation: Re -emphasis of the need to clearly communicate asset removals and
abandonments has been completed. The report to compare CMOM assets between GIS and
Maximo should be completed by 12/31/13.
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 9 FY 2013 AUDIT PLAN
OBSERVATIONS AND RECOMMENDATIONS
Issue 5 — "As-Builts"
Condition:
IA noted the following regarding "as-builts":
• The district receives "as -built" information from external contractors in various forms and
formats.
• Instances occur in which the "as -built" survey data and related inspection field notes
differ from the "as -built" drawings as submitted to GIS.
Recommendation:
IA recommends the following:
• To improve the accuracy and efficiency of the process to input the "as -built" data into the
District's systems, require or request the contractors to submit "as -built" information in an
electronic form utilizing the MSD standardized format.
▪ Perform a reconciliation process between the survey data and field inspection notes and
the "as -built" drawings prior to submission to GIS, reducing or eliminating the potential
for any discrepancies in the data.
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process Level: Moderate
Process Owner Response:
By July 2013, the GIS Group will develop As -Built Drawing cadd standards. The urstrict will
direct its consultants to follow these standards on all projects awarded in Fiscal Year 2014. For
projects currently in design or construction the District will modify the consultant contract to
include this additional requirement, provided there are adequate uncommitted funds available to
address any increase in cost to the consultant for this work. With regard to as -built drawings
generated for Private Development projects, the District will notify the engineering community of
these new standards and request that those firms that use Cadd software comply with these
standards.
Engineering staff have been directed to ensure that the field inspection notes, the pay
application information and the as -built survey data are all reconciled prior to sending the as -
built drawings to the GIS Group for map updating.
Date of Implementation: 6/30/14
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 10 FY 2013 AUDIT PLAN
OPPORTUNITY FOR IMPROVEMENT
An opportunity for improvement, which requires no action plan from the process -owner, is as
follows:
Condition:
Currently, the crews develop the daily work routes from a large pool of work orders assigned by
the planners.
The crews are not in the best position to set up the routes in a manner that addresses the work
orders in a timeframe that is conducive to meeting the requirements of the Consent Decree.
Consider removing the responsibility of daily route development from the crews and add the
responsibility to that of the planners who are a better position to be able to schedule the work
orders in a manner that is conducive to meeting the requirements of the Consent Decree.
Process Owner Response:
The current process for planning and scheduling of worx for sewers starts witn Planners
developing routes for cleaning sewers in the upper reaches of the plant watershed. CCTV PM
work follows after cleaning of the reaches. These reaches are split among crews based on map
quadrants. Manhole inspections routes are similarly developed based on a list of areas
provided by the Engineering department. Routes are developed which include on average 1
month of work for a crew. A procedure was developed by the Planners and the crews trained to
look at the Maximo map and select their daily work from these routes by picking work orders in
closer proximity until the route is finished.
While our ultimate desire/goal is to have Planners schedule work day by day, for the volume of
assets and work orders involved in routes, there is no easy way for them to go into the map to
see what work is geographically close together and then schedule this work efficiently day by
day within Maximo. A number of discussion have occurred with IBM to see if Maximo can be
changed in future releases to facilitate daily scheduling. Additional discussions are planned in
the future regarding Maximo Scheduler software capabilities and future plans that might
facilitate MSD's daily scheduling needs.
This report is intended solely for the nse of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 11 FY 2013 AUDIT PLAN
ACKNOWLEDGEMENTS
Internal Audit Engagement Team:
MSD internal Audit:
Todd Loretta
Brown Smith Wallace:
Ron Steinkamp
Adam Rouse
Davis and Associates, Inc.:
Judy Dooling
We would like to thank District personnel for their excellent cooperation and assistance
during this engagement.
Specifically, we would like to express our gratitude to the following:
Operations:
Jonathon Sprague
Gene Stinnett
Ed Cope
Mike Merli
Rob Brand
Brian Hollandsworth
Rob Daly
Jim Foster
Sheri Sayles
Rick Parrott
Dan Heitert
Joe Anselmo
Engineering:
Marie Collins
Rich Unverferth
Madhukar Mohan
Evan Montgomery
Bob Miller
This report is intended solely for the use of The Metropolitan St. Louis Sewer District ("MSD") and is not intended to be and should not
be used by any other parties without the prior written consent of MSD.
Review of the CMOM Program 12 FY 2013 AUDIT PLAN