HomeMy Public PortalAboutExhibit MSD 70G - Plant-Pump Cybersecurity Audit Report - April -2019This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT FY 2019 AUDIT PLAN
Metropolitan St. Louis Sewer District
Plant/Pump Cybersecurity Audit
April 2019
Exhibit MSD 70G
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT FY 2019 AUDIT PLAN
The St. Louis Metropolitan Sewer District
Plant/Pump Cybersecurity Audit
April 2019
Table of Contents
Overview & Scope .................................................................................. 1
Objectives .............................................................................................. 4
Methodology ........................................................................................... 5
Overall Conclusion and Results ............................................................. 6
Opportunities for Improvement ............................................................... 7
Acknowledgements .............................................................................. 11
OVERVIEW & SCOPE
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 1 FY 2019 AUDIT PLAN
Overview:
To continue the efforts and commitment of the District to achieve proper organizational
governance, an engagement to perform a review of Plant/Pump Cybersecurity Controls was
included in the Audit Plan for Fiscal 2019. This is the first review of the Plant/Pump
Cybersecurity Controls since the completion of significant and recent changes to the District’s
overall business network infrastructure. This type of audit was last performed in 2011.
The District has seven (7) treatment plants:
1. Lemay
2. Bissell
3. Grand Glaize
4. Fenton
5. Coldwater
6. Meramec
7. Missouri River
In addition, the District has approximately 277 pump stations, which are divided into three
geographic service areas:
1. Lemay Pump
2. Bissell Pump
3. County Pump
These operational facilities rely on various Distributed Control Systems (DCS) and
Supervisory Control and Data Acquisition (SCADA) systems to provide continuous 24x7
operations. In general, these systems and applications are standardized at the various facilities,
providing a higher level of security assurance. All systems are Windows-based, locally
administered, and are on a separate network segment from the District’s business network.
The District utilizes two main applications Foxboro (a DCS) and iFix (a SCADA system) to
operate/control, monitor, and report on treatment plant and pump station operations. Each of these
products uses a proprietary application code and interface language and format. These applications
are coupled to Programmable Logic Controllers (PLCs) and field input and output interfaces that
adjust pumps, gates, flow rates, and collect readings and report conditions. Foxboro (a DCS) is the
main application used for the treatment plants. iFix (a SCADA system) is the main application
used for the pump stations.
The various facility hardware and software are supported by outside vendors: Foxboro, INS (iFix).
Ongoing daily maintenance activities, change request and program changes are performed by
Operations personnel that are part of a shared service group of support technicians. These same
technicians support the software, networking, interfaces, and sensor troubleshooting. The District’s
Information Systems (IS) group is responsible for providing secure connectivity between
facilities but is not responsible for managing any systems on the DCS/SCADA network.
OVERVIEW & SCOPE
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 2 FY 2019 AUDIT PLAN
Scope:
The scope of this engagement consisted of a review of the District’s cybersecurity controls
related to treatment plants and pump stations. This was a point-in-time audit and the majority of
on-site fieldwork was completed during the month of December 2018. IA performed facility
walkthroughs on a selection of plant and pump stations as indicated below:
# Visited/Tested Location Name Address Function *MGD DCS/SCADA
System
1 No Coldwater T.P. 13798 Old Halls Ferry Road
Florissant
Wastewater
Treatment 25 Foxboro
2 Yes Lemay T.P. 201 Hoffmeister Ave.
St. Louis
Wastewater
Treatment 121 Foxboro
3 Yes Bissell T.P. 10 E. Grand Ave.
St. Louis
Wastewater
Treatment 120 Foxboro
4 No Meramec T.P. 7849 Fine Road
Mehlville
Wastewater
Treatment 12 Foxboro
5 Yes Grand Glaize T.P. 1000 Grand Glaize Parkway
Valley Park
Wastewater
Treatment 15 Foxboro
6 No Mo River T.P. 3455 Creve Coeur Mill Road
St. Louis
Wastewater
Treatment 31 Foxboro
7 Yes Fenton T.P. 75 Opps Lane
Fenton
Wastewater
Treatment 4 GE iFix
8 Yes County Pump 1025 Grand Glaize Pkwy.,
Valley Park
Pump Station
SCADA N/A GE iFix
9 Yes Lemay Pump -
Lemay
8520 Virginia Ave.
St. Louis
Lemay ORS
SCADA N/A GE iFix
10 Yes Bissell Pump -
Bissell
10 E Grand Ave.
St. Louis
Bissell ORS
SCADA N/A GE iFix
* = Million gallons per day (MGD)
OVERVIEW & SCOPE
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 3 FY 2019 AUDIT PLAN
Risk Profile:
Cybersecurity can be categorized as a combination of
Confidentiality, Integrity, and Availability. Based on
discussions with the District’s Operations and I.S. personnel,
we determined that, as it relates to the District’s plant and
pump cybersecurity objectives, the following risk profile was
appropriate:
1. Availability: Plant and pump systems must be
available 24 hours a day, 7 days a week. System
availability is the most important priority and therefore
the highest risk to the District’s plant and pump
operations.
2. Integrity: Plant and pump systems must accurately report metrics (such as flow,
temperature, and pressure) to sustain operations. While not as important as system
availability, the District places an emphasis on accurate reporting by implementing
controls to prevent unauthorized system and data modification.
3. Confidentiality: Plant and pump systems do not store or transmit sensitive information.
Data confidentiality is not as high of a risk for the District’s plant and pump operations.
The District places emphasis on restrictive access controls.
OBJECTIVES
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 4 FY 2019 AUDIT PLAN
Objectives:
The overall objective of this engagement was to assess whether there are adequate
cybersecurity controls related to the systems at the District’s treatment plant and pump stations.
Specifically, IA assessed security controls related to:
Governance and Risk Management
Business Continuity and Disaster Recovery
Server and Workstation Hardening
Access Control
Application Security
Encryption
Telecommunications, Network Security, and Architecture
Physical Security of PCS Equipment
Service Level Agreements (SLAs)
Operations Security (OPSEC)
Education
Personnel Security
METHODOLOGY
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 5 FY 2019 AUDIT PLAN
Methodology:
Internal Audit (IA) leveraged guidance published in 2017 by the American Water Works
Association (AWWA). Specifically, IA aligned the best practices contained in the document,
Process Control System Security Guidance for the Water Sector, to the NIST
Cybersecurity Framework published in 2016.
Internal Audit (IA) accomplished the objectives by:
Determining scope/applicability of published security guidance with Operations personnel.
Conducting interviews and walkthroughs with District Operations personnel and
reviewing any documentation to develop an understanding of:
Treatment plant/pump station locations
Infrastructure and software to support DCS and SCADA systems
Key vendors to support DCS and SCADA systems
Obtaining and reviewing District policies and procedures related to plant/pump security.
Interviewing Operations personnel on business continuity and disaster recovery plans
and obtain related documented.
Interviewing Operations personnel and inspecting network diagrams to gain an
understanding of security architecture.
Interviewing Operations personnel on server and workstation security hardening
procedures.
Inspecting network and application user access lists to determine if users are authorized
to access DCS and SCADA systems.
Inspecting data encryption methods for data-at-rest and data-in-transit.
Performing physical walkthroughs for a representative sample of treatment plant/pump
stations to assess physical security of:
Deterrence (e.g., fencing, bollards, signage)
Detection (e.g., video surveillance, motion detection, staffed security)
Defense (e.g., staffed security, locked doors)
Performing limited physical penetration testing to assess effectiveness of physical
security features for a sample of plant/pump stations.
Inspecting vendor and service agreements that support plant/pump systems.
Inspecting system backup and recovery procedures.
Inspecting system patching and maintenance levels to identify potential system
vulnerabilities.
Interviewing Operations personnel on security and situational awareness related to the
District’s security awareness training program.
OVERALL CONCLUSION AND RESULTS
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 6 FY 2019 AUDIT PLAN
In the opinion of Internal Audit, in all significant respects, the cybersecurity controls implemented
pertaining to the District’s plant and pump facilities are effectively designed and implemented,
except for the current network design and procedures in place to reduce the vulnerability of the
DCS/SCADA systems to potential attacks from the internet. This matter and two additional
items are discussed in detail in the Opportunities for Improvement section of this report.
Initial Inherent Business Process Risk: Moderate Risk #
Overall Assessment of Engagement Results: Generally Satisfactory **
** Engagement results are evaluated as satisfactory, generally satisfactory, or unsatisfactory.
• Satisfactory (clean opinion) – No significant engagement findings or material
weaknesses were noted.
• Generally Satisfactory (qualified opinion, i.e., “except for”) – Results contain
significant engagement findings. No material weaknesses were noted.
• Unsatisfactory (adverse opinion, immediate Management attention required) –
Significant engagement findings and/or material weaknesses were noted.
^ DEFINITIONS
Engagement Finding (# Low Risk): An engagement finding is a condition that could adversely
affect the organization but is less severe than a significant engagement finding or significant
deficiency. Classification includes process or control deficiencies that are not significant
deficiencies as well as includes other low risk or low impact conditions.
Significant Engagement Finding (# Moderate to High Risk): A significant engagement finding is
a condition that could adversely affect the organization. Definition includes all types of findings,
such as irregularities, waste, ineffectiveness, conflicts of interest, illegal acts, errors, and
significant deficiencies in internal control over financial reporting as well as other significant
internal control weaknesses. A significant deficiency is defined as a deficiency, or a combination
of deficiencies, in internal control over financial reporting that is less severe than a material
weakness, yet important enough to merit attention by those charged with governance.
Material Weakness (# High Risk): A material weakness is a deficiency, or a combination of
deficiencies, in internal control, such that there is a reasonable possibility that a material
misstatement of the financial statements will not be prevented or detected and corrected in a
timely basis. For internal audit purposes, the definition also includes material and/or severe
irregularities, waste, ineffectiveness, conflicts of interest, illegal acts, errors, and other material
control weaknesses, etc.
(The term “material weakness” should be thought of as a serious category of significant engagement findings and/or
significant deficiencies. However, not all significant engagement findings and significant deficiencies are material
weaknesses.)
^ - Definitions are based on guidance from the IIA Standards, GAAS, and the PCAOB.
# - Risk is assessed at the District (Entity) Level. (Risk to the District as a whole)
OPPORTUNITIES FOR IMPROVEMENT
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 7 FY 2019 AUDIT PLAN
1. Limit Internet and Network Access to DCS/SCADA Network (Distributed Control
Systems/Supervisory Control and Data Acquisition Network)
Industry best practice recommends that organizations separate the DCS/SCADA network
from the business network and prevent the DCS/SCADA network from talking to the Internet.
Should the business network or the DCS/SCADA network be compromised by malware or a
coordinated attack, the network that was compromised should not be a direct threat to the
other network (the non-compromised network).
IA observed that the District’s network was designed to separate the DCS/SCADA and
business networks and to prevent the DCS/SCADA network from directly connecting to the
Internet. However, every DCS/SCADA server is configured to talk to the DCS/SCADA
network (to operate, monitor, and report on DCS/SCADA systems) and the business
network (to communicate with the centralized data repository). This effectively creates a
“bridge” between these two networks that were designed to be separated. This
configuration also allows the DCS/SCADA Servers to be directly connected to the internet
and the DCS/SCADA Network to be indirectly connected to the internet, which presents
significant risk. This also presents a risk to the business network as an attack originating
from the campus-wide DCS/SCADA network may go undetected with multiple points of entry
into the business network.
Recommendation:
Current State
Recommended State
DCS/SCADA
Network
Business
Network
Internet DCS/SCADA
Servers
DCS/SCADA
Network
Business
Network
Internet
Servers
New
Jump-
box
OPPORTUNITIES FOR IMPROVEMENT
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 8 FY 2019 AUDIT PLAN
IA recommends that District personnel from the Operations DCS/SCADA Systems group
and personnel from the Information Systems division work together to develop and
implement steps that ensure the needed DCS/SCADA equipment on the business network
resides on a dedicated VLAN (virtual LAN (local area network)). After ensuring that the
network is appropriately separated and defined, IA recommends that Information Systems
creates network access rules that would prevent access across the two networks except to
the “data historian” servers as needed.
To further minimize user access between these two networks, IA recommends that
Information Systems implement a dedicated secure management terminal (i.e., a “jumpbox”)
on the business network that Operations Staff connect through before accessing
DCS/SCADA equipment. This terminal will be a single point of access between the networks
and will provide assurance that the business and DCS/SCADA assets are protected from
unauthorized access. This will also allow Information Systems to enforce secure
management to the DCS/SCADA environment with manageable access controls while
keeping the two networks separate, minimizing the ability for an attacker to use either
network to attack the other.
As an additional security layer, IA recommends that Operations individually configure the
Windows firewall for each system in the DCS/SCADA network to prevent inbound and
outbound connections from any machine that is not on the DCS/SCADA network, including
any connections from the internet.
Risk Rating at District (Entity) Level: Moderate
Risk Rating at Business Process Level: High
Process Owner Response:
IT will assess and develop an implementation plan in support of this observation by
2nd Quarter of FY 2020. Testing and implementation be will completed by 4th
Quarter of FY 2020.
Date of Implementation: See above.
OPPORTUNITIES FOR IMPROVEMENT
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 9 FY 2019 AUDIT PLAN
2. Patch Facility Workstations
Security patches for the DCS/SCADA servers are applied when the vendor has tested and
released the patches to the District. However, IA observed that facility workstations that
connect to the DCS/SCADA servers are not patched for security vulnerabilities. Because
DCS/SCADA servers have access to both networks this presents an avenue for attackers to
move laterally through the network and attack District business resources through the
DCS/SCADA equipment network. Because these machines are not physically or logically
separated from the network and the internet, they present a serious risk and initial attack
vector (point).
Recommendation:
IA recommends that Operations work with their vendors to determine which security patches
can be applied to the facility workstations. Most importantly, our first recommendation in #1
above to enforce segmentation should also be implemented to refuse Internet connections
and therefore reduce the District’s exposure. A server in the DCS/SCADA network could be
used to automatically apply approved patches in an offline manner further assisting in
protecting these machines from being vulnerable in the future.
Risk Rating at District (Entity) Level: Moderate
Risk Rating at Business Process Level: High
Process Owner Response:
District Automation Team personnel will consult with vendor base on Plant DCS systems
(Schneider Electric) and attempt to collaborate with IS group in order to address this
concern. Issue seems to be confined to 6 work stations at Lemay and Bissel Treatment
Plants.
Date of Implementation:
By November 1, 2019
OPPORTUNITIES FOR IMPROVEMENT
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 10 FY 2019 AUDIT PLAN
3. Document Disaster Recovery Plan
Each facility’s servers are configured to perform an automatic backup, which is then
periodically copied to servers on the business network located in the REJIS data center.
While the servers could be recovered in the event of a disaster, the process and priority to
recover these systems has not been tested or documented. Without a disaster recovery
plan, the Operations group may run into inefficiencies that could lead to an unexpected time
to recover. This is a carryover recommendation from the 2011 audit (Recommendation has
not been completed/closed due to the District working on an overall disaster recovery
plan).
Recommendation:
IA recommends that Operations define, prepare, and document a disaster recovery plan by
specifying and detailing the following:
• Personnel involved along with their assigned duties (the who and the what)
• Timing of the procedures (the when)
• Locations of the procedures (the where)
• Methodologies to be used (the how)
• Management/oversight process (Supervisory, monitoring/review, and approval
procedures).
Risk Rating at District (Entity) Level: Low
Risk Rating at Business Process Level: Moderate
Process Owner Response:
Operations is working with IS group and vendor base to develop draft plans. Operations
intends to ensure our plan integrates with, and aligns with, I.S. generated overall plan in
terms of content and format.
Date of Implementation: By January 1, 2020
ACKNOWLEDGEMENTS
This report is intended solely for the use of The Metropolitan St. Louis Sewer District (MSD or the District) and is not
intended to be and should not be used by any other parties without the prior written consent of MSD.
PLANT/PUMP CYBERSECURITY AUDIT 11 FY 2019 AUDIT PLAN
Internal Audit Engagement Team:
MSD Internal Audit:
Todd Loretta
Brown Smith Wallace:
Ron Steinkamp
Bill Gogel
Zach Bayne
Doan Trieu
We would like to thank District personnel for their excellent cooperation and assistance during
this engagement.
Specifically, we would like to express our gratitude to the following:
Rob Daly – Operations Division Manager
Paul Milton - SCADA Systems Specialist
Carey O’Brien – SCADA Systems Specialist
Peter Lewis – SCADA Systems Specialist
Ginny Kienstra – I.S. Assistant Director
Jacque Weddington – Technical Support Manager