By Joseph Santamaria, Vice President - Information Technology and Chief Information Officer, PSEG Services And Timothy Weeks, Senior Director Application Support and Vendor Management, PSEG Services
Joseph Santamaria, Vice President - Information Technology and Chief Information Officer, PSEG Services
Every CIO has a responsibility to understand the threats and opportunities that digitization represents for their company. Those opportunities are different in each entity and range from eradicating disruptions to business models, to material changes in operating models. In most industries, the changes already are or soon will be material.
CIOs must lead these digital transformations to drive competitive advantage. Enterprise transformation requires vision, courage, talent, and credibility.
Credibility enables the CIO to be an evangelizer, painting a vision of the future and gathering resources to realize that vision. Building credibility changes based on elements such as culture, business model, or geography. In a business like PSEG, which operates a large gas, electric and generation infrastructure, the IT leader cannot build credibility without operating a digital environment as predictable and reliable as customers expect our services to be.
Four years ago, PSEG embarked on an IT transformation to provide services that could be counted upon anytime, under any conditions. For a system to be operational, it must meet criteria such as acceptable performance, functional and data integrity, and availability.
In 2012, PSEG experienced 1,162 minutes of unplanned outages across three SAP environments (SAP CRM, SAP Enterprise, SAP Human Resources, and Payroll) equating to 99.78 percent. While PSEG’s SAP deployment is one of the most extensive worldwide among energy companies (Call Center, Work Management, Payroll, Customer Account, Finance, and Supply Chain all run in SAP), this level of performance was not acceptable for the U.S. leader in electric reliability.
"The PSEG IT solution development life cycle now includes resiliency reviews for existing and new systems to address availability by design"
In 2016, PSEG experienced zero unplanned outage minutes across the same landscape. To accomplish these results, we focused on three areas: Engaging employees in defining a vision for a better future, deploying resilient process and architectural standards, and becoming a learning organization.
First, we made sure that employees understood that it would take everyone’s participation to reinvent our operational framework. Then we made our systems fit that framework, becoming more resilient. Finally, once systems were designed for resiliency, we reduced human errors.
Operational Excellence – A process for purpose not just for process
At one time, internal users were frustrated by performance issues and poor availability of critical SAP systems. Over four years, PSEG IT implemented initiatives that have completely reversed the situation.
We improved core IT processes, including life cycle planning, change management, patch management, production acceptance and solution development. PSEG sought to achieve CMMI Level 3 SCAMPI A certification within three years. Repeatable processes for excellence at every point in the solution-delivery life cycle are critical to maximizing performance and availability. An engaged workforce is powerful and includes employees and suppliers. With this in mind, PSEG established a Partner Ecosystem program that creates shared metrics and rewards, focused on critical systems. Suppliers agree to share common KPIs, put fees at risk and are rewarded for exceptional performance and given additional weighting when competing for PSEG’s strategic initiatives. These improvements are paying off in the reduction of defects, performance issues and unplanned outages for PSEG’s SAP systems.
Timothy Weeks, Senior Director Application Support and Vendor Management, PSEG Services
We needed to do more. Leadership focused the partner ecosystem (PSEG IT, business users, SAP, application support vendors) on chronic issues - the “CIO Top 10”. The list included poorly performing transactions, filing applications, and system availability. These were issues we had given up on, accepting poor performance. That attitude bred a culture of mediocrity that had to be changed. Once the first 10 items were resolved, a new list was created. None of these efforts required additional funds beyond labor expended and they helped redefine “acceptable performance” across the department.
An improvement initiative to formalize and strengthen the PSEG IT Root Cause Analysis (RCA) program was institutionalized using a Kepner- Treggoe-based process supported by ServiceNow. This disclosed that most unplanned outages were due to human error, infrastructure single points of failure, as well as poor development and testing practices. Teams were making mistakes; some were being repeated. The improved RCA program has transformed PSEG into a learning organization that shares lessons, avoiding mistakes and repeated failures. These are documented and shared with the entire IT ecosystem to further the culture of continuous learning.
Availability by Design
It’s said that projects don’t get into trouble; they start in trouble. IT systems must be properly designed and built to provide expected levels of performance and reliability. The PSEG IT solution development life cycle now includes resiliency reviews for existing and new systems to address availability by design. What we refer to as ‘N-1 or N-2’ analysis focuses on hardware, software, cables, communications, power, load balancers, etc. Resiliency reviews now are part of standard procedures for ensuring all end-to-end single points of failure are identified, acknowledged and either accepted or mitigated, based on business requirements for availability.
All system architecture components are stored in the CMDB with support expiration dates. Enterprise Architects ensure that required Life Cycle Planning (LCP) activities are identified and budgeted in future investment plans, ensuring the technology currency of each component.
No business can run and be successful without IT systems. User expectation for availability and performance is “always on, always fast.” Meeting those expectations requires the whole package described above; there are no silver bullets. It requires a culture of continuous improvement, a strong partner ecosystem and every process tuned to achieve excellence at every point.