DevOps in Banking and Financial Service — Challenges & Opportunity, Part I
Introduction
In 2009 when Patrick Debois and Andrew Shafer coined the term “DevOps” since then until 2013 DevOps had been always a story of internet unicorns — Etsy, Netfix, Facebook , Google, Amazon- all the engineering driven technology companies. And almost same time it started picking up momentum in Silicon Valley internet startups where cloud adoption and DevOps became the path to success in fail-fast business environment. However since 2011–12, other large companies including banks and financial services, to remain competitive in disruptive marketplace also started depending on software driven innovation and adopted new technologies of SMAC (Social-Mobile-Analytics & Cloud) . These companies were also influenced by popular DevOps success stories of internet unicorns — “Amazon deploying thousands of changes per day” and wanted to find a better and efficient way to scale their business and deliver new customer experience. Banking and financial services were no exception to this! In spite of having several regulatory compliance and governance obligations, hierarchical line of business (LoB) driven organization culture along with challenges of dealing with complex legacy systems, banks and financial services (BFS) companies started exploring DevOps — its ideas, practices, tools and technologies and cultural influence.
In this 2 parts article, based on my experience working with several BFS clients in cloud and DevOps adoption, I’ll describe the challenges and opportunities of DevOps adoption in BFS companies. In the first part I’ll talk about the challenges of DevOps adoption in global financial institutions where many of them have huge investments in legacy platform and technology in contrast to digital banking startup companies and online fintech disruptors.
Common Challenges
BFS enterprises share many common technology problems and business challenges with those of internet Technology Company and cloud service providers. All of them want to have new idea to develop innovative product, infrastructure and technology practices that would help them do that in speed with improved quality and reduced cost. And to achieve these all of them face problem of scale. Let’s take an example of any large global trading company — though they cannot compete with Facebook or Google in terms of number of users accessing the application, still they have systems and applications which executes very large volume of complex transactions with very high value in any trading day. It could even be at the scale of very high precision processing of millions of messages per second. Also, like online tech companies, business ( and IT) of BFS enterprises are under constant pressure to deliver continuous growth, meeting short-term targets and which needs support of software based products and needs scalable infrastructure which can handle increased performance requirement. But, all of these are required for BFS enterprises with one important and mandatory requirement — not at the expense of security and reliability of service. So, BFS enterprises needs DevOps way of working within the constraints of security, compliance, governance requirements and stringent performance SLAs.
BFS Industry specific challenges
We recently observe that many of our BFS clients are adopting agile to bring in cultural changes within their organization to improve agility and some of the Banks have already pioneered the Agile way of working. Some of them have not only reached at higher level of engineering practices maturity by building Continuous Delivery pipeline, but they have also started cultural transformation by forming Squad based team (following Spotify model) or recruiting Site Reliability Engineers (following Google) in their operations team. Still there are certain technical and cultural differences between working of a global bank and online eCommerce site, which needs to be understood and considered for successful adoption of DevOps. I’ll talk about 4 major observations of these differences from my experience, which needs to be understood and addressed properly during conversation with BFS clients in order to set their expectation for successful adoption of DevOps
1. BFS Systems and Application Complexity
BFS industry is the oldest and largest adopter of information technology. Many of the large global banks had hugely invested in legacy technology, which for majority of the large banks had been more complex over last decades due additional inheritance of legacy systems due to merger and acquisitions. These systems were not designed for rapid changes.
· Although mobile and web channels are prevalent in retail online banking, trading etc, it depends on B2B integration of financial systems based on industry standard messaging of SWIFT, FAST etc, in some cases using low latency proprietary APIs and sometimes on Mainframe transactions. Therefore the impact of “2 Speed IT” is more on overall agility of delivery BFS products in the market compared to those in other industry
· Having a complete stateless architecture in highly interconnected B2B financial systems that exchange thousands of messages per second at low latency could be a challenge
· As stateless architecture works well in one-click deployment which most of the online web company practices. However for BFS enterprises it might not be possible or might not be required for all applications. Continuous Delivery pipeline of BFS applications would make every changes ready to be deployed, but actual deployment of changes to production might be coordinated and controlled by operations and compliance team. In highly regulated environment of BFS there could be strict requirements of separation of duties and formal release approval process.
· 24/7 operations like eCommerce web portals might not be required for financial trading companies which operate in smaller window of a trading day. It means there could be a built-in window of schedule maintenance and upgrade after trading hours, as very large volume of transactions usually take place during that small window of operation.
2. High Cost of Failure
One of the principle of DevOps is to “Fail Fast”. Since the inception of DevOps culture almost all the online technology companies have preached “Move Fast and Break Things”, “Culture of Accepting Failure” etc. However, financial industry does not promote or accept this principle of accepting failure and they invest a lot to prevent failures to happen.
With lightning speed of changes rolling out to production in almost every 10 seconds by unicorns like Amazon, Netflix etc, a small defect in the changes can have disastrous effect on the entire business. In recent times there are real examples of failure/outage by AWS, Google where they lost online revenues. It also has softer aspect of losing customer confidence which in turn could lead to loss of greater revenue. In 2013 Amazon lost nearly $65K USD per minute due to its 30 minutes sudden outage.
However, if we compare the above impact of failure with any outage or failure of any mission critical financial applications, the above figure does not appear as a big loss as failure of the financial application might have impact on national or global economy. It can also badly impact the confidence of investors on that financial company which might have larger implication to its overall business. In addition to this there are other additional costs that the impacted financial company might require to bear — regulatory fines, law suites, bringing in expert to review systems and processes, replacement of technology etc. Remember the famous NASDAQ failure during launch of Facebook IPO. Therefore there in general perception in the technology world that in BFS the speed of development and delivery of critical financial applications are slowed down by the process of preventing and mitigating these failures.
3. Compliance Requirements
Fulfilling Regulatory compliance requirements is basic fact of life in BFS industry. It impacts almost every system and applications in terms of requirements, design, testing and operations including the way employees would execute their operational responsibility. While all the online eCommerce company follows stringent auditing requirements for PCI DSS compliance, every BFS companies similarly follow many of the industry and country specific auditing requirements for almost every activity. And often times these are overlapping and conflicting requirements to be followed by business and IT of BFS.
The most challenging part of these regulatory compliance requirements are that they are ever changing with the change of governments, change of socio-economic-political climate of specific country/geography etc. For example, due to regulatory ask, compliance department of any bank might enhance its Know Your Customer (KYC) program so that they can analyze the spending pattern of accounts to identify suspicious transaction patterns and remain compliant with anti-money-laundering (ALM) regulations. From the software engineering process standpoint it means that there has to be a review and approval process for these enhancement requirements, there would be auditing requirements to be implemented in system design along with other stuff and there would be a process to review and approval of testing outcomes along with independent manual audits on control process, before the KYC application goes live in production.
Sometimes the last part of the audit process as mentioned above might cause impediment to the overall agility as the auditors might want to see the evidence of security testing, meeting minutes or checklists as provided to/by Change Advisory Board (CAB) and change management policies and procedures etc. In short, to verify compliance and regulatory requirements auditors would like to see the separation of duties between business, development and operations teams which DevOps advocate would rather consider as raising wall of silos that should be torn down.
4. Stringent Security Requirements
We all know that security is fundamentally important for BFS industry. There are new cyber threats coming up every day and financial institutions are the primary targets of these unethical hackers. Today’s BFS applications are more vulnerable to such attacks as most of the applications are multi-channel app, accessible over web and mobile. As these apps are integrated and dependent on many back end financial applications, it automatically provides larger attack surface which when impacted by security attack might cause halt of an entire economy even for a while. One of the notorious example of such attack was NASDAQ hacking.
Because of these increased risks, security compliance team of BFS companies still rely on stage gate reviews which give them opportunity and time to do their security checks and asserting control over system and application changes.
As we see above that there are certain technology and industry specific challenges for DevOps adoption, still BFS clients understand that risk of not adopting DevOps are too big to ignore for them by not being able to deliver value to their customers quickly and losing them to online fintech startups powered by DevOps. In the part 2 of this article, I’ll share experience of driving DevOps adoption effectively in BFS.
