What We Need to Practice in 2022 to Prevent Log4j type Vulnerabilities of 2021 and SolarWinds Threat of 2020
We all agree that 2020 and 2021 had changed the course of human history in every aspect — social, economic, political, and technical. Both in 2020 and 2021, we were busy to keep ourselves safe from Corona Virus and its spread, eagerly waiting for the invention of its vaccine, rushing ourselves jabbed and being panicked again by constant genetic evolution of this virus into new strains. In 2022 inspite of lightning spread of Omicron we see ray of hope at the end of the tunnel as experts are saying “pandemic” will be converted to “endemic” very soon. Similarly, When I look back at last two years and think what events have shaken and challenged the technical supremacy of IT and Software Development industry, I find couple of very similar incident which occurred during end of 2020 and 2021. I find a striking similarity of “pandemic” type attack on software supply chain in both of these years, but I am not sure that this will not reappear as another new “strain” in 2022 and any vaccine would be invented for that (it is just not jabbing antivirus software into a computer!). There are certain software development practices, as experts suggest, if followed rigorously may reduce the probability of such security threats. Let me explain!
A month back on December 9 , 2021 when Apache released a super critical Log4j vulnerability which impacted all most all IT companies and their clients. As many of us are/were Java developers, designers, architects, we know how pervasive the usages of Log4j are in any business-critical IT applications for our clients. I’ll not get into details of these vulnerabilities and how to remediate it. There are bunch of good articles available which I think most of us by this time have gone thru.
Quick rewind to December 8, 2020 FireEye announced highly sophisticated intrusion of malware into U.S computer networks where multiple U.S government agencies were targeted . Attack originated from some software update from a company named SolarWind and hence it was also known as SolarWinds hack.
Though I am not a security expert, I’ll share some of my thoughts in this blog based on experience and study which I think is of utmost importance for all of us, who are practicing architects/designers/engineers/developers, to understand and practice certain things in order to be prepared for remediating such security risks and prevent them to happen for our clients. This is not rocket science, many of us may be practicing some of these things already! Nevertheless, this is still very important to reiterate these amongst our technical community.
1. In today’s cloud native modern application development, we all are using or consuming different open source software libraries, building container images with all of these and running these containers in production. To quickly roll out revenue generating products developers use these types of libraries to speed up their development processes which increase the threat surface of the entire software supply chain execution path and makes it vulnerable to Log4j type of security breach. Question is how we can get rid of this? Is distroless the right idea to be more secure? By distroless we mean eliminating OS package managers, shells, and libraries from container image as much as possible. Reducing size of the container images in the execution path reduces the threat surface and probability of attack, but to achieve that more important is the process of standardisation and quality of software chosen by any enterprise in the execution path of production code. In distroless OS, one good example would be Red Hat Universal Base Image (UBI) which is highly performant, secure and deployable onto many different hosts. Google’s distroless images are also good example in this regard. Even if it is a distroless OS chosen for container development, I think most important is the process of provenance to ensure what has been built, is built securely.
2. I was reading Google’s whitepaper on Binary Authorization of Borg, an engaging article on software engineering practices followed in Google for code review and its provenance process. I think it’s a must read for all of us! There is another excellent blog from Google on this is here which talks about a framework for securing supply chain levels of software artefacts or SLSA. In my experience I have seen that security is the least thought of area in the development process, although these days things have improved because of wider adoption of DevSecOps. Some of the practices mentioned in the Google articles, if followed, will improve security, and help in defending integrity threats which gives rise to SolarWinds kind of attack.
3. The way securing software supply chain as described above is important to prevent security breaches, container supply chain (which is also part of software supply chain ) is as equally important to be managed in very controlled fashion to prevent security attacks. In my experience with financial institutions and banks I have seen they strictly restrict developers to a set of container images that pass security, quality, and conformance criteria. They set up enterprise registry, configure containers to host inside the organization to pull images only from this registry, rather than any other external or public registry. Enterprise registry not only mitigate risk of using uncertified images, but it also helps in defining approval workflow to move validated images from development to production thru DevSecOps pipeline. Red Hat Quay Enterprise is an example of such container image registry with advance features such as security scanning, role-based access, organisation, and team management and auditing etc. Other similar products are from JFrog and Nexus.
4. Last but not the least I’ll mention about OpenSCAP, an open source project that develops tools for implementing and enforcing security policies using Security Content Automation Protocol (SCAP) standard. I am referring to SCAP because today’s hybrid and multi-cloud IT environments are interconnected with hundreds and thousands of interconnected systems, applications and services built using open source softwares and accessed by diverse set of devices, human users and applications. Therefore, it is very much required to maintain control over the security of this type of vast and diverse environment and device a standard way to scan systems for compliance with security policies. SCAP security guide can be installed in Red Hat Enterprise Linux and use oscap command to generate the HTML security guide for a specific profile. Also, GUI tool of SCAP workbench can be installed to perform configurations scan and perform remediation of the system.
Overall, I think more we are informed on some of the above aspects of security and more we practice these during the software development process, we will be able to build a more secured IT environment.
