Why DevOps guys have to care about cybersecurity
12 minutes reading
Introduction
Thanks to my dual role as Pentester and DevOps engineer, I've been able to accumulate a wealth of experience. I understand the importance of cybersecurity and what's at stake, but also the challenge of integrating it into the DevOps philosophy. So I was keen to give my opinion on the security approach that every DevOps engineer should have.
What does DevSecOps mean
A little background
Far far away from here, in the 70's years tech companies were stating to define methodology and model to organize teams and improve software development.
Waterfall model
The first model was the Waterfall model. The Waterfall Model is a software development methodology that follows a linear, sequential approach to software development. It consists of a series of phases, each one building on the previous one and leading to the next one. The waterfall model is known for its strict adherence to planning and documentation.
For example, System admins worked tirelessly to keep everything running smoothly and afloat. Developers build and add as many features as possible, and finally, Quality Assurance (QA) engineers test the system's functionality, ensuring everything works as expected. If anything troubles the servers or something needs deployment, sysadmins will jump on it. If it's a code problem, devs will put out the fire. If there is anything to do with testing functionality and feedback, Quality Assurance teams will take care of it. But what if there is a flaw? A bug? Who fixes it? These situations led to many blame games and passing the baton around that created friction, things would get backlogged, and the symbiosis between teams would end up not working anymore.
This popular problem-solving strategy and system became a root cause of ineffectiveness in flexibility and communication across teams.
Agile model
Somewhere early in the 2000's, The agile manifesto appeared and introduced a new way to handle software development flow. The main concept was to iterate over the application development, by fixing minor objectives and bet on the long-term vision with flexibility.
Companies now value team collaboration and rely on self-organizing teams, focusing on clients and plenty of room for change and flexibility.
DevOps model
DevOps is quite different from the previous methodologies because it focuses on driving “cultural change” to increase efficiency. It does this by uniting the magic of all teams working on a project, using integration and automation. DevOps builds a philosophy that emphases building trust and better liaising between developers and other teams (sysadmins, QA, etc.).
With these ingredients, you get a cross-integration across all departments, QA+sysadmins+developers. For example, ensuring developers can now be involved in deployment and sysadmins can now write scripts, QA can figure out how to fix flaws vs constantly testing for functionality. By introducing automation and integration of systems, these engineers can now have the same visibility at all times and interact accordingly.
Also, many tools appeared to empower the model and improve Developers experiences:
- CI/CD
- Infrastructure as Code
- Configuration management
- Orchestration
- Monitoring
- Microservices
Important to notice:
You might have heard of the concept "Shifting Left." This means that DevOps teams focus on installing security from the earliest stages in the development lifecycle and introducing a more collaborative culture between development and security.
Nowadays, this provisioning of infrastructure in the cloud is automated. This shift has improved development productivity and speed. However, this increased velocity can also spark security concerns and lead to flaws that can go unnoticed.
At this step, DevOps is saw as a revolution, minimizing risk, automating process and improving visibility on projects.
Blending security in DevOps would enhance the impact of DevOps and eliminate a lot of other bottlenecks that could arise otherwise. With a rise in the frequency of cyber threats and tightening regulations, adding security to DevOps is not a choice now but indeed an obligation.
Welcome to DevSecOps
First, we have to define what does DevSecOps mean.
DevSecOps is an approach that relies heavily on automation and platform design that integrates security as a shared responsibility. It is a culture-driven development style that normalizes security as a day-to-day operation.
The things to understand at this step is that culture is key. It does not work without open communication and trust. It only works with collective effort. DevSecOps should aim to bridge the security knowledge gaps between teams; for everyone to think and be accountable for security, they first need the tools and knowledge to drive this autonomy efficiently and confidently.
Challenges
It is common for many security teams to be left out of DevOps processes and portray security as a separate entity, where specialised people can only maintain and lead security practices. This situation creates a silo around security and prevents engineers from understanding the necessity of security or applying security measures from the beginning.
This is not scalable or flexible. Security should be a supportive function to help other teams scale and build security, without security teams being a blocker, but rather a ramp to promote secure solutions and decisions. The best practice is to share these responsibilities across all team members instead of having a specialised security engineer.
Developers need environments to test new software without common security limitations. These environments are known as "SandBox," which are temporarily isolated environments. These environments have no connection to any internal network and have no customer data.
Culture
Promote autonomy of team:
- automating process
- add security testing as automate test
The ratio of developers, platform, infrastructure engineers, etc., won't be the same as security engineers, and we must understand they can't be in every conversation. Maybe you already heard about the 100/10/1 rule, which states that for every 100 developers, there are 10 platform engineers, and 1 security engineer. This is a common ratio in many companies, and it's not scalable to have security engineers in every conversation. This is why we need to promote autonomy and education across all teams.
For example, if you present a check before merging code, and the review doesn't pass and shows a message saying "signature of possible code injection flaw detected, please remediate," the developer or engineer should have access to the tool that is flagging that message.
In this example, a developer role can be created so that they have access to more information. This promotes education and autonomy by extending transparency that, traditionally, was only accessible by security teams
Security of the pipeline
Now we understood what is the DevSecOps challenges and objectives. Theses can be achieved by implementing an Secure Software Development Lifecycle, let's deep dive into it !
What is SSDLC ?
First, I have to show a graph to underestand the problem that we are resolving.
A study conducted by the Systems and Sciences institute at IBM discovered that it costs six times more to fix a bug found during implementation than one identified as early as during the design phase. It also reported that it costs 15 times more if flaws are identified during testing and up to 100 times more costly if identified during the maintenance and operation phases. See the figure below:
This problem lead to establish an SSDLC flow, using 4 steps we can define a solution to reduce the cost. Let's see !
Step 1: Risk Assesment
The risk assessment is pretty simple to understand and establish. Of course, it highly depends on your company and business but here are the big lines:
- List out your factor of risk
- Determine the value of data to be stolen
- Map the accessibility of the target
Step 2: Threat modelling
Threat modelling is a structured representation of all the information that affects the security of an application.
There as some well-know model already used, I will cover the 3 mains by giving a brief description of each one. I won't go into details because it's not my goal here, but I will provide some links to go further.
STRIDE
Spoofing: It is an act of impersonation of a user by a malicious actor which violates the authentication principle from the perspective of the CIA triad. Common ways include ARP, IP, and DNS spoofing.
Tampering: The modification of information by an unauthorised user. It violates the integrity principle of the CIA triad.
Repudiation: Not taking responsibility for events where the actions are not attributed to the attacker. It violates the principle of non-repudiability. For example, the attacker clears up all the logs that could lead to leaving traces.
Information Disclosure: It is an act of violation of confidentiality of the CIA triad. A typical example is data breaches.
Denial of Service: Denial of Service occurs when an authorised user cannot access the service, assets, or system due to the exhaustion of network resources. DoS is a violation of the availability principle of the CIA triad.
Elevation/Escalation of Privilege: Escalating privileges to gain unauthorised access is a classic example of a violation of the authorisation principle of the CIA triad.
Learn more about STRIDE: https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats
DREAD
Damage: refers to the possible damage a threat could cause to the existing infrastructure or assets. It is based on a scale of 0–10. A score of 0 means no harm, 5 means Information Disclosure, 8 means user data is compromised, 9 means internal or administrative data is compromised, and 10 means unavailability of a service.
Reproducibility: it measures the complexity of the attack. So how easily a hacker can replicate a threat. A score of 0 means it is nearly impossible to copy, 5 stands for being complex but possible, 7.5 for an authenticated user and a score of 10 means the attacker can reproduce very quickly without any authentication.
Exploitability: Referring to the attack's sophistication or how easy it would be to launch the attack. A score of 2.5 implies it would require an advanced skill set of networking and programming skills; 5 means can be exploited with available tools, a score of 9 means we would need a simple web application proxy tool and a score of 10 means it can exploit through a web browser.
Affected Users: it describes the number of users affected by the successful exploitation of a vulnerability. A score of 0 would mean that there would be no affected users, 2.5 shall mean for an individual user, 6 would mean a small group of users, 9 would mean significant users like administrative users, and 10 would imply all users are affected.
Discoverability: The process of discovering the vulnerable points in the system. For example, the threat would be easily found in case of compromise. A score of 0 would mean it would be challenging to discover it, a score of 5 means that the threat can be discovered by analysis of HTTP requests, and 8 means it can be easily found as it's public-facing. A score of 10 would mean it's visible in the browser address bar.
Learn more about DREAD: https://www.eccouncil.org/cybersecurity-exchange/threat-intelligence/dread-threat-modeling-intro/
PASTA
PASTA is an approach that consider the technical characteristics and the business logic. It is more general and comprehensive for non-technical users.
Define Objectives: The first stage focuses on noting the structure and defining objectives. This makes the end goal a whole lot clearer and ensures the relevant assets are threat modelled by defining an asset scope.
Define Technical Scope: This is where architectural diagrams are defined, both logical and physical infrastructure. Helpful in mapping the attack surface and dependencies from the environment.
Decomposition & Analysis: Each asset will have a defined trust boundary that encompasses all its components and data in this stage. For example, mapping threat vectors for a payment service and evaluating which components underlying the service can be leveraged for an attack; components can be libraries, dependencies, modules or underlying services etc.
Threat Analysis: This refers to the extracted information obtained from threat intelligence. Useful to identify which applications are vulnerable to specific vectors; for example, a customer/public-facing application can be susceptible to DDOS, unauthorised data alteration etc.
Vulnerabilities & Weaknesses Analysis: it analyses the vulnerabilities of web application security controls. It identifies security flaws in the application and enumerates vulnerabilities. It is highly recommended to add mitigation to the identified threat in this stage. For example, when describing a past incident involving an exploit of a mail server, lessons learned or mitigation: lack of thorough testing before implementation and hardening the server.
Attack/Exploit Enumeration & Modelling: In this stage, we map out the possible threat landscape and the entire attack surface of our web application. We then map all potential attack vectors to the different nodes, identifying exploits and attack paths. This stage simulates all the enumerated information extracted from all of the previous steps; this helps security professionals determine the extent and probability of successfully launching the identified vulnerabilities.
Risk Impact Analysis: Based on the collective data from the previous stages, all of the scoped assets that have been affected are analysed and finally, based upon the risk analysis, recommended steps to mitigate the risks and eliminate all of the residual risks are documented.
Yeah, they didn't find a good acronym ^^
PASTA stands for Process for Attack Simulation and Threat Analysis. It is a threat modelling more oriented on the business logic.
Learn more about PASTA: https://owasp.org/www-pdf-archive/AppSecEU2012_PASTA.pdf
Step 3: Code Scanning
At this step, we have to implement some tools to scan our code and detect vulnerabilities. There are many tools available, I will list the most common type. We will prefer includings theses tools in the CI/CD for example, to automate the process and avoid human error or forgetfulness.
SCA - Software composition analysis
Scan dependencies for security vulnerabilities, track and analyze any open-source component brought into a project.
SAST - Static application security testing
SAST uses a testing methodology of analyzing a source code to detect any traces of vulnerabilities
DAST - Dynamic application security testing
Tool to scan any application to find security vulnerabilities. In general, it is used on web applications and working using a headless browser to simulate a user experience.
IAST - Interactive application security testing
Analyzes code for security vulnerabilities while the app is running. It is usually deployed side by side with the application.
Step 4: Security Assessment
In the Software Security Development Life Cycle (SSDLC) flow, a security assessment is a critical step for evaluating and analyzing the security of a software application or system. It involves a systematic examination to identify vulnerabilities, weaknesses, and potential threats that could compromise the software's confidentiality, integrity, and availability.
Security assessments are typically conducted by a team of professionals, commonly referred to as penetration testers or ethical hackers. Their objective is to simulate real-world attacks and attempt to exploit vulnerabilities within the software. This process helps assess the effectiveness of existing security controls and highlights areas that require improvement.
The security assessment process involves several stages. It begins with planning, where the scope of the assessment is defined, including the systems and functionalities to be tested. The next phase is reconnaissance, where information about the target system is gathered. This can be done through passive techniques like reviewing documentation or active scanning to identify open ports and services.
Finally, a comprehensive report is generated, detailing the vulnerabilities discovered, their impact, and recommendations for remediation. This report serves as a valuable resource for development teams to address the identified security gaps and improve the overall security of the software.
By integrating security assessments into the SSDLC flow, organizations can proactively identify and mitigate security weaknesses early in the software development process. This approach helps minimize the risk of security breaches, enhance customer trust, and safeguard sensitive data.
Security in the pipeline
To implement security in your pipeline, I will mentions technical issues grouped by "stages of the DevOps lifecycle". For each stage, I will list some must-have and best practices to improve your security.
Development
IDE Security plugins
Use tools to improve your code quality and security
Pre-commit hooks
Prevent secret leaks or information disclosure
Peer review
Include security audit in code review
Coding standards
Define and teach security standards to your team
Containerization
Rootless mode
Run the Docker daemon as a non-root user
Use trusted images
Use well-know and audited images, with specific tags
Isolate containers with a user namespace
For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user on the Docker host.
CI/CD
SCA
Audit your applications dependencies
SAST
Audit your code for vulnerabilities or common mistakes
Sign images
Sign image to prevent images from tampering
Provision secrets
Provide application configuration and sensible data at build time to avoid secret leaks
Kubernetes
Define policies
K8S policies engine to validate cluster resources and state
RBAC
Ensure that cluster users and workloads have only the access to resources required to execute their roles
Pod Security Admission
Three levels defined by the Pod Security Standards: privileged, baseline, and restricted
Services mesh
Provide a management layer that enforces zero trust principles by providing identity-based service networking
Infrastructure
Infra As Code
Manage infrastructure automatically, prevent human error, don’t know your password
DAST/IAST
Test your application on production like environment
Cloud configuration
Manage your configuration using cloud provider resource, avoid custom configuration
Penetration testing
Order professionals to test your application security
Conclusion
I hope you enjoyed this article, and it will help you to improve your DevSecOps approach.
For me, the most important thing to remember is that culture is key, I learned a lot by preparing this article, and I hope you too.
Never forget that security is a shared responsibility and that security is a process, not a product.