container_patching_cnscon_2023_castle_panther.pdf

《container_patching_cnscon_2023_castle_panther.pdf》由会员分享，可在线阅读，更多相关《container_patching_cnscon_2023_castle_panther.pdf（51页珍藏版）》请在三个皮匠报告上搜索。

1、Container PatchingMaking It Less Gross Than The Seattle Gum WallGreg Castlemrgcastle,gregcastleinfosec.exchangeGKE Security,Google CloudWeston PantherGKE Security,Google CloudSimple View Of Patching0 days Scanner detects15 days Maintainer patches30 days Production patchedSeverityFedRAMPTargetsCRITIC

2、AL/HIGH30 daysMedium90 daysLow180 daysA Trip Down Empathy Lane2 weeks Bi-weekly cluster scan2 weeks Hey web team,webfrontend is missing 2 CRITICAL patches3 weeks Friendly ping?3 weeks Not our code,maybe django base container?3 weeks Django container team,can you patch?4 weeks These vulns are in perl

3、,we dont even use perl,do we need to patch?A Trip Down Empathy Lane4 weeks Yes,or better yet,remove perl.OUT OF FedRAMP/PCI SLO5 weeks Patched acme-django:v2.1.1 5 weeks Hey web team,rebuild with acme-django:v2.1.16 weeks Done!7 weeks Still running the old version?A Trip Down Empathy Lane8 weeks For

4、got to update the K8s manifest.Done!9 weeks Still no?Also theres three new HIGH vulns,but lets get this done first.10 weeks Had to soak in QA first,updated for prod rollout.11 weeks Fixed!Who else runs django apps.?Why Its GrossHumans at every stepWhich layer needs patching?No inventoryPatching unus

5、ed codeVulns faster than patches=Slow,incomplete,unscalable patchingIs the majority of the industry doing better than this today?88%of respondents:“Challenging to ensure containerized applications are free from vulnerabilities”https:/www.slim.ai/blog/container-report-2022/GKE Container Patching Case

6、 StudyEnforcement PointsPrevent:minimal containersDetect:scanning capability/coverageFix:ownership,dependencies,releaseMonitor:dashboards,alerting,escalationsWhat Containers?Vendor/MSP containersContainers you rebuildK8s manifests you updateWhat Do We Know Anyway?Patching for 1000s of containers acr

7、oss GKE,Anthos and adjacent productsButour environment constraints help a lot:Mandatory use of compiled language Mandatory container repo Mandatory base images Control over code/config pre-submit Control over releaseContainer/K8s Delivery PipelineSourceContainerBuildPackageRunDeployStaging RepoProd

8、RepoInventoryDevGood Start:Runtime DetectionSourceContainerBuildPackageRunDeployStaging RepoProd RepoInventoryDevRuntime scanner:detection and inventoryBetter:Prevention Complementing DetectionSourceContainerBuildPackageRunDeployStaging RepoProd RepoInventoryDevPrevention requires integration with p

9、ipelinePreventPrevent:ProblemsSo many containersSo many dependenciesMeeting SLO is hard without reducing volume#PreventPrevent:Strategy Standardize base containers Minimal containers:Less code,less vulns,less patching Remove unused code:separate build and runtime images Two approaches:Start small:Sc

10、ratch,Distroless,Wolfi/Chainguard Images Slim down:SlimToolkit Challenge:apply consistently everywhere#PreventOur Solution Standardize on Distroless Just enough to run golang binaries All containers in a single registry Inventory Availability#PreventOur SolutionSourceDevPre-Submit Checkimage=gcr.io/

11、gke-release/*deployment.yamlContainer/etc/os-releaseHOME_URL=https:/ distrolessAlternativesContainerBuildPackageRunDeployStaging RepoProd RepoIs distrolessimage=gcr.io/gke-release/*#PreventAlternatives:AdmissionContainerBuildRunDeployProd RepoisDistroless attestationAdmissionimage=gcr.io/gke-release

12、/*Verify isDistroless attestation#PreventOn GKE:Use Binary AuthorizationDemo:Admission#PreventPrevent:Summary Identify and use enforcement points Standardize on patchable base containers Standardize on container registries for inventory#PreventDetectDetect:Problems Which containers to scan?Which sca

13、nner?Different coverage Different vuln sources Duplicate handling Filtering noise Which layer has the vuln?#DetectAll Container ImagesWhich Container?Our SolutionSourceDevList of Containers in ProductionScannerPre-Submit CheckImage Fully Patched#DetectWhich Container?AlternativesInventoryList of Con

14、tainers In ProductionScannerIn registry,but what is running in production?DaemonsetProd RepoRun#DetectLanguage Pack ScanningSBOM ConsumptionVEX SupportSupplemental CVE SourcesScan programs in your container:Rust Cargo.lockPython egg filesGo binaries/go.modetc.Scanners are starting to support SBOMsFi

15、lter out remediated vulnerabilities based on VEXMore vulns from more placesOS vendor feedsGithub Advisories DatabaseLanguage-specific DBs(vuln.go.dev)#DetectWhich Scanner?Base Image DetectionReachability AnalysisAdditional ScansTry to determine the base images:From metadata in image manifestFrom the

16、 DockerfileTry to figure out if the code is actually in use:Typically uses sourceCan use symbol table in a binaryScan all the things:CIS benchmarksHardcoded keysMisconfigurations(root user,host volume mounts,etc.)#DetectWhich Scanner?False Positives vs.Coverage#DetectWhich one is correct?Vulnerable

17、module(golang.org/x/crypto/ssh)Built with old golang version(1.18.1)On old debian base(buster-20210208)Which Scanner?Our SolutionPublic containers:probably“more than one”Identify gaps and false positivesSee what our customers see#DetectDetect:Noise CVEs that will never be patched(debian CVE-2004-097

18、1,CVE-2005-2541,CVE-2010-4756)Ancient low priority vulns without patches(debian CVE-2011-4116,CVE-2016-2781)OS vendor has a lower rating than NVD(debian CVE-2022-37434)CVE is for a different architecture(golang CVE-2021-38297)CVEs that are clearly overrated(CVE-2020-29363:9.8 down to 7.5)#DetectUser

19、 ControlScanner Control Codepath is unused Recent CVEs with no patchNoise:Golang Specific Problem:all vulns in whole version or module detected Solution:govulncheck can report only reachable vulnerabilities go.dev/blog/vuln#DetectDemo:govulncheck#DetectDetect:Summary Take advantage of new advances i

20、n coverage Look to your scanner vendor to help with noise Use silence/ignore where it fits threat model#DetectFixProblems:Multi-layer Complex ProcessCVE 2023-123 in kube-proxyScannerThis Container?Find OwnerFile Bug/Send PRWait ForRebuildDependent Containers?DoneFind Parentyesnonoyes#FixOur Solution

21、:Base Images#Fixgke-distrolessdebian-basedebian-iptablesv1v1v1v2v2v3v31.Scan the latest base images2.If fixable vulns,rebuild3.Repeat for eternityv4Our Solution:Ownership#FixFind OwnerSourceDevPre-Submit CheckImage Ownership Filedeployment.yamlOur Solution:Simplified ProcessCVE 2023-123 in kube-prox

22、yScannerFind ownerFile Bug/Send PRWait forrebuildDone#FixSummary:Fix Track container parent-child relationships Automate patching base images Comprehensive inventory and ownership Use existing ticket systems to track#FixMonitorMonitor:Problems#MonitorContainer isnt patched-who is watching?Who do we

23、escalate to?Which containers have the CVE?Which applications use this container?Are we meeting our SLOs?What are the gaps and pain points?Is CVE-123 patched?Has it rolled out everywhere?What gets measuredOur Solution#MonitorNew Finding?Scanner Detects CVEFind Existing BugyesNearing SLO?EscalatenoBug

24、 FiledAdd CommentPast SLO?yesyesCVEsContainersApplicationsGKE VersionMonitor:Composition#MonitorCVE-2021-44228?Monitor:Visibility#MonitorNew Image CreatedPR MergedManifest UpdatedQualification StartsRollout BeginsPatchedClock startsDashboards provide status at-a-glanceTrack progress with metrics ove

25、r timeMeasure each step to find pain pointsActive CVE Count By ImageImage NameImage Tag#Fixable CVEsfake-imagev1.0.155fake-imagev1.0.345demo-imagev3.520nginx1.22.110Monitor:AlternativesInventory:ScannersComposition:Lyft:Cartography graph databaseSBOMs/GUACIgnore layers,just patch:copacetic,crane reb

26、aseSLO:Bug management softwareTrack commits and rollouts#MonitorSummary:Monitor Track SLOs over time Track patch/release stages to identify bottlenecks Use existing systems for escalation/dashboarding#MonitorSummary Standardize on registries and minimal containers Enforce as far left as possible Sca

27、nners for inventory+visibility Record ownership of containers Auto-patch if possible Tickets to track/escalatePrefer automation(doing)over tellingLinksDemo codeSlim.ai container reportLyft patching blogpostSeparate build and runtime imagesSmall images:Scratch,Distroless,Wolfi/Chainguard ImagesSlimTo

28、olkitAllowedRepos Gatekeeper policySigstore:signing,policy controllerGKE Binary Authorization:attestations,image policyOpensource scanners:trivy,clairGoogle Container AnalysisGUACThe Seattle Gum WallFeedbackmrgcastlegregcastleinfosec.exchangeAppendix:Feature Request Wishlist Idea#Detect Prisma Cloud

29、 has received 150 reports that dispute this severity Aqua Security has received 231 reports that dispute this severity Google Container Analysis has received 109 reports that dispute this severityIf enough users report a critical vuln as inaccurate,the scanner manually evaluates,updates the severity for all their users,and works with NIST to correct NVD#Detect