1、 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Z U R I C H|1 1 S E P T E M B E R 2 0 2 5 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Try again:The tools and techniques behind resilient systemsAhmed GamaleldinA P P 3 0 9(he/him)Principal Technical Account M
2、anagerAmazon Web ServicesReem Gaafar(she/her)Snr Technical Account ManagerAmazon Web Services 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.AgendaThe trouble with retries,and what to do insteadBreaking circuitsErasure coding vs.the tailA theory of stable systems 2025,Amazon Web
3、Services,Inc.or its affiliates.All rights reserved.Rethinking retries 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.ClientTimeout3 retriesExponential backoffJitterClientClientServerLatency=A+B*concurrency 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.ClientT
4、imeout3 retriesExponential backoffJitterClientClientServerLatency=A+B*concurrency 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.Transient failure
5、sCommon in redundant systems,caused by individual component failuresRetries helpSystemic failuresCaused by load,correlated behavior,software bugs,operational issues,and so onRetries harm 2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.2025,Amazon Web Services,Inc.or its affiliates
6、.All rights reserved.Try callAdd 0.01 tokensIs there a token in the bucket?Are we below the max try count?2025,Amazon Web Services,Inc.or its affiliates.All rights reserved.No retries:Great for systemic,no help for transient 3 retries:Terrible for systemic,great for transient Adaptive:Great for syst