A Scenario-Based Benchmark for Assessing AI Safety Controls in Banking Operations: Metrics for Containment, False Intervention, and Recovery
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V5I3P124Keywords:
AI Governance, AI Safety, Banking Operations, Benchmark, Containment, False Positives, Operational Resilience, Circuit Breaker, Human-In-The-Loop, Model Risk ManagementAbstract
Banks Are Moving Ai Agents From Advisory Roles Into Operations, Where They Release Payments, Change Credit Limits, Settle Disputes, Triage Anti-Money-Laundering (Aml) Alerts, And Move Liquidity. Supervisors Expect Such Systems To Be Validated, Monitored, And Controllable, And Institutions Respond With Layered Safety Controls: Rule Limits, Anomaly Monitors, Human Review, And Circuit Breakers. Yet There Is No Standard, Repeatable Way To Measure How Well Those Controls Work, Or What They Cost When Nothing Is Wrong. This Paper Introduces A Scenario-Based Benchmark That Treats The Control Stack, Not The Model, As The System Under Test. It Contributes (I) A Failure Taxonomy For Operational Banking Ai; (Ii) A Library Of Ten Seeded Scenarios Across Five Banking Domains, Including Two Benign-Stress Scenarios That Expose Over-Intervention; (Iii) Three Families Of Formally Defined Metrics: Containment (Exposure Averted And Time To Containment), False Intervention (Wrongful Holds, Blocks, False Alarms And False Halts On Legitimate Activity), And Recovery (Time To Restore Safe Service And Share Of Leaked Exposure Reversed); And (Iv) A Paired, Seed-Controlled Evaluation Protocol With Bootstrap Confidence Intervals. A Reference Implementation Evaluates Six Control Configurations Over 200 Paired Episodes Per Scenario. Adding A Circuit Breaker With Automated Rollback Raised Exposure-Weighted Containment From 66.6% To 94.3% (Paired Difference 27.7 Percentage Points, 95% Ci [26.3, 29.1]) And Cut Median Time To Containment From 115 To 13 Minutes, But Roughly Doubled Median Time To Recovery (92 To 183 Minutes) And Halted Service In Every Benign Holiday-Surge Episode. A Calendar-Aware Volume Baseline Removed Those False Halts Without Reducing Containment. The Results Show Why The Three Metric Families Must Be Reported Together: A Control That Looks Best On One Can Be Worst On Another. All Results Come From A Calibrated Simulation And Are Evidence About Control Behavior, Not Predictions Of Production Loss.
References
[1] R. J. Bolton and D. J. Hand, “Statistical fraud detection: A review,” Statistical Science, vol. 17, no. 3, pp. 235–255, 2002.
[2] A. Dal Pozzolo, G. Boracchi, O. Caelen, C. Alippi, and G. Bontempi, “Credit card fraud detection: A realistic modeling and a novel learning strategy,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 8, pp. 3784–3797, 2018.
[3] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” ACM Computing Surveys, vol. 46, no. 4, pp. 1–37, 2014.
[4] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023, pp. 79–90.
[5] Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency, “Supervisory guidance on model risk management,” SR Letter 11-7, Apr. 2011.
[6] National Institute of Standards and Technology, “Artificial intelligence risk management framework (AI RMF 1.0),” NIST AI 100-1, Jan. 2023.
[7] European Commission, “Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” COM(2021) 206 final, Brussels, Apr. 2021.
[8] Basel Committee on Banking Supervision, “Principles for operational resilience,” Bank for International Settlements, Mar. 2021.
[9] European Parliament and Council, “Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA),” Official Journal of the European Union, L 333, 2022.
[10] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, “Concrete problems in AI safety,” arXiv:1606.06565, 2016.
[11] D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt, “Unsolved problems in ML safety,” arXiv:2109.13916, 2021.
[12] A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,” OpenAI, Tech. Rep., 2019.
[13] P. Liang et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023.
[14] X. Liu et al., “AgentBench: Evaluating LLMs as agents,” arXiv:2308.03688, 2023.
[15] Y. Ruan et al., “Identifying the risks of LM agents with an LM-emulated sandbox,” arXiv:2309.15817, 2023.
[16] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” in NeurIPS ML Safety Workshop, 2022, arXiv:2211.09527.
[17] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” in Proc. AAAI Conf. Artificial Intelligence, 2018, pp. 2669–2678.
[18] R. Parasuraman and V. Riley, “Humans and automation: Use, misuse, disuse, abuse,” Human Factors, vol. 39, no. 2, pp. 230–253, 1997.
[19] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006.
[20] E. S. Page, “Continuous inspection schemes,” Biometrika, vol. 41, no. 1/2, pp. 100–115, 1954.
[21] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering: How Google Runs Production Systems. Sebastopol, CA, USA: O’Reilly Media, 2016.
[22] International Organization for Standardization, “Information technology Artificial intelligence Management system,” ISO/IEC 42001:2023, 2023.
[23] I. D. Raji et al., “Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing,” in Proc. ACM Conf. Fairness, Accountability, and Transparency (FAT*), 2020, pp. 33–44.
