Next-Generation Penetration Testing: A Cross-Domain Review of Challenges, Trends, and Taxonomy for Urban Digital Ecosystems

Penetration testing has become a vital aspect of modern cybersecurity as digital ecosystems continue to grow and diversify, particularly in the context of urban digital ecosystems where interconnected infrastructures face unique risks. This study provides a detailed analysis of penetration testing, following its evolution from manual, traditional methods to automated, AI-assisted frameworks designed for emerging technologies such as blockchain, quantum computing, IoT/CPS, cloud computing, and artificial intelligence (AI). By analyzing the progression of testing methods over time, we illustrate the increasing need for adaptable, scalable, and explainable testing solutions. The review classifies existing methods, evaluates their applicability across domains, and identifies important research gaps, including lack of real-world validation, fragmented evaluation standards, and disconnection from existing DevSecOps strategies. To fill this void, we provide a set of practical recommendations and propose a conceptual framework to guide the development of next-generation penetration testing tools. This paper aims to assist researchers and practitioners in developing more resilient, context-aware security methods for dynamic and hybrid digital infrastructures.