2024.1.31：Continually Verifying and Validating Compliance as ADAS Features and AV Performance Are Developed (Commentary)

Key Takeaways

• Successful adoption of ADAS and AV requires trust―trust by the end users and trust in the knowledge and solutions used to design, develop, and validate the vehicles.
• Continuous verification and validation of compliance to regulatory requirements while engineering for performance improves the successful introduction of new ADAS features and AV products.
• Modern virtual engineering ecosystems utilizing scenarios learned from actual product usage enable continuous verification and validation as design changes are considered.
• Siemens Digital Industries Software solutions enable on demand, continuous verification and validation even as regulatory and customer requirements evolve over a product’s lifecycle.

Introduction

Continuous verification and validation (V&V) of Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AVs) is a critical narrative ensuring a product’s safety and performance throughout its useful life. Understanding how to satisfy changing requirements, whether consumer-driven or regulatory, is faster and easier when using computer-based design and powerful simulation ecosystems. A systematic approach to continuously test, assess, and validate the functionality and safety of ADAS features and AV as they evolve is essential to remain competitive.[1]

Deploying new ADAS features to existing products requires earned customer trust. Trust grows when using a virtual engineering ecosystem to frequently evaluate actual usage patterns (aka scenarios). Enabling continuous V&V and keeping regulatory compliance always visible during evaluation of product performance is needed. Continuous V&V testing is affordable due to decreasing database and compute server costs.

Automakers perform verification and validation as a step after detailed design of components and subsystems and then the integration of product prototypes containing them. Modern databases and on-going learned load cases (driving scenarios and the physical loads they excite) make frequent evaluation possible and can be done in parallel with development. This shortens the repair cycle when flaws are discovered.

This systematic assessment method, employing continuous verification and validation, requires several capabilities including data collection from continuous monitoring, scenario-based testing, earlier algorithm validation, product software updating, regulatory compliance, machine learning for model validation, user feedback during incident analysis, redundancy for fail-safe mechanisms, and necessary reporting. This commentary discusses these capabilities and explains why each is integral to achieving continuous verification and validation to regulatory and customer requirements while system performance development and upgrades proceed.

Assuring Compliance During Development and Use

The replay of customer usages captured as recorded scenarios combine vehicle motion with sensing scripts and forms the basis for continuous ADAS and AV systems V&V. Waiting solely on physical prototypes for V&V is no longer required. And while a final physical validation is still likely, the development process and evaluations of customer reported anomalies improves with trusted models used within closed-loop simulations. Trusted models come from correlation to actual product usage experiences. Trusted models are adjusted to reflect the latest usage measurements. More accurate models create a verification and validation environment where new ideas can be assessed quickly, including their effects on regulatory compliance. Using continuous verification and validation for complex ADAS and AV electro-mechanical systems is needed. 

Figure 1 shows broader, environmentally driven experiences with an existing product in more extreme weather conditions. The evolving ADAS and AV product can be assessed quickly when new usage scenarios are captured and synthesized based on physical understanding and knowledge. Continuous V&V improves with the newest learned scenarios, both measured and synthesized. Keeping regulatory requirements and industry standards visible in the context of the evolving product design shortens the validation cycle. This effective continuous V&V approach is applicable to electro-mechanical systems as software based models of mechanical mechanisms replace physical prototypes.

Fast-paced features design, verification, and validation are a key to successful competitiveness when developing ADAS and AV products. Continuous V&V with trusted systems models that evolve as knowledge is discovered makes engineering more responsive to the market and changing operating environments. Robust, compliant products with reliable upgrades are becoming the hallmark for great companies, keeping customer safety and satisfaction in the forefront.

Meeting all the requirements, standards, and performance objectives as the AV design evolves and while it is used is needed by the AV OEMs. The massive number of user scenarios needs to be managed. Keeping requirements traceable, visible, and assessing the AV systems’ ability to meet them is essential. V&V tasks can be done physically or virtually. When done virtually the only limit to continuous execution is compute machine power and database capacity. Advances in both continue to make virtual engineering more and more affordable and faster―making continuous V&V possible.

Siemens’ Solutions Enable Continuous Verification and Validation

By identifying and providing continuous verification and validation capabilities, Siemens has assembled a requirements traceability ecosystem that can enable automakers to competitively bring new ADAS features and AVs to market. Making the achievement of requirements visible as engineering development proceeds is insightful. Siemens shows this in a video by reporting the requirements coverage as each scenario is evaluated. This assessment helps engineers understand the level of requirements coverage achieved. 

Furthermore, it provides insight into which scenarios need enhancements or expansion to improve the requirements coverage. Siemens has provided CIMdata detailed examples which show the power this method brings to the systems and validation engineers.

Siemens’ scenario management combined with interactive requirements coverage made contextually visible within accomplishment dashboards and the 3D product mockup enables continuous verification and validation. The scenario database learns from and adapts to synthesized data based on a known understanding of the physical world. New, measured data augments the classical understanding, often leading to new insights on coupled physics. Being able to simulate these new phenomena builds trust in the predictive performance models. Compliance, through comprehensive traceability, is always visible from the start. The value this provides automakers is clear―optimized products that meet and anticipate mobility market desires.

Specific capabilities Siemens is providing to automakers that enable the continuous V&V narrative include:

• Data Collection and Monitoring
  Keeping the scenarios comprehensive―covering the wide range of possibilities, is essential. Continuous data collection from different sensors (e.g., cameras, LiDAR, radar, and other sources) measures how the system perceives the environment. Real-time monitoring of system performance allows for immediate detection of anomalies, malfunctions, and deviations from expected behavior. The definition of various driving scenarios, including a variety of environmental conditions and traffic situations, is driven by actual customer use.
• Scenario-Based Testing
  Scenarios must be regularly updated and expanded to cover a changing, wide range of real-world driving situations. Real world situations include sensor soiling, fog, rain, bright light, and traffic congestion. By using simulation, both offline and in real-time, a system’s responses to scenarios, including those that are difficult to physically recreate, can be thoroughly evaluated.
• Earlier Algorithm Validation
  Virtual engineering solutions using trusted simulation models enable earlier algorithm validation and software verification while ensuring that new features or improvements do not introduce unexpected behaviors or vulnerabilities.
• Dynamic Updating
  An iterative over-the-air (OTA) solution delivery helps keep ADAS and AV systems up to date with evolving customer expectations, usages, road conditions, and changing regulations. Continuous V&V provides the evaluation ecosystem as these incremental solutions are crafted, improving the refined systems’ behavior confidence while reducing risks.
• Regulatory Compliance
  Automakers must stay abreast of evolving ADAS and AV regulations and standards to ensure compliance with local, national, and international safety regulations. A continuous V&V ecosystem keeps requirements traceability and compliance visible all the time.
• Machine Learning and AI Model Validation
  Trusted models need continuous training to remain valid through machine learning using fresh data to adapt to changing road conditions and user behaviors. By monitoring model correlation accuracy, retraining is done when needed to improve their trustworthiness and applicability.
• User Feedback and Incident Analysis
  Involving humans with their unique driving variations in the feedback process is vital. The ADAS and AV systems must protect humans, both drivers and pedestrians, from unsafe operations. The integration of user feedback and all incident investigation analyses within the continuous V&V framework improves requirements traceability.
• Redundancy and Fail-Safe Mechanisms
  Using frequent tests and validation to examine and explore critical components’ redundancy ensures reliable fail-safe operation. Comparison of alternative fail-safe mechanisms is faster and more effective in virtual engineering trade studies.
• Reporting
  Easy to produce on-demand reports and dashboards of all testing, validation, and incidents, including correlation studies establishing model trust enable personnel at all levels to monitor the system’s safety and performance progress.

Several CIMdata articles summarize the expansion of MBSE driven, multi-discipline capabilities that Siemens has developed. Siemens’ solutions are already enabling advances in electric vehicle engineering, systems and software engineering, executable digital twins, manufacturing planning, and operations, all contributing to accelerating every aspect of product development, production, and use. Siemens is now focusing those capabilities on ADAS feature and AV product continuous V&V by managing massive numbers of scenarios in all the different development environments from model-in-the-loop to software-in-the-loop, to hardware-in-the-loop, and finally systems-in-the-loop. By using the same scenarios repository, verification and validation are always using the latest insights from the customer while assessing requirements coverage. Siemens’ effective scenarios management and affordable computing enables continuous verification and validation.

Conclusion

Keeping and building trust with customers is crucial. Fast ADAS feature and AV product verification and validation is a key to success. Continuous V&V from Siemens can be a competitive advantage. Siemens is developing a growing set of capabilities for data capture, scenario-based analysis, critical scenario creation, and seamless system optimization. This can improve the continuous V&V of AV/ADAS features.

CIMdata believes Siemens continuous V&V framework will encourage and enable more rapid ADAS features and AV product development in new ways, expanding virtual engineering and evolving evaluation practices. By combining the ADAS/AV scenarios management, virtual engineering broadens, and trust grows―the trust needed for a completely autonomous transportation future. CIMdata recommends automakers should consider Siemens’s Xcelerator technology platforms when evaluating solutions for continuous V&V during ADAS and AV development.

CIMdata believes that Siemens’ focus on the narratives, shown in sky blue in Figure 3, will help enable improving system optimization and efficient verification and validation based on a complete usage database comprehending relevant scenarios as they are discovered. Advances in computing make continuous verification and validation possible even when new scenarios learned from the latest customer usage are discovered. 

To learn more about Siemens capabilities for engineering ADAS features and AV products, see CIMdata’s previous papers and commentaries in this series. CIMdata has written about the general challenges for ADAS and AV engineering work in a white paper and has explored the need for scenario management in a recent optimization commentary. The use of simulation during earliest design concept with a capability Siemens calls front loading has also been published.

[1] Research for this commentary was partially supported by Siemens Digital Industries Software.