• The researchers assessed the AI’s capacity to detect potential security vulnerabilities across seven commonly observed types.
  • While economic impacts are significant, advocates prioritize environmental conservation and Indigenous rights, setting the stage for potential conflict.

Although AI, such as ChatGPT, shows impressive capabilities in code generation and analysis, there are doubts about its effectiveness as a reliable security auditor.Highlighting the importance of utilizing both specialized tools and human judgement to conduct comprehensive evaluations of potential vulnerabilities.

Assessing AI systems for common security vulnerabilities

As the capabilities of artificial intelligence (AI) continue to expand, researchers from Salus Security recently put GPT-4 and other AI systems to the test in detecting common security vulnerabilities. The study, conducted by a pair of experts from Salus Security, a prominent blockchain security company with operations in North America, Europe, and Asia, aimed to assess GPT-4’s proficiency in parsing and auditing smart contracts.

Also read: A look at Alphabet’s Gemini, the AI model aiming to challenge ChatGPT-4 

Exploring the dual nature of AI

The research revealed that while AI, including ChatGPT, demonstrates a commendable aptitude for generating and analyzing code, its reliability as a security auditor remains questionable. Leveraging a dataset of 35 smart contracts known as the SolidiFI-benchmark vulnerability library, which encompassed a total of 732 vulnerabilities, the researchers evaluated the AI’s ability to identify potential security weaknesses across seven prevalent types of vulnerabilities.

The results indicated that ChatGPT exhibited a high precision rate of over 80% in detecting true positives, representing actual vulnerabilities that warrant further investigation outside of a controlled environment. However, the AI system encountered challenges in producing false negatives, as evidenced by a low recall rate of only 11% in the Salus team’s experiments. This led to the conclusion that GPT-4’s vulnerability detection capabilities are limited, with its highest accuracy reaching a mere 33%.

Also read: ChatGPT adds access to other GPTs using ‘@’

Human judgement and specialized tools in vulnerability assessments

In light of these findings, the researchers recommended relying on dedicated auditing tools and human expertise for scrutinizing smart contracts until AI systems such as GPT-4 can catch up in terms of efficacy. This perspective underscores the need for a cautious approach to integrating AI into sensitive areas like security auditing, emphasizing the complementary role of human judgement and specialized tools in ensuring thorough assessments of potential vulnerabilities.

The study’s outcomes shed light on the evolving landscape of AI applications within security and underscore the importance of continually refining AI systems to meet the stringent demands of security auditing. As AI technologies continue to advance, it becomes increasingly vital for organizations to strike a balance between leveraging AI’s capabilities and recognizing its current limitations in critical areas such as security assessment.

The insights gleaned from this research serve as a valuable contribution to the ongoing discourse surrounding the intersection of AI, prompting industry stakeholders to consider the nuanced implications of integrating AI into security protocols. Going forward, this study may catalyze further dialogue and innovation aimed at enhancing the effectiveness of AI-driven security auditing tools, ultimately fortifying the resilience of digital ecosystems against emerging threats.