Trends

Why should we balance AI capabilities with human expertise?

AuthorBTW Editorial

Reading Time3 min

PublishedFebruary 21, 2024

Primary DomainN/A

Content TypeN/A

TopicN/A

EntityN/A

RegionN/A

Time HorizonN/A

ImpactN/A

ConfidenceN/A

Headline

Context

Although AI, such as ChatGPT, shows impressive capabilities in code generation and analysis, there are doubts about its effectiveness as a reliable security auditor.Highlighting the importance of utilizing both specialized tools and human judgement to conduct comprehensive evaluations of potential vulnerabilities. As the capabilities of artificial intelligence (AI) continue to expand, researchers from Salus Security recently put GPT-4 and other AI systems to the test in detecting common security vulnerabilities. The study, conducted by a pair of experts from Salus Security, a prominent blockchain security company with operations in North America, Europe, and Asia, aimed to assess GPT-4’s proficiency in parsing and auditing smart contracts.

Evidence

Pending intelligence enrichment.

Analysis

Also read: A look at Alphabet’s Gemini, the AI model aiming to challenge ChatGPT-4 The research revealed that while AI, including ChatGPT, demonstrates a commendable aptitude for generating and analyzing code, its reliability as a security auditor remains questionable. Leveraging a dataset of 35 smart contracts known as the SolidiFI-benchmark vulnerability library, which encompassed a total of 732 vulnerabilities, the researchers evaluated the AI’s ability to identify potential security weaknesses across seven prevalent types of vulnerabilities. The results indicated that ChatGPT exhibited a high precision rate of over 80% in detecting true positives, representing actual vulnerabilities that warrant further investigation outside of a controlled environment. However, the AI system encountered challenges in producing false negatives, as evidenced by a low recall rate of only 11% in the Salus team’s experiments. This led to the conclusion that GPT-4’s vulnerability detection capabilities are limited, with its highest accuracy reaching a mere 33%. Also read: ChatGPT adds access to other GPTs using ‘@’

Key Points

The researchers assessed the AI’s capacity to detect potential security vulnerabilities across seven commonly observed types.
While economic impacts are significant, advocates prioritize environmental conservation and Indigenous rights, setting the stage for potential conflict.