Vibing Drupal: Using AI to hammer at the Webform module's security issues

I came up with the title for this blog post while working on 20+ Webform security issues, because there were moments when I used Codex to hammer out a particularly complex issue. I couldn't help but find it ironic to use something as advanced as AI to hammer at a problem or challenge.

Some security issues were so complex to reproduce that I had to push Codex to replicate the problem, and it occasionally generated sloppy code. Still, even with Codex generating AI-slop, it helped me understand the root causes and solutions for security issues that had lingered for years.

Before I go any further, let's step back and talk about the challenge of maintaining the Webform module and addressing security issues.

The bulk of the Webform module was created a decade ago, when I had more time and motivation to make a sizable contribution to Drupal. Drupal contributors and their contributions come in all shapes and sizes. The current codebase is stable and extendable, with "extendable" as the keyword, because people and AIs can alter and create Webform features and behaviors as needed using contributed modules or custom code. It is ironic that all the example webforms included in the Webform module, intended to help humans, have proven incredibly useful for AIs in understanding and extending webforms.

Though I am willing to say the code is stable, the fact that a webform is generally public and accepts input leaves the Webform module open to security issues. In other words, malicious actors, including AI, will target webforms to exploit XSS vulnerabilities or expose data.

It is worth recognizing and praising Drupal's security team for establishing and continually improving the process for reporting, triaging, and releasing security fixes for Drupal core and contributed modules. I also have to thank the security team for their patience while I worked through the Webform security issues.

The list of people who helped is huge. One of the most irksome things I found with Codex was its difficulty in attributing the right people to a security advisory. Still, Codex was a lifesaver when it came to addressing a Webform security issue that had lingered for years.

When I began working through the security issue backlog, I started with the oldest ticket from 2022, which concerned a SPAM protection bypass. I found it difficult to conceptualize how to reproduce it. When I pointed Codex at the ticket using the in-app browser, which required me to log in, Codex understood the issue and created a regression test that clearly showed a specifically crafted URL bypassing SPAM protection. First, creating a test that replicates the issue allows us to verify the fix. From there, Codex proposed a straightforward solution to prevent it from recurring.

When I started to see the potential of AI to help me work through 20+ security issues, I realized there needed to be a process to organize and work through each issue, and, more specifically, an agent skill.

I feel people are overthinking agent skills. I'm guilty of trying to find or create the perfect skill to solve all my AI challenges. Recently, I've started to view AI as a co-worker that needs some direction and review. My instructions don't need to be perfect. So my workflow is to do some work with an agent and, at the end, have it create or update a skill related to that work. If the skill does a reasonable job, it basically points the AI in the right direction. I will leave it alone. I created the "webform-security" skill to review and track all open Webform security issues.

Our "webform-security" skill created a private local list of markdown files for issue tracking, and we gradually worked through each issue. At some point, I realized so many issues were being resolved that I needed to create a meta ticket to coordinate a massive security release for the Webform module. Codex had a lot of fun creating a table of every ticket it was working on and gradually marking them as done.

Here is where AI becomes indispensable, handling repetitive tasks that require some knowledge and expertise. For the Webform security issues, I used AI to coordinate and test 20+ MRs across two versions of the Webform module. My prompt asked Codex to apply each MR to Webform 6.3.x, then cherry-pick them into 6.2.x. If a cherry-pick fails, create a dedicated MR for 6.2.x.

Once this was done, I had AI create a bash script to merge and stage 20+ issues in a private GitLab repository, where Drupal.org's CI/CD tested all the changes at once. Of course, regressions appeared, and AI helped fix them.

In the end, AI made a task that felt impossible possible. Yet it's worth being open about AI's successes and failures.

I found that AI is really good at organizing tasks and testing code. The "webform-security" skill showed me that I could use AI to iteratively review and fix issues. Everyone can benefit from having AI help organize their tasks.

I simply can't say enough about the value of using test-driven development (TDD) with AI. There were issues I couldn't understand without a test that replicates the problem. For a series of XSS-related issues in JavaScript, there was no initial test coverage, and AI had no problem creating a reasonable baseline test. JavaScript isn't my area of expertise, but the Webform module requires a lot of it. I appreciate that AI can augment my JavaScript skills by reviewing my code and writing tests.

Yep, I definitely ran into some AI-slop and hallucinations while working through these issues. For me, AIs are becoming increasingly human-like because they are trained on our online comments and interactions. In other words, AIs like to bullshit you, and I’ll even say mansplain things.

If an AI doesn't know an answer, it will make one up or come up with a ridiculously complex solution to a simple problem. If you give a slightly off instruction, it will run with it and then blame you. I find it ironic that I could almost accuse the AI of AI-splaining something to me. Yet are we surprised to see a pattern-recognition tool replicate what it sees on the internet?

Right now, the Webform module has no known security vulnerabilities, but with AI, we have to assume malicious actors will use it to find new ones. AI is a double-edged sword: it has helped me resolve issues, but it will also continue to reveal vulnerabilities that malicious actors will seek to exploit. With the help of the security team, the Drupal community, decent test coverage, and a helpful AI assistant, we should be able to resolve reported security issues immediately and no longer let them linger.

For the past few months, I've addressed the Webform security backlog while other maintainers, including Liam Morland, tagged a stable release for D11. Continuing to improve the stability and reliability of the Webform module is the right direction for now. There is a significant backlog of bugs and issues that should be addressed. I'm going to start to use AI to review, diagnose, and resolve some of these issues.

Maybe, just maybe, we can start thinking about what comes next. I don’t yet know what the next version of the Webform module will look like with AI assisting with planning and building.

I already praised the security team in this post, and I want to address my use of AI to resolve Webform security issues, which may have come across as aggressive to those who are cautious about AI. I want to end by recognizing that AI is rapidly upending things and by appreciating the Drupal security team's patience. We are all grappling with how AI affects our careers and code, and I value the Drupal community's core values, which include empathy, understanding, patience, and a willingness to take on and explore new challenges, such as AI.