Return to site

Webform module now supports importing submissions

March 18, 2019
Problem

How do you migrate webforms and submissions from previous versions of Drupal and other form builders?

The answer is Drupal's Migrate API, which is incredibly powerful but can feel overwhelming. When I migrated MSKCC.org from Drupal 6 to Drupal 8, the Migrate API was just being introduced into Drupal 8 core, and I felt more comfortable writing a custom migration script instead of using code that was still under development. Migrate API is now stable and if you are an experienced Drupal developer, you should use it.

Certain aspects of Drupal require some expertise.

The level of expertise required to build and maintain a Drupal 8 website has changed from Drupal 7, mainly because we are creating more ambitious digital experiences. The Drupal community struggles to simplify our flexible and sometimes complex product. My approach is to make the Webform module as flexible and robust as possible, while not forgetting that people need a simple way to start building a form. This is exactly why I include an introduction video on the Webform module's main page. Besides making the Webform module an awesome tool for experienced Drupal site builders, the Webform module needs to be welcoming to new users and make it easy for them to move their existing forms to Drupal.

Either an organization is starting from scratch and building a new Drupal site, or more commonly an organization has decided they need to provide a more ambitious digital experience and they have chosen to switch to Drupal. In both situations, we need to make it easy for someone to switch from other form builders to Webform.

The problem that needs to be addressed is…

How can we make it easy for an organization to migrate existing forms with submissions to the Webform module?

Solution

The simplest way to migrate to the Webform module is to rebuild an external form and then import the existing data. Building a webform is fun and easy, forms are a critical aspect to most websites; it is worth taking the time needed to rebuild an existing form, and take full advantage of all the cool elements and features available in the Webform module. The only missing feature from the Webform module is an easy and fast way to import existing submission data.

The Webform module already makes it easy to export submissions. There is a 'Download' tab under the Results section that generates a CSV (comma separated values) which can be opened in Excel or Google Sheets.

The solution is to make it extremely easy for anyone to import submissions using a CSV.

Importing a CSV containing submission data should be a simple process where a site builder uploads a CSV, which is prepared, reviewed, validated, and submitted to a webform.

Below are the steps which one might take to import CSV data:

  • Build a webform
  • Populate webform with some test submissions
  • Download the test submissions using the CSV export
  • Examine the CSV export
  • Populate the CSV export with external data
  • Upload and import the CSV export
  • Review imported data

Being able to export and import submission data as needed has the added bonus that site builders can download all the existing submissions, bulk edit them using Excel or Google Sheets, and then upload them back into the database. Submissions can now be migrated from one server to another. There is only one caveat, and it is not required, but it helps to make sure all exported and imported CSV records have a UUID (universally unique identifier).

Notes

The below notes are pulled directly for the Webform module's CSV upload form:

  • All submission properties and data is optional.
  • If UUIDs are included, existing submissions will always be updated.
  • If UUIDs are not included, already imported and unchanged records will not create duplication submissions.
  • File uploads must use publicly access URLs which begin with http:// or https://.
  • Entity references can use UUIDs or entity IDs.
  • Composite (single) values are annotated using double underscores. (e.g. ELEMENT_KEY__SUB_ELEMENT_KEY)
  • Multiple values are comma delimited with any nested commas URI escaped (%2E).
  • Multiple composite values are formatted using inline YAML.
  • Import maximum execution time limit is 6 hours.

For those hardcore developers out there, the answer is…

Yes!!! There is even a drush:webform:import command.

Demo

To make sure no one feels overwhelmed, I created the below video to help people feel more comfortable with this new functionality.

Supporting organization

This awesome new feature was sponsored by Kennesaw State University. Kaleem Clarkson (kclarkson), Operations Manager @ Center for Excellence in Teaching & Learning at Kennesaw State University, and I scoped out this ticket on Drupal.org via Issue #2902977: Provide a straight forward tool for importing submissions from a CSV document.

I previously talked about Coming to an agreement within the Drupal community and sponsoring a Webform feature, and it is challenging working out the process for a paid feature. I have discovered that scoping out all paid Webform feature requests on the Drupal.org has worked out to be the most transparent process. Even when a potentially sponsored feature agreement falls through, the scoped out work remains in the Webform issues queue. I feel better knowing the research and work I did for the feature request is not a complete loss. I have even seen a feature request sit in the queue for a few months and then someone like Michael Feranda (mferanda) appears out of nowhere and helps complete a feature like Issue #2888862: Provide a mechanism to lock a webform submission.

Getting Involved

Kaleem Clarkson (kclarkson) and Michael Feranda (mferanda) contributions show there are multiple ways to get involved in helping to support the Webform module and Drupal. Michael found a task in the Webform issue queue and started contributing code back to the Drupal community. Kaleem is not a coder, he is more of a 'master' site builder and community leader, so he found a requirement from his project, convinced Kennesaw State University about the value of contributing back to the Drupal, and they sponsored a pretty awesome improvement, which helps everyone who is adopting the Webform module and Drupal.

Sponsor a Feature

Sponsored a feature is a challenging but needed enhancement to the process of building and maintaining the Webform module. I have had some success implementing a few paid features.

Open Collective provides a transparent platform that might make 'Sponsor a feature' more natural and accessible for the Drupal community. Paying for a feature using Open Collective's payment system which provides standardized invoices could make it easier and justifiable for large organizations to financially contribute to Webform, Drupal, and Open Source.

At the same time, I still feel it is also okay to not pay Open Collective's 13-14% overhead and execute a paid feature request via a direct transaction. Either way, I am absolutely committed to scoping out all feature requests transparently on Drupal.org.

Backing the Webform module

Open Collective is providing us, Drupal and Open Source, with a platform to experiment and improve Open Source sustainability. If you appreciate and value what you are getting from the Webform module, please consider becoming a backer of the Webform module's Open Collective.