Of Webpacker Config and Failed Rails App Deploys

Earlier this year, deploys of my team’s main application began failing with a mysterious error.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

This is a Rails app with an AngularJS front-end currently being converted to React. In the months leading up to those failures, deploy times had steadily increased. Before they began failing, our longest deploys took 24+ minutes. 😱 Here’s how we fixed the issue and what I learned about the cause.

Attempted Fixes

The --max_old_space_size Setting

We increased Node.js’s memory limit to 2GB by setting --max_old_space_size=2048 as recommended in several Stack Overflow posts and Github issues. While this worked for many others, it did not solve our problem. Deploys continued to fail.

Node.js Upgrade

We next upgraded the app’s Node.js version from 8 to 12 to take advantage of this feature:

This update will configure the JavaScript heap size based on available memory instead of using defaults that were set by V8 for use with browsers. In previous releases, unless configured, V8 defaulted to limiting the max heap size to 700 MB or 1400MB on 32 and 64-bit platforms respectively. Configuring the heap size based on available memory ensures that Node.js does not try to use more memory than is available and terminating when its memory is exhausted. (source)

Upgrading Node.js unblocked our deploys for several weeks. However, during that time, we continued converting our AngularJS code to React and added new features in React. Deploys took longer and longer, and after a while, they began failing again.

The Fix

Given the attempted fixes above and with the help of infrastructure monitoring already in place, we were pretty sure that we weren’t running out of memory on our deploy server. As it turns out, the root cause for this issue was in our Webpacker configuration.

Our webpacker.yml contained this:

default: &default
  source_path: app-web
  source_entry_path: react
...

Because of the way our app is structured, this meant we were telling Webpacker to process ALL of our React and Redux-related files, which were increasing in number with every sprint. As I researched the deploy failures, I learned about a helpful of rule of thumb about Webpacker from Ross Kaffenberger’s blog:

If any file in Webpacker’s “packs” directory does not also have a corresponding javascript_pack_tag in your application, then you’re overpacking.

Based on this rule, I should’ve seen just one file in our packs directory. What I saw, though, was essentially a replica of the entire structure of our /app-web/react directory. We were overpacking.

Ultimately, we moved only the two necessary files into a startup directory and reconfigured webpacker.yml to use that as its entry point:

default: &default
  source_path: app-web
  source_entry_path: react/startup
...

What I Learned

What is Webpacker, and what does it do?

Webpacker is a gem which allows Rails apps to use webpack to process and bundle assets, particularly JavaScript.

According to its documentation, webpack “is a static module bundler for modern JavaScript applications. When webpack processes your application, it internally builds a dependency graph which maps every module your project needs and generates one or more bundles.”

Okay, cool. But what does that actually mean?

Webpack basically does the work of figuring out what depends on what in your application to generate the minimum “bundles” of assets required to run your app. You include these minimum packs in your application - in Rails, like below - so the app can load with the necessary assets already compiled.

<%= javascript_pack_tag 'application' %>

See this article for a much more in-depth intro to what webpack actually does and why module bundlers are needed.

Why was our configuration wrong?

Since webpack builds a dependency graph based on a specified entry point, the greater the number of items in that entry point, the more processing time and resources needed. Because our configuration told Webpacker to process ALL of our React files, this required more time and server resources as we added more files to the React directory.

So, basically, the idea was to not ask Webpacker to process every single file in our React application, but only the entry points to the React app (aka the files that have corresponding javascript_pack_tags), so that they and their immediate dependencies would be ready on initial application load.

Impact

This fix unblocked our deploys and dramatically reduced our deploy times and resource usage on our deploy server.

So, lesson learned - don’t overpack with Webpacker! 🧳