Earlier this year, deploys of my team’s main application began failing with this error:
This is a Rails app with an AngularJS front-end currently being converted to React. In the months leading up to those failures, deploy times had steadily increased. Before they began failing, our longest deploys took 24+ minutes. 😱 Here’s how we fixed the issue and what I learned about the cause.
We increased Node.js’s memory limit to 2GB by setting
--max_old_space_size=2048 as recommended in several Stack Overflow posts and Github issues. While this worked for many others, it did not solve our problem. Deploys continued to fail.
We next upgraded the app’s Node.js version from 8 to 12 to take advantage of this feature:
Upgrading Node.js unblocked our deploys for several weeks. However, during that time, we continued converting our AngularJS code to React and added new features in React. Deploys took longer and longer, and after a while, they began failing again.
Given the attempted fixes above and with the help of infrastructure monitoring already in place, we were pretty sure that we weren’t running out of memory on our deploy server. As it turns out, the root cause for this issue was in our Webpacker configuration.
webpacker.yml contained this:
default: &default source_path: app-web source_entry_path: react ...
Because of the way our app is structured, this meant we were telling Webpacker to process ALL of our React and Redux-related files, which were increasing in number with every sprint. As I researched the deploy failures, I learned about a helpful of rule of thumb about Webpacker from Ross Kaffenberger’s blog:
If any file in Webpacker’s “packs” directory does not also have a corresponding
Based on this rule, I should’ve seen just one file in our
packs directory. What I saw, though, was essentially a replica of the entire structure of our
/app-web/react directory. We were overpacking.
Ultimately, we moved only the two necessary files into a
startup directory and reconfigured
webpacker.yml to use that as its entry point:
default: &default source_path: app-web source_entry_path: react/startup ...
What I Learned
What is Webpacker, and what does it do?
Okay, cool. But what does that actually mean?
Webpack basically does the work of figuring out what depends on what in your application to generate the minimum “bundles” of assets required to run your app. You include these minimum packs in your application - in Rails, like below - so the app can load with the necessary assets already compiled.
See this article for a much more in-depth intro to what webpack actually does and why module bundlers are needed.
Why was our configuration wrong?
Since webpack builds a dependency graph based on a specified entry point, the greater the number of items in that entry point, the more processing time and resources needed. Because our configuration told Webpacker to process ALL of our React files, this required more time and server resources as we added more files to the React directory.
So, basically, the idea was to not ask Webpacker to process every single file in our React application, but only the entry points to the React app (aka the files that have corresponding
This fix unblocked our deploys and dramatically reduced our deploy times and resource usage on our deploy server.
|Deploy Time||Max Deploy CPU Usage||Max Deploy Memory Usage|
|Before Fix||> 24 min||~90%||~2.2GB|
|After Fix||10 min||~60%||~0.28GB|
So, lesson learned - don’t overpack with Webpacker! 🧳