Nodejs on App Service Linux and why to avoid installing packages in startup scripts

9 minute read | By Anthony Salemo

This post is about why to avoid using a custom startup script on App Service Linux with Node.js to install your npm or yarn packages and dependencies at runtime. This post will be focused on Node Blessed Images.

Why should you avoid it?

Applications with a lot of logic being executed in initialization on restarts, or during “cold start” events, such as if being moved across instances can already introduce long start or restart times.

Adding a startup script into the equation - one that reinstalls packages, rebuilds, compiles, or others - at runtime, which gets executed on each container start - adds even more time. Regardless though, even for small applications, this can cause increased startup times.

If possible, the solution would be to move the logic contained in this, to more appropriate parts of development. Such as into the build itself, or into a post or pre deployment script.

Exceptions

However, there may be times that a specific shared library (.so files) are missing, for example:

error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

In these scenarios, if not wanting to - or not able to use a custom Docker Image, using a custom startup script in these scenarios will be appropriate, since you may not be able to install these Linux based packages during build time - when build against Oryx directly.

However, if installing numerous Linux Based packages for missing .so files, one may still encounter longer start up times for the same reason as explained above.

Issues

Long start times using npm (or yarn) install, build, etc. in startup scripts

This may consume the most time on startup. Take the below example. This is a Nuxt.js application which has the following startup script:

Node Startup Script

npm i && npm run start

2022-10-14T18:36:48.351733102Z PATH="$PATH:/home/site/wwwroot" npm i && npm start
2022-10-14T18:36:48.390043448Z Found tar.gz based node_modules.
2022-10-14T18:36:48.390381662Z Removing existing modules directory from root...
2022-10-14T18:36:48.486176677Z Extracting modules...
2022-10-14T18:37:20.693090060Z Done.
2022-10-14T18:37:26.721965911Z npm info it worked if it ends with ok
2022-10-14T18:37:26.722261524Z npm info using npm@6.14.15
2022-10-14T18:37:26.722274824Z npm info using node@v16.14.2
2022-10-14T18:37:33.238288716Z npm info lifecycle azure-webapps-linux-node-nuxt-basic@1.0.0~preinstall: azure-webapps-linux-node-nuxt-basic@1.0.0
2022-10-14T18:37:55.606857813Z npm timing stage:loadCurrentTree Completed in 22337ms
2022-10-14T18:37:55.789792493Z npm timing stage:loadIdealTree:cloneCurrentTree Completed in 158ms
2022-10-14T18:37:55.891106936Z npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
2022-10-14T18:38:00.581919680Z npm timing stage:loadIdealTree:loadShrinkwrap Completed in 4793ms
2022-10-14T18:38:07.395425690Z npm http fetch GET 200 https://registry.npmjs.org/nan 457ms
2022-10-14T18:38:07.403550534Z npm http fetch GET 200 https://registry.npmjs.org/bindings 469ms
2022-10-14T18:38:07.613075315Z npm http fetch GET 200 https://registry.npmjs.org/nan/-/nan-2.17.0.tgz 196ms
2022-10-14T18:38:07.629404508Z npm http fetch GET 200 https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz 178ms
2022-10-14T18:38:07.805022351Z npm http fetch GET 200 https://registry.npmjs.org/file-uri-to-path 143ms
2022-10-14T18:38:07.906724162Z npm http fetch GET 200 https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz 82ms
.....removed lines for readability
...
..
2022-10-14T18:40:35.836717604Z npm http fetch GET 200 https://registry.npmjs.org/source-map/-/source-map-0.7.4.tgz 7322ms
2022-10-14T18:40:38.636325042Z npm http fetch GET 200 https://registry.npmjs.org/@nodelib/fs.stat/-/fs.stat-2.0.5.tgz 2840ms
.....removed lines for readability
...
..

When we look at Log Stream or Application Logs in Diagnose and Solve, we see that it’s reinstalling all of our packages on startup, before running the site. This doesn’t include typical application builds (ex., generating bundles). We can see in this example this ultimately took almost 4 minutes just to install node_modules.

This is actually completely redundant, because as through essentially any depoyment method (Local Git, ZipDeploy with Oryx as the builder, ZipDeploy without Oryx (we expect everything to be pre-built prior to deploying), DevOps pipelines, GitHub Actions, etc.) we’re either:

  • Building with Oryx against the Kudu site, which will contain all installed packages since it uses this approach for build and run logic
  • Pushing an artifact from a pipeline (DevOps, GitHub Actions with GitHub as the builder) as a zip which is extracted and contains all needed depedencies (since any installs and builds should have been done on the pipeline already)
  • Manually pushing files (which we expected all expected dependencies are pushed as well)

So when we introduce a startup command like this - using the above deployment methods. We’re essentially unnecessarily duplicating our deployment efforts.

Solution

Ensure that any typical Node.js package install or build is done during build time and not at runtime.

Below are some examples which include Local Git, Github Actions and DevOps pipelines

For certain situations when using Oryx and the expected build files in the project root are in a different folder, you can use KuduScript to generate a quickstart custom deployment script to customize the behavior of the install or build.

NOTE: This will much likely require customizing what is generated by default and is up to the developer to do this to fit their needs

For other situations where the entrypoint .js file may be in a non-typical location expected by Oryx, a custom startup command can still be used as this ideally is just to run the application.

Permissions issues, OOM kills (JavaScript Heap OOM), Maximum call stack size exceeded and others

It is not recommended to install modules using npm install inside /home through WebSSH or using a custom startup script.

The file system of your application is a mounted network share (Windows fileshare) and certain npm packages look for chown permissions which will not apply for the remote home storage. The permissions on this are 777. You cannot change these permissions, even if you attempt to do so from an initialization script or from SSH. This will also apply if you have Web App for Containers using an App Setting WEBSITES_APP_SERVICE_ENABLE_STORAGE=true .

You can see errors like this:

EPERM

npm ERR! code EPERM
npm ERR! syscall lchown
npm ERR! path /home/node_modules/.bin/is-docker
npm ERR! errno -1
npm ERR! Error: EPERM: operation not permitted, lchown '/home/node_modules/.bin/is-docker'
npm ERR!  [OperationalError: EPERM: operation not permitted, lchown '/home/node_modules/.bin/is-docker'] {
npm ERR!   cause: [Error: EPERM: operation not permitted, lchown '/home/node_modules/.bin/is-docker'] {
npm ERR!     errno: -1,
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'lchown',
npm ERR!     path: '/home/node_modules/.bin/is-docker'
npm ERR!   },
npm ERR!   errno: -1,
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'lchown',
npm ERR!   path: '/home/node_modules/.bin/is-docker'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.
npm timing npm Completed in 74867ms

ERR! Maximum call stack size exceeded

root@251a3b9444f8:/home/site/wwwroot# npm i webpack
npm info it worked if it ends with ok
npm info using npm@6.14.15
npm info using node@v14.21.2
npm http fetch GET 200 https://registry.npmjs.org/webpack 187ms
npm http fetch GET 200 https://registry.npmjs.org/webpack/-/webpack-5.76.3.tgz 148ms
npm ERR! Maximum call stack size exceeded
npm timing npm Completed in 1187ms

Or JavaScript out of memory (OOM):

  • Allocation failed - JavaScript heap out of memory

Make sure that App Service Logs are enabled to view these failures.

Solution

The solution for any of these scenarios, will be to leverage App Service Build (Oryx) to build your code and avoid rebuilding at runtime/startup. Any custom startup script that is doing npm install can experience the same issue.

Missing shared libraries: xxx.so.1: and other Linux specific dependencies

Some Node packages may require C library based Linux packages being available on the filesystem. These libraries are normally .so files, shared libaries, that can be installed via Linux distribution based package managers (eg., apt-get, apk).

There are times these may be missing, and may present itself in various forms at runtime like the below:

node: /lib/x86_64-linux-gnu/xxx.so.6: version `xxx′ not found

Solution

The solution for these specific missing packages can go two general ways:

  • Add a custom startup script and install the needed packages via package manager (apt, apt-get, or apk, depending on the distribution). Note that if needing to install numerous Linux packages or the packages are large, you’ll end up running into the same issue described in this blog post.
  • Use a Custom Docker Image. Sometimes, it makes more sense to use a custom Image that you can build yourself to what your project needs. If the Linux package sizes that you’re installing at runtime is adding a large amount of container startup time - then it may make more sense to include these in the custom Docker Image itself.

You may also enjoy

Tomcat HTTP 404’s on App Service Linux

12 minute read

This post will go over general troubleshooting for HTTP 404’s when using Java on App Service Linux with .war-based applications, which uses Tomcat.