Node Deployments and ‘node_modules.tar.gz: cannot open: no such file or directory’

This post will cover application failures with Node on App Service Linux - specifically regarding the 'node_modules.tar.gz: cannot open: no such file or directory' message.

Prerequisites

IMPORTANT: Make sure App Service Logs are enabled first

• LogStream
• Retrieving logs directly from the Kudu site, or directly view/download via an FTP client
• Diagnose and Solve Problems -> Application Logs detector, Container Crash detector, or Container Issues detector

If App Service Logs are not enabled, only docker.log files will be generated, which will not show application related stdout / stderr and will make troubleshooting these kinds of issues more complicated.

Overview

NOTE: This applies to Node “Blessed Images” and not custom Docker Images.

This issue will manifest in scenarios where application content was or is being deployed with Oryx as the builder.

When encountering this error, it will show as your typical Application Error : ( message when trying to browse your application. This can be post-deployment, or, if certain files (explained below) are changed or removed.

The Application Error message shows, in this case, due to the fact the container is crashing.

This is due to the main following errors, which can be viewed in your App Service Logs, specifically default_docker.log:

• node_modules.tar.gz: Cannot open: No such file or directory

A secondary error that you’ll see is one of the dependencies in your package.json appearing as missing, such as the below:

Error: Cannot find module 'express'
Require stack:
- /home/site/wwwroot/server.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1026:15)
    at Function.Module._load (node:internal/modules/cjs/loader:871:27)
    at Module.require (node:internal/modules/cjs/loader:1098:19)
    at require (node:internal/modules/cjs/helpers:108:18)
    at Object.<anonymous> (/home/site/wwwroot/server.js:1:17)
    at Module._compile (node:internal/modules/cjs/loader:1196:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10)
    at Module.load (node:internal/modules/cjs/loader:1074:32)
    at Function.Module._load (node:internal/modules/cjs/loader:909:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [ '/home/site/wwwroot/server.js' ]
}

The secondary error, regarding the missing module, can appear in various other issues, but the main error of node_modules.tar.gz: Cannot open: No such file or directory is what should be focused on.

What is the relation to node_modules.tar.gz

This error will manifest when Oryx is used to build the application. This is because of the logic defined here:

As part of the application startup process, we extract the contents of node_modules.tar.gz to /node_modules, which is outside of the volume share. Since the Node.js runtime looks for packages inside directories called node_modules starting at the application directory (/home/site/wwwroot) all the way to /, it is able to find the extracted packages.

This behavior with node_modules.tar.gz is done in the following scenarios:

• Deployments:
  • Local Git
  • Visual Studio Code Deployment with the Azure Extension
  • Zip Deploy with Oryx Builder (Setting SCM_DO_BUILD_DURING_DEPLOYMENT to true)
    • CI/CD with Azure DevOps (built-in deployment tasks) or GitHub Actions both use Zip Deploy as their default deployment mechanism on deployment tasks.
    • You can enable or disable Oryx as a builder with the above App Setting, even with using this CI/CD approach.
    • Using Azure DevOps, Oryx builder may still be used by default unless disabled via SCM_DO_BUILD_DURING_DEPLOYMENT to false

If App Service Logs are enabled, and after a deployment of content with Oryx, viewing your default_docker.log will show the below - which validates what’s described here:

Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
......
....
echo Found tar.gz based node_modules.
extractionCommand="tar -xzf node_modules.tar.gz -C /node_modules"
echo "Removing existing modules directory from root..."
rm -fr /node_modules
mkdir -p /node_modules
echo Extracting modules...
$extractionCommand
export NODE_PATH="/node_modules":$NODE_PATH
export PATH=/node_modules/.bin:$PATH
if [ -d node_modules ]; then
    mv -f node_modules _del_node_modules || true
    fi

if [ -d /node_modules ]; then
    ln -sfn /node_modules ./node_modules 
fi
echo "Done."
npm start
Found tar.gz based node_modules.
Removing existing modules directory from root...
Extracting modules...
Done.

Furthermore, if you SSH into the application container and run cat /opt/startup/startup.sh you’ll see the full logic of the startup script. This script is executed on every start of the container.

If we run ls -lrta under /home/site/wwwroot, we’ll notice the symlink created between the node_modules under /home/site/wwwroot to the node_modules under /. From the logic explained in the earlier link, and shown in the startup logic above, the node_modules used by our application is extracted from the node_modules.tar.gz placed on /home/site/wwwroot. Then the /node_modules under the root path is then symlinked to /home/site/wwwroot.

root@5b3ec5f3aa14:/home/site/wwwroot# ls -lrta
total 17841
-rwxrwxrwx 1 nobody nogroup     3269 May 15 17:52 hostingstart.html
-rwxrwxrwx 1 nobody nogroup     1714 May 15 19:11 .gitignore
-rwxrwxrwx 1 nobody nogroup       44 May 15 19:11 .deployment
-rwxrwxrwx 1 nobody nogroup   166132 May 15 19:11 package-lock.json
-rwxrwxrwx 1 nobody nogroup      281 May 15 19:11 server.js
-rwxrwxrwx 1 nobody nogroup      639 May 15 19:11 package.json
drwxrwxrwx 2 nobody nogroup        0 May 15 19:11 controllers
lrwxrwxrwx 1 nobody nogroup       13 May 15 19:11 _del_node_modules -> /node_modules
-rwxrwxrwx 1 nobody nogroup 18065754 May 15 19:56 node_modules.tar.gz
-rwxrwxrwx 1 nobody nogroup       14 May 15 19:56 .ostype
drwxrwxrwx 2 nobody nogroup     4096 May 15 20:57 ..
-rwxrwxrwx 1 nobody nogroup      303 May 15 21:05 oryx-manifest.toml
lrwxrwxrwx 1 nobody nogroup       13 May 15 21:20 node_modules -> /node_modules
drwxrwxrwx 2 nobody nogroup     4096 May 15 21:20 .

Now knowing this, the secondary error we talked about earlier may make more sense - since if this node_modules.tar.gz file cannot be accessed, then the node_modules directory our Node application relies on will never be created - thus causing the module not found error for our dependencies.

What is oryx-manifest.toml?

Another important file here, which can also cause potential issues like what’s described in this post, is the oryx-manifest.toml file in /home/site/wwwroot.

If you are not using Oryx as a builder, you’ll likely see this:

Cound not find build manifest file at '/home/site/wwwroot/oryx-manifest.toml'

You’ll likely also notice the lack of stdout from the startup script ran, because of not using Oryx as the builder. This is normal and can be ignored.

This file’s presence is relevant in conjunction when using Oryx as the builder.

If we cat oryx-manifest.toml, we’ll see the below contents:

NodeVersion="16.20.0"
nodeBuildCommandsFile="/tmp/zipdeploy/extracted/oryx-build-commands.txt"
Frameworks="Express"
compressedNodeModulesFile="node_modules.tar.gz"
OperationId="2d96022a2e0a10be"
SourceDirectoryInBuildContainer="/tmp/8db557e33dbe5d7"
PlatformName="nodejs"
CompressDestinationDir="false"

This contains important metadata in regards to Oryx, and in this case, the compressedNodeModulesFile property which points to the node_modules tar.

What causes the error?

After covering the logic around how node_modules.tar.gz is generated, and how important this is in regards to creating node_modules for our application when using Oryx as the builder, let’s cover some scenarios where node_modules.tar.gz may be unaccessible.

Deleting the file

The most obvious way to cause this is to delete node_modules.tar.gz. This change on the file system will be picked up after a restart, which is when you’ll see the error occur.

Resolution:

• To resolve this, redeploy the application using Oryx.
• In certain scenarios (if redeplyoment via Oryx does not resolve this), you may need to delete oryx-manifest.toml and regenerate node_modules and placing this under /home/site/wwwroot using another build or deploy method.

Renaming or moving the file

If you rename, or, move node_modules.tar.gz to another location, this will also break the extraction logic by default.

If for some reason it was absolutely needed, you can edit oryx-manifest.toml to point compressedNodeModulesFile to the location of node_modules.tar.gz., such as:

compressedNodeModulesFile="/home/site/node_modules.tar.gz"

You’ll notice in startup stdout that the extractionCommand is now updated:

extractionCommand="tar -xzf /home/site/node_modules.tar.gz -C /node_modules"

This is not recommended and advised to not do this. There is no gaurantee of this consistently working.

Resolution:

• The simplest and most advisable solution and recommendation is to not move or rename both node_modules.tar.gz and oryx-manifest.toml.
• If these files were already changed, redeploy the application - and/or - move said files back to their default location.
• In certain scenarios (if redeplyoment via Oryx does not resolve this), you may need to delete oryx-manifest.toml and regenerate node_modules and placing this under /home/site/wwwroot using another build or deploy method.

Multiple instances

In very rare scenarios, when scaled out to more than one (1) instance, there may be an occurrence of a condition where a particular instance of the application tries to start when the node_modules directory under root is deleted and recreated (see /opt/startup/startup.sh).

Through Diagnose and Solve Problems, this can be narrowed down to a particular instance for easier troubleshooting through either the Application Logs or Container Crash / Container Issues detectors.

Resolution:

• If this is the case, consider using Advanced Application Restart to restart the container on the failing instance.
• Optionally, you can use a build and deploy method that does not use Oryx by design, such as Azure DevOps, GitHub Actions, or Zip Deploy (without Oryx / with Oryx disabled)
  • This also includes using a custom Docker Image, which additionally (by default, unless opt-in) has no reliance on platform mounted /home storage