Overview

On Azure App Service, the underlying infrastructure fleet includes workers with different CPU vendors - specifically Intel and AMD processors. App Service does not guarantee a specific CPU vendor or instruction set for any given worker. Over time, stamps may transition between hardware generations, and instance movements (due to scaling, platform maintenance, or rebalancing) can place your application on a worker with a different CPU architecture than the one it was previously running on.

If your container image was compiled with CPU-specific instructions (for example, using -march=native on an Intel build machine, or linking against libraries that use Intel-specific instruction sets like AVX-512), the container may crash immediately when placed on an AMD worker - or vice versa.

This crash presents itself as exit code 132, which corresponds to signal 4 (SIGILL - Illegal Instruction). The container typically exits so quickly that no application logs (stdout/stderr) are produced.

What does exit code 132 mean?

Exit code 132 is the result of the Linux kernel sending SIGILL (signal 4) to a process. This signal is raised when the CPU encounters an instruction it does not recognize or support.

The formula is: 128 + signal number = exit code. So 128 + 4 = 132.

Common reasons for SIGILL:

• The binary was compiled with -march=native on an Intel machine, which may enable AVX-512, SSE4.2, or other Intel-specific instructions that AMD processors do not support (or vice versa)
• Native extensions or shared libraries (.so files) were built targeting a specific CPU microarchitecture
• Compiled languages such as C, C++, Rust, or Go (with assembly) may embed architecture-specific instructions at build time
• Python packages with native C extensions (like numpy, scipy, cryptography, etc.) may have been compiled from source on a specific CPU architecture

How this manifests on App Service

A typical scenario looks like this:

1. Your application runs without any issues on a worker with an Intel CPU
2. An instance movement occurs - this could be due to platform maintenance, scaling events, or worker rebalancing
3. Your container is placed on a worker with an AMD CPU
4. The container starts, but crashes immediately with exit code 132
5. The container enters a crash loop - every restart attempt results in the same exit code 132
6. Since the crash happens so fast, no application logs are produced - you may see messages like Failed to get container logs in diagnostic logging
7. HTTP traffic returns 503 errors since the container never becomes healthy

If you look at Diagnose and Solve Problems or App Service Logs, you’ll see entries similar to:

Container [containerName] for site [siteName] has exited, exit code: 132

This will repeat for every restart attempt. The container never successfully starts.

NOTE: This issue is classified as an application-level issue, even though it is triggered by an infrastructure change (instance movement). App Service does not guarantee a specific CPU vendor, and applications should be built to run on any supported x86-64 processor.

Identifying the issue

To confirm this is a CPU architecture mismatch:

1. Check the exit code - Exit code 132 specifically indicates SIGILL. This is different from other common exit codes like 137 (OOM kill) or 139 (segfault)
2. Check if an instance movement occurred - Look at App Service diagnostic logs or Diagnose and Solve Problems to see if the worker changed around the time the crashes started
3. Check if the application was previously healthy - If the same image was running without issues and then suddenly started crashing with exit code 132 after a worker change, this strongly suggests a CPU architecture mismatch
4. No application logs - The process is killed by the kernel before it can write any output. If you see empty logs or “Failed to get container logs”, combined with exit code 132, this is characteristic of SIGILL

Resolution

The fix is to rebuild your container image so that it does not rely on CPU-specific instructions. The following steps should be taken:

1. Use architecture-agnostic compiler flags

If you’re compiling code (C, C++, Rust, Go with assembly, etc.), use generic x86-64 baseline target flags instead of architecture-specific ones:

# Instead of this (targets the build machine's exact CPU):
RUN gcc -march=native -O2 -o myapp myapp.c

# Use this (targets the generic x86-64 baseline):
RUN gcc -march=x86-64 -O2 -o myapp myapp.c

# Or use x86-64-v2 for a slightly newer baseline (SSE4.2, SSSE3, POPCNT):
RUN gcc -march=x86-64-v2 -O2 -o myapp myapp.c

2. Check for SIMD intrinsics

If your application or its dependencies use SIMD (Single Instruction, Multiple Data) intrinsics, ensure runtime CPU feature detection is in place rather than compile-time assumptions. Many modern libraries support this - for example, checking for AVX support at runtime before using AVX instructions.

3. Python with native extensions

If you’re using Python with packages that have native C extensions (such as numpy, scipy, cryptography, pillow, etc.):

• Use pre-built wheels from PyPI instead of building from source. Pre-built wheels target the generic x86-64 baseline
• If you must build from source, ensure the build does not use -march=native
• Consider using packages from conda-forge, which are also built for generic x86-64

4. Rust applications

For Rust, ensure your .cargo/config.toml or build command does not specify a CPU-specific target:

# Avoid this:
[build]
rustflags = ["-C", "target-cpu=native"]

# Use this instead (or simply omit the target-cpu flag):
[build]
rustflags = ["-C", "target-cpu=x86-64"]

5. Go applications

Go applications are generally safe, as Go compiles to a generic x86-64 target by default. However, if you’re using assembly files (.s files) or cgo with C code that targets a specific CPU, verify those components are architecture-agnostic.

6. Push a new image and restart

After rebuilding your image:

1. Push the new image to your container registry with a new tag
2. Update the Web App for Containers configuration to use the new image tag
3. The new image will be pulled

Additional considerations

• Multi-stage builds: If you use multi-stage Docker builds, ensure the build stage uses generic compiler flags, not just the final stage
• Base images: Some base images may include pre-compiled binaries that target specific architectures. If you’re using a niche or heavily optimized base image, verify it’s compatible with both Intel and AMD x86-64 processors
• Third-party binaries: If your Dockerfile downloads pre-compiled binaries (rather than building from source), ensure those binaries target the generic x86-64 baseline
• Testing: To test locally, you can use objdump -d <binary> | grep -i avx512 (or other instruction sets) to check if your binary contains architecture-specific instructions

This article provides the steps to reset a WordPress admin password for a site hosted on Azure App Service using SSH and WP-CLI.

This method is particularly useful when access to the WordPress admin portal (/wp-admin) is not available, or you do not have access to the MySQL database.

Scope

• Azure App Service (Linux) hosting WordPress
• WordPress deployments with WP-CLI available
• Scenarios where password reset via email or UI is not possible

Prerequisites

• Access to the Azure Portal
• Sufficient permissions to the target Web App
• SSH access enabled on the App Service
• Basic familiarity with command-line usage

Procedure

1. Access Azure Portal

• Sign in to the Azure Portal: https://portal.azure.com
• Navigate to your WordPress Web App

2. Open SSH Console

• In the left-hand menu, select:
  Development Tools → SSH
• Click Go to open the SSH session

3. Navigate to WordPress Root Directory

Run the following command:

cd /home/site/wwwroot

4. List WordPress Users

To identify the user account:

wp user list --allow-root

This will display:

• User ID
• Username(user_login)
• Display name
• Email(user_email)
• date of registration(user_registered)
• Roles

5. Reset the Password

wp user update <USER_ID> --user_pass='<NEW_PASSWORD>' --allow-root

Example:

wp user update 1 --user_pass='P@ssw0rd123!' --allow-root

Expected Result

• The password is updated immediately
• No restart of the App Service is required

Verification Steps

1. Navigate to:

   https://<your-site>/wp-admin
   

2. Log in using:
   • Username
   • New password

Important Notes

• The –allow-root flag is required in Azure App Service environments
• Always use a strong and secure password:
  • At least 12 characters recommended
  • Combination of uppercase, lowercase, numbers, and symbols
• Avoid sharing credentials in plain text

Troubleshooting

Issue: wp: command not found

• WP-CLI may not be installed or available in the environment
• Verify your WordPress image or installation

  Issue: Permission errors

• Ensure you are in the correct directory:

  cd /home/site/wwwroot
  

  Issue: Unable to identify correct user

• Re-run:

  wp user list --allow-root
  

• Verify the correct Id before updating

Overview

This post provides a practical overview of load testing applications deployed on Azure PaaS services such as Azure App Service and Azure Container Apps.

Azure Load Testing is a fully managed service for testing and evaluating application performance. It is especially valuable before production deployments because it helps predict application behavior at scale and under varying traffic patterns.

The following are the core components of the service:

1. Test - The overall configuration of the test including endpoints, rules, and metrics.
2. Test run - One execution of a test.
3. Test engine - Managed compute that generates traffic and executes test runs. You can scale tests by increasing engine instances and users.
4. App components - The resources to monitor during load (for example CPU, memory, latency, and HTTP failures).
5. Engine instances - Up to a maximum of 400 per test, with a maximum of 1000 concurrent instances across tests (subject to change).
6. Users - On the Azure portal, users per engine instance are limited (up to 250, subject to change). This can be customized further in JMX based tests.

JMeter

Azure load tests are executed by Apache JMeter under the hood. More information on JMeter is available here. You can upload an existing JMeter script (.jmx) to the test engine. While Azure Load Testing users are analogous to JMeter threads and Azure engine instances are similar to JMeter nodes, Azure Load Testing includes built in Azure Monitor integration. This makes it easier to benchmark Azure resource metrics without additional plugins, integrations, or credential setup. JMeter still offers advanced customizations that are not exposed directly through the Azure Load Testing UI or API.

Test Types: URL vs JMX

Before proceeding, it helps to understand the two common test types.

URL tests are lightweight and can be configured directly in the portal or by using a JSON request file (for example requests.json) to define one or more HTTP requests. They are useful for quick API checks.

JMX tests use a full Apache JMeter test plan (.jmx). Choose this when you need advanced behavior such as authentication flows, parameterization, and CSV data driven testing.

In practice, start with URL for fast validation and move to JMX as your scenario complexity grows.

Creating Tests in Portal vs Azure CLI

You can create and run tests from either the Azure portal or Azure CLI. The CLI option is better for repeatable workflows, source control, and CI/CD automation.

Below are corresponding Test Definition samples.

Sample YAML for JMX deployment

version: v0.1
testId: YourUniqueTestID
displayName: Your Readable Test Name
description: Load test website home page
testPlan: YourTestPlan.jmx
testType: JMX #URL is the other type
engineInstances: 1
subnetId: /subscriptions/<subid>/resourceGroups/rgname/providers/Microsoft.Network/virtualNetworks/vnetname/subnets/subnetid #optional
configurationFiles:
  - 'sampledata.csv'
zipArtifacts:
  - largedata.zip 
#Large configuration/data files under 50MB, up to 5 zip files - only for JMX
splitAllCSVs: true 
#Splits CSV files per engine. Useful to split test data across engines or to avoid collisions such as with logins
failureCriteria: 
#examples
  - avg(response_time_ms) > 300
  - percentage(error) > 50
  - YourJMeterSampler: avg(latency) > 200
autoStop:
  errorPercentage: 80
  timeWindow: 60 #seconds
env: 
#Env variable referenced by the script
    BASE_URL: https://app.yourdomain.com  
    ENVIRONMENT: prod

Sample YAML for URL type deployment

version: v0.1
testId: YourUniqueTestID
displayName: Your Readable Test Name
description: Simple URL load tests
testType: URL
testPlan: requests.json
engineInstances: 2
failureCriteria:
  - avg(response_time_ms) > 500
  - percentage(error) > 5
autoStop:
  errorPercentage: 20
  timeWindow: 60 #seconds

Sample JSON (required) configuration for URL type test

{
  "requests": [
    {
      "requestName": "HomePage",
      "method": "GET",
      "url": "https://yourdomain.com/"
    },
    {
      "requestName": "HealthCheck",
      "method": "GET",
      "url": "https://yourdomain.com/health"
    }
  ]
}

Configuring and executing tests from the Azure Portal

The first step is to create the resources, which can be done quickly from the portal. Create an Azure Load Testing resource, select Tests, then choose Create and follow the guided options. The following areas are the most important to configure.

1. Load Select the number of engine instances and the type of load pattern. It is recommended to create separate tests to test all the load patterns (Linear, Spike and Step) to replicate production scenarios and your use cases.
   • Linear: traffic increases over time. Allow an initial ramp up time.
   • Spike: traffic rises rapidly to simulate seasonal or occasional spikes.
   • Step: traffic increases in defined plateaus. Select the number of concurrent users per engine. On the portal, this is commonly limited to 250 users per engine instance. Select the number of engines, test duration, and ramp-up time. More information on limits is available here.

2. Monitoring Attach key Azure dependencies being tested, such as App Service and Azure databases. Otherwise, testing is limited to client side metrics. It is recommended to add relevant components such as App Service, App Service Plan, upstream dependencies, and Application Insights.

3. Test criteria Specify test criteria for client side metrics as well as Azure resource metrics. For client side analysis, useful metrics include response times, latency, and errors. For server side analysis, useful metrics include CPU, memory, scaling events, requests, and responses.

Additionally, if your workload is private, configure subnet/VNet integration and validate DNS/routing before runs.

As an example, to simulate 5,000 users, you could configure 250 virtual users with 20 engine instances, set ramp up time to one minute, and run separate tests for linear, spike, and step scenarios.

After configuring the test, review the resources and settings, start the run, monitor live metrics, and then save and compare outcomes across runs.

Configuring and executing tests with Azure CLI and GitHub Actions

Azure CLI is the recommended path for repeatability, source control, and CI/CD integration. In a typical operating model, you keep YAML/JMX/JSON/CSV files in the repository, create or update tests from YAML, and trigger runs in release stages. This provides traceable configuration history and consistent execution across environments.

Example invocation with Azure CLI

# Install the Azure Load Testing CLI extension if needed
az extension add --name load --upgrade
# Create or update a test from YAML
az load test create \
  --load-test-resource <your-load-test-resource-name> \
  --resource-group <your-load-test-rg> \
  --test-id <test-id> \
  --load-test-config-file loadtest-config.yaml
# Start a test run
az load test-run create \
  --load-test-resource <your-load-test-resource-name> \
  --resource-group <your-load-test-rg> \
  --test-id <test-id> \
  --test-run-id <test-run-id>

This can be automated with Github Actions as part of pull request or release workflows. The pipeline authenticates to Azure, runs the test with the repository configuration, and publishes result files as build artifacts for auditing and comparison.

Example workflow with GitHub Actions

name: your-load-test-name
on:
  workflow_dispatch:
  push:
    branches: [ "main" ]
jobs:
  run-load-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Azure Login
        uses: azure/login@v2
        with:
          creds: $
      - name: Run Azure Load Testing
        uses: azure/load-testing@v1
        with:
          loadTestConfigFile: loadtest-config.yaml
          loadTestResource: <your-load-test-resource-name>
          resourceGroup: <your-load-test-rg>
      - name: Publish results artifact
        uses: actions/upload-artifact@v4
        with:
          name: load-test-results
          path: |
            loadTest/*.json
            loadTest/*.csv

An example appraoch would be to run short tests with every pull requests and more comprehensive tests before releases. Save the test results to compare and review trends over time corresponding to application releases.

Readme: https://github.com/azureossd/networking-troubleshooting-utility/blob/main/README.md

Overview

Outbound connectivity issues in cloud environments can stem from many causes and manifest in different ways. Applications may experience intermittent or persistent failures when reaching a specific host or API. Sometimes only one external endpoint is affected while others remain accessible, and in other cases all outbound traffic fails.

Common causes include DNS resolution errors, routing or firewall misconfigurations, upstream service failures, and platform constraints such as SNAT port limits in Azure App Services. These problems typically appear as latency, timeouts, or refused connections, often surfacing as HTTP 5xx errors downstream or at the client.

Effective troubleshooting requires isolating the failure domain, whether DNS, network path, platform limits, or application behavior. The following tools and approaches can help systematically diagnose and resolve these issues.

Environment Setup and Prerequisites

Before troubleshooting, confirm which tools are available. Managed runtimes (Azure App Services, Conatainer Apps etc) or IaaS services (ex:Azure VMs) may not include all utilities by default.

In Azure App Service (Kudu SSH console): curl, dig, nc, and tcpdump are pre-installed in the sandbox. nmap, tshark, and zeek are absent and must be side-loaded as static binaries into /home (persistent storage) if needed. Note that as a non-root user, you cannot install these tools in the kudu container. Additionally, to be able to install these tools in the runtime container, it needs to be up and running. And with custom docker containers on App Services, ssh would need to be enabled. Refer here for steps to enable SSH.

Below can be run as a quick test to see which (common) tools are already installed.

# Inventory what is available
for tool in curl wget dig nslookup nc nmap tcpdump tshark zeek ss netstat ip traceroute mtr iftop tcpping; do
    command -v "$tool" &>/dev/null && echo "✓  $tool" || echo "✗  $tool (not found)"
done

DNS

One of the first steps in troubleshooting upstream connectivity failures is validating name resolution. In Linux based environments, it is important to review /etc/hosts and /etc/resolv.conf, as these files may contain custom entries. Always validate DNS before moving to reachability tests.

When an application, command line tool such as curl, browser, or runtime attempts to resolve a hostname, it invokes the operating system resolver through a system call. Typically (as determined by the resolutin order), /etc/hosts is checked first followed by DNS servers defined in /etc/resolv.conf. When any process resolves a hostname the OS resolver follows an order controlled by /etc/nsswitch.conf — typically files first, then dns:

A couple of tools(nslookup and dig) and their example usage is below.

dig

Basic resolution:

dig microsoft.com

Query a specific DNS server (bypass the system resolver):

dig @8.8.8.8        microsoft.com   # Google public DNS
dig @168.63.129.16  microsoft.com   # Azure platform resolver
dig @10.1.0.4       microsoft.com   # An example custom DNS server in your VNet

Follow the full delegation chain:

dig +trace microsoft.com

Reverse DNS lookup:

dig -x 20.112.52.29

Query specific record types:

dig microsoft.com A       # IPv4
dig microsoft.com CNAME   # Canonical name alias
dig microsoft.com TXT     # SPF, DMARC, ACME challenges
dig microsoft.com NS      # Authoritative nameservers
dig microsoft.com SOA     # Start of Authority

Measure DNS query time and check TTL:

dig microsoft.com | grep -E "Query time|ANSWER SECTION|IN\s+A"
# ;; Query time: 4 msec
# microsoft.com.  30  IN  A  20.112.52.29

A low TTL (30 seconds in this case) means results expire quickly. A cached NXDOMAIN with a TTL of 300 will persist for 5 minutes even after a DNS fix is deployed. Account for this when validating a fix.

nslookup

nslookup microsoft.com
nslookup microsoft.com 168.63.129.16   # query a specific server

TCP Reachability and Connectivity Testing

If DNS resolution fails, review /etc/resolv.conf, test with an alternate DNS server, and confirm there are no local blockers such as an incorrect hosts file entry or a corporate proxy interfering with name resolution. Check custom DNS servers, forwarding, and Firewall rules.

Once name resolution succeeds, the next step is to validate reachability. These tests determine whether traffic can physically traverse the network path to the destination over TCP.

While several tools are available for this purpose, utilities such as nmap and nc are commonly present in Linux environments and offer many useful capabilities including port scanning, custom packet testing, and basic data transfer.

Basic TCP connectivity test:

nc

nc -zv -w 5 microsoft.com 443

Probe multiple ports in a single command:

nc -zv microsoft.com 80 443 8080

Send a raw HTTP request and inspect the response

printf "GET / HTTP/1.0\r\nHost: microsoft.com\r\n\r\n" | nc microsoft.com 80

nmap Port and Service Discovery

nmap provides richer results than nc, including service identification and the distinction between a filtered (firewall dropped) port and a closed (RST received or unavailable) port. Single port test:

nmap -p 443 80 microsoft.com

TCP SYN scan:

nmap -sS -p 80,443 microsoft.com

Service and version detection: Very handy

nmap -sV -p 443 microsoft.com
# 443/tcp open  ssl/http Microsoft IIS httpd

Show why each port has a given state (filtered vs. closed): Equally, very useful

nmap -p 443 --reason microsoft.com
# 443/tcp open      syn-ack    — port open, accepting connections
# 443/tcp filtered  no-response — firewall/NSG silently dropping packets
# 443/tcp closed    reset       — host reachable, nothing listening on that port

curl

curl is a useful tool for end-to-end n/w validation as well as for downloading packages or artifacts. Simple request with granular timing breakdown:

curl -o /dev/null -s -w \
"DNS lookup:    %{time_namelookup}s\n\
TCP connect:   %{time_connect}s\n\
TLS handshake: %{time_appconnect}s\n\
TTFB:          %{time_starttransfer}s\n\
Total:         %{time_total}s\n\
HTTP status:   %{http_code}\n" \
https://microsoft.com

Very verbose — includes the full TLS certificate chain:

curl -vvv https://microsoft.com 2>&1 | head -100

Test with internal or custom CA certificate

curl --cacert /etc/ssl/certs/custom-ca.crt \
  https://custom-api.mycompany.com

ss or netstat. Note that ss is a newer replacement.

# Show aoo connections
netstat -tunp
# Connections to a specific remote host
ss -tnp dst microsoft.com
# Show details including MTU and MSS for active connections
ss -tin

Check Interface MTU

# Show all interfaces and their MTU
ip link show
# Single interface
ip link show eth0 | grep mtu

Other utilities

Additional tools are available to quickly troubleshoot connectivity, review bandwidth usage, latency, and overall network performance. Common examples include iftop, iptraf-ng, and nethogs, all of which provide real time visibility into network activity on Linux systems.

After installation, these utilities can be launched directly from the console to display active connections in real time.

iftop

Iftop also launches a UI which shows live traffic and can be useful to view current and active outbound connections and bandwidth usage.

iftop -i eth0             # live bandwidth by connection pair
iftop -i eth0 -f "port 443"   # filter to HTTPS only

nethogs

nethogs eth0              # bandwidth by PID
nethogs -d 2 eth0         # refresh every 2 seconds

iptraf-ng UI based tool similar to iftop nad nethogs with additional utility

Example view below showing outbound connections

Capturing Network traces

If reachability or connectivity tests fail, or if the issues are intermittent, capturing and analyzing a network trace is the next step. tcpdump is a powerful tool for this purpose, allowing you to record traffic that can be analyzed directly in the CLI using tcpdump or tshark, or externally with tools like Wireshark.

Full Capture with tcdump

tcpdump -i any -s 0 -tttt -U -nn -w “trace.pcap” 

This command captures full TCP packets (-s 0) on all interfaces (-i any), includes timestamps (-tttt), avoids resolving hostnames and ports (-nn), outputs in verbose mode (-vv) (not needed), and writes packets immediately to disk (-U) to the specified file (-w).

Filters can be applied during capture, often to reduce noise. However, it may be beneficial to capture all traffic on primary (or all) interfaces and apply filters later during analysis.

# Rolling capture with 100 MB file rotation, keeping 5 files
tcpdump -i any -s 0 -nn -w trace_%Y%m%d_%H%M%S.pcap -C 100 -W 5

# Traffic to or from a specific host
tcpdump -i any -s 0 -nn -w trace.pcap host microsoft.com

# Traffic on a specific port
tcpdump -i any -s 0 -nn -w trace.pcap port 443

# Traffic between two specific IP addresses
tcpdump -i any -s 0 -nn -w trace.pcap \
  'src host 10.1.0.10 and dst host 20.112.52.29'

# TCP RST and FIN packets only (connection termination events)
tcpdump -i any -nn -s 0 -w resets.pcap \
  'tcp[tcpflags] & (tcp-rst|tcp-fin) != 0'

# DNS traffic only
tcpdump -i any -s 0 -nn -w dns.pcap port 53

Analyzing network trace

The packet capture file can be analyzed in several ways right away with tcpdump or tshark.

With tcpdump

# Verbose output
tcpdump -v -r trace.pcap

# Show ASCII payload (useful for HTTP headers and plain-text protocols)
tcpdump -v -A -r trace.pcap

# Filter to traffic involving microsoft.com's IP
tcpdump -nn -r trace.pcap 'host 20.112.52.29 and port 443'

# Show DNS queries and responses
tcpdump -A -nn -r trace.pcap 'port 53'

# All unique destination ports contacted (from SYN packets)
tcpdump -nn -r "trace.pcap" \
  'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' | \
  grep -oP '\d+\.\d+\.\d+\.\d+\.(\d+)' | \
  grep -oP '\d+$' | sort -n | uniq -c | sort -rn

Deeper Analysis with tshark

tshark provides a more powerful command-line option for analyzing capture files. A practical approach is to take a quick trace while troubleshooting an issue live and analyze it immediately with tcpdump or tshark. If the results are inconclusive, the capture file can be examined externally using tools such as Wireshark. For more complex scenarios, a comprehensive, long running (with file rotation) network trace can be captured to provide deeper insights.

# With absolute UTC timestamps
tshark -r trace.pcap -t ud
# All DNS records
tshark -r trace.pcap -Y "dns"
# TCP RST packets
tshark -r trace.pcap -Y "tcp.flags.reset == 1"
# TCP retransmissions (tshark expert analysis field — reliable)
tshark -r trace.pcap -Y "tcp.analysis.retransmission"
# IP endpoint statistics (top talkers by packet and byte count)
tshark -r trace.pcap -q -z endpoints,ip

In real-world environments, it is common to iterate through multiple analysis methods to isolate the issue. To simplify this process, below is a handy OSS script that wraps these native Linux tools into a single interface. It can be downloaded with a simple curl command and run either interactively or by specifying the destination IP and/or port.

Usage

Download and install the script with the below curl command

curl -fsSL https://raw.githubusercontent.com/azureossd/networking-troubleshooting-utility/refs/heads/main/nwutils_install.sh | bash

Install all tools (only) and run the commands manually

nwutils install

Run interactively for dynamic ports

nwutils run

Or pass the target directly

nwutils myapi.com 4000

Simply run the script and it installs the necessary tools, runs through the diagnostics, collects network trace (default 60s), analysis the trace and generates a logfile and a html report with a summary.

Overview

There are times that a Container App Environment may fail to delete. Although this is rare. Potential symptons of this after invoking a delete command (via portal, CLI, or else) may show something like the below:

• Fail fast, it may fail after a few seconds with an error about a failure to delete
• Timeout - the command/action may run for tens-of-minutes or more, also ending with a message about failure to delete

At this time of this post (02/16/2026) the direct root cause reason isn’t surfaced back. But we can infer two general reasons for why this may happen:

• Resource locks - This is especially common with environments that use a VNET and a lock is placed on the VNET infrastructure / managed-resource group. But this can happen on environments that don’t have a VNET.
• Unhealthy environment/cluster: A cluster that is in a failed state may cause delete (or in general, create/update/delete options from succeeding). This has it’s own potential causes too.

Resource locks

Resource locks on any of the following (but not limited to) would cause this failure to delete:

• On the Container App Environment
  • Potentially on any resources inside of it, such as an a Container App
• On the managed resource group that’s created as part of the environment when a custom VNET is used at creation time
  • This resource group gets deleted when the environment gets deleted - so if there is a lock on this resource group or anything in this resource group (eg. Azure Load Balancer, public IP, etc.), then the deletion operation will fail since it cannot fully clean up all resources

These resource groups will look like the following (unless it was customized at creation time):

• MC_some-env-name-rg_region (MC_ prefixed is used for Consumption-only environments)
• ME_some-env-name-rg_region (ME_ prefixed is used for Workload profile environments)

A quick way to see these resource groups is to go to the Subscription -> Resource groups blade in the Azure Portal.

Under Settings -> Locks, you can check if a lock is in place. By default, the Container Apps platform does not add locks.

NOTE: In real-world scenarios, this is likely added by deployment automation in companies/organizations/teams, etc. This automation may also be wide-spread to various resources, and may/may not also be tied to an Azure Policy.

If a user-created lock exists for any resource, you’ll see it in that blade. Below, we can see a lock named prevent-delete was added to this managed resource group.

In our case, a lock was created at the subscription level, which cascades to resources within it.

Now when you try to delete it, you’ll see a popup after a few minutes in the top left of the Azure Portal (or returned through your deployment client) with something like the following:

[resource_name]: The scope '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/some-rg/providers/Microsoft.App/managedEnvironments/some-env' cannot perform delete operation because following scope(s) are locked: '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'. Please remove the lock and try again. (Code: ScopeLocked)

The important part of this error is because following scope(s) are locked: '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - since this tells you what scope the lock is for. In our case, it’s a subscription level lock. If it was a resource group lock, it woud look something like /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/some-rg. If it was an individual resource lock, it would target the resource directly. Use this to then go to the Locks blade on the relevant resource, remove it - and then try the delete operation again.

More information on locks, including finding locks in ways not described above, review Azure Resource Locks.

Unhealthy environment/cluster

This can be more tricky to troubleshoot - you can get an indication something may be wrong with the environment/cluster state by the type of error returned during CRUD operations - as it may report back a message stating something like managedEnvironment provisioning state is failed. Use the below guidance to understand if this is related to deletion failures.

You can see this in a more clear way by going to Diagnose and Solve Problems -> Container App Down and look for the following:

• powerState - Cluster power state
• environmentProvisioningState - Container App Environment provisioning state
• managedEnvironmentProvisioningState - Managed environment provisioning state

The below is a non-exhaustive list for reasons why environment/cluster states may be unhealthy

If powerState is failed, then create/update/delete operations will likely not succeed.

If environmentProvisioningState not “Succeeded”, this doesn’t immediately mean a deletion or other operations will fail. Take into account the below:

• If environmentProvisioningState is shown as “Updating” for multiple hours, then this means this is not a “terminal” state, so create/update/deletion operations will potentially fail. The environmentProvisioningState should be at a terminal state, which essentially not “Updating” - if it reaches most other states, you should still be able to delete it
  • environmentProvisioningState being unhealthy could be due to a previous delete failure due to resource locks
  • This could also mean that the environment was attempted to be upgraded through typical Platform Upgrades at a prior point in time but you have an Azure Policy preventing updates. See this documentation on policies with Azure Container Apps for more information

managedEnvironmentProvisioningState may be unhealthy for a few reasons:

• Initial failure upon deployment. This again could be Azure Policy related. Eg. creating a resource without meeting certain policy demands could cause deployments to fail as only a subset of resources failed. Policy errors, like above, may not be returned to the user. You’d have to investigate what policies are set for these resources that are being created within this subscription
• This could also fail due to Platform Upgrades being blocked due to the above
• This could fail due to unhealthy infrastructure

TIP: If the suspected issue was due to an Azure Policy in please blocking upgrades or preventing resource changes - and you have not seen environmentProvisioningState or managedEnvironmentProvisioningState move to a succeeded state, then add OR update a tag on the Container App Environment to reconcile the cluster

NOTE: For Consumption-only environments - these are much more sensitive to customer-brought networking, especially for UDR’s and NAT gateways. Blocking any part of underlying AKS traffic (eg. kubeapi-server) will cause the cluster to enter a failed state. Which in turn would cause the above provisioning states to be failed, and your create/update/delete operations to fail.

• This is called out in Securing a virtual network in Azure Container Apps, through Required outbound network rules and FQDNs for AKS clusters. The fact UDR’s are only supported on Workload Profiles are also mentioned here. This also applies to NAT gateways. The short of it is, avoid using this - or send all traffic to the internet through a route, to avoid breaking the environment.
  • Ideally, it is heavily recommended to migrate to and use workload profile environments which support this as a feature.

Overview

In some App Service Logs, with a App Service Linux - Node.js “Blessed” image (not a custom image, eg. Web App for Containers), you may see a reference to a directory named _del_node_modules.

This directory is created on startup through logic defined in the Node.js Blessed Image’s container entrypoint and startup behavior. You can see the logic invoked to create it if App Service Logs are enabled, and if you look in default_docker.log.

2026-01-30T21:06:43.8273864Z export NODE_PATH="/node_modules":$NODE_PATH
2026-01-30T21:06:43.8273897Z export PATH=/node_modules/.bin:$PATH
2026-01-30T21:06:43.8273929Z if [ -d node_modules ]; then
2026-01-30T21:06:43.8273962Z     mv -f node_modules _del_node_modules || true
2026-01-30T21:06:43.8273993Z fi
2026-01-30T21:06:43.8274023Z 
2026-01-30T21:06:43.8274065Z if [ -d /node_modules ]; then
2026-01-30T21:06:43.8274098Z     ln -sfn /node_modules ./node_modules 
2026-01-30T21:06:43.8274130Z fi
2026-01-30T21:06:43.8274162Z 
2026-01-30T21:06:43.8274192Z echo "Done."
2026-01-30T21:06:43.8274224Z node server.js

Call outs about this directory

• It is essentially transparent. During deployments, or, mostly any node_modules related aspects, this should not be touched or worried about. This doesn’t directly affect the application.
• This directory has the same package versions as the ones referenced in your package.json and also the node_modules being used by your application. In some cases, if you have a cloud security scanner, it may flag _del_node_modules as having outdated versions.
  • You can SSH into the application container and look at the package.json for your packages in either /node_modules, /home/site/wwwroot/node_modules, or _del_node_modules and cross compare these
  • If there is a scenario where _del_node_modules is being flagged as having an older version of packages. Delete the directory, restart, and then redeploy. However, you should ensure the applications package.json (or package-lock.json, yarn.lock) should NOT be the one to contain the package version causing this to be flagged in the first place.
    • Testing with build automation enabled (Oryx) and without (which nowadays most sites will use the same node_module compression logic that Oryx uses as called out in Improved Node.js Deployment Performance on Azure App Service) shows that _del_node_modules and node_modules will have mirroring package versions

The only directory, in most cases, that you should share about, is the ones described in here: Improved Node.js Deployment Performance on Azure App Service.

Overview

This post will cover the error version 'GLIBC_x.xx' not found. This error signature may vary depending on the language of the application, but it generally has the same syntax. This can happen for any language - the context of this post applies to “Blessed Images” but this general information can apply to anywhere this happens on Linux machines.

A full error in a real-world setting is below, where the language is Python:

• ImportError: /lib/x86_64-linux-gpu/libc.so.6: version 'GLIBC_2.33' not found (required by /home/site/wwwroot/antenv/lib/python3.12/site-pcackages/cryptography/hazmat/bindings/_rust.abi3.so)

Deconstructing the error

The main part of the error is version 'GLIBC_2.33' not found. Which is saying that a particular version of glibc is not found at runtime. Since App Service Linux is talked about here, the runtime is the container. The container is ran from an image in which a certain version of glibc is included in, normally based on the distribution and version.

This version of glibc is being requested by a specific peice of code. Normally this is going to be something like C bindings or shared object libraries. We see this occasionally with packages that are primarily built upon C, such as cryptography, bcrypt and other packages like this.

How can you check the glibc version?

Go into SSH and run ldd --version. Do this in the application container, not the Kudu container.

NOTE: The application must be running to go into SSH. If it’s crashing you cannot SSH in. You cannot SSH into a container that is crashing, in general.

NOTE: For custom images/custom containers (Web App for Containers), you need to enable SSH in your image. See Enabling SSH on Linux Web App for Containers

What is “glibc?”: This is the C standard library (GNU) which is a foundational aspect in Linux. There is also musl C, which Alpine uses.

Important points and troubleshooting

Do not try to manually change the libc version. This means attempting things like trying to install/update glibc or things along this path because:

• You may break other aspects of the container. There is a large amount of tooling builtin to container images that normally rely on C. Changing this may cause adverse/unexpected behavior at runtime
• You may waste a large amount of time. Even though you can alter some behavior via startup commands/scripts - due to complexity of trying to change it while troubleshooting the initial error, you may end up being unsuccessful and ending frusturated

Typical path to resolution:

• Identify the package that wants a specific version of glibc. You may be able to downgrade or upgrade these. The package repository (or general online searching) should point to information that shows what version of glibc a package may use
• Or, If you can’t down/upgrade - then for “Blessed Images”, see if you can upgrade to a higher version of the image you’re on. For example, at the time of writing this blog post and in relation to the above error that’s ran within a Python “Blessed” image, the following glibc versions are utilized:
  • Python 3.11 image: glibc 2.21
  • Python 3.12 image: glibc 2.21
  • Python 3.13 image: glibc 2.36
  • Python 3.14 image: glibc 2.39
  • The error assumes that the application is using an image that’s 3.12 or below. Therefor a potential resolution is to use the Python 3.13 image or higher.
  • This same concept should apply to other Blessed Images. Of course, there may be differences in which glibc versions are used across language versions of the other images. Use ldd --version to check the glibc version while in SSH
• Or, Use a custom image through Web App for Containers. There may be packages that require a specific version that may not be compatible with the current distribution version or other aspects in Blessed Images. This is an occasionally recommended path for things like this is to use a custom image since you have complete control over how the image is built.

Overview

A “webhook” in this context is an endpoint exposed on the Kudu/.scm. site for a Web App for Containers resource that offeres another way to restart the site, which will also initiate an image pull, as apart of the site restart operations.

This endpoint is exposed at: https://[username]:[password]@mysite-bxx8bgaxxxxx.scm.eastus-01.azurewebsites.net/api/registry/webhook

• [username] is the username for “Basic Auth” credentials that are enabled/disabled in the Configuration blade within the portal
• [password]` is the password “Basic Auth” credentials

These credentials are found in Deployment Center under the Application-scope section by default.

Mechanics of the webhook

This is touched on above, but to be more specific, this endpoint offers another way to trigger an image pull.

In most cases, users will update the images tag set for an image used on an application (in this case, Web App for Containers), to something unique - typically incremental over time as updates are done to an applications image, eg:

• myacr.azurecr.io/myimage:2025-12-16-v1.0.0.1
• myacr.azurecr.io/myimage:2025-12-16-v1.0.0.2
• myacr.azurecr.io/myimage:2025-12-16-v1.0.0.3

Since these are site configuration level updates, and implicitly also will cause a restart - which the “restart” causes the image to be repulled, you’ll have your new image downloaded with all relevant layer changes. This normally fits most use cases. Another scenario that may happen is users pushing changes to the same tag, which is not recommended - in this case, you’d have to manually restart the application for changes to any layers on that same tag to be repulled.

• Note: App Service, like most other PaaS services, do not “watch” or have any awareness that you would have pushed a change to the same tag. A restart is the only way these new changed layers would be repulled with the container runtime.

A webhook, on the other hand, does have “awareness” that an event happened for a specific tag, defined by what action and scope. This webhook is created on Azure Container Registry - it is not created on App Service, only the endpoint exposed to allow restarts for a notification event is exposed by App Service. The only relevancy this has to App Service is the Service URI, in which a “notification” is posted (eg. to restart)

More information on Azure Container Registry Webhooks can be found at Using Azure Container Registry webhooks

Use cases - pushing changes to the same tag

Although not recommended for reasons explained in the overall container community, you can push changes to the same tag - and not have to manually restart (or write some automation/operation to restart the site) when pushing changes to the same tag being used by your Web App for Container.

To be more specific and clear on “pushing changes to the same tag”, this means that you’re just only pushing changes to your Azure Container Registry. You also do not need to have some type of deployment event (like a CI/CD pipeline, or other explicit CLI commands or IaC set up) that would need to handle deployment. This can be seen as another benefit in certain situations.

By enabling a webhook (below), and keeping the scope to that same image:tag in use, you’ll see that by reviewing docker.log under /home/LogFiles in App Service Logs (or other areas like Log stream), that the application will be restarted, which will repull image layers, and thus your changes.

• From the time the webhook notification is posted to the App Service webhook endpoint, to the time it takes to start the image pull, may not be immediate
• Meaning, there may be a couple of seconds or so gap, which is expected

Pushing to a webhook

After setting up your webhook, you can use the UI on the Azure Container Registry side to look at webhook events. In this example, we pushed an image with the tag of latest to Azure Container Registry - which fired this webhook.

Click into the webhook to see if any events fired, for example:

If we look at docker.log (or generally container lifecycle events on this Web App for Container), you’ll see a restart was triggered at the same time:

Note: 11:52AM EST = 16:52 UTC

Creating a webhook

If you want to create a webhook for a registry that is not Azure Container Registry, you need to follow the below option to manually create it on your Azure Container Registry.

On Azure Container Registry

On your Azure Container Registry, go to Services > Webhooks. Click “Add”. Then fill out the required information.

On App Service

Go to Deployment Center > Check the Continuous deployment box, and click Save. This will create a ready-to-go webhook in your Azure Container Registry under webhooks. This will be created with a scope of your current image:tag set on the Web App for Container.

Other information

• push is the default action, which should be kept
• scope is also important, in-line with the above information in this post, keep this to a specific image:tag you’ll be pushing changes to. scope is what the webhook is “watching” changes for. This can be a specific tag, or all tags in a repository.
  • Note: Doing something like image:* to watch any tags under an image repository won’t do anything in App Service’s case since for Web App for Containers, this isn’t going to change the images tag on App Service to one that was updated (or anything similiar to that). Forthat effect, you need to do change the image tag on App Service explicitly.

Troubleshooting

If you notice no restart is happening, go to Azure Container Registry and look at your webhook events. Typically, a HTTP 4xx or 5xx status would be indicative of an issue. Below are examples of HTTP 401’s

HTTP 401:

• Your username and/or password is incorrect. If manually setting this up on the ACR side, double check your credentials
• As a test, set this up via the App Service side since it’ll automatically fill this information in for you
• You also may have Basic Auth publishing disabled. This must be enabled for webhooks to work.

HTTP 403

• Your site likely has a Private Endpoint or Access Restrictions
  • If this is due to Access Restrictions, you will need to whitelist the registry IP
  • For Private Endpoints, ensure the registry has access to the subnet App Service is in

HTTP 500

• The kudu container is experiencing an issue. Try scaling (up/down) to land on a new instance and have the kudu container restart

HTTP 503

• The kudu container is likely crashing. If using Bring Your Own Storage - when the volume is failing to mount, this will also bring down the kudu container. Review if this is the case. See How to troubleshooting Bring Your Own Storage (BYOS) issues on App Service Linux
• Otherwise, try scaling (up/down) to land on a new instance and have the kudu container restart

No HTTP response is seen/no notification is posted

• There are times inbound network traffic may block a webhook event, which may not return a HTTP response from Kudu (since its blocked before this)
• Review inbound NSG’s and any networking flows towards App Service to ensure this traffic is allowed

Azure Container Apps (ACA) simplifies container orchestration, but capacity planning often confuses developers. Questions like “How do replicas consume node resources?”, “When does ACA add nodes?”, and “How should I model limits and requests?”. This guide pairs official ACA guidance with practical examples to demystify workload profiles, autoscaling, and resource modelling.

Understanding Workload Profiles in Azure Container Apps

ACA offers three profile types:

Consumption

• Scales to zero
• Platform decides node size
• Billing per replica execution time

Dedicated

• Choose VM SKU (e.g., D4 → 4 vCPU, 16 GiB RAM)
• Billing per node

Flex (Preview)

• Combines dedicated isolation with consumption-like billing

Each profile defines node-level resources. For Example: D4 → 4 vCPU, 16 GiB RAM per node.

2. How Replicas Consume Node Resources

ACA runs on managed Kubernetes.

• Node = VM with fixed resources
• Replica = Pod scheduled on a node
• Replicas share node resources; ACA packs replicas until node capacity is full.

Example

Node: D4 (4 vCPU, 16 GiB RAM)
Replica requests: 1 vCPU, 2 GiB
5 replicas → Needs 5 vCPU, 10 GiB

ACA places 4 replicas on Node 1 and adds Node 2 for replica 5

3. When ACA Adds Nodes

ACA adds nodes when:

• Pending replicas cannot fit on existing nodes
• Resource requests exceed available capacity

ACA uses Kubernetes scheduling principles. Nodes scale out when pods are not schedulable due to CPU/memory constrains.

4. Practical Sizing Strategy

1. Identify peak load → translate to CPU/memory per replica
2. Choose workload profile SKU (e.g., D4)
3. Calculate packing: node capacity ÷ replica request = max replica node
4. Add buffer ( e.g 20% ) for headroom
5. Configure autoscaling:
   • Min replicas for HA.
   • Max replicas for burst.
   • Min/Max nodes for cost control.

5. Common Misconceptions

Myth: “Replicas have dedicated CPU/RAM per container automatically.” Reality: Not exactly.They consume from the node pool based on your configured CPU & memory. Multiple replicas compete for the same node until capacity is exhausted.

Myth: “ACA node scaling is CPU-time based.” Reality: ACA node scaling is driven by unschedulable replicas (cannot place due to configured resources). Triggers for replica scaling are KEDA rules (HTTP, queue, CPU/memory %, etc.), but node scale follows from replica placement pressure.

6. Key Takeaways

• Model per-node packing before setting replica counts
• Plan for zero-downtime upgrades (double replicas temporarily)
• Monitor autoscaling behavior; defaults may not fit every workload

Overview

As of writing this, documentation for mounting volumes with Azure Container Apps is mostly cli-based. This post will cover a basic NFS set up from scratch. Current documentation for NFS volumes and Azure Container Apps can be found at Use storage mounts in Azure Container Apps

Creating resources

Storage

To start, you’ll need a Premium tier Storage Account. An NFS share cannot be created on a Standard one if that’s being used.

1. In the Azure Portal, click on ‘Create a Resource’ and search for Storage Account

   

2. Set the following options on the Storage Account basics
   • Subscription: Choose your subscription to create the Storage Account in. Use the same subscription as the Container App Environment
   • Resource Group: Choose with Resource Group to create this in
   • Storage Account Name: Choose a name for the account
   • Region: Choose a region for the account
   • Preferred storage type: Set this to Azure Files
   • Performance: Set this to Premium
   • File share billing and Redundancy: Set this to your preferred options

   

3. Select Review and create to create the Storage Account. The rest of the tabs during the creation process can be left the same.

Create the NFS share

1. On the Storage Account, go to the Data Storage blade and then select File Shares
2. Click on “+ File Share” to create a new share
3. In the File Share creation window, set the following:
   • Name: Name of your NFS share
   • Protocol: NFS
   • Root Squash: Keep this defaulted to No Root Squash
   • The rest of the options under Provisioned storage and Performance can be left alone unless desired to be changed.

   

4. Click Review + create to create the share.

Remove “Secure Transfer Required”

On the Storage Accountt, go to Settings > Configuration and disable Secure transfer required. If this is not done, you’ll see a volume mount failure later on with a message of mount.nfs: access denied by server while mounting in Container App system logs

Container App Environment

At this point, if creating the Storage Account first, you’ll see the message about any access to the NFS share is through a Virtual Network. For the Storage Account, VNET integration can be done later on after the fact, post-creation.

With a Container App Environment, the Virtual Network needs to be specified at creation time. You cannot enable a VNET post-creation - so if you’re following along and your environment does not have a VNET, you’ll need to create a net-new one.

Create a Container App Environment and Container App (with a VNET)

1. In the Azure Portal, click on ‘Create a Resource’ and search for Container App

   

2. On the Basics blade:
   • Subscription: Choose a subscription that the Storage Account from earlier exists i n
   • Resource group: Choose a Resource Group
   • Container app name: Choose a name for your Container App
   • For Optimize for Azure Functions, leave this as-is (unless you’re deploying an Azure Function)
   • Deployment Source: Container image (unless you’re deploying from source - eg. source-to-cloud)
   • Container Apps environment:
     • Region: Select a region for the envinronment
     • Container apps environment: Click on “Create new environment”
       • Environment name: Give a name for the environment
       • Networking blade: In the networking blade, select Use your own virtual network as “Yes” under the Virtual network section
         • Select Create new if you do not already have an existing VNET and an empty subnet
         • Leave Virtual IP as external (for the sake of this blog post)
         • Leave Infrastructure resource group empty so it takes the default name

         • All the other blades for Workload profiles and Monitoring can be left-as unless you want to enable Workload Profiles or change your logging destination during the creation process

           

         • Then, select Create
3. On the Container blade
   • You can either use the quickstart image, by checking the Use quickstart image image checkbox
   • Or, bring your own image - NOTE: Failures can happen post-creation, which, at this point would have nothing to do with the storage volume. Always review error logs - follow (Applications (and revisions) stuck in activating state on Azure Container Apps)[https://azureossd.github.io/2025/05/05/Applications-(and-revisions)-stuck-in-activating-state-on-Azure-Container-Apps/index.html] for common reasons why revisions may end up in a degraded or failed state.

   

4. Ingress blade
   • If using your own container, and if this application expects external traffic, ensure the Target port matches the application listening port
5. Click Review and create to create the environment with the selected VNET, as well as the Container App.

(Option 1) Service Endpoint - Allow the Storage Account access from the ACA VNET

Since the VNET is now created - go back to your Storage Account from earlier.

1. Go to the File Share. Under Overview will show the following:

   

2. For ease of setup in terms of this blog post, select Service Endpoint
3. Use the following options:
   • Public network access: Enabled from selected virtual networks and IP addresses
   • Virtual networks: Select Add existing virtual network and then select the VNET and subnet from earlier. This will prompt to enable a Service Endpoint, which you want to do. Select Enable

   

   • After this process completes while in the same blade, click Add
   • Then, click Save

4. At this point, Endpoint Status should be Enabled and we can now move on to adding the storage resource and volume on the environment and application

   

(Option 2) Private Endpoint - Allow the Storage Account access from the ACA VNET

Another option is using a Private Endpoint. You’ll need an empty subnet (different than the one given to the Container App Environment) to proceed. You can create an empty subnet in your Virtual Network with the defaults provided and then move back to the below blade on the Storage Account - or - just follow Create a private endpoint for Azure Files

1. Go to the File Share. Under Overview will show the following:

   

2. Select Private Endpoint. This will take you through the creation blade for a Private Endpoint. Follow the step form to create this:
   • Basics:
     • Subscription: Create this in the same subscription as the Container App Environment and Storage account
       • Resource Group: Choose a resouce group
     • Name: Give the Private Endpoint a name
     • Network Interface Name: This will defaul to [name]-nic, leave this as the default
   • Resource: Leave the defaults
   • Virtual Network:
     • Virtual Network: Select the VNET from earlier
     • Subnet: Select the empty subnet created (mentioned above)
     • Leave the rest of the defaults for all following tabs
   • Click through and then click Create
3. After Private Endpoint creation, you should see Private link resource set to your Storage Account. If you nslookup your file share FQDN, you should see this now also has the private link alias associated with it:

$ nslookup myfsstorageaccount.file.core.windows.net
Server:  some.thing.com
Address:  xxx.xx.xx.xx

Non-authoritative answer:
Name:    file.xxxxxxxxxx.store.core.windows.net
Address:  xx.xx.xxx.x
Aliases:  myfsstorageaccount.file.core.windows.net
          myfsstorageaccount.privatelink.file.core.windows.net

Create a Storage Resource on the environment

NOTE: Irregardless if you used a Service Endpoint or Private Endpoint, the normal FQDN for the file share (eg. somefileshare.file.core.windows.net will be used below - don’t use the Private Link alias if a Private Endpoint was used)

1. Go back to the Container App Environment and select Azure Files under Settings
2. Select Add and then Network File System (NFS)
3. In the pop-out blade, set the following:
   • Name: Set a name for the resource
   • Server: Set this to yourstorageaccount.file.core.windows.net
   • File share name: Set this to /yourstorageaccount/yournfssharename
   • Access Mode: Set your access mode to read/write or read-only

   IMPORTANT: Make sure to include the forward slashes, this syntax is expected.

   

NOTE: If you go back to your NFS file share, you’ll see this info in which you can pull from

1. Select Add and then Save

Add a volume to the Container App

1. Go to your Azure Container App that you created and go to Application > Volumes
2. Select Add to add a volume
   • Volume type: Azure file volume
   • Name: Choose a name for the volume
   • File share name: Choose the name of the storage resource created earlier above on the Container App Environment
   • Mount options: Omit this unless you know which options to pass in. Allowed mount options are also the ones listed on the NFS “man page” - nfs(5) - Linux man page

   

3. Lastly, click Add. Then click Save as new revision

Mount the volume to the Container App

NOTE: You can either explicitly create a new revision through the Revisions and replicas or do this through the Containers blade

1. Go to your Azure Container App that you created and go to Application > Revisions and replicas
2. Click Create new revision
3. Select the container to mount a volume to, then click Edit
4. On the popout window, go to the Volume mounts tab
   • Volume name: Select the Volume created just before on the Container App
   • Mount path: Specify a directory to mount this volume to

5. Click Save, then click Create

   

Confirm the volume is mounted

If the Revision is not in a degraded or failed state after enabling the volume mount, this already would imply this is successful.

This is is now Failed or Degraded, ensure your Logs destination is set to either Log Analytics or Azure Monitor and query the system logs table for ContainerAppSystemLogs (Azure Monitor) or ContainerAppSystemLogs_CL (Log Analytics) and review the errors seen there.

Otherwise, go to the Console blade, and run df -a or df -h. You can also use the mount command, assuming it’s installed in your container image and available in the running container, to use the command mount | grep "nfs" to see additional arguments and version information

If using a Private Endpoint, you’ll still see the normal FQDN for the file share in the above output. But if you nslookup this from within the container, you should see this resolves to the IP of the NIC for the Private Endpoint

We know this works because if this was failing, console access would be unavailable due to the fact the volume monut happens early on in the container lifecycle (within a pod) - a container is not yet created and running by that point in time.

FAQs and other information

FAQs

• You cannot pass mount options like -t, --types through the “mount options” field. -t is only set when you either choose SMB (mount.cifs) or NFS (mount.nfs). Using the above Console method, you can see that the NFS version used is nsf4. This cannot be changed (eg. to aznfs)
• SMB and NFS have different methods of setting file/directory permissions which can be seen here: Container Apps - Setting storage directory permissions
• SMB requires Access Keys to set up a storage resource and mount a volume, NFS does not
• NFS requires a VNET on the Container App Environment and integrated with the Storage Account to make all of this possible
• This blog post only covered a “typical” Private Endpoint set up - if your VNET has custom DNS, ensure the file share FQDN can be resolved against your DNS servers. You may need to add Azure DNS (168.63.129.16) to “resolve unresolved queries” on your DNS servers
• If a volume mount is failing, no “application” / “console” logs will be available, only system logs. This is because a volume mount happens very early on in a pod lifecycle, prior to a running container. A container never gets to the point of being successfully created and running, which means no application process ever starts (and thus, no application logs)

Troubleshooting

• For general storage mount troubleshooting, review Troubleshooting volume mount issues on Azure Container Apps
• You also should review Azure Files security compatability on Container Apps. This by default is set to “maximum capability” on the file share - if this is changed to options other than this, you risk having the mount fail with permission denied (13)
• Forgetting to disable “Secure Transfer Required” on the Storage Account will also cause permission denied

Aside from system logs, going to Diagnose and Solve Problems > Storage Mount Failures will also display storage troubleshooting information for mount failure on the Container App