How to write Retry Logic for Failed Commands in Bash

Retry Logic for Failed Commands in Bash

In Bash scripting, sometimes commands may fail due to temporary issues, such as network outages, server downtime, or other intermittent errors.

Instead of allowing the script to fail immediately, you can implement retry logic, which attempts to rerun the failed command a specified number of times before giving up. This approach increases the resilience of your scripts, particularly when dealing with unreliable resources or external dependencies.

In this tutorial, we’ll go through why use retry logic for failed commands, the syntax for retry logic, and a few examples to understand this concept.

---

Why Use Retry Logic in Bash?

Retry logic is beneficial when you encounter commands that might fail temporarily but are expected to succeed if retried. Common scenarios include:

• Network requests (e.g., downloading a file or making an API call)
• Database queries or connections
• File operations (e.g., accessing a file that is temporarily locked)
• Commands interacting with external services or APIs

---

Basic Syntax for Retry Logic

The basic structure for implementing retry logic in Bash involves using a loop to repeatedly attempt the command, checking its exit status after each attempt.

You can control the maximum number of retries and the delay between attempts.

while [ $attempt -le $max_attempts ]
do
    command
    if [ $? -eq 0 ]; then
        break
    fi
    attempt=$((attempt + 1))
    sleep $delay
done

In this syntax:

• attempt: The current retry attempt count.
• max_attempts: The maximum number of retry attempts allowed.
• delay: The time (in seconds) to wait between retry attempts.

We have used While loop in the syntax.

---

Examples of Retry Logic in Bash

Let’s look at some examples to illustrate how to implement retry logic in different scenarios.

---

1. Retry a Command with a Fixed Number of Attempts

In this example, we try to download a file using curl. If the download fails, the script retries up to 3 times with a 5-second delay between attempts.

example.sh

#!/bin/bash

url="https://example.com/file.txt"
max_attempts=10
attempt=1
delay=5

while [ $attempt -le $max_attempts ]
do
    echo "Attempt $attempt: Downloading file..."
    curl -O $url
    if [ $? -eq 0 ]; then
        echo "File downloaded successfully."
        break
    fi
    echo "Download failed. Retrying in $delay seconds..."
    attempt=$((attempt + 1))
    sleep $delay
done

if [ $attempt -gt $max_attempts ]; then
    echo "Failed to download file after $max_attempts attempts."
    exit 1
fi

Output

In this script, if the download fails, it retries up to 3 times. If it still fails after the maximum number of attempts, the script exits with an error message.

How did we do this, we disconnected the internet and run the script. Then during second attempt we switched on the the internet. And when it attempted to download for the third time, it could download the file successfully.

---

2. Retry a Command with Exponential Backoff

Exponential backoff is a strategy where the delay between retry attempts increases exponentially. This approach reduces the load on external systems during failures. In this example, we use exponential backoff for retrying a network request.

example.sh

#!/bin/bash

max_attempts=5
attempt=1
delay=2

while [ $attempt -le $max_attempts ]
do
    echo "Attempt $attempt: Pinging google.com..."
    ping -c 1 google.com
    if [ $? -eq 0 ]; then
        echo "Ping successful."
        break
    fi
    echo "Ping failed. Retrying in $delay seconds..."
    attempt=$((attempt + 1))
    sleep $delay
    delay=$((delay * 2))
done

if [ $attempt -gt $max_attempts ]; then
    echo "Failed to ping google.com after $max_attempts attempts."
    exit 1
fi

Output

In this script, the delay between retries doubles after each failed attempt, implementing an exponential backoff strategy. This method helps avoid overwhelming the network or server during outages.

---

Best Practices for Implementing Retry Logic

• Limit the number of retries: Set a reasonable maximum number of attempts to avoid infinite loops and resource wastage.
• Use delays between retries: Adding a delay between attempts prevents overwhelming the external service and allows time for issues to resolve.
• Implement exponential backoff: Increase the delay after each failed attempt to reduce the load on the system during extended outages.
• Log each attempt: Keep track of retry attempts in logs for easier debugging and monitoring.
• Check exit status codes: Always verify the exit status of commands to determine whether the retry is necessary.

By following these best practices, you can make your Bash scripts more resilient and capable of handling temporary errors gracefully.