Application does not recover after a short outage

Your application ran for weeks without a problem. Then the database server was unavailable for a few minutes, or a server was restarted during maintenance, and the website stayed down for hours — with HTTP Error 500.30 – ASP.NET Core app failed to start or 503 Service Unavailable — until you clicked Restart application in the Control Panel.

The short outage was not caused by your code. Staying down afterwards was. This article explains why a few minutes of outage become hours, which patterns in the code are responsible, and how to write an application that recovers on its own.

Why a short outage becomes a long one

ASP.NET Core applications run InProcess on our servers: your code is loaded directly into the IIS worker process. This is the default and the recommended hosting model — it is fast and uses the least memory. It has one consequence you have to know about.

A failed start is final for that worker process. When your Program.cs throws an exception — for example because the database is not reachable — the ASP.NET Core Module remembers that the application failed and answers every following request with the 500.30 page. It does not try to start the application again. The only way out is a new worker process, which is created by:

• Restart application in the Control Panel, a deployment or a configuration change,
• the idle timeout of 3 hours — but every request, even one answered with 500.30, resets it. A website that receives webhooks, API calls or monitoring requests never becomes idle, so it never restarts.

In other words: the database is back after 10 minutes, but your application keeps saying 500.30 until somebody restarts it.

Three paths lead into this state:

1. The application starts while a dependency is down. Typical during maintenance: the web server comes back a little earlier than the database server, or the application is restarted exactly when the database is unreachable. Any code in Program.cs that touches the database (migrations, seeding, loading settings) throws, and the start fails.
2. A background service throws. Since .NET 6 an unhandled exception in a BackgroundService stops the whole application by default. The application shuts down, IIS immediately starts a new one, the new one runs the start-up code against the still unreachable database, and it fails as described above.
3. The process crashes repeatedly. An exception on a thread that nobody awaits — a timer callback, async void, a raw Thread — terminates the process. If this happens several times within a few minutes, the web server stops the application pool to protect the server and the website returns 503 until the pool is started again. See AppPool is not enabled on server.

A real example

The database server of a Blazor application was restarted during maintenance and was unavailable for 5 minutes.

Time	What happened
02:10	The database server stops responding. Requests fail, but the application is still running.
02:12	A background service that checks the database every few minutes throws SqlException outside a try/catch. The application shuts itself down. IIS starts a new worker process.
02:12	Program.cs runs the identity seeder against the database, throws, and the start fails. From now on every request is answered with 500.30 in 12 ms.
02:15	The database server is back. Nothing happens — nobody retries the start.
02:15 – 08:40	Users, webhooks and API clients hit the site and get 500.30 every time.
08:41	The customer notices, restarts the application in the Control Panel, and the start succeeds.

Five minutes of maintenance became six and a half hours of outage for the customer, and it would have lasted until the next deployment if nobody had noticed.

Cause 1: database work in Program.cs

The most frequent reason. Code like this runs on every start and has no protection at all:

var app = builder.Build();

using (var scope = app.Services.CreateScope())
{
    var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    db.Database.Migrate();                        // throws when the database is unreachable
    await Seeder.SeedAsync(scope.ServiceProvider); // and so does this
}

app.Run();

Any Migrate(), EnsureCreated(), seeding, cache warm-up or "load settings from the database" in Program.cs turns a database outage into a failed start.

Option A (recommended): do not run migrations at start-up. Apply them during deployment instead — with a SQL script generated by EF Core that you run in SQL Server Management Studio, or with a migration bundle (dotnet ef migrations bundle) run from your deployment pipeline. Production start-up then never writes to the schema, which is also the safer option when more than one instance or a deployment and a user hit the database at the same time.

Option B: run the start-up work in a hosted service with retries. The application starts immediately, serves static files and error pages, and keeps trying to reach the database in the background until it succeeds:

public sealed class StartupTasks : BackgroundService
{
    private readonly IServiceScopeFactory _scopes;
    private readonly ILogger<StartupTasks> _log;

    public StartupTasks(IServiceScopeFactory scopes, ILogger<StartupTasks> log)
    {
        _scopes = scopes;
        _log = log;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var delay = TimeSpan.FromSeconds(5);

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope = _scopes.CreateScope();
                var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

                await db.Database.MigrateAsync(stoppingToken);
                await Seeder.SeedAsync(scope.ServiceProvider, stoppingToken);

                _log.LogInformation("Start-up tasks finished");
                return;                                   // done, the service ends
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                return;                                   // the application is stopping
            }
            catch (Exception ex)
            {
                _log.LogError(ex, "Start-up tasks failed, retrying in {Delay}", delay);
                await Task.Delay(delay, stoppingToken);
                delay = TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds * 2, 120)); // back off, at most 2 minutes
            }
        }
    }
}

Register it in Program.cs and remove the database code from there:

builder.Services.AddHostedService<StartupTasks>();

var app = builder.Build();
app.Run();

Requests that need the database before the tasks finish fail with an error page, which is acceptable: the moment the database is back, the next request works — without a restart.

Cause 2: a background service without try/catch

Code like this stops the whole application the first time the database hiccups:

public class SessionMonitor : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope = _scopes.CreateScope();
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

            var sessions = db.Sessions.ToList();          // SqlException here ends the application
            // ...
            await db.SaveChangesAsync(stoppingToken);

            await Task.Delay(TimeSpan.FromMinutes(13), stoppingToken);
        }
    }
}

The exception leaves ExecuteAsync, the host logs BackgroundService failed and — because BackgroundServiceExceptionBehavior is StopHost by default since .NET 6 — shuts the application down. From the outside it looks like the website restarted for no reason, and if the database is still down, the restart fails as in cause 1.

Every loop in a background service needs a try/catch around one iteration:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        try
        {
            using var scope = _scopes.CreateScope();
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

            var sessions = await db.Sessions.ToListAsync(stoppingToken);
            // ...
            await db.SaveChangesAsync(stoppingToken);
        }
        catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
        {
            break;                                        // normal shutdown
        }
        catch (Exception ex)
        {
            _log.LogError(ex, "Session check failed, will try again");
        }

        try
        {
            await Task.Delay(TimeSpan.FromMinutes(13), stoppingToken);
        }
        catch (OperationCanceledException)
        {
            break;
        }
    }
}

As a safety net for services you forgot, tell the host not to stop when a background service fails:

builder.Services.Configure<HostOptions>(options =>
{
    options.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.Ignore;
});

With Ignore the failed service is simply gone until the next restart — the website keeps running, but the job does not. The try/catch loop above is still the real fix; the option only prevents the worst case.

Cause 3: no retries on the database connection

A connection attempt that fails once should not fail the whole operation. Both EF Core and SqlClient can retry for you; by default they do not.

Entity Framework Core — enable the retrying execution strategy:

builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(connectionString, sql =>
        sql.EnableRetryOnFailure(
            maxRetryCount: 5,
            maxRetryDelay: TimeSpan.FromSeconds(10),
            errorNumbersToAdd: null)));

Connection failures, timeouts and transient SQL Server errors are retried up to five times with growing delays before the exception reaches your code. If you use explicit transactions (BeginTransaction), wrap them in CreateExecutionStrategy().ExecuteAsync(...) as described in the EF Core documentation.

Plain SqlClient, Dapper and other libraries — add connection retries to the connection string:

Server=dbXXXX.databaseasp.net; Database=dbXXXX; User Id=dbXXXX; Password=...; Encrypt=False; ConnectRetryCount=3; ConnectRetryInterval=10;

ConnectRetryCount (default 1) and ConnectRetryInterval (seconds, default 10) tell the driver how often to retry opening a connection before it gives up.

Cause 4: exceptions on threads nobody awaits

An exception that escapes a thread the runtime does not observe ends the process immediately — there is no request to return an error to, so the worker process dies. The event log then shows an Application Error entry and, after several crashes in a short time, the application pool is stopped. Watch out for:

• async void methods (anything except event handlers),
• callbacks of System.Threading.Timer and System.Timers.Timer,
• new Thread(...) and ThreadPool.QueueUserWorkItem(...).

// Crashes the process when SendReminders throws
_timer = new Timer(_ => SendReminders(), null, TimeSpan.Zero, TimeSpan.FromMinutes(5));

// Safe
_timer = new Timer(_ =>
{
    try { SendReminders(); }
    catch (Exception ex) { _log.LogError(ex, "Reminder job failed"); }
}, null, TimeSpan.Zero, TimeSpan.FromMinutes(5));

Better still, replace timers and fire-and-forget threads with a BackgroundService and PeriodicTimer, written as in cause 2. Note that Task.Run(...) without await is the opposite problem: its exceptions are silently swallowed and the job just stops working. Await your tasks or catch inside them.

Cause 5: other dependencies at start-up

The same rules apply to everything your application contacts while starting: an external API that returns the exchange rates, an SMTP server, Redis, a licence server, a storage account. Give each call a timeout, catch the exception, log it, and let the application start without the data. A website that shows yesterday's exchange rates is better than a website that shows 500.30.

A start-up that survives an outage

Putting it together, Program.cs of a resilient application looks like this:

var builder = WebApplication.CreateBuilder(args);

var connectionString = builder.Configuration.GetConnectionString("DefaultConnection")
    ?? throw new InvalidOperationException("Connection string 'DefaultConnection' is missing."); // a restart will not fix this - fail fast

builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(connectionString, sql => sql.EnableRetryOnFailure()));

builder.Services.Configure<HostOptions>(options =>
    options.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.Ignore);

builder.Services.AddHostedService<StartupTasks>();   // migrations and seeding with retries, see cause 1
builder.Services.AddHostedService<SessionMonitor>(); // every loop iteration in try/catch, see cause 2

builder.Services.AddHealthChecks()
    .AddDbContextCheck<AppDbContext>();              // package Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore

builder.Services.AddRazorPages();

var app = builder.Build();

app.UseExceptionHandler("/Error");
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.MapRazorPages();
app.MapHealthChecks("/health");

app.Run();                                            // nothing above this line talks to the database

With this structure the database can disappear at any moment: requests fail while it is away, and the first request after it returns succeeds. No restart, no ticket, no waiting for someone to notice.

The /health endpoint returns 200 Healthy when the database is reachable and 503 Unhealthy when it is not. Point an uptime monitor at it and you learn about an outage from a message instead of from your users.

Test it before it happens

You do not have to wait for maintenance to find out how your application behaves:

Related

• HTTP Error 500 – the application failed to start
• Application keeps restarting or loses sessions
• AppPool is not enabled on server (HTTP 503)
• Cannot connect to the database
• Hosting model: InProcess and OutOfProcess
• Keep your application always running