DEV3 Technologies home

Uptime check job with alert debouncing

A queued HTTP monitor check that only raises an alert after several consecutive failures and sends one recovery notice when the site returns.

LaravelQueue jobsmonitoringqueuehttpalerts

Code

<?php

namespace App\Jobs;

use App\Models\Monitor;
use App\Notifications\MonitorDown;
use App\Notifications\MonitorRecovered;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Support\Facades\Http;
use Throwable;

class CheckMonitor implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable;

    public int $tries = 1;

    public function __construct(public int $monitorId) {}

    /** One check per monitor at a time, even if the scheduler fires twice. */
    public function middleware(): array
    {
        return [(new WithoutOverlapping((string) $this->monitorId))->dontRelease()->expireAfter(120)];
    }

    public function handle(): void
    {
        $monitor = Monitor::findOrFail($this->monitorId);
        $started = hrtime(true);

        try {
            $res = Http::timeout(10)->withoutRedirecting()->get($monitor->url);
            $up = $res->status() < 400;
            $note = 'HTTP '.$res->status();
        } catch (Throwable $e) {
            $up = false;
            $note = class_basename($e);
        }

        $monitor->checks()->create([
            'up' => $up,
            'note' => $note,
            'ms' => (int) ((hrtime(true) - $started) / 1e6),
        ]);

        $up ? $this->onSuccess($monitor) : $this->onFailure($monitor);
    }

    private function onFailure(Monitor $m): void
    {
        $m->increment('consecutive_failures');
        $m->refresh();

        // Debounce: alert once, when the streak first reaches the threshold.
        if ($m->consecutive_failures >= $m->alert_after && ! $m->alerted_at) {
            $m->forceFill(['alerted_at' => now()])->save();
            $m->owner->notify(new MonitorDown($m));
        }
    }

    private function onSuccess(Monitor $m): void
    {
        $wasAlerted = $m->alerted_at !== null;
        $m->forceFill(['consecutive_failures' => 0, 'alerted_at' => null])->save();

        if ($wasAlerted) {
            $m->owner->notify(new MonitorRecovered($m));
        }
    }
}

How it works

What it solves

A single failed request is rarely an outage: a dropped packet, a deploy restart or a slow DNS answer will all trip a naive monitor and wake somebody for nothing. This job separates "a check failed" from "the site is down" and makes sure each incident produces exactly one alert and one recovery message.

How it works

Each run makes a 10 second request without following redirects, records a check row with status and response time, then updates a streak counter on the monitor. Transport errors and any HTTP status of 400 or above count as failures.

Alerting is a tiny state machine held in two columns. consecutive_failures grows with each failed check. When it reaches alert_after and alerted_at is still null, the job stamps alerted_at and sends the down notification. Further failures keep incrementing but cannot re-alert because the stamp is set. The first successful check resets the streak, clears the stamp and, only if a stamp existed, sends one recovery notification.

WithoutOverlapping keyed by the monitor id prevents two workers racing on the same counters.

Adapting it

  • Store alert_after per monitor (2 or 3 is typical) and add interval for the scheduler.
  • Add a rule for slow responses by treating ms above a limit as a failure.
  • Swap notify() for your own channel dispatcher.

Pitfalls

  • Count the streak across fresh checks only; do not retry inside the job or alerts arrive later than configured.
  • increment() followed by refresh() avoids reading a stale counter if two checks overlap.

How to use it

  1. Create Monitor (url, alert_after, consecutive_failures, alerted_at, owner relation) and a checks relation with up, note, ms.
  2. Create the two notifications MonitorDown and MonitorRecovered.
  3. Put the job in app/Jobs/CheckMonitor.php.
  4. Schedule it: Schedule::call(fn () => Monitor::where('active', true)->each(fn ($m) => CheckMonitor::dispatch($m->id)))->everyMinute();
  5. Run a worker with php artisan queue:work.

Licence: DEV3 Snippet Licence. Free to use, change and ship in personal and commercial projects. Attribution is appreciated, not required. Please do not redistribute snippets as a collection.

Related snippets