Uptime check job with alert debouncing
A queued HTTP monitor check that only raises an alert after several consecutive failures and sends one recovery notice when the site returns.
LaravelQueue jobsmonitoringqueuehttpalerts
Code
<?php
namespace App\Jobs;
use App\Models\Monitor;
use App\Notifications\MonitorDown;
use App\Notifications\MonitorRecovered;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Support\Facades\Http;
use Throwable;
class CheckMonitor implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable;
public int $tries = 1;
public function __construct(public int $monitorId) {}
/** One check per monitor at a time, even if the scheduler fires twice. */
public function middleware(): array
{
return [(new WithoutOverlapping((string) $this->monitorId))->dontRelease()->expireAfter(120)];
}
public function handle(): void
{
$monitor = Monitor::findOrFail($this->monitorId);
$started = hrtime(true);
try {
$res = Http::timeout(10)->withoutRedirecting()->get($monitor->url);
$up = $res->status() < 400;
$note = 'HTTP '.$res->status();
} catch (Throwable $e) {
$up = false;
$note = class_basename($e);
}
$monitor->checks()->create([
'up' => $up,
'note' => $note,
'ms' => (int) ((hrtime(true) - $started) / 1e6),
]);
$up ? $this->onSuccess($monitor) : $this->onFailure($monitor);
}
private function onFailure(Monitor $m): void
{
$m->increment('consecutive_failures');
$m->refresh();
// Debounce: alert once, when the streak first reaches the threshold.
if ($m->consecutive_failures >= $m->alert_after && ! $m->alerted_at) {
$m->forceFill(['alerted_at' => now()])->save();
$m->owner->notify(new MonitorDown($m));
}
}
private function onSuccess(Monitor $m): void
{
$wasAlerted = $m->alerted_at !== null;
$m->forceFill(['consecutive_failures' => 0, 'alerted_at' => null])->save();
if ($wasAlerted) {
$m->owner->notify(new MonitorRecovered($m));
}
}
}
How it works
What it solves
A single failed request is rarely an outage: a dropped packet, a deploy restart or a slow DNS answer will all trip a naive monitor and wake somebody for nothing. This job separates "a check failed" from "the site is down" and makes sure each incident produces exactly one alert and one recovery message.
How it works
Each run makes a 10 second request without following redirects, records a check row with status and response time, then updates a streak counter on the monitor. Transport errors and any HTTP status of 400 or above count as failures.
Alerting is a tiny state machine held in two columns. consecutive_failures grows with each failed check. When it reaches alert_after and alerted_at is still null, the job stamps alerted_at and sends the down notification. Further failures keep incrementing but cannot re-alert because the stamp is set. The first successful check resets the streak, clears the stamp and, only if a stamp existed, sends one recovery notification.
WithoutOverlapping keyed by the monitor id prevents two workers racing on the same counters.
Adapting it
- Store
alert_afterper monitor (2 or 3 is typical) and addintervalfor the scheduler. - Add a rule for slow responses by treating
msabove a limit as a failure. - Swap
notify()for your own channel dispatcher.
Pitfalls
- Count the streak across fresh checks only; do not retry inside the job or alerts arrive later than configured.
increment()followed byrefresh()avoids reading a stale counter if two checks overlap.
How to use it
- Create
Monitor(url, alert_after, consecutive_failures, alerted_at, owner relation) and achecksrelation withup,note,ms. - Create the two notifications
MonitorDownandMonitorRecovered. - Put the job in
app/Jobs/CheckMonitor.php. - Schedule it:
Schedule::call(fn () => Monitor::where('active', true)->each(fn ($m) => CheckMonitor::dispatch($m->id)))->everyMinute(); - Run a worker with
php artisan queue:work.
Licence: DEV3 Snippet Licence. Free to use, change and ship in personal and commercial projects. Attribution is appreciated, not required. Please do not redistribute snippets as a collection.