Alerts

Alerts surface operational conditions in Tagnet that need attention: an integration endpoint that has stopped responding, a delivery queue backing up, deliveries piling up as dead letters. An alert is not a log line for something that already happened. It is the current status of a problem, and it stays open for as long as the problem does.

An alert is a condition, not an event

The distinction matters because of how Tagnet handles repeats. If a connection fails once, that is one alert. If it fails a thousand times in a row before anyone notices, that is still one alert, with an occurrence count of a thousand, not a thousand separate rows. Tagnet keeps exactly one open alert per distinct condition (its alert type together with the specific thing it is about, such as one connection profile) and folds every new occurrence into it: the occurrence count goes up, and the "last occurred" timestamp moves forward.

Some alert types are raised by a periodic check rather than by a failure: Queue backing up, Consumer stalled, Dead letters aging, the System Health types other than Audit write failed and Database was unreachable, and the Device types. Their check re-raises the alert on every pass while the condition holds, so a count would only say how many times it was checked, not how many things are affected. For these, notifications read "Ongoing since" with the time Tagnet first detected the condition, the alert's detail page leaves out the occurrence count, and the Alerts list shows a dash in its place. How many things are affected, when it matters, is in the alert's own detail text.

This also means an open alert's details reflect the condition as it stands right now, not as it stood when the alert first opened. If a condition escalates while its alert is open (occasional failures turning into constant ones, say), the title and detail text update to the latest occurrence's wording, and severity moves in one direction: it rises immediately when the condition worsens, and an escalation past what you were already notified about re-notifies the routed audience right away, even on an acknowledged alert. Severity never moves back down while the alert stays open, so the audience a Critical escalation brought in keeps hearing about it until it actually resolves.

Once an alert resolves, the slate is clean: if the same condition happens again later, it opens as a fresh alert with its own occurrence count starting at 1, not a reopening of the old one.

Where alerts come from today

Three areas of Tagnet raise alerts in this release: the Integration Outbox, the system health monitor, and the device monitor that watches your readers.

Integration Outbox

The Integration Outbox is Tagnet's durable delivery system for outbound HTTP calls to your ERP, WMS, or other external systems. Six conditions can raise an alert there:

Alert type	Severity	What it means
Endpoint unavailable	Warning	Repeated delivery failures to one connection. Resolves itself on the next successful delivery.
Authentication failure	Critical	Token acquisition or authorization is failing for a connection and will not self-heal.
Deliveries dead-lettered	Warning	Deliveries exhausted their retries and need requeuing or abandoning. Resolves itself once the connection has no dead letters left.
Queue backing up	Warning	The oldest waiting delivery is older than the configured age.
Consumer stalled	Critical	Deliveries are queued but nothing is sending them.
Dead letters aging	Critical	Dead-lettered deliveries have sat unhandled for days; retention will eventually purge them. Requeue or abandon them to resolve it.

System Health

Tagnet watches the installation itself and raises these conditions:

Alert type	Severity	What it means
Storage running low	Warning, escalating to Critical	Free disk space fell below the threshold: the database volume on SQLite installs, the application server's volume on SQL Server.
Memory pressure	Warning, escalating to Critical	The process is close to the memory available to it.
CPU sustained high	Warning	The Tagnet process held the CPU near its limit across several samples.
Request processing backed up	Warning	Work is queuing inside the app faster than it can be processed.
Database near its size limit	Warning, escalating to Critical	The database is approaching its size quota; writes fail when it is reached.
Background job failed	Warning	A background job's most recent run failed; retention failing silently means unbounded growth.
Email delivery failing	Warning	Outbound emails exhausted their retries and sit dead-lettered.
Audit write failed	Warning	A change succeeded but its operational audit entry could not be saved.
Database was unreachable	Warning, or Critical from five minutes	Tagnet could not reach its database for a while and has recovered. Resolves itself once sent.

Devices

Tagnet watches every active reader and raises these conditions, so a reader that goes quiet is reported instead of discovered days later when inventory stops moving:

Alert type	Severity	What it means
Reader offline	Critical	An active reader has not checked in for longer than the offline threshold (10 minutes by default).
Antenna disconnected	Warning	An antenna port that was connected has reported disconnected for longer than the antenna threshold (5 minutes by default).

Reader offline is Critical, so the seeded catch-all rule sends it to administrators with no setup. Antenna disconnected is a Warning: it shows in the Alerts inbox but notifies nobody until a routing rule covers it. To be told about antenna problems, add the Devices category to a rule in Alert Routing, or edit the catch-all to include it.

More areas of Tagnet will raise alerts through this same system as the product grows, so what you see in the inbox and in Alert Routing's type picker will expand over time without changing how alerts, routing, or the inbox work.

How each condition is raised and cleared

The first three Integration Outbox types are per-connection: they key on a single connection profile, so two endpoints failing at the same time give you two separate alerts, each with its own occurrence count. The last three describe the outbox as a whole. The System Health types come from a periodic background sweep and describe the installation as a whole, except Background job failed, which is per-job: two failing jobs give you two alerts. The Device types come from their own sweep (every minute by default) and are per-reader, and the antenna alert per port: two readers down give you two alerts, and each alert links to its reader.

• Endpoint unavailable: raised after several delivery attempts to one connection fail in a row (5 by default). Only attempts that actually went out on the wire count, and any answered response resets the streak, so occasional failures spread across hours never accumulate into an alert. Failures caused by authentication problems do not count either; those raise the authentication alert instead. The alert reads <connection> is not accepting deliveries. Clears automatically on the next successful delivery to that connection. It reports on the endpoint, not on any delivery, so abandoning the deliveries that failed does not clear it, and the abandon confirmation says so. If the connection will never be sent to again (a retired system, say), resolve the alert by hand: it opens again only if another delivery to that connection fails.
• Authentication failure: raised on the first occurrence, with no threshold, because credential problems never fix themselves. Three shapes raise it: token acquisition failing, the connection's authentication failing to resolve at all (a deleted profile or a missing secret), and the endpoint rejecting fresh credentials with HTTP 401 or 403. Clears automatically on the next successful delivery to that connection. Like the endpoint alert, abandoning deliveries does not clear it.
• Deliveries dead-lettered: raised when a delivery exhausts its retry budget, one occurrence per dead-lettered delivery, so the occurrence count is the number of deliveries needing triage. While an endpoint-unavailable alert is open for the same connection, dead letters that outage explains fold into the outage's story instead of raising separately. Dead letters are data waiting on a decision, so the alert stays open while that connection has any. It clears automatically once none are left: requeue the ones that should still be sent and abandon the ones that must not be, and it resolves on the next recovery sweep (every 30 seconds by default). Dead letters that retention purges also count as gone; the Dead letters aging alert is the warning before that happens.
• Queue backing up: raised when the oldest delivery that has never been attempted is older than the queue age threshold (15 minutes by default). Deliveries in routine retry backoff do not count, however old, so this fires only when new work is not being picked up at all. Clears automatically once nothing waits past the threshold.
• Consumer stalled: raised when deliveries sit queued with no worker sending them, continuously, for longer than the stall threshold (5 minutes by default). Clears automatically as soon as sending resumes, without waiting for the backlog to drain.
• Dead letters aging: raised when any dead-lettered deliveries have sat unhandled longer than the aging threshold (7 days by default), counted from the most recent requeue when there was one. It exists because failed deliveries are eventually purged by retention; this is the last call to requeue or abandon them while that is still possible. Clears automatically once no dead letter is over the threshold, which includes abandoning the ones you know must not be sent.
• Storage running low: raised when free space on the monitored volume falls below the warning threshold (10 GB by default), and the same open alert escalates to Critical below the critical threshold (2 GB by default): one worsening condition, not two alerts. Clears automatically once free space climbs well clear of the warning line, so a value hovering at the threshold holds one alert instead of flapping. When the database is on SQL Server, the monitor samples the application server's own volume instead and the alert says so; SQL Server's own storage, wherever it lives, needs its own monitoring. Because this type escalates, a routing rule with a Critical minimum still matches it, but only its escalated occurrences.
• Memory pressure: raised when the Tagnet process's own memory use stays at or above the warning percentage (85% by default) of what is available to it across consecutive samples, escalating to Critical at the critical percentage (95% by default). On hosted instances "available" means the container's own memory limit, which is the number that predicts trouble; memory used by other software on the machine never trips it. Clears once usage drops well clear of the warning line; hovering just under it neither re-raises nor clears.
• CPU sustained high: raised when the Tagnet process holds the CPU at or above the threshold (90% of all cores by default) across several consecutive samples, so a busy burst never pages anyone. Clears once usage drops well clear of the line.
• Request processing backed up: raised when work queues inside the application across consecutive samples instead of being processed, the symptom behind pages and API calls stalling. Clears the moment the queue drains.
• Database near its size limit: raised at 80% of the database's size quota and escalating to Critical at 95%. The quota is the one configured on the policy, or the provider's own cap when none is configured (Azure SQL always has one); a database with no cap anywhere never raises this, since growth into the disk is the storage alert's territory. Clears when usage drops back below the thresholds.
• Background job failed: raised while a background job's most recent completed run failed, one alert per job. A run an operator cancelled neither raises nor clears it. Clears automatically when the job's next run completes.
• Email delivery failing: raised while any outbound emails sit dead-lettered after exhausting their retries. Route this one to the in-app channel: email itself is the failing path, so an email notification about it may never arrive. Clears automatically when the dead letters are requeued or purged. Resolving it by hand writes off the emails dead-lettered so far: the alert stays quiet unless another email dead-letters after that. Use it when you have fixed the cause but do not want to resend the backlog. The write-off is remembered on the resolved alert itself, so it lasts only while that resolved alert is retained (see How long alerts stick around); with default retention the written-off emails are purged long before that matters.
• Database was unreachable: raised when the database comes back after Tagnet could not reach it for a minute or more: Warning under five minutes, Critical from five. Shorter blips are only logged. It reports an incident that is already over, from when to when and the last error seen, so Tagnet resolves it itself once it has been sent (or after an hour when no rule sent it): you get one notification and it stays in the inbox's history, without repeating. A rule set to notify on resolve also sends its usual resolved notice. Each outage is its own alert. See When the database is unavailable for what happens while it lasts.
• Reader offline: raised when an active reader that has checked in before goes longer than the offline threshold without checking in. Any contact counts: heartbeats, tag reads, and, for an R700 that Tagnet manages directly, the status poll it makes every five minutes. Readers that have never checked in are not reported, so a reader you are still setting up never pages anyone, and neither is a reader in the middle of a deployment, since a deployment restarts it. An R700 whose event delivery is paused is allowed at least 15 minutes, because only that five-minute poll hears from it. After Tagnet restarts (an upgrade, say) or its database comes back from an outage, readers could not reach it in the meantime either, so each one gets a full threshold to check in again before it is reported. An offline alert that was already open stays open through that wait. Clears automatically on the reader's next check-in. To stop watching a reader you have taken out of service, deactivate it: that also resolves its open alert.
• Antenna disconnected: raised when an antenna port that reported connected in at least two heartbeats in a row has since reported disconnected, over at least two heartbeats, for longer than the antenna threshold. Requiring two connected heartbeats first is what keeps empty ports quiet: some readers report an unused port as connected for a single heartbeat every few minutes, and a blip like that never qualifies. An antenna that is plugged back in after an outage qualifies again as soon as it has been connected for two heartbeats, so pulling it again raises a fresh alert. This uses the antenna status the reader itself reports in its heartbeats, which today means Zebra FX readers. An R700 does not report antenna connections, so it never raises this alert. Clears automatically when the port reports connected again, or when the reader stops listing the port in its heartbeats at all. While a reader is offline, or has stopped sending heartbeats even though its reads still arrive, its antenna alerts neither raise nor clear, since its antenna state is unknown. If you removed an antenna on purpose, resolve the alert by hand on the Alerts page: it stays resolved for as long as the port stays disconnected, and only a fresh drop after the port has reconnected raises it again. Turning the port off in the reader's configuration does not clear the alert, because the reader keeps reporting whether each port has an antenna attached either way.
• Audit write failed: raised when a configuration change succeeds but its entry in the operational audit log could not be saved, leaving a gap in the change history. The change itself is not affected. The usual cause is the database being briefly unavailable; the application log names each change that was missed. Repeated failures add to one open alert rather than raising a new one each time. It does not clear on its own: resolve it once you have reviewed the log.

When the database is unavailable

Alerts, routing rules and their recipients are all kept in the database, so while Tagnet cannot reach its database no alert can be raised or sent the usual way. Tagnet covers that gap on its own:

• It keeps checking. Every 30 seconds Tagnet tries its SQL Server database, including while it is still starting up, and the service log records when an outage began (after a minute) and when it ended. While it lasts, the errors that only repeat that the database is unavailable are left out of the log, so a long outage does not fill it; the line recording the end says how many were left out.
• It emails directly once an outage passes five minutes. One message, [Critical] Tagnet on <server> cannot reach its database, goes to everyone a Critical alert reaches by email, sent straight through your mail settings rather than the alert system. With the default routing that is every active administrator. If mail is not configured, no email is sent and the service log says so.
• It raises the alert when the database is back: Database was unreachable, above, through your normal routing.

To know who to email while it cannot read the routing, Tagnet saves the email addresses a Critical alert reaches to .alert-recipients.json in the data subfolder of its data folder (C:\ProgramData\Stratum Global\Tagnet\data), refreshed every 15 minutes while the database is available. Changes to routing, groups or users therefore reach this email within 15 minutes. The file holds email addresses only, and that subfolder is readable only by the Tagnet service account, SYSTEM and administrators.

The email cannot help when the mail server is unreachable too, or when the Tagnet server itself is down; watching the service from outside Tagnet covers those. This applies to SQL Server installations: on SQLite the database is a file on the Tagnet server, and a full disk raises Storage running low instead.

For the whole picture (what users and readers see, how long an outage your readers can absorb, and the health endpoints for external monitoring), see Database Outages.

The Integration Outbox thresholds (consecutive failures, queue age, stall, aging) are adjustable on the Integration / Outbox policy, the storage, memory, CPU, and database-quota thresholds on the Alerting / SystemHealth policy, and the reader offline and antenna thresholds on the Alerting / Devices policy, all on the Policies page.

Severity

Every alert carries one of three severities, set by the condition that raised it:

Severity	Meaning
Info	Worth knowing, no action required.
Warning	Degraded or heading toward failure; act soon.
Critical	Broken and will not self-heal; act now.

The alert lifecycle

An alert moves through up to three states: Active, Acknowledged, and Resolved.

• Active means the condition is firing and nobody has claimed it. If it matches a routing rule, notifications keep going out on that rule's repeat interval (see Noise controls) for as long as it stays Active.
• Acknowledged means a person clicked Acknowledge, claiming ownership of the problem. Acknowledging stops repeat notifications: Tagnet assumes the acknowledging person is on it, so the rest of the routed audience is not paged again for the same alert, even if the condition keeps firing in the background (occurrences still count up, they just stop generating notifications).
• Resolved means the alert is closed. That happens two ways:
  • Automatically, when the code that raised the alert reports the condition cleared on its own, such as a delivery to a formerly unavailable endpoint succeeding again. No user is attached to this kind of resolution.
  • Manually, when someone with the Manage Alerts permission clicks Resolve, closing the alert as handled regardless of whether Tagnet has independently confirmed the condition cleared. If the condition is detected again later, a fresh alert opens. To quiet notifications while a condition is still present, acknowledge instead of resolving.

A resolved alert's detail page always shows which of the two happened: either who resolved it, or "automatically (condition cleared)."

If a routing rule matching the alert opted into recovery notices, resolving it (either way) can also trigger a "Resolved:" notification to that rule's audience. See Sending notifications for how that opt-in works.

How long alerts stick around

Active and Acknowledged alerts are never automatically removed, no matter how old they are: an alert nobody has dealt with is exactly the row that must not quietly disappear. Only Resolved alerts are cleaned up, and only after a configurable retention window (90 days by default) measured from when they resolved, not when they were first raised. This is what keeps a flapping condition, one that fails and recovers every few minutes, from growing the alerts table forever with old, closed rows. An administrator can adjust the retention window on the Policies page.

The Alerts inbox

The Alerts page has three tabs: Alerts (the inbox described here), Alert Routing, and Notification Groups. The last two are for users with Manage Alerts. For users with View Alerts, the Alerts entry in the navigation menu carries a red dot while any alert is active, so a problem is visible without opening the page; hover it to see how many. The dot reloads each time you navigate or change the scope filter rather than on a timer.

The Alerts tab lists every alert, most recent activity first. Across the top, a state summary strip shows a count for each state; clicking a chip filters the list to it, click again to clear. The filter panel adds a minimum severity filter (matches that severity and above) alongside free-text search across the alert type, title, and condition key.

Each row shows the state and severity as colored chips, the alert's title and its underlying alert type, the occurrence count (a dash for the periodically checked types described above), and when it first and last occurred. For users with Manage Alerts, the row's action menu offers Acknowledge / Resolve on any alert not already Resolved. It opens a chooser that explains what each option means before you commit: Acknowledge (stop repeat notifications while the problem is worked on) appears only while the alert is Active, and Resolve (close the alert; a re-detected condition opens a fresh one) on anything still open.

Clicking a row opens its detail page, with three sections:

• Alert: the detail text, state and severity, the alert type id, the condition key that ties repeat occurrences together, a link to the related view when one exists (such as the connection profile the alert is about), and any correlation id.
• Timeline: when the condition first and last occurred, the occurrence count (except for periodically checked types), and who acknowledged or resolved it and when.
• Notifications: when the routed audience was first notified, last notified, and, if applicable, when a recovery notice went out. This reads "nothing sent yet" until a routing rule actually matches the alert.

Permissions

Permission	Grants
View Alerts	View the alerts inbox and each alert's history.
Manage Alerts	Everything View Alerts grants, plus acknowledging and resolving alerts, and managing Alert Routing rules and Notification Groups.

Both permissions are granted only to the Admin role today. Alerts are a system-wide operational surface in this release, not scoped to a site or department the way inventory and locations are.

Where to go next

• Alert Routing: decide who hears about which alerts, through which channels, and how often.
• Notification Groups: build the reusable recipient lists that routing rules point at.