Queues

A queue is a durable to-do list your system writes to now and processes later. Each project can create standard SQS queues in your AWS account, namespaced as shovelbase-<ref>--<name>. Manage them in the portal's Queues page or from the CLI:

Receiving third-party webhooks

Stripe, GitHub, and Twilio all retry — and eventually give up — if your endpoint is slow. The inbound webhook URL below writes straight to SQS with no compute in the path, so you accept in milliseconds and process afterwards. Deliveries survive even a full portal outage.

Slow work triggered by a user action

Image resizing, PDF generation, sending a welcome email. Insert the row, enqueue the job, return immediately — the user isn't waiting on an email provider.

Smoothing out spikes

A launch or a cron-driven stampede produces more work per second than your downstream can take. The queue holds the backlog and the consumer drains it at a steady rate, instead of every request timing out at once.

Fanning one event out to several handlers

One “order placed” message can run a receipt email, a warehouse sync, and an analytics roll-up — a trigger on the queue can list several functions, and each gets every message.

Managing queues

shovelbase queues create jobs
shovelbase queues send jobs '{"kind": "resize", "id": 42}'   # smoke test
shovelbase queues list
# jobs    messages=1    in_flight=0    delayed=0
shovelbase queues delete jobs --yes

The portal shows the three SQS gauges per queue — messages (visible), in flight (received, not yet deleted), and delayed — plus the queue URL. Counts are SQS approximations and can lag by a few seconds.

shovelbase manages the queues; your producers and consumers talk to SQS directly with the queue URL and standard AWS SDKs/credentials (an edge function, a worker on your own infra, a Lambda). Deleting a project deletes all of its queues. Note SQS refuses to recreate a queue with the same name for 60 seconds after deletion.

Management actions (create, delete, test sends) appear under Logs → Queues; 24-hour send/receive throughput from CloudWatch is under Observability → Queues.

Triggers

A trigger is the link between a queue and the edge function(s) that run for its messages — the queue's consumer, managed for you. Create one by naming a queue and the function(s) to invoke, from the Queues → Triggers tab or the CLI:

shovelbase functions deploy handle-payment
shovelbase triggers create process-jobs --queue jobs --function handle-payment
# link several functions by repeating --function (or a comma-separated list):
shovelbase triggers create on-order --queue jobs \
  --function send-receipt --function warehouse-sync --function analytics-roll-up
shovelbase triggers list
shovelbase triggers disable process-jobs      # pause without deleting
shovelbase triggers delete process-jobs --yes

The portal polls every triggered queue about every 10 seconds and POSTs each message's raw body to every function the trigger lists, the same way a schedule invokes a function on a cron. A message is only removed from the queue once every listed function returns success — a failing function leaves the message for SQS's normal visibility-timeout retry (there's no dead-letter queue, so a message that can never succeed retries until it ages out under the queue's normal retention).

A trigger fires on any message the queue receives, no matter how it got there — a webhook, shovelbase queues send, or your own producers all trigger the same functions. The queue→function relationship is many-to-many: list several functions on one trigger to fan a message out, or create several triggers pointing at the same function to feed it from different queues. Disabling a trigger stops the polling (messages pile up on the queue until you re-enable it); deleting it leaves the queue in place but stops running functions. Triggers provision no AWS resources of their own.

Webhooks

A webhook gives a queue a public inbound URL so outside systems can put messages on it — nothing more. It runs no functions on its own; to process what it delivers, add a trigger on the same queue. Create one from the Queues → Webhooks tab or the CLI by naming an existing queue:

shovelbase webhooks create stripe-events --queue jobs
# → https://<ref>.shovelbase.com/webhooks/stripe-events/<token>
shovelbase webhooks list
shovelbase webhooks delete stripe-events --yes

The URL is on your project's own host but is served, behind the scenes, straight by AWS API Gateway with a direct SQS integration — the /webhooks/* path is routed to it at the CDN edge, so POSTing writes the raw request body straight onto the queue as a message with no shovelbase compute in the request path (it works even if the portal itself is down). Point any third-party webhook sender (Stripe, GitHub, …) at it directly; no AWS credentials or shovelbase auth needed, since the unguessable <token> in the URL is the only credential. Deleting a webhook tears down its API Gateway route, so the old URL stops accepting requests — the queue and any triggers on it are untouched.

Writing the consumer

A triggered function receives the message's raw body as its request body. The contract that matters is the response: 2xx means “done, delete it” and anything else means “retry later”. Since a retry can redeliver a message you already partly handled, make the handler idempotent — key the work on something stable from the payload so processing it twice is harmless.

const URL = Deno.env.get('SHOVELBASE_URL')!;
const SERVICE_KEY = Deno.env.get('SHOVELBASE_SERVICE_ROLE_KEY')!;   // bypasses RLS
 
Deno.serve(async (req: Request) => {
  const event = await req.json();
 
  // Idempotency: the provider's own event id is the natural key. With a
  // unique index on event_id, "merge-duplicates" turns a redelivery into a
  // no-op instead of a double charge.
  const res = await fetch(`${URL}/rest/v1/payments`, {
    method: 'POST',
    headers: {
      apikey: SERVICE_KEY,
      Authorization: `Bearer ${SERVICE_KEY}`,
      'Content-Type': 'application/json',
      Prefer: 'resolution=merge-duplicates',
    },
    body: JSON.stringify({ event_id: event.id, amount: event.amount }),
  });
 
  // Non-2xx leaves the message on the queue: SQS redelivers after the
  // visibility timeout, so a transient database blip fixes itself.
  if (!res.ok) {
    console.error('payment upsert failed', res.status, await res.text());
    return new Response('retry', { status: 500 });
  }
 
  return new Response('ok');   // 2xx → message deleted
});

Because there is no dead-letter queue, a message that can never succeed — a malformed payload, say — will retry until it ages out of the queue's retention window. If a payload is permanently bad, log it and return 2xx to drop it deliberately, rather than letting it retry for days.