[2026-04-20 02:45:00] WARNING jobs_health_check: marked 1 image(s) as failed (job idle past cutoff; NATS consumer num_pending=0 num_ack_pending=0 num_redelivered=1). IDs: ['24763090'] [2026-04-20 02:45:00] INFO Cleaned up NATS resources for job 1322 [2026-04-20 02:45:00] INFO Deleted NATS stream job_1322 for job '1322' [2026-04-20 02:45:00] INFO Deleted NATS consumer job-1322-consumer for job '1322' [2026-04-20 02:45:00] INFO Finalizing NATS consumer job-1322-consumer before deletion (delivered=2 ack_floor=0 num_pending=0 num_ack_pending=0 num_redelivered=1) [2026-04-20 02:45:00] INFO Cleaned up Redis state for job 1322 [2026-04-20 02:45:00] INFO is_complete()=True after stage='results' update; firing cleanup. Stages: collect=100.0% SUCCESS, process=100.0% FAILURE, results=100.0% FAILURE [2026-04-20 02:45:00] INFO Job 1322 throughput: elapsed=12m 17s, processed=1/1, rate=0.1 imgs/min, ETA=0m 00s [2026-04-20 02:45:00] INFO Updated job 1322 progress in stage 'results' to 100.0% [2026-04-20 02:45:00] INFO Job 1322 throughput: elapsed=12m 16s, processed=1/1, rate=0.1 imgs/min, ETA=0m 00s [2026-04-20 02:45:00] INFO Updated job 1322 progress in stage 'process' to 100.0% [2026-04-20 02:33:15] INFO Tasks fetched: requested=8, delivered=1, user=antenna+processingservice-01@insectai.org, token_id=389306fc… [2026-04-20 02:32:43] INFO Tasks fetched: requested=8, delivered=1, user=antenna+processingservice-01@insectai.org, token_id=389306fc… [2026-04-20 02:32:43] INFO Waiting for workers to pick up tasks for pipeline 'quebec_vermont_moths_2023' (1/1 online recently) [2026-04-20 02:32:43] INFO Successfully queued 1/1 images to stream for job '1322' [2026-04-20 02:32:43] INFO Created NATS consumer job-1322-consumer (max_deliver=2, ack_wait=30.0s, max_ack_pending=1000, deliver_policy=all, ack_policy=explicit) [2026-04-20 02:32:43] INFO Created NATS stream job_1322 [2026-04-20 02:32:43] INFO Initialized task state tracking for 1 images [2026-04-20 02:32:43] INFO Queuing 1 images to NATS stream for job '1322' [2026-04-20 02:32:43] INFO Changing status of job 1322 from PENDING to STARTED [2026-04-20 02:32:43] INFO Running job #1322 "Capture #24763090" (PENDING)
