Data Retention & Cleanup Policies

To prevent database bloat and ensure GDPR/PII compliance, operational data must be purged systematically.

Retention Lifecycles

1. Temporary Upload Files

• Uploaded CSV/XLSX files reside in /tmp/tenant_id/ during chunking.
• Cleanup Strategy: Deleted immediately via fs.promises.unlink upon successful chunking.
• Failsafe: A cron job (@Cron(CronExpression.EVERY_HOUR)) clears /tmp/tenant_id/ files older than 24 hours.

2. Redis Replay Cache

• Webhook nonces (NX) and idempotency keys expire automatically via Redis TTLs.
• TTL Configuration: 48 hours for webhooks, 24 hours for API idempotency keys.

3. Contact Import Jobs & Previews

• ContactImportChunk JSON payloads (staged valid data/errors) consume significant space.
• Cleanup Strategy: If a user abandons an import in PENDING_REVIEW for > 7 days, the ContactImportJob and all related ContactImportChunk records are cascaded deleted.

4. Queue Dead Letter (DLQ)

• Jobs in BullMQ DLQ are purged after 14 days.

Lifecycle Architecture

---

5. Application Log Files

Winston log files are managed by winston-daily-rotate-file with environment-variable-driven retention:

Variable	Staging Default	Local Default	Effect
LOG_MAX_FILES	14d	3d	Delete log files older than N days
LOG_MAX_SIZE	20m	5m	Rotate when file reaches N MB

All rotated files are gzip-compressed automatically.

A host-level logrotate config provides a safety net to hard-delete any logs older than 30 days.

Log file locations on server: /srv/apps/kloyst-core/logs/

6. Loki Log Retention

The Grafana Loki instance is configured with:

• retention_period: 720h (30 days)
• Auto-compaction every 10 minutes
• Old chunks deleted 2 hours after expiry (delete delay)