{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibqjw3fey7bf3ekcy4lovo2fpxj6jborwtm5xzezez4jr7cor7nsq",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpkg5u2viai2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreiatrd6go55ixdscqxyuuelkk64y7c7oj7flqbr5y6i3wwzi4feqsa"
    },
    "mimeType": "image/webp",
    "size": 254144
  },
  "path": "/samson_tanimawo/log-management-at-scale-how-we-cut-costs-70-without-losing-signal-1k1j",
  "publishedAt": "2026-07-01T00:55:38.000Z",
  "site": "https://dev.to",
  "tags": [
    "logging",
    "observability",
    "devops",
    "sre",
    "Nova AI Ops",
    "https://novaaiops.com",
    "@app.post"
  ],
  "textContent": "##  $12,000/Month for Logs Nobody Reads\n\nOur logging bill was $12,000/month. We were ingesting 2TB/day. When I asked the team what percentage of logs they actually looked at during incidents, the answer was embarrassing: about 5%.\n\nWe were paying to store 95% noise.\n\n##  The Log Audit\n\nFirst, I categorized all log sources by value:\n\n\n\n    High value (always need during incidents):\n      Application errors (stack traces)\n      Authentication events\n      Business transactions\n      External API calls with responses\n      Health check failures\n\n    Medium value (sometimes useful):\n      Request/response logs (sampled)\n      Performance metrics in logs\n      Deployment events\n      Configuration changes\n\n    Low value (almost never needed):\n      Debug/trace level logs\n      Health check successes\n      Static asset requests\n      Heartbeat messages\n      Verbose framework logs\n\n\n##  Strategy 1: Log Levels as a Service\n\nWe made log levels dynamic. In production, default is WARN. During incidents, flip to DEBUG for the affected service:\n\n\n\n    import os\n    import logging\n\n    # Log level from environment variable, changeable at runtime\n    LOG_LEVEL = os.environ.get('LOG_LEVEL', 'WARNING')\n    logging.basicConfig(level=getattr(logging, LOG_LEVEL))\n\n    # Endpoint to change log level without restart\n    @app.post('/admin/log-level')\n    async def set_log_level(level: str):\n        logging.getLogger().setLevel(getattr(logging, level.upper()))\n        return {'status': 'ok', 'level': level}\n\n\nIn Kubernetes:\n\n\n\n    # Normal operation\n    kubectl set env deployment/api LOG_LEVEL=WARNING\n\n    # During incident\n    kubectl set env deployment/api LOG_LEVEL=DEBUG\n\n    # After incident\n    kubectl set env deployment/api LOG_LEVEL=WARNING\n\n\n##  Strategy 2: Tiered Retention\n\n\n    retention_policy:\n      hot_storage:  # Fast search, expensive\n        duration: 7 days\n        filter: \"level >= WARN OR tag:business_event\"\n\n      warm_storage:  # Slower search, cheaper\n        duration: 30 days\n        filter: \"level >= INFO\"\n\n      cold_storage:  # Archive only, cheapest\n        duration: 365 days\n        filter: \"tag:audit OR tag:compliance\"\n\n      drop:  # Don't store at all\n        filter: \"level = DEBUG OR source:health_check\"\n\n\n##  Strategy 3: Structured Logging\n\nUnstructured logs are expensive to parse. Structured logs are cheap to query:\n\n\n\n    # Bad: Unstructured\n    logger.info(f\"User {user_id} purchased {product_id} for ${amount}\")\n    # Parsing this requires regex, which costs compute\n\n    # Good: Structured\n    logger.info(\"purchase_completed\", extra={\n        'user_id': user_id,\n        'product_id': product_id,\n        'amount': amount,\n        'currency': 'USD'\n    })\n    # Output: {\"message\": \"purchase_completed\", \"user_id\": \"u123\", ...}\n    # Queryable without parsing\n\n\n##  Strategy 4: Sample Verbose Logs\n\n\n    import random\n\n    def should_log_request(request):\n        # Always log errors\n        if request.status_code >= 400:\n            return True\n        # Always log slow requests\n        if request.duration_ms > 1000:\n            return True\n        # Sample 10% of successful requests\n        return random.random() < 0.10\n\n\n##  The Results\n\n\n    Before:\n      Daily ingestion: 2 TB\n      Monthly cost: $12,000\n      Useful data: ~5%\n\n    After:\n      Daily ingestion: 400 GB\n      Monthly cost: $3,600\n      Useful data: ~70%\n\n\nWe cut costs by 70% AND improved signal quality. Searches are faster because there's less noise. Incidents resolve quicker because relevant logs surface immediately.\n\n##  The Rule\n\nBefore adding a log statement, ask: \"Will someone look at this during an incident?\" If the answer is no, it's DEBUG level at most.\n\nIf you're spending too much on logs and want smarter log management, check out what we're building at Nova AI Ops.\n\n**Written by Dr. Samson Tanimawo**\nBSc · MSc · MBA · PhD\nFounder & CEO, Nova AI Ops. https://novaaiops.com",
  "title": "Log Management at Scale: How We Cut Costs 70% Without Losing Signal"
}