If you work in cloud security or vulnerability scanning, chances are high that you rely on ProjectDiscovery Nuclei —the gold standard open-source vulnerability scanner powered by YAML templates.

While Nuclei performs exceptionally well as a standalone CLI tool, embedding it as an underlying SDK engine inside long-running microservices or continuous scanning workers introduces unique architectural challenges: memory bloat and goroutine leaks over extended execution loops.

Recently, I investigated and resolved these exact engine lifecycle leaks in Nuclei Issue #7503 and submitted Pull Request #7508. Here is a breakdown of what I discovered under the hood and how I fixed it in Go.

🔍 The Problem: Unbounded State & Orphaned Goroutines

When embedding NucleiEngine into a long-running application loop (where engines are instantiated and closed dynamically per scan target), I noticed that memory consumption climbed steadily over time, and orphaned goroutines remained active long after calling engine.Close().

Upon profiling the engine lifecycle in Go, I identified three primary memory leaks:

1. Unboundedsync.Map in HTTP-to-HTTPS Port Tracker: The HTTPToHTTPSPortTracker stored host port mapping states in an unbounded sync.Map. Over thousands of target scans, this map grew infinitely without eviction.
2. Orphaned Per-Host Rate Limiter Goroutines: Global protocol state maintained per-execution rate limit pools (PerHostRateLimitPool). When an engine execution finished, worker background routines were not cleanly shut down or purged.
3. Cached Template Parsers: Compiled template ASTs (parsedTemplatesCache and compiledTemplatesCache) retained parsed representations in memory between engine instances without an explicit cache purging mechanism during engine teardown.

🛠️ The Solution: Architecture & Code Fixes

1. Bounded Expirable LRU Caching

Instead of holding unbounded host entries in a sync.Map, I replaced the storage structure with an expirable LRU (Least Recently Used) cache configured with a strict capacity bound (4,096 entries) and a 24-hour TTL:

// Replacing unbounded sync.Map with bounded expirable LRU cache
type HTTPToHTTPSPortTracker struct {
    cache *expirable.LRU[string, struct{}]
}

func NewHTTPToHTTPSPortTracker() *HTTPToHTTPSPortTracker {
    return &HTTPToHTTPSPortTracker{
        cache: expirable.NewLRU[string, struct{}](4096, nil, 24*time.Hour),
    }
}

This guarantees that host mappings automatically expire and memory remains strictly bounded regardless of how many millions of URLs are scanned.

2. Lifecycle Cleanup in protocolstate.Close()

I updated the global protocol state tear-down procedure in pkg/protocols/common/protocolstate/state.go to release rate limiter worker routines and purge trackers upon Close():

func Close(executionID string) {
    stateLock.Lock()
    defer stateLock.Unlock()

    if state, ok := globalStateMap[executionID]; ok {
        // Release per-host rate limiters and background goroutines
        if state.PerHostRateLimitPool != nil {
            state.PerHostRateLimitPool.Close()
        }
        // Purge HTTP to HTTPS tracker entries
        if state.HTTPToHTTPSPortTracker != nil {
            state.HTTPToHTTPSPortTracker.Purge()
        }
        delete(globalStateMap, executionID)
    }
}

3. Engine Cache Purging Interface

Finally, I added a thread-safe Purge() method to the template parser struct and invoked interface type assertions during NucleiEngine.Close():

// Safely purge compiled template caches on engine close
func (e *NucleiEngine) closeInternal() error {
    if e.parser != nil {
        e.parser.Purge()
    }
    if purger, ok := e.executerOpts.Parser.(interface{ Purge() }); ok {
        purger.Purge()
    }
    return nil
}

⚖️ Technical Trade-offs & Potential Criticisms

When designing solutions for large open-source codebases, evaluating architectural trade-offs is essential:

1. Fixed LRU Capacity vs. Configuration: Setting a hardcoded 4,096 capacity works as a balanced default for standard worker memory limits. However, in enterprise environments scanning millions of domains concurrently, exposing this bound as a configurable parameter (Options.HTTPToHTTPSCacheSize) would be a clean future addition.
2. Runtime Interface Assertion: Using runtime type assertions (interface{ Purge() }) keeps the codebase decoupled and preserves backward compatibility for third-party SDK consumers using custom parsers without breaking their implementations.
3. Memory Reclamation vs. Re-parsing Overhead: Purging compiled template caches on engine teardown prioritizes memory stability over template compilation caching across separate engine instances.

🧪 Results & Verification

I validated these fixes across Nuclei unit test packages (httpclientpool, protocolstate, templates, and lib), verifying 100% success with zero memory accumulation between consecutive engine shutdowns.

    fix(engine): resolve memory and goroutine leaks in embedded engine usage (#7503)
  

#7508

ThryLox posted on Jun 27, 2026

Summary

Fixes #7503 by implementing the required leak-prevention cleanup mechanisms outlined in #7502 for long-running embedded engines.

Key Changes

1. Size-Bounded HTTP-to-HTTPS Tracker: Replaced the unbounded sync.Map in HTTPToHTTPSPortTracker (pkg/protocols/http/httpclientpool/http_to_https_tracker.go) with a size-bounded expirable LRU cache (4096 entries max, 24h TTL) and added Purge().
2. Per-Host Rate Limiter Pool Cleanup: Updated protocolstate.Close() (pkg/protocols/common/protocolstate/state.go) to release per-host rate-limit pool goroutines and purge the HTTP-to-HTTPS tracker on shutdown.
3. Template Cache Purging: Updated NucleiEngine.Close() / closeInternal() (lib/sdk.go) and Parser (pkg/templates/parser.go) to purge parsed and compiled template caches on engine close.

View on GitHub

    Memory and goroutine leaks in long-running embedded engine usage
  

#7503

coderabbitai[bot] posted on Jun 24, 2026

Summary

The embedded engine can leak memory and goroutines over time during long-running usage.

Required changes

Implement the leak-prevention work described in #7502:

• bound the HTTPToHTTPS tracker with an LRU
• release the per-host rate-limit pool goroutines on close
• purge the template caches on engine close

Rationale

Without explicit cleanup and bounded caching, long-running embedders can accumulate memory usage and leave background goroutines running indefinitely.

Affected areas

• HTTPToHTTPS tracking / redirect bookkeeping
• per-host rate limit pool lifecycle and shutdown
• template cache lifecycle during engine close

Acceptance criteria

• The HTTPToHTTPS tracker is size-bounded and evicts old entries.
• Per-host rate-limit pool goroutines are released when the engine closes.
• Template caches are purged on engine close.
• Long-running embedded usage no longer shows continued growth from these resources.

Backlinks

• Pull request: https://github.com/projectdiscovery/nuclei/pull/7502
• Request comment: https://github.com/projectdiscovery/nuclei/pull/7502#issuecomment-4794470056
• Requested by: @Mzack9999

Additional context

PR title: fix leaks

View on GitHub

💡 Key Takeaways for Go Developers

1. Beware of Unboundedsync.Map in Long-Running Apps: While sync.Map is convenient, it lacks eviction policies. Use LRU caches with TTLs for dynamic lookup tables.
2. Explicit Teardown Interfaces: When building Go SDKs meant to be embedded, always provide clean Close() / Purge() methods to release background channels and goroutines.
3. Decoupled Lifecycle Hooks: Interface checks like if purger, ok := obj.(interface{ Purge() }); ok enable clean resource cleanup without introducing rigid package dependencies.

Written by @Thrylox. Connect with me on GitHub!