Skip to content

fix: defer boot_agent until after eventlet monkey_patch to prevent recursion error - #903

Open
CagriYonca wants to merge 1 commit into
mainfrom
fix-eventlet
Open

CagriYonca wants to merge 1 commit into
mainfrom
fix-eventlet

Conversation

@CagriYonca

@CagriYonca CagriYonca commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Problem & Root Cause

When running under Gunicorn with the eventlet worker class, Instana's early auto-instrumentation eagerly imports third-party libraries in the master process before eventlet.monkey_patch executes.

Modules such as urllib3 and pymongo capture module-level references to the unpatched ssl.SSLContext. When Eventlet subsequently patches the ssl module, ssl.SSLContext is replaced with its green subclass. Initializing SSL connections later causes super lookups inside property setters to resolve the global green subclass instead of the original class, resulting in an infinite recursion and a RecursionError.

Solution

This PR defers boot_agent execution when eventlet is present in the environment:

  • Wraps eventlet.monkey_patch using wrapt.
  • Defers boot_agent and all eager instrumentation imports until after eventlet.monkey_patch completes in the forked worker process.
  • Suppresses sensor boot in the Gunicorn arbiter process, ensuring instrumentation runs only in worker processes.

Changes

  • src/instana/__init__.py: Added _defer_boot_until_eventlet_patch to defer agent boot until worker post-patch.
  • tests/frameworks/test_eventlet_autotrace.py: Added unit tests covering deferred boot registration, worker execution, and arbiter suppression.

@CagriYonca
CagriYonca marked this pull request as ready for review September 16, 2026 12:51
@CagriYonca
CagriYonca requested a review from a team as a code owner September 16, 2026 12:51

@pvital pvital left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @CagriYonca.
I have a few considerations.

Comment thread src/instana/__init__.py Outdated
Comment thread src/instana/__init__.py Outdated
Comment on lines +308 to +317
# Eventlet deferred boot: if eventlet is present in the environment,
# defer boot_agent() until after eventlet.monkey_patch() has been called
# (which happens in each worker process after fork). This prevents the
# ssl.SSLContext global-rebind RecursionError seen with gunicorn eventlet workers.
if (
(is_autowrapt_instrumented() or is_webhook_instrumented())
and "INSTANA_DISABLE_AUTO_INSTR" not in os.environ
and importlib_util.find_spec("eventlet")
):
_defer_boot_until_eventlet_patch()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me understand here. If gevent and eventlet are used at the same time (I don't know if that's possible or not), first we apply the gevent monkey patch and then the eventlet monkey patch?

Anyway, I see some repetitive code here; I guess we can reduce this to something like (consider the change starting on line 299):

Suggested change
# Eventlet deferred boot: if eventlet is present in the environment,
# defer boot_agent() until after eventlet.monkey_patch() has been called
# (which happens in each worker process after fork). This prevents the
# ssl.SSLContext global-rebind RecursionError seen with gunicorn eventlet workers.
if (
(is_autowrapt_instrumented() or is_webhook_instrumented())
and "INSTANA_DISABLE_AUTO_INSTR" not in os.environ
and importlib_util.find_spec("eventlet")
):
_defer_boot_until_eventlet_patch()
if (
(is_autowrapt_instrumented() or is_webhook_instrumented())
and "INSTANA_DISABLE_AUTO_INSTR" not in os.environ
):
# Automatic gevent monkey patching
# unless auto instrumentation is off, then the customer should do manual gevent monkey patching
if importlib_util.find_spec("gevent"):
apply_gevent_monkey_patch()
# Eventlet deferred boot: if eventlet is present in the environment,
# defer boot_agent() until after eventlet.monkey_patch() has been called
# (which happens in each worker process after fork). This prevents the
# ssl.SSLContext global-rebind RecursionError seen with gunicorn eventlet workers.
if importlib_util.find_spec("eventlet"):
_defer_boot_until_eventlet_patch()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've refactored the block as you suggested, the shared condition is now extracted into a single outer block, with gevent and eventlet checks nested inside.

gevent and eventlet are not used simultaneously, these are Gunicorn's worker classes and they serve the same purpose. The two inner checks are independent branches for each case.

Comment thread src/instana/__init__.py Outdated
…cursion error

Signed-off-by: Cagri Yonca <cagri@ibm.com>
Comment thread src/instana/__init__.py
if is_truthy(os.environ.get("INSTANA_AUTOPROFILE", None)):
_start_profiler()

boot_agent()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If is_worker is True, boot_agent() is called twice

Comment thread src/instana/__init__.py
if is_worker:
if is_truthy(os.environ.get("INSTANA_AUTOPROFILE", None)):
_start_profiler()
boot_agent()

@GSVarsha GSVarsha Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as here

Comment thread src/instana/__init__.py
args: tuple[object, ...],
kwargs: dict[str, object],
) -> Optional[object]:
result = wrapped(*args, **kwargs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eventlet.monkey_patch() returns None

Suggested change
result = wrapped(*args, **kwargs)
wrapped(*args, **kwargs)

Comment thread src/instana/__init__.py
if is_truthy(os.environ.get("INSTANA_AUTOPROFILE", None)):
_start_profiler()
boot_agent()
return result

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return result

Reason: same as here

Comment thread src/instana/__init__.py
instance: object,
args: tuple[object, ...],
kwargs: dict[str, object],
) -> Optional[object]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
) -> Optional[object]:
) -> None:

Reason: same as here

Comment thread src/instana/__init__.py
import sys
from importlib import util as importlib_util
from typing import Tuple
from typing import Optional

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from typing import Optional

Reason: same as here

@GSVarsha GSVarsha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants