[wip] Python 3 - #762
Conversation
Todo: - Tests still not ported - So far only tested MySQL collector and hostedgraphite handler
|
Btw, since this is running fine in our environment I won't be working more on this. Any additional PR's and help is greatly appreciated. |
|
Can you split out the diamond relative to full from /import statements like at the top of the handler files to a separate PR? Those changes are non-breaking and I can merge it right away. Also include the three lines changed in bin/diamond. |
|
@shortdudey123 I won't have time to do this anytime soon, sorry! Feel free to take over this branch or if any future readers see this please feel free to work off this branch 🙏 |
|
Thank you very much for the effort @erkie ! Based on your change I introduced more fixes to make diamond work in my environment: kt97679@8fee22d I would like to emphasize that I had to change config processing in the src/diamond/collector.py: kt97679@8fee22d#diff-65d20bb86255c9b2b68200bc4532b54b0c3fea580cf4ccaa0181c8259b2c2f6aL199-R208 I noticed that subsections using double square brackets are not parsed correctly so I switched from to Other changes are mostly automatic from the |
- ref: #1 - `ERROR:awesome_lint.py: Diamond: last updated -1019 days, 1:39:42.695981 ago, older than 365 days` - python-diamond/Diamond#762
- ref: #1 - `ERROR:awesome_lint.py: Diamond: last updated -1019 days, 1:39:42.695981 ago, older than 365 days` - python-diamond/Diamond#762
Ubuntu 26.04 ships Python 3.14, where the modules Diamond imported at startup no longer exist, so `diamond` died on import. - imp -> importlib.util for loading collectors from a file path. The module is registered in sys.modules before exec_module so collectors can import themselves, and dropped again if exec fails. - pkg_resources -> importlib.metadata.entry_points(group=...), which needs Python 3.10 (the oldest in the fleet). - setup.py: import setuptools unconditionally, distutils is gone in 3.12 and USE_SETUPTOOLS only ever selected between the two. - pyproject.toml: declare distro as a build requirement, setup.py imports it at build time and pip's isolated build env would not have it. Verified on Python 3.14 / Ubuntu 26.04: builds, `diamond --help` runs and all 145 collectors load, MySQLCollector included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver. Diamond passes live collector and handler objects to its children, and those hold thread locks and open sockets, so forkserver's pickling of the arguments blew up at startup with "TypeError: cannot pickle '_thread.lock' object". Pin an explicit fork context for the Manager and both Process call sites instead of setting the global start method, so embedding diamond in another program does not change that program's behaviour. Falls back to the platform default where fork is unavailable (Windows). Verified on Python 3.14 / Ubuntu 26.04: the service stays up with its main, SyncManager and Handlers processes, and collectors report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On MySQL 8.4 both replication status queries error, get_db_stats swallows the error into an empty tuple, and every slave/master metric silently stops - which is how a Grafana no-data alert became the first sign that gra-db1-new had upgraded. Query the 8.0.22+ forms first (quietly, since they are expected to fail on ancient servers), fall back to the legacy forms on empty, and translate the renamed Source/Replica columns back to their Master/Slave names so Seconds_Behind_Master and friends keep their identities in Graphite: every dashboard and alert predates the rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 participants
It felt kinda weird that there was absolutely nothing in this repo about how to run Diamond using Python 3. Like many others I recently upgraded to Ubuntu 20.04 which does not ship with a fully formed python 2.7 environment so I gave a stab at making it work with Python 3.
This is very much a WIP but I got it running fully for our setup.
How to install for you:
Since
platform.distro()was deprecated in 3.8 whichsetup.pyrelied on it now uses distro, which is a pip module that needs to be installed before running pip install diamond.Todo:
References #396