feat(kingfisher): Add scrapyd_norestart pillar flag to deploy without restarting Scrapyd

AGENTS.md

@@ -0,0 +1 @@
+Follow the style guides in `docs/develop/style/` when editing Salt states, Pillar files, Jinja templates and configuration files.

docs/deploy/deploy.rst

@@ -29,6 +29,18 @@ The ``state.apply`` function often completes within one minute.
 
       ./run.py --state-output=changes 'mytarget-dev' state.apply pillar='{"python_apps":{"myapp":{"git":{"branch":"BRANCH_NAME"}}}}'
 
+.. _deploy-norestart:
+
+.. tip::
+
+   To not restart Scrapyd (for example, if spiders are running), run, for example:
+
+   .. code-block:: bash
+
+      ./run.py --state-output=changes 'registry' state.apply pillar='{"scrapyd_norestart":true}'
+
+   This skips all changes to Scrapyd, which the next deployment without ``scrapyd_norestart`` will apply.
+
 Deploy part of a service
 ~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/deploy/setup.rst

@@ -1,12 +1,12 @@
 Setup
 =====
 
-Before performing any deployment task, run the *Setup* tasks below. Once done, run the *Cleanup* tasks below. If you haven't already, please follow the :doc:`../develop/get_started` guide.
+Before deploying, run the tasks below. If you haven't already, follow the :doc:`../develop/get_started` guide.
 
 1. Update deploy repositories
 -----------------------------
 
-Ensure the ``deploy``, ``pillar/private`` and ``salt/private`` repositories are on the default branch and are up-to-date. You can run this convenience script to run the appropriate ``git`` commands:
+Ensure the ``deploy``, ``pillar/private`` and ``salt/private`` repositories are up-to-date and on the default branch:
 
 .. code-block:: bash
 
@@ -16,21 +16,29 @@ Check the output in case there are any issues switching to the default branch or
 
 .. _check-if-kingfisher-is-busy:
 
-2. Check if Kingfisher is busy
+2. Check if the server is busy
 ------------------------------
 
 .. note::
 
-   Skip this step unless you're working on Kingfisher.
+   Skip this step for servers other than ``kingfisher-main`` and ``registry``.
 
-#. :ref:`Access Scrapyd's web interface<access-scrapyd-web-service>`, click *Jobs* and look under *Running*. If any spiders are running, don't deploy without the consent of data support managers.
-#. :doc:`SSH<../use/ssh>` into ``kingfisher-main`` as the ``root`` user.
-#. Check if any :ref:`long-running tasks<tmux>` are running, by attaching to each session in ``tmux`` to see which commands are running. If any commands would be interrupted by the deployment, don't deploy without the consent of the data support managers, who should be identified by the session names.
+#. Check whether any spiders are running in Scrapyd, by clicking *Jobs* and looking under *Running*:
 
-   To list all sessions:
+   -  ``kingfisher-main``: :ref:`Access Scrapyd's web interface<access-scrapyd-web-service>`
+   -  ``registry``: https://collect.data.open-contracting.org/jobs
 
-   .. code-block:: bash
+   If any spiders are running, :ref:`deploy without restarting Scrapyd<deploy-norestart>`. On ``kingfisher-main``, you can alternatively get the consent of the data team.
 
-      for i in root $(ls -1 /home); do echo $i; su $i -c "tmux ls"; done
+#. On ``kingfisher-main`` only:
 
-#. If the ``postgres`` service would be restarted by the deployment (for example, due to a configuration change or a package upgrade), check if any :ref:`long-running queries<pg-stat-activity>` are running. If there are queries with a ``state`` of ``active`` and a ``time`` greater than an hour, don't deploy without the consent of the data support managers, who should be identified by the ``usename``, ``client_addr`` or comment at the start of ``query``.
+   #. :doc:`SSH<../use/ssh>` into ``kingfisher-main`` as the ``root`` user.
+   #. Check if any :ref:`long-running tasks<tmux>` are running, by attaching to each session in ``tmux`` to see which commands are running. If any commands would be interrupted by the deployment, don't deploy without the consent of the data team, who should be identified by the session names.
+
+      To list all sessions:
+
+      .. code-block:: bash
+
+         for i in root $(ls -1 /home); do echo $i; su $i -c "tmux ls"; done
+
+   #. If the ``postgres`` service would be restarted (for example, due to a configuration change or a package upgrade), check if any :ref:`long-running queries<pg-stat-activity>` are running. If there are queries with a ``state`` of ``active`` and a ``time`` greater than an hour, don't deploy without the consent of the data team, who should be identified by the ``usename``, ``client_addr`` or comment at the start of ``query``.

docs/develop/style/language.rst

@@ -48,22 +48,33 @@ To concatenate strings, use the ``~`` operator, which converts its operands to s
 
       \+ ['"]|['"] \+
 
+Optional boolean keys
+~~~~~~~~~~~~~~~~~~~~~
+
+To test whether an optional boolean is true, use the ``.get()`` method:
+
+.. code-block:: jinja
+
+   {% if pillar.parent.get('enabled') %}
+
+Add a new boolean key to the regular expression under *Optional mapping keys*.
+
 Optional mapping keys
 ~~~~~~~~~~~~~~~~~~~~~
 
-To test whether a key is present in a mapping, use the ``in`` operator:
+To test whether a non-boolean key is present in a mapping, use the ``in`` operator:
 
 .. code-block:: jinja
 
    {% if 'child' in pillar.parent %}
 
 .. note::
 
-   Maintainers can check this style rule with this regular expression:
+   Maintainers can check this style rule with this regular expression, which excludes boolean keys:
 
    .. code-block:: none
 
-      if .*get\(.(?!(?:autoremove|compilemessages|enabled|public_access|replication|smartmon|summarystats)\b)
+      if .*get\(.(?!(?:ENABLE_CHECKER|autoremove|cardinal|enabled|filter|public_access|replication|reports|scrapyd_norestart|site_logs|smartmon|summarystats|syslog_logging|write)\b)
 
 To iterate over an optional mapping:
 
@@ -100,15 +111,6 @@ Or:
 
    {% for value in salt['pillar.get']('parent:child', []) %}
 
-Optional boolean keys
-~~~~~~~~~~~~~~~~~~~~~
-
-To test whether an optional boolean is true, use the ``.get()`` method:
-
-.. code-block:: jinja
-
-   {% if pillar.parent.get('enabled') %}
-
 Optional keys
 ~~~~~~~~~~~~~
 

docs/develop/update/wordpress.rst

@@ -182,7 +182,7 @@ WordPress
         sites:
           USERNAME:
             checks:
-              enabled:
+              scheduled:
                 - integrity
                 - updates
                 - silent

pillar/coalition.sls

@@ -53,7 +53,7 @@ wordpress:
       context:
         FATHOM_ANALYTICS_ID: LNRZMMVR
       checks:
-        enabled:
+        scheduled:
           - integrity
           - updates
           - silent

pillar/corporate.sls

@@ -52,7 +52,7 @@ wordpress:
         GTM_CONTAINER_ID: GTM-WMV9C5F
         ITEMS_PER_PAGE: 10
       checks:
-        enabled:
+        scheduled:
           - integrity
         premium_plugins:
           - acfml

salt/aws/backup.sls

@@ -21,7 +21,7 @@ include:
       - file: /home/sysadmin-tools/bin/site-backup-to-s3.sh
 
 {% for directory, options in pillar.backup.directories|items %}
-{% if options and options.get('exclude') %}
+{% if options and 'exclude' in options %}
 set BACKUP_EXCLUDE setting for {{ directory }}:
   file.keyvalue:
     - name: /home/sysadmin-tools/aws-settings.local

salt/cms/init.sls

@@ -110,14 +110,14 @@ set {{ constant }} in {{ user }} wp-config.php:
 {% endfor %}
 
 {% set checks = entry.checks|default({}) %}
-{% set enabled = checks.enabled|default([]) %}
-{% set parts = ['updates', 'silent']|select('in', enabled)|list %}
+{% set scheduled = checks.scheduled|default([]) %}
+{% set parts = ['updates', 'silent']|select('in', scheduled)|list %}
 {% set wp = '/usr/local/bin/wp --no-color --path=' ~ userdir ~ '/public_html' %}
 # The must-use plugins above have no checksums at wordpress.org.
 {% set premium = checks.premium_plugins|default([]) %}
 {% set exclude = premium + mu_plugins|map('regex_replace', '^', 'opencontracting-')|list %}
 
-{% if 'integrity' in enabled %}
+{% if 'integrity' in scheduled %}
 /usr/local/lib/wp-cli/integrity-check-{{ user }}.sh:
   file.managed:
     - mode: 755
@@ -140,7 +140,7 @@ set {{ constant }} in {{ user }} wp-config.php:
 {% endif %}
 
 WordPress integrity check for {{ user }}:
-  cron.{{ 'present' if 'integrity' in enabled else 'absent' }}:
+  cron.{{ 'present' if 'integrity' in scheduled else 'absent' }}:
     - name: /usr/local/lib/wp-cli/integrity-check-{{ user }}.sh
     - identifier: WORDPRESS_INTEGRITY_CHECK
     - user: {{ user }}

salt/docker_apps/files/.env

@@ -1,5 +1,3 @@
-{%- if entry.get('env') %}
-{%- for key, value in entry.env|items %}
+{%- for key, value in entry.env|default({})|items %}
 {{ key }}='{{ value }}'
 {%- endfor %}
-{%- endif %}

salt/grafana/init.sls

@@ -36,10 +36,10 @@ grafana:
           http_addr: '127.0.0.1'
           root_url: 'https://{{ pillar.apache.sites.grafana.servername }}'
         users:
-          allow_sign_up: true
+          allow_sign_up: True
         auth.proxy:
-          enabled: true
-          auto_sign_up: false
+          enabled: True
+          auto_sign_up: False
           whitelist: 127.0.0.1
     - watch_in:
       - service: grafana

salt/kingfisher/collect/init.sls

@@ -45,6 +45,12 @@ allow {{ userdir }} access:
       - user: {{ pillar.kingfisher_collect.user }}_user_exists
       - user: {{ pillar.kingfisher_collect.group }}_user_exists
 
+# Skip the states that restart Scrapyd, so that a later deployment applies their changes and restarts Scrapyd.
+{% if pillar.get('scrapyd_norestart') %}
+scrapyd-norestart:
+  test.show_notification:
+    - text: Skipped Scrapyd states, because scrapyd_norestart is set.
+{% else %}
 {{ directory }}/requirements.txt:
   file.managed:
     - source: https://raw.githubusercontent.com/open-contracting/kingfisher-collect/{{ pillar.kingfisher_collect.ref|default('main') }}/requirements.txt
@@ -87,6 +93,7 @@ allow {{ userdir }} access:
       - service: scrapyd
 
 {{ systemd({'service': 'scrapyd', 'user': user, 'group': group, 'appdir': directory}) }}
+{% endif %}
 
 {{ set_cron_env(user, 'MAILTO', 'sysadmin@open-contracting.org', 'kingfisher.collect') }}