Skip to content

Commit c928650

Browse files
authored
Merge branch 'main' into feature/system-tests
2 parents ef312ca + 702d3fc commit c928650

File tree

19 files changed

+480
-25
lines changed

19 files changed

+480
-25
lines changed

.bumpversion.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 0.9.0
2+
current_version = 0.10.0
33
commit = True
44
tag = True
55

AUTHORS.rst

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,7 @@
22
Credits
33
=======
44

5-
Development Lead
6-
----------------
7-
5+
* Daniel Hatton
86
* Markus Gerstel
9-
10-
Contributors
11-
------------
12-
13-
None yet. Why not be the first?
7+
* Richard Gildea
8+
* Stu Fisher

HISTORY.rst

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,22 @@
22
History
33
=======
44

5+
Unreleased
6+
----------
7+
8+
0.10.0 (2021-10-04)
9+
-------------------
10+
* New ``zocalo.shutdown`` command to shutdown Zocalo services
11+
* New ``zocalo.queue_drain`` command to drain one queue into another in a controlled manner
12+
* New ``zocalo.util.rabbitmq.http_api_request()`` utility function to return a
13+
``urllib.request.Request`` object to query the RabbitMQ API using the credentials
14+
specified via ``zocalo.configuration``.
15+
* ``zocalo.wrap`` now emits tracebacks on hard crashes and ``SIGUSR2`` signals
16+
17+
0.9.1 (2021-08-18)
18+
------------------
19+
* Expand ~ in paths in configuration files
20+
521
0.9.0 (2021-08-18)
622
------------------
723
* Removed --live/--test command line arguments, use -e/--environment instead

README.rst

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ Zocalo as a wider whole is made up of two repositories (plus some private intern
6161

6262
As mentioned, Zocalo is currently built on top of ActiveMQ. ActiveMQ is an apache project that provides a `message broker <https://en.wikipedia.org/wiki/Message_broker>`_ server, acting as a central dispatch that allows various services to communicate. Messages are plaintext, but from the Zocalo point of view it's passing aroung python objects (json dictionaries). Every message sent has a destination to help the message broker route. Messages may either be sent to a specific queue or broadcast to multiple queues. These queues are subscribed to by the services that run in Zocalo. In developing with Zocalo, you may have to interact with ActiveMQ or RabbitMQ, but it is unlikely that you will have to configure it.
6363

64-
Zocalo allows for the monitoring of jobs executing ``python-workflows`` services or recipe wrappers. The ``python-workflows`` package contains most of the infrastructure required for the jobs themselves and more detailed documentation of its components can be found in the ``python-workflows`` `GitHub repository <https://github.com/DiamondLightSource/python-workflows/>`_ and `the Zocalo documentation <https://zocalo.readthedocs.io>`_.
64+
Zocalo allows for the monitoring of jobs executing ``python-workflows`` services or recipe wrappers. The ``python-workflows`` package contains most of the infrastructure required for the jobs themselves and more detailed documentation of its components can be found in the ``python-workflows`` `GitHub repository <https://github.com/DiamondLightSource/python-workflows/>`_ and `the Zocalo documentation <https://zocalo.readthedocs.io>`_.
6565

6666
.. _ActiveMQ: http://activemq.apache.org/
6767
.. _STOMP: https://stomp.github.io/
@@ -94,29 +94,36 @@ The only public Zocalo service at present is ``Schlockmeister``, a garbage colle
9494
Working with Zocalo
9595
-------------------
9696

97-
`Graylog <https://www.graylog.org/>`_ is used to manage the logs produced by Zocalo. Once Graylog and the message broker server are running then services and wrappers can be launched with Zocalo.
97+
`Graylog <https://www.graylog.org/>`_ is used to manage the logs produced by Zocalo. Once Graylog and the message broker server are running then services and wrappers can be launched with Zocalo.
9898

99-
Zocalo provides some command line tools. These tools are ``zocalo.go``, ``zocalo.wrap`` and ``zocalo.service``: the first triggers the processing of a recipe and the second runs a command while exposing its status to Zocalo so that it can be tracked. Services are available through ``zocalo.service`` if they are linked through the ``workflows.services`` entry point in ``setup.py``. For example, to start a Schlockmeister service:
99+
Zocalo provides the following command line tools::
100+
* ``zocalo.go``: trigger the processing of a recipe
101+
* ``zocalo.wrap``: run a command while exposing its status to Zocalo so that it can be tracked
102+
* ``zocalo.service``: start a new instance of a service
103+
* ``zocalo.shutdown``: shutdown either specific instances of Zocalo services or all instances for a given type of service
104+
* ``zocalo.queue_drain``: drain one queue into another in a controlled manner
105+
106+
Services are available through ``zocalo.service`` if they are linked through the ``workflows.services`` entry point in ``setup.py``. For example, to start a Schlockmeister service:
100107

101108
.. code:: bash
102109
103110
$ zocalo.service -s Schlockmeister
104111
105-
.. list-table::
112+
.. list-table::
106113
:widths: 100
107114
:header-rows: 1
108115

109116
* - Q: How are services started?
110117
* - A: Zocalo itself is agnostic on this point. Some of the services are self-propagating and employ simple scaling behaviour - in particular the per-image-analysis services. The services in general all run on cluster nodes, although this means that they can not be long lived - beyond a couple of hours there is a high risk of the service cluster jobs being terminated or pre-empted. This also helps encourage programming more robust services if they could be killed.
111118

112-
.. list-table::
119+
.. list-table::
113120
:widths: 100
114121
:header-rows: 1
115122

116123
* - Q: So if a service is terminated in the middle of processing it will still get processed?
117124
* - A: Yes, messages are handled in transactions - while a service is processing a message, it's marked as "in-progress" but isn't completely dropped. If the service doesn't process the message, or it's connection to ActiveMQ gets dropped, then it get's requeued so that another instance of the service can pick it up.
118125

119-
Repeat Message Failure
126+
Repeat Message Failure
120127
----------------------
121128

122129
How are repeat errors handled? This is a problem with the system - if e.g. an image or malformed message kills a service then it will get requeued, and will eventually kill all instances of the service running (which will get re-spawned, and then die, and so forth).

docs/conf.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@
5656
# the built documents.
5757
#
5858
# The short X.Y version.
59-
version = "0.9.0"
59+
version = "0.10.0"
6060
# The full version, including alpha/beta/rc tags.
6161
release = version
6262

requirements_dev.txt

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
PyYAML==5.4.1
22
graypy==2.1.0
33
marshmallow==3.13.0
4-
pytest-cov==2.12.1
5-
pytest==6.2.4
6-
setuptools==58.0.4
7-
workflows==2.12
4+
pytest-cov==3.0.0
5+
pytest-mock
6+
pytest==6.2.5
7+
setuptools==58.2.0
8+
workflows==2.13
89
junit-xml==1.9

requirements_doc.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
Sphinx==4.1.2
1+
Sphinx==4.2.0
22
sphinx-rtd-theme==1.0.0

setup.cfg

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
22
name = zocalo
3-
version = 0.9.0
3+
version = 0.10.0
44
description = Infrastructure components for automated data processing at Diamond Light Source
55
author = Diamond Light Source - Scientific Software et al.
66
author_email = scientificsoftware@diamond.ac.uk
@@ -41,12 +41,16 @@ zip_safe = False
4141
[options.entry_points]
4242
console_scripts =
4343
zocalo.go = zocalo.cli.go:run
44+
zocalo.queue_drain = zocalo.cli.queue_drain:run
4445
zocalo.service = zocalo.service:start_service
46+
zocalo.shutdown = zocalo.cli.shutdown:run
4547
zocalo.wrap = zocalo.cli.wrap:run
4648
zocalo.run_system_tests = zocalo.cli.system_test:run
4749
libtbx.dispatcher.script =
4850
zocalo.go = zocalo.go
51+
zocalo.queue_drain = zocalo.queue_drain
4952
zocalo.service = zocalo.service
53+
zocalo.shutdown = zocalo.shutdown
5054
zocalo.wrap = zocalo.wrap
5155
libtbx.precommit =
5256
zocalo = zocalo

src/zocalo/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
__author__ = "Markus Gerstel"
1010
__email__ = "scientificsoftware@diamond.ac.uk"
11-
__version__ = "0.9.0"
11+
__version__ = "0.10.0"
1212

1313
logging.getLogger("zocalo").addHandler(logging.NullHandler())
1414

src/zocalo/cli/queue_drain.py

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
#
2+
# zocalo.queue_drain
3+
# Drain one queue into another in a controlled manner
4+
#
5+
6+
7+
import argparse
8+
import queue
9+
import sys
10+
import time
11+
from datetime import datetime
12+
13+
import workflows.recipe.wrapper
14+
import workflows.transport
15+
16+
import zocalo.configuration
17+
18+
19+
def show_cluster_info(step):
20+
try:
21+
print("Beamline " + step["parameters"]["cluster_project"].upper())
22+
except Exception:
23+
pass
24+
try:
25+
print("Working directory " + step["parameters"]["workingdir"])
26+
except Exception:
27+
pass
28+
29+
30+
show_additional_info = {"cluster.submission": show_cluster_info}
31+
32+
33+
def run(args=None):
34+
35+
# Load configuration
36+
zc = zocalo.configuration.from_file()
37+
zc.activate()
38+
39+
parser = argparse.ArgumentParser(
40+
usage="zocalo.queue_drain [options] source destination"
41+
)
42+
43+
default_transport = workflows.transport.default_transport
44+
if (
45+
zc.storage
46+
and zc.storage.get("zocalo.default_transport")
47+
in workflows.transport.get_known_transports()
48+
):
49+
default_transport = zc.storage["zocalo.default_transport"]
50+
51+
parser.add_argument("-?", action="help", help=argparse.SUPPRESS)
52+
parser.add_argument("SOURCE", type=str, help="Source queue name")
53+
parser.add_argument("DEST", type=str, help="Destination queue name")
54+
parser.add_argument(
55+
"--wait",
56+
action="store",
57+
dest="wait",
58+
type=float,
59+
default=5,
60+
help="Wait this many seconds between deliveries",
61+
)
62+
parser.add_argument(
63+
"--stop",
64+
action="store",
65+
dest="stop",
66+
type=float,
67+
default=60,
68+
help="Stop if no message seen for this many seconds (0 = forever)",
69+
)
70+
parser.add_argument(
71+
"-t",
72+
"--transport",
73+
dest="transport",
74+
metavar="TRN",
75+
default=default_transport,
76+
help="Transport mechanism. Known mechanisms: "
77+
+ ", ".join(workflows.transport.get_known_transports())
78+
+ f" (default: {default_transport})",
79+
)
80+
zc.add_command_line_options(parser)
81+
workflows.transport.add_command_line_options(parser)
82+
args = parser.parse_args(args)
83+
84+
transport = workflows.transport.lookup(args.transport)()
85+
transport.connect()
86+
87+
messages = queue.Queue()
88+
89+
def receive_message(header, message):
90+
messages.put((header, message))
91+
92+
print("Reading messages from " + args.SOURCE)
93+
transport.subscribe(args.SOURCE, receive_message, acknowledgement=True)
94+
95+
message_count = 0
96+
header_filter = frozenset(
97+
{
98+
"content-length",
99+
"destination",
100+
"expires",
101+
"message-id",
102+
"original-destination",
103+
"originalExpiration",
104+
"subscription",
105+
"timestamp",
106+
"redelivered",
107+
}
108+
)
109+
drain_start = time.time()
110+
idle_time = 0
111+
try:
112+
while True:
113+
try:
114+
header, message = messages.get(True, 0.1)
115+
except queue.Empty:
116+
idle_time = idle_time + 0.1
117+
if args.stop and idle_time > args.stop:
118+
break
119+
continue
120+
idle_time = 0
121+
print()
122+
try:
123+
print(
124+
"Message date: {:%Y-%m-%d %H:%M:%S}".format(
125+
datetime.fromtimestamp(int(header["timestamp"]) / 1000)
126+
)
127+
)
128+
except Exception:
129+
pass
130+
try:
131+
print("Recipe ID: {}".format(message["environment"]["ID"]))
132+
r = workflows.recipe.wrapper.RecipeWrapper(message=message)
133+
show_additional_info.get(
134+
args.DEST, show_additional_info.get(r.recipe_step["queue"])
135+
)(r.recipe_step)
136+
except Exception:
137+
pass
138+
139+
new_headers = {
140+
key: header[key] for key in header if key not in header_filter
141+
}
142+
txn = transport.transaction_begin()
143+
transport.send(args.DEST, message, headers=new_headers, transaction=txn)
144+
transport.ack(header, transaction=txn)
145+
transport.transaction_commit(txn)
146+
message_count = message_count + 1
147+
print(
148+
"%4d message(s) drained in %.1f seconds"
149+
% (message_count, time.time() - drain_start)
150+
)
151+
time.sleep(args.wait)
152+
except KeyboardInterrupt:
153+
sys.exit(
154+
"\nCancelling, %d message(s) drained, %d message(s) unprocessed in memory"
155+
% (message_count, messages.qsize())
156+
)
157+
print(
158+
"%d message(s) drained, no message seen for %.1f seconds"
159+
% (message_count, idle_time)
160+
)

0 commit comments

Comments
 (0)