The ethel servers will come in two flavours. Currently only the server is available. These are:

• server - this version contains all the server components
• worker - this version does not contain the runs service or api and can be configured to run just the tasks or system services

Who's who in the ethel service zoo

The server has three main modules:

• api - handles the REST API
• core - contains the services that manage the runs and tasks
• system - contains services for managing background tasks like monitoring, notifications, etc.

The key module is the core module with the following services:

• RunnersService - used by the task runners to communicate with the system
• RunsService - controls which run is running
• TasksService - monitors each queue and makes sure the queues are filled with waiting tasks

Service communication

Most comunication in the system happens via events and message. There are three types of messages, run, queue and system messages. The events, the service that raises them and the services that listen to them are described below.

Outside the events approach the RunnerService has the queues object injected by the TasksService at initiation. The RunnerService uses this to enqueue tasks with the relevant queue.

Message/Event	Listener	Emitters
queue - drained	RunsService	The TasksService will emit a message when a queue has no more tasks to process.
queue - purge	TasksService	The SystemController will emit a message if the client sends a request. The RunsService will emit a message if a running run is suspeneded, stopped or deleted. The change in run status can happen internally or via the client
queue - reconfigured	TasksService	The SystemController will emit a message when a queue configuration has been updated. The TasksService will then reconfigure the queue adapter.
run - created	RunsService	The RunsController will emit a message when a new run is added via the API. The RunnerService will emit a message when a new run is created via batch configuration.
run - started	TasksService	The RunsService will emit a message when a run is started or restarted.
run - suspended	RunsService NotificationService	The RunsController will emit a message when a run is suspended via the API. The RunsService will emit a message when a new run has been inactive for 15 minutes. The TasksService used to emit a message when a run had too many errors. This is currently disabled, but may return.
run - restarted	RunsService	The RunsController will emit a message when a run is restarted via the API. The TasksService will emit a message when a task error retry is due, if the run was suspended while waiting for the retry.
run - stopped	RunsService	The RunsController will emit a message when a run is stopped via the API.
run - completed	NotificationService	The RunsService will emit a message when a run is complete.
run - deleted	RunsService	The RunsController will emit a message when a run is deleted via the API.
system - shutdown	SystemService	The SystemController will emit a message when the client sends a request.
system - updateRunData	RunsService	The SystemService schedules a job to emit this message very 10 minutes so that the run ETA is updated.

There are some edge cases with runs and tasks hanging, generally on restart. Haven't worked out yet how to reproduce these problems.

In most cases it is because either the queue drained or run started message isn't being raised. It could either be because the RunsService doesn't know to check whether a run has been completed, or the TasksService doesn't know a queue is empty.
There's a bit of a chicken and egg problem when looking at these problems. The RunsService relies on receiving a queue drained message so that it can check if any running run is completed. This then allows it to emit the run started message that will prompt the TasksService to enqueue tasks with the appropriate queue. So look at whether these two events are being raised at the right time.