Ethel Docs

Database runners

These runners read from, or write to, databases.

The runners are:

Extractor batch configuration

In some migrations a custom database may be used to control the status of the migration, rather than the ethel run and batch configuration. When this happens you may want to change two configuration options:

  • disableSkip - by default, batches will skip the number of items already processed in the run. If your process removes these processed items from the query results you will want to set disableSkip to true to stop the automatic skipping functionality of the batch configuration
  • hashName - if the status of each item is being recorded in the database there may be no need for extract hash lookups. In this case do not specify a hash name.

PostgresExtractor

Tested

This runner extracts data from a Postgres database

Configuration options
Option Data type Default Description
batchConfig number | BatchConfig, mandatory none See batches
cancelCheckNumber number 100 How often, in tasks raised, to check if the extract should be cancelled
connection LookupDetails, mandatory none The name and key to the lookup with the connection details.
disableSkip boolean false Whether to ignore the skip value from the batch config or not.
filters string | string[] None The extract items to be ignored or included. See Ignoring extract items for more information.
lock string none Query to be run before the mandatory query
hashName string none The lookup name for the extract hash
quarantinedLookupName string quarantinedPostgresExtractor Name value for the quarantine lookup
query string, mandatory none The query to be run
unlock string none Query to be run after the mandatory query

The connection value should include these values:

  • database
  • user
  • password
  • host - the server running Postgres. Defaults to localhost
  • port - the port where Postgres is listening. Defaults to 5432
Message attributes

It adds the following attributes to the message object of raised tasks:

  • extractItem the row extracted from the database

PostgresLoader

Tested

This runner loads data into a Postgres database

Configuration options
Option Data type Default Description
addHash boolean true Whether to save the extractHash lookup
connection LookupDetails, mandatory none The name and key to the lookup with the connection details.
query string, mandatory none The insert/update to be run
queryValues string queryValues The message attribute containing any values used by the query
queryValuesTemplate string none The query values used by the query using template literals. If populated queryValues is ignored.

The connection value should include these values:

  • database
  • user
  • password
  • host - the server running Postgres. Defaults to localhost
  • port - the port where Postgres is listening. Defaults to 5432
Message attributes
Option Data type Description
extractHash Lookup The extract hash object
queryValues any[] The values used in the query.

Message attribute names can be changed in the configuration options.

This runner does not update the message object.

SQLiteExtractor

Production

This runner extracts data from a SQLite database

Configuration options
Option Data type Default Description
batchConfig number | BatchConfig, mandatory none See batches
cancelCheckNumber number 100 How often, in tasks raised, to check if the extract should be cancelled
dbPath string, mandatory none The path to the database file.
disableSkip boolean false Whether to ignore the skip value from the batch config or not.
filters string | string[] None The extract items to be ignored or included. See Ignoring extract items for more information.
hashName string none The lookup name for the extract hash
lock string none Query to be run before the mandatory query
quarantinedLookupName string quarantinedSQLiteExtractor Name value for the quarantine lookup
query string, mandatory none The query to be run
unlock string none Query to be run after the mandatory query
Message attributes

It adds the following attributes to the message object of raised tasks:

  • extractItem the row extracted from the database

SQLiteLoader

Production

This runner loads data into a SQLite database

Configuration options
Option Data type Default Description
addHash boolean true Whether to save the extractHash lookup
dbPath string, mandatory none The path to the database file
query string, mandatory none The insert/update to be run.
queryValues string queryValues The message attribute containing any values used by the query
queryValuesTemplate string none The query values used by the query using template literals. If populated queryValues is ignored.
Message attributes
Option Data type Description
extractHash Lookup The extract hash object
queryValues any[] The values used in the query.

Message attribute names can be changed in the configuration options.

This runner does not update the message object.

SQLServerExtractor

Tested

This runner extracts data from a SQLServer database

Configuration options
Option Data type Default Description
batchConfig number | BatchConfig, mandatory none See batches
cancelCheckNumber number 100 How often, in tasks raised, to check if the extract should be cancelled
connection LookupDetails, mandatory none The name and key to the lookup with the connection details
disableSkip boolean false Whether to ignore the skip value from the batch config or not.
filters string | string[] None The extract items to be ignored or included. See Ignoring extract items for more information.
hashName string none The lookup name for the extract hash
lock string none Query to be run before the mandatory query
quarantinedLookupName string quarantinedSQLServerExtractor Name value for the quarantine lookup name
query string, mandatory none The query to be run
unlock string none Query to be run after the mandatory query

Information on the connection details in the mssql project

Message attributes

It adds the following attributes to the message object of raised tasks:

  • extractItem the row extracted from the database

SQLServerLoader

Tested

This runner loads data into a SQLServer database

Configuration options
Option Data type Default Description
addHash boolean true Whether to save the extractHash lookup
connection LookupDetails, mandatory none The name and key to the lookup with the connection details.
query string, mandatory none The insert/update to be run.
queryValues string queryValues The message attribute containing any values used by the query
queryValuesTemplate string none The query values used by the query using template literals. If populated queryValues is ignored.
Message attributes
Option Data type Description
extractHash Lookup The extract hash object
queryValues { name: string; type: SQLServerLoaderDataType; value: any; }[] The values used in the query.

Message attribute names can be changed in the configuration options.

This runner does not update the message object.

SQLServerLoaderDataType values:

'Bit'
'BigInt'
'Decimal'
'Float'
'Int'
'Money'
'Numeric'
'SmallInt'
'SmallMoney'
'Real'
'TinyInt'
'Char'
'NChar'
'Text'
'NText'
'VarChar'
'NVarChar'
'Xml'
'Time'
'Date'
'DateTime'
'DateTime2'
'DateTimeOffset'
'SmallDateTime'
'UniqueIdentifier'
'Variant'
'Binary'
'VarBinary'
'Image'
'UDT'
'Geography'
'Geometry'
Driver Lane © 2020-2025