Configuration Reference

Given here are all configuration settings for zappend.

Target Outline

`append_dim`

Type string. The name of the variadic append dimension. Defaults to "time".

`append_step`

If set, enforces a step size in the append dimension between two slices or just enforces a direction. Must be one of the following:

Arbitrary step size or not applicable. Its value is null.
Monotonically increasing. Its value is "+".
Monotonically decreasing. Its value is "-".
Type string. A positive or negative time delta value, such as 12h, 2D, -1D.
Type number. A positive or negative numerical delta value.

Defaults to null.

`fixed_dims`

Type object. Specifies the fixed dimensions of the target dataset. Keys are dimension names, values are dimension sizes. The object's values are of type integer.

`included_variables`

Type array. Specifies the names of variables to be included in the target dataset. Defaults to all variables found in the first contributing dataset. The items of the list are of type string.

`excluded_variables`

Type array. Specifies the names of individual variables to be excluded from all contributing datasets. The items of the list are of type string.

`variables`

Type object. Defines dimensions, encoding, and attributes for variables in the target dataset. Object property names refer to variable names. The special name * refers to all variables, which is useful for defining common values. The object's values are of type object. Variable metadata.

dims: The names of the variable's dimensions in the given order. Each dimension must exist in contributing datasets. The items of the list are of type string.
encoding: Variable Zarr storage encoding. Settings given here overwrite the encoding settings of the first contributing dataset.
- dtype: Must be one of "int8", "uint8", "int16", "uint16", "int32", "uint32", "int64", "uint64", "float32", "float64".
- chunks: Must be one of the following:
  - Type array. Chunk sizes for each dimension of the variable. The items of the list must be one of the following:
    - Type integer. Dimension is chunked using given size.
    - Disable chunking in this dimension. Its value is null.
  - Disable chunking in all dimensions. Its value is null.
- shards: Zarr v3 shard sizes for each dimension of the variable. Each shard size must be a multiple of the corresponding chunk size. Ignored with a warning for zarr_format equal to 2. The items of the list are positive integers.
- fill_value: Must be one of the following:
  - Type number. A number of type and unit of the given storage dtype.
  - Not-a-number. Can be used only if storage dtype is float32 or float64. Its value is "NaN".
  - No fill value. Its value is null.
- scale_factor: Scale factor for computing the in-memory value: memory_value = scale_factor * storage_value + add_offset.
- add_offset: Add offset for computing the in-memory value: memory_value = scale_factor * storage_value + add_offset.
- units: Units of the storage data type if memory data type is date/time.
- calendar: The calendar to be used if memory data type is date/time.
- compressor: Compressor definition. Set to null to disable data compression. Allowed parameters depend on the value of id. The key id is required.
  - id: Compressor identifier
- filters: List of filters. Set to null to not use filters. The items of the list are of type object. Filter definition. Allowed parameters depend on the value of id. The key id is required.
  - id: Compressor identifier
attrs: Arbitrary variable metadata attributes.

`group_variables`

Type object. Optional variable-configuration overrides for individual xarray.DataTree groups. Keys are absolute Zarr group paths such as "/group_0" or "/group_1"; each value uses the same syntax as variables.

For a group, its configuration is merged over the global variables configuration. If a group has no entry here, normal zappend behavior applies: the global variables setting is used if present, otherwise encoding such as chunk sizes is taken from that group's first slice.

group_variables:
  "/group_0":
    var_0:
      encoding:
        chunks: [31, 450, 450]
  "/group_1":
    var_0:
      encoding:
        chunks: [31, 450, 450]

`attrs`

Type object. Arbitrary dataset attributes. If permit_eval is set to true, string values may include Python expressions enclosed in {{ and }} to dynamically compute attribute values; in the expression, the current dataset is named ds. Refer to the user guide for more information.

`attrs_update_mode`

The mode used update target attributes from slice attributes. Independently of this setting, extra attributes configured by the attrs setting will finally be used to update the resulting target attributes. Must be one of the following:

Use attributes from first slice dataset and keep them. Its value is "keep".
Replace existing attributes by attributes of last slice dataset. Its value is "replace".
Update existing attributes by attributes of last slice dataset. Its value is "update".
Ignore attributes from slice datasets. Its value is "ignore".

Defaults to "keep".

`zarr_format`

The Zarr format to be used. Required unless the deprecated zarr_version: 2 setting is supplied. Allowed values are 2 and 3.

`zarr_version`

Deprecated legacy setting for Zarr v2 only. Use zarr_format in new configurations. Its only allowed value is 2.

Data I/O - Target

`target_dir`

Type string. The URI or local path of the target Zarr dataset. Must specify a directory whose parent directory must exist.

`target_storage_options`

Type object. Options for the filesystem given by the URI of target_dir.

`force_new`

Type boolean. Force creation of a new target dataset. An existing target dataset (and its lock) will be permanently deleted before appending of slice datasets begins. WARNING: the deletion cannot be rolled back. Defaults to false.

Data I/O - Slices

`slice_storage_options`

Type object. Options for the filesystem given by the protocol of the URIs of contributing datasets.

`slice_engine`

Type string. The name of the engine to be used for opening contributing datasets. Refer to the engine argument of the function xarray.open_dataset().

`slice_polling`

Defines how to poll for contributing datasets. Must be one of the following:

No polling, fail immediately if dataset is not available. Its value is false.
Poll using default values. Its value is true.
Type object. Polling parameters. The keys interval, timeout are required.
- interval: Polling interval in seconds. Defaults to 2.
- timeout: Polling timeout in seconds. Defaults to 60.

`slice_source`

Type string. The fully qualified name of a class or function that receives a slice item as argument(s) and provides a slice object. If a class is given, it must be derived from zappend.api.SliceSource. If the function is a context manager, it must yield an xarray.Dataset or xarray.DataTree. If a plain function is given, it must return any valid slice item type. Refer to the user guide for more information.

`slice_source_kwargs`

Type object. Extra keyword-arguments passed to a configured slice_source together with each slice item.

`persist_mem_slices`

Type boolean. Persist in-memory slices and reopen from a temporary Zarr before appending them to the target dataset. This can prevent expensive re-computation of dask chunks at the cost of additional i/o. Defaults to false.

Data I/O - Transactions

`temp_dir`

Type string. The URI or local path of the directory that will be used to temporarily store rollback information.

`temp_storage_options`

Type object. Options for the filesystem given by the protocol of temp_dir.

`disable_rollback`

Type boolean. Disable rolling back dataset changes on failure. Effectively disables transactional dataset modifications, so use this setting with care. Defaults to false.

Miscellaneous

`version`

Configuration schema version. Allows the schema to evolve while still preserving backwards compatibility. Its value is 1. Defaults to 1.

`dry_run`

Type boolean. If true, log only what would have been done, but don't apply any changes. Defaults to false.

`permit_eval`

Type boolean. Allow for dynamically computed values in dataset attributes attrs using the syntax {{ expression }}. Executing arbitrary Python expressions is a security risk, therefore this must be explicitly enabled. Refer to the user guide for more information. Defaults to false.

`extra`

Type object. Extra settings. Intended use is by a slice_source that expects an argument named ctx to access the extra settings and other configuration.

`profiling`

Profiling configuration. Allows for runtime profiling of the processing. Must be one of the following:

Type boolean. If set, profiling is enabled and output is logged using level "INFO". Otherwise, profiling is disabled.
Type string. Profile path. Enables profiling and writes a profiling report to given path.
Type object. Detailed profiling configuration.
- enabled: Enable or disable profiling.
- path: Local file path for profiling output.
- log_level: Defaults to "INFO". Must be one of "CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET".
- keys: Sort output according to the supplied column names. Refer to Stats.sort_stats(*keys). Defaults to ["tottime"]. Must be one of "calls", "cumulative", "cumtime", "file", "filename", "module", "ncalls", "pcalls", "line", "name", "nfl", "stdname", "time", "tottime".
- restrictions: Used to limit the list down to the significant entries in the profiling report. Refer to Stats.print_stats(*restrictions). The items of the list must be one of the following:
  - Type integer. Select a count of lines.
  - Type number. Select a percentage of lines.
  - Type string. Pattern-match the standard name that is printed.

`logging`

Logging configuration. Must be one of the following:

Type boolean. Shortform that enables logging to the console using log level "INFO".
Shortform that enables logging to the console using the specified log level. Must be one of "CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET".
Type object. Detailed logging configuration. For details refer to the dictionary schema of the Python module logging.config. The key version is required.
- version: Its value is 1.
- formatters: Formatter definitions. Each key is a formatter id and each value is an object describing how to configure the corresponding formatter. The object's values are of type object. Formatter configuration.
  - format: Format string in the given style. Defaults to "%(message)s".
  - datefmt: Format string in the given style for the date/time portion. Defaults to "%Y-%m-%d %H:%M:%S,uuu".
  - style: Must be one of "%", "{", "$".
- filters: Filter definitions. Each key is a filter id and each value is a dict describing how to configure the corresponding filter. The object's values are of type object. Filter configuration.
- handlers: Handler definitions. Each key is a handler id and each value is an object describing how to configure the corresponding handler. The object's values are of type object. Handler configuration. All keys other than the following are passed through as keyword arguments to the handler's constructor. The key class is required.
  - class: The fully qualified name of the handler class. See logging handlers.
  - level: Must be one of "CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET".
  - formatter: The id of the formatter for this handler.
  - filters: A list of ids of the filters for this logger. The items of the list are of type string.
- loggers: Logger definitions. Each key is a logger name and each value is an object describing how to configure the corresponding logger. The tool's logger has the id 'zappend'. The object's values are of type object. Logger configuration.
  - level: Must be one of "CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET".
  - propagate: The propagation setting of the logger.
  - filters: A list of ids of the filters for this logger. The items of the list are of type string.
  - handlers: A list of ids of the handlers for this logger. The items of the list are of type string.