2.2.1   Purpose and Structure¶

Avro alert packets are composed of opaque binary data. Interpretation of this data is done according to a schema, which defines the type and structure of each field within the packet. Strict adherence to the schema ensures that data will be correctly interpreted upon receipt.

Avro schemas can be composed of nested sub-schemas under a top level namespace. Nesting simplifies what would otherwise be monolithic schemas as new fields are added. For example, the base alert schema (lsst.alert) is of type “record” and includes previous detections of DIA sources as an array of type lsst.alert.diaSource.

The current schemas contain all fields specified by the LSST Data Products Definition Document ([5]). At this stage in construction, this schema should be regarded as exploratory and subject to rapid change; as we move closer to the operational era, a change control process will be implemented.

The current schema proposed for use with LSST alerts is stored in the lsst/alert_packet repository. This primary purpose of this package is to enable generation of alerts during Prompt Processing. Versioning of this package with the rest of the Science Pipelines software ensures reproducability. During operations, users will access the production schemas using other tools (see 2.2.2   Management and Evolution).

2.2.2   Management and Evolution¶

While a schema is required to interpret Avro’s binary data, to avoid duplication many industrial users of Kafka ship Avro payloads without including the schema. Since the schema can be lengthy, and since each alert will be packaged for shipping separately, we expect to distribute LSST alerts to community alert brokers without including the associated schema.

The recipient of each packet requires a schema to interpret the data they have been sent. In the simplest model, the producer (the LSST Alert Distribution System) and the consumer (the remote broker) simply agree on the schema prior to the start of the stream. In practice, however, this is impractical: over the course of LSST operations, we anticipate that the alert schema will evolve in response to changing technical and scientific requirements. At the same time, this change must be managed so as to cause minimum disruption to consumers.

Given the concerns above, we expect to adopt the following protocol:

• The LSST Project will make available a registry of all alert schemas ever used operationally by LSST. Schemas may be retrieved from this registry by some convenient interface given a four-byte schema ID. We plan to use the Confluent Schema Registry, although it is possible an alternative implementation will be deployed in practice.
• Alerts will be transmitted following the Confluent Wire Format. That is, the alert data encoded in Avro format will be prepended with a “magic byte” indicating the version of the wire format in use and the four-byte schema ID.
• On receipt of an alert packet, an alert broker can retrieve the appropriate schema from the registry before attempting to interpret the packet. (Consumers are expected to cache the schema, rather than requesting a fresh copy of it for every packet received!)

LSST alert schemas will follow a MAJOR.MINOR versioning scheme.

Within a given MAJOR version, schemas will follow the FORWARD_TRANSITIVE type of Confluent compatibility model. In this model, data produced by a newer schema can be interpreted by a consumer using an older schema. The producer may add fields to the schema (which will not be seen by the consumer) and may delete optional fields (in which case the consumer will see the default value). In this way, LSST may add to or augment the contents of alert packets without impacting consumers (of course, consumers who wish to take advantage of the new information available will have to upgrade their systems to match the new schema). All such additions or augmentations to the schema will result in a new MINOR version being generated.

In some cases, a break with compatibility may be required (for example, when some particular data product is rendered obsolete, deleting the corresponding field from the alert schema will break the FORWARD_TRANSITIVE compatibility guarantee). Such a break with compatibility will be signified by the release of a new MAJOR version of the schema. Issuing a new MAJOR version of the schema will require action on the part of consumers: some data on which they may be relying is no longer available. Consumers will have to update their systems to continue following the alert stream.

The Confluent Schema Registry makes it possible for schemas to evolve within a given subject name while enforcing the specified compatibility model. Thus, a given MAJOR version of the schema may be published with a particular subject name (for example, lsst-alert-N for MAJOR version N); releasing a new major version will necessitate defining a new subject.

Archival disk space is somewhat less constrained than the outbound bandwidth required for the real-time alert stream. To ensure that alerts can be read independently of the Project’s Schema Registry, all alerts that we store on disk will include the schema they were written with. For convenience and efficiency we will frequently store many alerts together in single Avro files sharing a single schema. Users can then read these files directly with existing Avro libraries.

2.2.3   Example Data¶

At present, Avro files populated with precursor data following the published schema are available at locations specified in the lsst-dm/sample_alert_info repository. Although we expect to continue to make example alert data available for the indefinite future, the contents, format, and location is subject to change with time.