Channel Access Protocol Specification

1. Introduction

1.1. Implementation Requirements

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", "OPTIONAL" in this document are to be interpreted as described in RFC 2119: Key words for use in RFCs to Indicate Requirement Levels[3].

An implementation is not compliant if it fails to satisfy one or more of the MUST or REQUIRED level requirements for the protocols it implements. An implementation that satisfies all the MUST or REQUIRED level and all the SHOULD level requirements for its protocols is said to be "unconditionally compliant"; one that satisfies all the MUST level requirements but not all the SHOULD level requirements for its protocols is said to be "conditionally compliant."

This document is based on the original CA protocol implementation that accompanies EPICS. Wherever the MAY, OPTIONAL or RECOMMENDED is specified, it implies that experience has shown such behaviour to improve either performance, portability or memory consumption.

A protocol implementation that is not unconditionally compliant MUST NOT be used in any production environment or used with EPICS databases controling physical devices.

1.2. Basic Concepts

1.2.1. Process Variables

A Process Variable (PV) is representation of a single value within an EPICS host.

1.2.2. Channels

A channel is created whenever a PV is connected using CA.

From implementation point of view, a channel is a connection, established over virtual circuit, between server and client through which a single PV is accessed. Both the client and the server will provide a way of uniquely identifying channels. These identifiers are explained in section Message Identifiers (3.2.).

1.2.3. Monitors

A monitor is created on a channel as a means of registering for asynchronous change notifications. CA protocol defines the subscription mechanism through which clients register for notifications. These changes will then be provided through monitor object or callback, depending on implementation and environment. Monitors may be filtered to receive only a subset of events, such as value or alarm changes. Several different monitors may be created for each channel.

1.2.4. Channel Access Client Library

A client library implements channel access protocol and exposes it through a programmer-friendly API. How the protocol and its mechanisms are implemented in the library depends on the programming language and no specific requirements are made with respect to this.

1.2.5. Repeater

Repeater is a standalone process that allows several clients on one machine to share a predefined UDP port for recieving broadcast messages. Broadcast messages are sent without prior knowledge of which port the CA clients and hosts are listening at, so predefined port numbers are used. Since most network stack implementations do not allow multiple clients to listen on the same port, repeater is installed to listen on a single port, used for broadcast notifications. Repeater will fan-out unmodified incoming datagram messages to all the clients on the same host, that have previously registered with the repeater.

Typically, a client library will attempt to register with the repeater during startup. If repeater cannot be found or is not available, the library will attempt to spawn a new repeater process. Repeater and client communicate using a minimal set of messages over UDP.

1.2.6. Server Beacons

Server beacons are simple protocol messages that allow servers to broadcast their availability. These messages are sent out periodically to announce their presence. Additionally, these beacons are used to restore virtual circuits that have lost connections.

1.2.7. Virtual Circuit

Channel Access protocol is designed to minimize resources used on both client and server. Virtual circuits minimize number of TCP connections used between clients and servers. Each client will have exactly one active and open TCP connection to an individual server, regardless of how many channels it accesses over it. This helps to ensure that servers do not get overwhelmed by too many connections.

The life-cycle of a virtual circuit is defined in section Virtual Circuit Operation (10.).

1.2.8. Message Buffering

When sending packets over network, channel access allows huge savings to be made by grouping individual messages in a single TCP packet. This is performed by maintaining per virtual circuit buffers that collect all outgoing messages. These buffers are flushed upon application's explicit request. This mechanism is independent of the TCP/IP stack, which maintains its own send queue and defines a maximum frame size. Messages sent over virtual circuit may be larger than this and will not always be aligned on frame boundary. Implementation should also be aware that messages received may not yet be available entirely or will be split over several TCP frames.

1.2.9. Version compatibility

Certain aspects of Channel Access protocol have changed between releases. In this document, Channel Access versions are identified using CA_VXYY, where X represents single-digit major version number and YY represents a single- or double-digit minor version number. Stating that a feature is available in CA_VXYY implies that any client supporting version XYY must support the feature. Implementation must be backward compatible with all versions up to and including its declared supported minor version number.

Example: CA_V43, denotes version 4.3 (major version 4, minor version 3).

Currently, all protocol definitions are assumed to have major version 4. Minor version ranges from 1 through 11.

1.2.10. Exceptions

This document uses the concept of exceptions to refer to the mechanism of reporting errors that occur during command request execution. An exception defines reporting of error condition on the server. This can occur either due to client problem or due to some unexpected condition on the server (such as running out of resources). Exceptions are reported either within the command's response or using a specialized message. Actual form and method (response or asynhronously received message) depends on circumstances where the exception occurs. Individual commands and messages determine which method is used.

1.3. Overall Operation

This section is intended as a quick overview of channel access functionality and behaviour. For more information, please refer to Channel Access Reference Manual[1].

The goal of channel access (CA) is to provide remote access to records and fields managed by IOC, including search and discovery of hosts and minimal flow contol. Protocol itself is designed to provide minimal overhead and maximize network throughput for transfer of large number of small data packets. Additionaly, implementation overhead of the protocol can be kept very small, to allow operation with limited resources.

All commands and events in CA are encapsulated in predefined messages, which can be sent in one of three forms:

• As a beacon, which requires no confirmation. Used for host discovery and keep-alive notification.
• As a request/response pair. Most commands use this method.
• As a subscriber notification, where the client registers with the host and receives updates. Event notification uses this method to report value changes.

Communication between server and client is performed by sending command messages over UDP and TCP. Client will use UDP to search for hosts and PVs, server will use them to notify its startup and shutdown. Once client requests a specific PV (by specifying its name), UDP message will be broadcast to either a subnet or a list of predefined addresses, and the server which hosts requested PV will respond.

Data exchange between client and server is performed over TCP. After locating the PV, the client will establish a TCP connection to the server. If more that one PV is found to be on the same server, client will use existing TCP connection. Reusable TCP connection between client and server is called Virtual circuit.

Once a virtual circuit is established or already available, channels can be created to PVs.

Let's assume the following setup of PVs and hosts:

• PV "A" is located on host IOC1.
• PV "B" is located on host IOC2.
• PV "C" does not exist on either host.
• PV "D" is located both hosts, IOC1 and IOC2.

A typical scenario would be similar to this:

1. User application starts and initializes channel access client library. During initialization, client library might spawn a new repeater process or register with an existing one. If successful, the library is ready to start using the channel access.
2. Application will start searching for PVs named "A", "B", "C" and "D". Client will send UDP search packets with "A", "B", "C" and "D" PV names, and wait for an application-specified amount of time. To which network hosts (broadcast or specific IP) these packets are actually sent depends on configuration of the client.
3. All hosts that receive this message will check their database and if they know about any of these PV names, they will respond using UDP search response message. Thus, IOC1 will respond with "A" and "D", whereas IOC2 will respond with "B" and "D".
4. For each search packet sent, the client will wait for a predefined ammount of time or until response is received.
5. In this case (assuming no packet loss in the network), the client will establish a virtual circuit (TCP connection) to IOC1 and IOC2. PV "C" will be assumed to be unavailable at this point and will be ignored. Since "D" is located on both hosts, client will be unable to distinguish between them. First response received will be considered to be proper answer and any additional responses will be considered erroneous and reported to user.
6. Once virtual circuits are established, client can access values of "A", "B" and "D". From now on, all messages for either of the PVs will be sent via a corresponding virtual circuit.
7. If virtual circuit looses TCP connection or the host disconnects, client will notify application appropriately. Client will also listen for server beacon messages (sent whenever IOC host starts), to be able to restore any lost connections. Beacons are also used to detect when a host stops responding.
8. Once the application is done using the channel access library, it will gracefully close any open connections.

2. Data Types

This section defines all primitive data types employed by the CA, as well as their C/C++ equivalents. These data types are referred to in the subsequent sections.

All values are transmitted over the network in big-endian (network) order. For example: UINT32 3145 (0x00000C49) would be sent over the network represented as 00 00 0C 49.

3. Messages

3.1. Message Structure

All channel access messages are composed of a header, followed by the payload.

Header is always present. Its structure is fixed, and contains predefined fields. At the very least, this will be command ID and payload size. Other header fields may carry command-specific meaning. If a field is not used within a certain message, its value must be 0 (0x00).

Payload is a sequence of bytes, which are command and version dependent. Implementation is required to provide proper payload with respect to individual command specification.

Total size of an individual message is limited. With CA versions older than CA_V49, the maximum message size is limited to 16384 (0x4000) bytes. Out of these, header has a fixed size of 16 (0x10) bytes, with the payload having a maximum size of 16368 (0x3ff0) bytes.

Versions CA_V49 and higher may use the extended message form, which allows for larger payloads. The extended message form is indicated by the header fields Payload Size and Data Count being set to 0xffff and 0, respectively. Real payload size and data count are then given as UINT32 type values immediately following the header. Maximum message size is limited by 32-bit unsigned integer representation, 4294967295 (0xffffffff). Maximum payload size is limited to 4294967255 (0xffffffe7).

Extended message form should only be used if payload size actualy exceeds the pre-CA_V49 message size limit of 16368 bytes.

3.1.1. Header

Names of header fields are based on their most common use. Certain messages will use individual fields for purposes other than those described here. These variations are documented for each message individually. All of values in header are unsigned integers.

The common header present in all messages is structured as follows:

The extended message header (CA_V49 and newer) has the following structure:

3.1.2. Payload

The structure of the payload depends on the type of the message. The size of the payload matches the Payload Size header field.

If the payload size is not a multiple of 8, zero-padding must be performed to achieve 8-byte alignment.

3.2. Message Identifiers

Some fields in messages serve as identifiers, which must be properly handled by the implementation. These fields serve as identification tokens in asynchronous communication and must be unique within the context of the client library. Recommended scheme for allocating these values is to create them sequentially starting at 0. All IDs are represented with UINT32.

Overflow must be anticipated! Although it is unlikely that an application will need more than 2³² different channel IDs, such an overflow can occur with the IOID identifier (see below).

3.2.1. CID - Client ID

CID identifies individual channel within client library context. When a message requires the CID, this value must be passed. CID for a given channel may not change within the lifetime of the channel.

Recommended way of allocating CIDs is sequential 0-based indexing. First client library channel reference that is created has value of 0, second 1, etc. Duplicate CIDs within a single active instance of client library are not allowed.

CID value is passed to the server when the channel is actually connected.

Overflow should be potentially handled, since an application with expected long lifetime and frequent channel allocation and destruction might increase the index frequently. The best way to handle it is to ignore the problem until the first overflow. When that occurs, old CIDs should be no longer relevant, and CID value can be restarted from 0, but excluding any already used values.

3.2.2. SID - Server ID

Performs the same function as CID, but is provided by the server when the channel is connected. SID of channel will never change, unless the channel represented by particular SID is removed from the server during runtime. In that case, the channel will be no longer available.

To handle overflows, the same method could be used as for the Client Channel ID.

3.2.3. Subscription ID

When a subscription is created on a channel (monitor), a unique subscription ID must be provided. Client library will use this ID to identify different subscriptions on the same channel. This value will allow the client library to dispatch monitors appropriately.

Any subscription ID is only valid during the subscription's lifespan. After that, IDs may be reused.

3.2.4. IOID

Unique ID is given to all read and write messages sent by the client library.

ID passed with the request will be returned in the matching response.

Properly handling the wrapping of identifiers is vital to IOID, since an individual ID is used each time a request is made.

4. Commands (TCP and UDP)

The following commands are sent as either UDP datagrams or TCP messages. Some of the messages are also used within the context of a Virtual Circuit.

4.0. CA_PROTO_VERSION

Command: CA_PROTO_VERSION

ID: 0 (0x00)

Description: Exchanges client and server protocol versions and desired circuit priority. This is the first message sent when a new TCP (Virtual Circuit) connection is established. Must be sent before any other exchange between client, server and repeater. The communication is not strictly request response, but will be perceived as such by the implementation. When a new TCP connection is established by the client, CA_PROTO_VERSION is sent. Likewise, the server will accept the connection and send the response form back. Sent over UDP or TCP.

4.0.1. Request

Header

Compatibility

Comments

• Priority indicates the server's dispatch scheduling priority which might be implemented by a circuit dedicated thread's scheduling priority in a preemptive scheduled OS.

4.0.2. Response

Header

Compatibility

4.6. CA_PROTO_SEARCH

Command: CA_PROTO_SEARCH

ID: 6 (0x06)

Description: Searches for a given channel name. Sent over UDP or TCP.

4.6.1. Request

Header

Payload

Comments

• Sent as a UDP datagram.
• It is illegal to specify DO_REPLY flag whenever the message is sending as UDP datagram, regardless of whether broadcast or multicast is used.
• CID will be allocated by the client before this message is sent.
• CID field value is duplicated.
• Reply flag will be generally DONT_REPLY when searching using broadcast and DO_REPLY when searching using unicast. When DO_REPLY is set, server will send a CA_PROTO_NOT_FOUND (4.14.) message indicating it does not have the requested channel.

4.6.2. Response

Header

Payload

Comments

• Received as UDP datagram.
• Client channel ID field value (CID) is copied from the request.
• Port number of the server indicates on which port the server that replied listens to. This is used in case more than one server share the same IP.
• Server-assigned channel ID will have value of 0xffffffff, since the SID is not known at this time. Real value will be returned when channel is actually created using CA_CREATE_CHAN.
• In CA_V411 the server's IP address is optionally returned in the CID field of the header allowing a CA server to act as a directory service. If not, then the CID field is set to 0xffffffff.
• The port number included in the header is the *TCP* port of the server. Two servers on the same host can share a UDP port number, but not a TCP port number. Therefore, the port the client needs to connect to in that situation may not be the same as expected if this field in the response is not used.

4.14. CA_PROTO_NOT_FOUND

Command: CA_PROTO_NOT_FOUND

ID: 14 (0x0E)

Description: Indicates that a channel with requested name does not exist. Sent in response to CA_PROTO_SEARCH (4.6.), but only when its DO_REPLY flag was set. Sent over UDP.

4.14.1. Response

Header

Comments

• Contents of the header are copied from the request.
• CID fields are diplicated.
• Original request payload is not returned with the response.

4.23. CA_PROTO_ECHO

Command: CA_PROTO_ECHO

ID: 23 (0x17)

Description: Connection verify used by CA_V43. Sent over TCP.

4.23.1. Request

Header

4.23.2. Response

Header

5. Commands (UDP)

The following commands are sent as UDP datagrams.

5.13. CA_PROTO_RSRV_IS_UP

Command: CA_PROTO_RSRV_IS_UP

ID: 13 (0x0D)

Description: Beacon sent by a server when it becomes available. Beacons are also sent out periodically to announce the server is still alive. Another function of beacons is to allow detection of changes in network topology. Sent over UDP.

5.13.1. Response

Header

Comments

• IP field may contain IP of the server. If IP is not present (field Address value is 0), then IP may be substituted by the receiver of the packet (usually repeater) if it is capable of identifying where this packet came from. Any non-zero address must be interpreted as server's IP address.
• BeaconIDs are useful in detecting network topology changes. In certain cases, same packet may be routed using two different routes, causing problems with datagrams. If multiple beacons are received from the same server with same BeaconID, multiple routes are the cause.
• If a server is restarted, it will most likely start sending BeaconID values from beggining (0). Such situation must be anticipated.

5.17. CA_REPEATER_CONFIRM

Command: CA_REPEATER_CONFIRM

ID: 17 (0x11)

Description: Confirms successful client registration with repeater. Sent over UDP.

5.17.1. Response

Header

Comments

• Since repeater can bind to different local address, its IP is reported in Repeater address.. This address will be either 0.0.0.0 or 127.0.0.1.

5.24. CA_REPEATER_REGISTER

Command: CA_REPEATER_REGISTER

ID: 24 (0x18)

Description: Requests registration with the repeater. Repeater will confirm successful registration using CA_REPEATER_CONFIRM. Sent over TCP.

5.24.1. Request

Header

6. Commands (TCP)

The following commands are used within the context of Virtual Circuit and are sent using TCP.

6.1. CA_PROTO_EVENT_ADD

Command: CA_PROTO_EVENT_ADD

ID: 1 (0x01)

Description: Creates a subscription on a channel, allowing the client to be notified of changes in value. A request will produce at least one response. Sent over TCP.

6.1.1. Request

Header

Payload

Comments

• All payload fields except Mask are initialized to 0 and are present only for backward compatibility.
• Successful subscription will result in an immediate response with the current value. Additional responses will be sent as the change occurs based on the Mask parameter.
• Mask defines a filter on which events will be sent.
• A subscription should be destroyed when no longer needed to reduce load on server. See CA_PROTO_EVENT_CANCEL (6.2.).

6.1.2. Response

Header

Payload

Comments

• Response data type and count match that of the request.
• To confirm successful subscription, first response will be sent immediately. Additional responses will be sent as the change occurs based on mask parameters.

6.2. CA_PROTO_EVENT_CANCEL

Command: CA_PROTO_EVENT_CANCEL

ID: 2 (0x02)

Description: Clears event subscription. This message will stop event updates for specified channel. Sent over TCP.

6.2.1. Request

Header

Comments

• Both SID and SubscriptionID are used to identify which subscription on which monitor to destroy.
• Actual data type and count values are not important, but should be the same as used with corresponding CA_PROTO_EVENT_ADD (6.1.).

6.2.2. Response

Header

Comments

• Notice that the response has CA_PROTO_EVENT_ADD (6.1.) command identifier!
• Regardless of data type and count, this response has no payload.

6.3. CA_PROTO_READ

Command: CA_PROTO_READ

ID: 3 (0x03)

Description:

Read value of a channel. Sent over TCP.

Deprecated since protocol version 3.13.

6.3.1. Request

Header

Comments

• Channel from which to read is identified using SID.
• Response will contain the same IOID as the request, making it possible to distinguish multiple responses.

6.3.2. Response

Header

Payload

6.4. CA_PROTO_WRITE

Command: CA_PROTO_WRITE

ID: 4 (0x04)

Description: Writes new channel value. Sent over TCP.

6.4.1. Request

Header

Payload

Comments

• There is no response to this command.

6.5. CA_PROTO_SNAPSHOT

Command: CA_PROTO_SNAPSHOT

ID: 5 (0x05)

Description: Obsolete.

6.7. CA_PROTO_BUILD

Command: CA_PROTO_BUILD

ID: 7 (0x07)

Description: Obsolete.

6.8. CA_PROTO_EVENTS_OFF

Command: CA_PROTO_EVENTS_OFF

ID: 8 (0x08)

Description: Disables a server from sending any subscription updates over this virtual circuit. Sent over TCP. This mechanism is used by clients with slow CPU to prevent congestion when they are unable to handle all updates recived. Effective automated handling of flow control is beyond the scope of this document.

6.8.1. Request

Header

Comments

• This request will disable sending of subscription updates on the server to which it is sent.
• Command applies to a single virtual circuit, so having multiple priority virtual circuit connections to the server would only affect the one on which the message is sent.
• No response will be sent for this request.

6.9. CA_PROTO_EVENTS_ON

Command: CA_PROTO_EVENTS_ON

ID: 9 (0x09)

Description: Enables the server to resume sending subscription updates for this virtual circuit. Sent over TCP. This mechanism is used by clients with slow CPU to prevent congestion when they are unable to handle all updates recived. Effective automated handling of flow control is beyond the scope of this document.

6.9.1. Request

Header

Comments

• This request will enable sending of subscription updates on the server to which it is sent.
• Command applies to a single virtual circuit, so having multiple priority virtual circuit connections to the server would only affect the one on which the message is sent.
• No response will be sent for this request.

6.10. CA_PROTO_READ_SYNC

Command: CA_PROTO_READ_SYNC

ID: 10 (0x0A)

Description: Deprecated since protocol version 3.13.

6.10.1. Request

Header

6.11. CA_PROTO_ERROR

Command: CA_PROTO_ERROR

ID: 11 (0x0B)

Description: Sends error message and code. This message is only sent from server to client in response to any request that fails and does not include error code in response. This applies to all asynchronous commands. Error message will contain a copy of original request and textual description of the error. Sent over UDP.

6.11.1. Response

Header

Payload

Comments

• Complete exception report is returned. This includes error message code, CID of channel on which the request failed, original request and string description of the message.
• CID value depends on original request and may not actually identify a channel.
• First part of payload is original request header with the same structure as sent. Any payload that was part of this request is not included. Textual error message starts immediately after the header.

6.12. CA_PROTO_CLEAR_CHANNEL

Command: CA_PROTO_CLEAR_CHANNEL

ID: 12 (0x0C)

Description: Clears a channel. This command will cause server to release the associated channel resources and no longer accept any requests for this SID/CID.

6.12.1. Request

Header

6.12.2. Response

Header

Comments

• Server responds immediately and only then releases channel resources.
• Once a channel with a given SID has been cleared, any request sent with this SID will fail.
• Sent over TCP.

6.15. CA_PROTO_READ_NOTIFY

Command: CA_PROTO_READ_NOTIFY

ID: 15 (0x0F)

Description: Read value of a channel. Sent over TCP.

6.15.1. Request

Header

Comments

• Channel from which to read is identified using SID.
• Response will contain the same IOID as the request, making it possible to distinguish multiple responses.

6.15.2. Response