HPACKGoogle, Incfenix@google.comCanon CRFherve.ruellan@crf.canon.fr
Applications
HTTPbis Working GroupHTTPHeader
This document describes HPACK, a format adapted to efficiently
represent HTTP headers in the context of HTTP/2.0.
This document describes HPACK, a format adapted to efficiently
represent HTTP headers in the context of HTTP/2.0.
In HTTP/1.X, HTTP headers, which are necessary for the
functioning of the protocol, are transmitted with no
transformations. Unfortunately, the amount of redundancy in
both the keys and the values of these headers is high, and is
the cause of increased latency on lower bandwidth links. This
indicates that an alternate more compact encoding for headers
would be beneficial to latency, and that is what is proposed
here.
As shown by SPDY, Deflate
compresses HTTP very effectively. However, the use of a
compression scheme which allows for arbitrary matches against
the previously encoded data (such as Deflate) exposes users to
security issues.
In particular, the compression of sensitive data, together
with other data controlled by an attacker, may lead to leakage
of that sensitive data, even when the resultant bytes are
transmitted over an encrypted channel.
Another consideration is that processing and memory costs of a
compressor such as Deflate may also be too high for some
classes of devices, for example when doing forward or reverse
proxying.
The HTTP header encoding described in this document
is based on a header table that map (name, value) pairs to index values.
This scheme is believed to be safe for all known attacks
against the compression context today. Header tables are
incrementally updated during the HTTP/2.0 session.
The encoder is responsible for deciding which headers to
insert as new entries in the header table. The
decoder then does exactly what the encoder prescribes,
ending in a state that exactly matches the encoder's
state. This enables decoders to remain simple and
understand a wide variety of encoders.
As two consecutive sets of headers often have headers in
common, each set of headers is coded as a difference from
the previous set of headers. The goal is to only encode
the changes (headers present in one of the set and not in
the other) between the two sets of headers.
An example illustrating the use of these different
mechanisms to represent headers is available in .
The encoding and decoding of headers relies on some
components and concepts. The set of components used form
an encoding context.
The header table (see ) is a component used
to associate headers to index values.
The reference set (see ) is a component
containing a group of headers used as a reference
for the differential encoding of a new set of
headers.
A header set (see ) is
a group of headers that are encoded jointly. A
complete set of key-value pairs as encoded in
an HTTP request or response is a header set.
A header can be represented in encoded form either
as a literal or as an index (see ). The indexed
representation is based on the header table.
When decoding a set of headers, some operations
emit a header (see ). An emitted header
is added to the set of headers. Once emitted, a
header can't be removed from the set of headers.
The set of components used to encode or decode a
header set form an encoding context: an encoding
context contains a header table and a reference set.
Using HTTP, messages are exchanged between a client
and a server in both direction. To keep the encoding
of headers in each direction independent from the
other direction, there is one encoding context for
each direction.
The headers contained in a PUSH_PROMISE frame sent by
a server to a client are encoded within the same
context as the headers contained in the HEADERS frame
corresponding to a response sent from the server to
the client.
A header table consists of an ordered list of (name,
value) pairs. The first entry of a header table is
assigned the index 0.
A header can be represented by an entry of the header
table if they match. A header and an entry match if
both their name and their value match. A header name
and an entry name match if they are equal using a
character-based, case insensitive
comparison (the case insensitive comparison is used
because HTTP header names are defined in a case
insensitive way). A header value and an entry value
match if they are equal using a character-based,
case sensitive comparison.
Generally, the header table will not contain duplicate
entries. However, implementations MUST be prepared to
accept duplicates without signalling an error.
Initially, a header table contains a list of common
headers. Two initial lists of header are provided in
. One list is for
headers transmitted from a client to a server, the
other for the reverse direction.
A header table is modified by either adding a new
entry at the end of the table, or by replacing an
existing entry.
The encoder decides how to update the header table and
as such can control how much memory is used by the
header table. To limit the memory requirements on the
decoder side, the header table size is bounded (see
the SETTINGS_MAX_BUFFER_SIZE in ).
The size of an entry is the sum of its name's length
in bytes (as defined in ), of its
value's length in bytes () and of
32 bytes. The 32 bytes are an accounting for the entry
structure overhead. For example, an entry structure
using two 64-bits pointers to reference the name and
the value and the entry, and two 64-bits integer for
counting the number of references to these name and
value would use 32 bytes.
The size of a header table is the sum of the size of
its entries.
A reference set is defined as an unordered set of
references to entries of the header table.
The initial reference set is the empty set.
The reference set is updated during the processing of
a set of headers.
Using the differential encoding, a header that is not
present in the reference set can be encoded either
with an indexed representation (if the header is
present in the header table), or with a literal
representation (if the header is not present in the
header table).
A header that is to be removed from the reference set
is encoded with an indexed representation.
A header set is a group of header fields that are
encoded as a whole. Each header field is a (name,
value) pair.
A header set is encoded using an ordered list of zero
or more header representations. All the header
representations describing a header set a grouped into
a header block.
A header can be represented either as a literal or as
an index.
A literal representation defines a new
header. The header name is represented either
literally or as a reference to an entry of the
header table. The header value is represented
literally.
Three different literal representations are provided:
A literal representation that does not
add the header to the header table
(see ).
A literal representation that
adds the header at the end of the
header table
(see ).
A literal representation that
uses the header to replace an existing
entry of the header table
(see ).
The indexed representation defines a header as
a reference in the header table (see ).
The emission of header is the process of adding a
header to the current set of headers. Once an header
is emitted, it can't be removed from the current set
of headers.
The concept of header emission allows a decoder to
know when it can pass a header safely to a higher
level on the receiver side. This allows a decoder to
be implemented in a streaming way, and as such to only
keep in memory the header table and the reference set.
With such an implementation, the amount of memory used
by the decoder is bounded, even in presence of a very
large set of headers. The management of memory for
handling very large sets of headers can therefore be
deferred to the application, which may be able to emit
the header to the wire and thus free up memory
quickly.
The processing of an encoded header set to obtain a list
of headers is defined in this section. To ensure a
correct decoding of a header set, a decoder MUST obey the
following rules.
All the header representations contained in a header
block are processed in the order in which they are
presented, as specified below.
An indexed representation corresponding
to an entry not present in the
reference set entails the following actions:
The header corresponding to the entry is
emitted.The entry is added to the reference set.
An indexed representation corresponding
to an entry present in the
reference set entails the following actions:
The entry is removed from the reference set.
A literal representation that is
not added to the header table entails
the following action:
The header is emitted.
A literal representation that is
added to the header table entails
the following actions:
The header is emitted.The header is added to the header table, at the
location defined by the representation.The new entry is added to the reference set.
Once all the representations contained in a header
block have been processed, the headers that are in
common with the previous header set are emitted,
during the reference set emission.
For the reference set emission, each header contained
in the reference set that has not been emitted during
the processing of the header block is emitted.
Once all of the header representations have been
processed, and the remaining items in the reference
set have been emitted, the header set is complete.
The header table can be modified by either adding a
new entry to it or by replacing an existing one.
Before doing such a modification, it has to be ensured
that the header table size will stay lower than or
equal to the SETTINGS_MAX_BUFFER_SIZE limit (see ). To achieve
this, repeatedly, the first entry of the header table
is removed, until enough space is available for the
modification.
A consequence of removing one or more entries at the
beginning of the header table is that the remaining
entries are renumbered. The first entry of the header
table is always associated to the index 0.
When the modification of the header table is the
replacement of an existing entry, the replaced entry
is the one indicated in the literal representation
before any entry is removed from the header table. If
the entry to be replaced is removed from the header
table when performing the size adjustment, the
replacement entry is inserted at the beginning of the
header table.
The addition of a new entry with a size greater than
the SETTINGS_MAX_BUFFER_SIZE limit causes all the
entries from the header table to be dropped and the
new entry not to be added to the header table. The
replacement of an existing entry with a new entry with
a size greater than the SETTINGS_MAX_BUFFER_SIZE has
the same consequences.
Three occurrences of the same indexed representation,
corresponding to an entry not present in the reference
set, emit the associated header twice:
The first occurrence emits the header a first
time and adds the corresponding entry to the
reference set.
The second occurrence removes the header's
entry from the reference set.
The third occurrence emits the header a second
time and adds again its entry to the reference
set.
This allows for headers sets which include duplicate
header entries to be encoded efficiently and faithfully.
The first occurrence of the indexed representation can
be replaced by a literal representation creating an
entry for the header.
Integers are used to represent name indexes, pair
indexes or string lengths. To allow for optimized
processing, an integer representation always finishes
at the end of a byte.
An integer is represented in two parts: a prefix that
fills the current byte and an optional list of bytes
that are used if the integer value does not fit in the
prefix. The number of bits of the prefix (called N)
is a parameter of the integer representation.
The N-bit prefix allows filling the current byte. If
the value is small enough (strictly less than 2^N-1),
it is encoded within the N-bit prefix. Otherwise all
the bits of the prefix are set to 1 and the value is
encoded using an
unsigned variable length integer
representation.
The algorithm to represent an integer I is as follows:
The value 10 is to be encoded with a 5-bit prefix.
10 is less than 31 (= 2^5 - 1) and is
represented using the 5-bit prefix.
The value I=1337 is to be encoded with a 5-bit
prefix.
1337 is greater than 31 (= 2^5 - 1).The 5-bit prefix is filled with its
max value (31).I = 1337 - (2^5 - 1) = 1306.I (1306) is greater than or equal
to 128, the while loop body
executes:I % 128 == 2626 + 128 == 154154 is encoded in 8 bits as:
10011010I is set to 10 (1306 / 128 ==
10)I is no longer greater than or
equal to 128, the while loop
terminates.
I, now 10, is encoded on 8 bits as: 00001010
The process ends.
Header names are sequences of ASCII characters that
MUST conform to the following header-name ABNF
construction:
They are encoded in two parts:
The length of the text, defined as the number of
octets of storage required to store the text,
represented as a
variable-length-quantity .
The specific sequence of ASCII octets
Header values are encoded as sequences of UTF-8 encoded
text. They are encoded in two parts:
The length of the text, defined as the number of
octets of storage required to store the text,
represented as a
variable-length-quantity .
The specific sequence of octets representing the
UTF-8 text.
Invalid UTF-8 octet sequences, "over-long" UTF-8
encodings, and UTF-8 octets that represent
invalid Unicode Codepoints MUST NOT be used.
This representation starts with the '1' 1-bit pattern,
followed by the index of the matching pair, represented as
an integer with a 7-bit prefix.
This representation, which does not involve updating
the header table, starts with the '011' 3-bit pattern.
If the header name matches the header name of a (name,
value) pair stored in the Header Table, the index of
the pair increased by one (index + 1) is represented
as an integer with a 5-bit prefix. Note that if the
index is strictly below 31, one byte is used.
If the header name does not match a header name entry,
the value 0 is represented on 5 bits followed by the
header name ().
Header name representation is followed by the header
value represented as a literal string as described in
.
This representation starts with the '010' 3-bit
pattern.
If the header name matches the header name of a (name,
value) pair stored in the Header Table, the index of
the pair increased by one (index + 1) is represented
as an integer with a 5-bit prefix. Note that if the
index is strictly below 31, one byte is used.
If the header name does not match a header name entry,
the value 0 is represented on 5 bits followed by the
header name ().
Header name representation is followed by the header
value represented as a literal string as described in
.
This representation starts with the '00' 2-bit
pattern.
If the header name matches the header name of a (name,
value) pair stored in the Header Table, the index of
the pair increased by one (index + 1) is represented
as an integer with a 6-bit prefix. Note that if the
index is strictly below 62, one byte is used.
If the header name does not match a header name entry,
the value 0 is represented on 6 bits followed by the
header name ().
The index of the substituted (name, value) pair is
inserted after the header name representation as a
0-bit prefix integer.
The index of the substituted pair MUST correspond to a
position in the header table containing a non-void
entry. An index for the substituted pair that
corresponds to empty position in the header table MUST
be treated as an error.
This index is followed by the header
value represented as a literal string as described in
.
A few parameters can be used to accommodate client and server
processing and memory requirements.
These settings are currently not supported as they have
not been integrated in the main specification. Therefore,
the maximum buffer size for the header table is fixed at
4096 bytes.
Allows the sender to inform the remote endpoint of the
maximum size it accepts for the header table.
The default value is 4096 bytes.
Is this default value OK? Do we need a maximum size? Do we want to allow infinite buffer?
When the remote endpoint receives a SETTINGS frame
containing a SETTINGS_MAX_BUFFER_SIZE setting with a
value smaller than the one currently in use, it MUST
send as soon as possible a HEADER frame with a stream
identifier of 0x0 containing a value smaller than or
equal to the received setting value.
This changes slightly the behaviour of the
HEADERS frame, which should be updated as follows:
A HEADER frame with a stream identifier of 0x0
indicates that the sender has reduced the maximum size
of the header table. The new maximum size of the
header table is encoded on 32-bit. The decoder MUST
reduce its own header table by dropping entries from
it until the size of the header table is lower than or
equal to the transmitted maximum size.
This compressor exists to solve security issues present in
stream compressors such as DEFLATE whereby the compression
context can be efficiently probed to reveal secrets.
A conformant implementation of this specification should be
fairly safe against that kind of attack, as the reaping of any
information from the compression context requires more work than
guessing and verifying the plaintext data directly with the
server. As with any secret, however, the longer the length
of the secret, the more difficult the secret is to guess. It
is inadvisable to have short cookies that are relied upon to
remain secret for any duration of time.
A proper security-conscious implementation will also need to
prevent timing attacks by ensuring that the amount of time it
takes to do string comparisons is always a function of the
total length of the strings, and not a function of the number
of matched characters.
Another common security problem is when the remote endpoint
successfully causes the local endpoint to exhaust its memory.
This compressor attempts to deal with the most obvious ways
that this could occur by limiting both the peak and the
steady-state amount of memory consumed in the compressor
state, by providing ways for the application to consume/flush
the emitted headers in small chunks, and by considering
overhead in the state size calculation. Implementors must
still be careful in the creation of APIs to an implementation
of this compressor by ensuring that header keys and values are
either emitted as a stream, or that the compression
implementation have a limit on the maximum size of a key or
value. Failure to implement these kinds of safeguards may
still result in a scenario where the local endpoint exhausts
its memory.
This memo includes no request to IANA.SPDY ProtocolTwistGoogle
Refactored of Header Encoding Section: split
definitions and processing rule.
Backward incompatible change: Updated
reference set management as per issue #214. This
changes how the interaction between the reference
set and eviction works. This also changes the
working of the reference set in some specific
cases.
Backward incompatible change: modified initial
header list, as per issue #188.
Added example of 32 bytes entry structure (issue
#191).
Added Header Set Completion section. Reflowed
some text. Clarified some writing which was
akward. Added text about duplicate header entry
encoding. Clarified some language w.r.t Header
Set. Changed x-my-header to mynewheader. Added
text in the HeaderEmission section indicating that
the application may also be able to free up memory
more quickly. Added information in Security
Considerations section.
The tables in this section should be updated based on
statistical analysis of header names frequency and specific
HTTP 2.0 header rules (like removal of some headers).
These tables are not adapted for headers contained in
PUSH_PROMISE frames. Either the tables can be merged, or
the table for responses can be updated.
The following table lists the pre-defined headers that
make-up the initial header table user to represent
requests sent from a client to a server.
IndexHeader NameHeader Value0:schemehttp1:schemehttps2:host3:path/4:methodGET5accept6accept-charset7accept-encoding8accept-language9cookie10if-modified-since11user-agent12referer13authorization14allow15cache-control16connection17content-length18content-type19date20expect21from22if-match23if-none-match24if-range25if-unmodified-since26max-forwards27proxy-authorization28range29via
The following table lists the pre-defined headers that
make-up the initial header table used to represent
responses sent from a server to a client. The same header
table is also used to represent request headers sent from
a server to a client in a PUSH_PROMISE frame.
IndexHeader NameHeader Value0:status2001age2cache-control3content-length4content-type5date6etag7expires8last-modified9server10set-cookie11vary12via13access-control-allow-origin14accept-ranges15allow16connection17content-disposition18content-encoding19content-language20content-location21content-range22link23location24proxy-authenticate25refresh26retry-after27strict-transport-security28transfer-encoding29www-authenticate
Here is an example that illustrates different representations
and how tables are updated.
This section needs to be updated to better reflect the new processing of header fields, and include more examples.
The first header set to represent is the following:
The header table is empty, all headers are represented as
literal headers with indexing. The 'mynewheader' header
name is not in the header name table and is encoded
literally. This gives the following representation:
The header table is as follows after the processing of
these headers:
As all the headers in the first header set are indexed in
the header table, all are kept in the reference
set of headers, which is:
The second header set to represent is the following:
Comparing this second header set to the reference set, the
first and third headers are from the reference set are not
present in this second header set and must be removed. In
addition, in this new set, the first and third headers
have to be encoded.
The path header is represented as a literal header with
substitution indexing. The mynewheader will be
represented as a literal header with incremental indexing.
The header table is updated as follow:
All the headers in this second header set are indexed in
the header table, therefore, all are kept in the reference
set of headers, which becomes: