[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [IPFIX] WG last call on draft-ietf-ipfix-file-02
Gerhard,
Thanks for your detailed review and comments. I'll address these by
type.
I. Comments related to the length and level of detail of the document:
The IPFIX File draft differs from other IPFIX drafts in that it 1.
contains its own requirements (analogous to 3917 and 5101 being the
same document) and 2. contains more motivating text than other drafts,
due to its somewhat unusual nature within the IETF (standardizing a
file format without binding it to some specific transport). The
authors believe that the inclusion and length of the requirements and
motivation sections (sections 4 and 5) is warranted by these
considerations.
II. Typographical, editorial and stylistic comments:
Thanks for these, there were quite a few typos/capitalization issues
we'd not caught yet. Stylistic suggestions have been incorporated
where they improve the clarity of the document.
III. Terminology confusion:
There are a couple of terms invented here in order to describe
concepts 1. for which there is no accepted terminology but 2. which
appear only for purposes of background. One of these is "IPFIX Message
format", which is intended to mean "the encoding and framing of IPFIX
Messages defined by RFC 5101 as used with the information model
defined by RFC 5102", or less formally, "IPFIX off the wire", _not_
the format of an individual message. Your comments indicate some
confusion here, and I would appeal to the working group to suggest an
alternative if this is a problem.
Similarly, an "IPFIX Message stream" is a collection of IPFIX Messages
in some loose order. This is applying the word "stream" as it is used
when talking about storage, where there is no connotation in English
of an unbounded sequence. I'm going to suggest we leave this term, as
the suggested alternatives have their own problems: "Sequence"
connotes strict ordering (not the case; IPFIX Messages can be
reordered within a File containing a Message stream for e.g. template
consistency.) "Series" has a strict and nonapplicable mathematical
meaning.
Alternately, a question to the WG: should we define "IPFIX Message
stream" and "IPFIX Message format" as intermediate terms in the
Terminology section of the document?
The issues regarding the interchangeability of "IPFIX File" and "IPFIX
Transport Session" given the fact that Sessions may span Files have
been fixed; many thanks for pointing this one out. V9-related
terminology has been fixed, too; I got a bit confused in a couple of
places switching between V9 and IPFIX.
IV. Technical issues:
Time window resolution finer than seconds: Perhaps. We were trying to
avoid the explosion of timestamp IEs we have elsewhere in the
information model. Keep in mind that the Time Window is meant, in
multi-file environments, to determine whether a file must be scanned
to find information about a specific moment in time. Going below the
second-level resolution, at least in my experience, is not necessary
_for this use case_.
Applicability of opaqueOctets: Indeed, one could create an enterprise-
specific IE and not export ET info about it, then it's opaque. But 1.
we still need to address the how to encap non-IPFIX info in an IPFIX
stream and 2. don't necessarily want to tell people go get a PEN to do
it. I'm inclined to keep opaqueOctets.
Thanks again!
- Brian
On Aug 15, 2008, at 6:59 PM, Gerhard Muenz wrote:
>
> Dear all,
>
> Here is a review for draft-ietf-ipfix-file-02.
> It is a very lengthy document. Many things are repeated. Maybe you can
> shorten it a little bit and make it more concise.
> Also, there are many unnecessary filler words like "note that". I do
> not
> really like that in a standard document.
>
> More comments inline.
>
> Gerhard
>
>
>> Abstract
>>
>> This document describes a file format for the storage of flow data
>> based upon the IPFIX Message format. It proposes a set of
>> requirements for flat-file, binary flow data file formats, then
>
> flat, binary flow data file formats?
>
>> applies the IPFIX message format to these requirements to build a
>> new
>> file format. This IPFIX-based file format is designed to
>> facilitate
>> interoperability and reusability among a wide variety of flow
>> storage, processing, and analysis tools.
>
>> 1.1. IPFIX Documents Overview
>>
>> "Specification of the IPFIX Protocol for the Exchange of IP Traffic
>> Flow Information" [RFC5101] and its associated documents define the
>
> Which are the associated documents? Ah, you mention them in the next
> paragraph. In this section, the division into paragraphs seems to be
> arbitrary.
>
>> IPFIX Protocol, which provides network engineers and administrators
>> with access to IP traffic flow information.
>>
>> "Architecture for IP Flow Information Export" [I-D.ietf-ipfix-arch]
>> defines the architecture for the export of measured IP flow
>> information out of an IPFIX Exporting Process to an IPFIX
>> Collecting
>> Process, and the basic terminology used to describe the elements of
>> this architecture, per the requirements defined in "Requirements
>> for
>> IP Flow Information Export" [RFC3917]. [RFC5101] then covers the
>
> Confusing as you already mentioned RFC 5101 before.
> I suggest modifying the introductory paragraph of this section such
> that
> it does not refer to RFC 5101 - or do not repeat it here.
>
>> details of the method for transporting IPFIX Data Records and
>> Templates via a congestion-aware transport protocol from an IPFIX
>> Exporting Process to an IPFIX Collecting Process.
>
> UDP is not congestion-aware.
>
>> "Information Model for IP Flow Information Export" [RFC5102]
>> describes the Information Elements used by IPFIX, including details
>> on Information Element naming, numbering, and data type encoding.
>> Finally, "IPFIX Applicability" [I-D.ietf-ipfix-as] describes the
>> various applications of the IPFIX protocol and their use of
>> information exported via IPFIX, and relates the IPFIX
>> architecture to
>> other measurement architectures and frameworks.
>>
>> In addition, "Exporting Type Information for IPFIX Information
>> Elements" [I-D.ietf-ipfix-exporting-type] specifies a method for
>> encoding Information Model properties within an IPFIX Message
>> stream.
>>
>> This document references [RFC5101] and the architecture document
>> for
>> terminology, defines IPFIX File Writer and IPFIX File Reader in
>> terms
>> of the IPFIX Exporting Processes and IPFIX Collecting Process
>> definitions from [RFC5101], and extends the IPFIX Information Model
>> defined in [RFC5102] to provide new Information Elements for IPFIX
>> File metadata. It uses the method described in "Exporting Type
>> Information for IPFIX Information Elements"
>> [I-D.ietf-ipfix-exporting-type] document to support the self-
>> description of IPFIX Files containing enterprise-specific
>> Information
>> Elements.
>>
>>
>> 2. Terminology
>>
>> Terms used in this document that are defined in the Terminology
>> section of [RFC5101] are to be interpreted as defined there.
>
> One more sentence explaining that you define additional terms below.
>
>> IPFIX File: An IPFIX File is a serialized stream of IPFIX
>> Messages
>
> A stream usually is continuous while a file has a beginning and an
> end.
> Consider: series or sequence of IPFIX Messages.
>
>> stored on a filesystem. Any IPFIX Message stream that would be
>> considered valid when transported one or more of the specified
>> IPFIX transports (SCTP, TCP, or UDP) as defined in [RFC5101] is
>> considered an IPFIX File for purposes of this document; however,
>
> What do you mean by "for purposes of this document"?
> Possibly remove it?
>
>> this document extends that definition with recommendations on
>> the
>> construction of IPFIX Files that meet the requirements
>> identified
>> herein.
>
> Where is "herein"? in the file? in the document?
>
>> IPFIX File Reader: An IPFIX File Reader is a Process which reads
>> IPFIX Files from a filesystem, and is analogous to an IPFIX
>
> I'm not a native English speaker, yet "analogous to" sounds very vague
> as long as you do not say in which respect the analogy holds.
> I do not have a good idea how to express what you want to say. As a
> starting point, consider something like: an IPFIX File Reader is
> used as
> a replacement of an IPFIX Collection Process.
>
>> Collecting Process. An IPFIX File Reader MUST behave as an
>> IPFIX
>> Collecting Process as outlined in [RFC5101], except as
>> modified by
>> this document.
>>
>> IPFIX File Writer: An IPFIX File Writer is a process which writes
>> IPFIX Files to a filesystem, and is analogous to an IPFIX
>
> same as above
>
>> Exporting Process. An IPFIX File Writer MUST behave as an IPFIX
>> Exporting Process as outlined in [RFC5101], except as modified
>> by
>> this document.
>
>> 3. Design Overview
>>
>> An IPFIX File, as defined by this document, is simply a stream
>> containing one or more IPFIX Messages serialized to some
>> filesystem.
>
> What is simple? Again, consider series or sequence instead of stream.
>
>> Though any set of valid IPFIX Messages can be serialized into an
>> IPFIX File, the specification proposes guidelines designed to ease
>> storage and retrieval of flow data using the format.
>
> Which format?
>
>> IPFIX Files contain only IPFIX Messages; any file metadata such as
>> checksums or export session details are stored using Options within
>> the IPFIX Message. This design has several advantages, including
>> complete compatibility with the IPFIX Protocol on the wire and free
>> manipulability of IPFIX Files through concatenation, appending, and
>> splitting (on IPFIX Message boundaries). A schematic of a typical
>> file is shown below:
>>
>>
>> +=======================================+
>> | IPFIX File |
>> | +===================================+ |
>> | | IPFIX Message | |
>> | | +-------------------------------+ | |
>> | | | Options Template Set | | |
>> | | | Options Template Record | | |
>> | | | . . . | | |
>> | | +-------------------------------+ | |
>> | | +-------------------------------+ | |
>> | | | Template Set | | |
>> | | | Template Record | | |
>> | | | . . . | | |
>> | | +-------------------------------+ | |
>> | +===================================+ |
>> | | IPFIX Message | |
>> | | +-------------------------------+ | |
>> | | | Data Set | | |
>> | | | Data Record | | |
>> | | | . . . | | |
>> | | +-------------------------------+ | |
>> | | +-------------------------------+ | |
>> | | | Data Set | | |
>> | | | Data Record | | |
>> | | | . . . | | |
>> | | +-------------------------------+ | |
>> | | . . . | |
>> | +===================================+ |
>> | . . . |
>> +=======================================+
>>
>> Figure 1: Typical File Structure
>
> Looks a little bit as if the IPFIX File encapsulates the Messages with
> header and trailer.
>
> Consider:
>
> ^ +===================================+
> : | IPFIX Message |
> : | +-------------------------------+ |
> : | | Options Template Set | |
> : | | Options Template Record | |
> : | | . . . | |
> : | +-------------------------------+ |
> : | +-------------------------------+ |
> : | | Template Set | |
> : | | Template Record | |
> : | | . . . | |
> : | +-------------------------------+ |
> : +===================================+
> : | IPFIX Message |
> : | +-------------------------------+ |
> : | | Data Set | |
> : | | Data Record | |
> IPFIX File | | . . . | |
> : | +-------------------------------+ |
> : | +-------------------------------+ |
> : | | Data Set | |
> : | | Data Record | |
> : | | . . . | |
> : | +-------------------------------+ |
> : | . . . |
> : +===================================+
> : . . .
> V
>
>
>> 4. Motivation
>
>> The variety of applications of flow data, and the variety of
>> presently deployed storage approaches, would seem to indicate the
>
> "would seem to indicate" => indicate
>
>> need for a standard approach to flow storage with applicability
>> across the continuum of time scales over which flow data is stored.
>> A storage format based around flat files would best address the
>> variety of storage requirements. While much work has been done on
>> structured storage via RDBMS, relational database systems are not a
>> good basis for format standardization owing to the fact that their
>> internal data structures are generally private to a single
>> implementation and subject to change for internal reasons. Also,
>> there are a wide variety of operations available on flat files, and
>> external tools and standards can be leveraged to meet file-based
>> flow
>> storage requirements. Further, flow data is often not very
>> semantically complicated, and is managed in very high volume;
>> therefore, an RDBMS-based flow storage system would not benefit
>> much
>> from the advantages of relational database technology.
>
> I would remove the last sentence. There is no need to argue against
> the
> usage of RDBMS.
>
>> Over the past decade XML markup has emerged as a new "universal"
>> representation format for structured data. It is intended to be
>> human-readable; indeed, that is one reason for its rapid adoption.
>> However XML has limited usefulness for representing network flow
>> data. Network flow data has a simple, repetitive, non-hierarchical
>> structure that does not benefit much from XML. An XML
>> representation
>> of flow data would be an essentially flat list of the attributes
>> and
>> their values for each flow record.
>>
>> The XML approach to data encoding is very heavyweight when compared
>> to binary flow encoding. XML's use of start- and end-tags, and
>> plain-text encoding of the actual values, leads to significant
>> inefficiency in encoding size. Typical network flow datasets can
>> contain millions or billions of flows per hour of traffic
>> represented. Any increase in storage size per record can have
>> dramatic impact on flow data storage and transfer sizes. While
>> data
>> compression algorithms can partially remove the redundancy
>> introduced
>> by XML encoding, they introduce additional overhead of their own.
>>
>> A further problem is that XML processing tools require a full XML
>> parser. XML parsers are fully general and therefore complex,
>> resource-intensive and relatively slow, introducing significant
>> processing time overhead for large network-flow datasets. In
>> contrast, parsers for typical binary flow data encodings are simply
>> structured, since they only need to parse a very small header and
>> then have complete knowledge of all following fields for the
>> particular flow. These can then be read in a very efficient linear
>> fashion.
>
> I suggest shortening the text of the three paragraphs above.
> The argument that parsing binary IPFIX Messages is simpler than
> parsing
> XML-encoded flow data is not convincing.
> A better argument is that parsers for IPFIX Messages exist anyway and
> can be reused.
>
>> This leads us to propose the IPFIX Message format as the basis
>> for a
>> new flow data file format. The IPFIX working group, in defining
>> the
>> IPFIX protocol, has already defined an information model and data
>> formatting rules for representation of flow data. Especially at
>> shorter time scales, when a file is a unit of data interchange, the
>> filesystem may be viewed as simply another IPFIX Message transport
>> between processes. This format is especially well suited to
>> representing flow data, as it was designed specifically for flow
>> data
>> export; it is easily extensible unlike ad-hoc serialization, and
>> compact unlike XML. In addition, IPFIX is an IETF standard for the
>> export and collection of flow data; using a common format for
>> storage
>> and analysis at the collection side allows implementors to use
>> substantially the same information model and data formatting
>> implementation for transport as well as storage.
>
>> 5.2. Self Description
>
>> The IPFIX Message format is partially self-describing; that is,
>> IPFIX
>> Templates containing only IANA-assigned Information Elements can be
>> completely interpreted according to the IPFIX Information Model
>> without additional external data.
>
> I do not think that the IPFIX Message format is self-describing
> because
> a single IPFIX Message usually is not self-describing. You need all
> messages sequence of a Transport Session to get all the required
> Templates etc.
>
>> However, Templates containing private information elements lack
>> detailed type and semantic information; a Collecting Process
>> receiving data described by a template containing private
>> Information
>> Elements it does not understand can only treat the data contained
>> within those Information Elements as octet arrays. To be fully
>> self-
>> describing, enterprise-specific Information Elements must be
>
> Unless this was a MUST, I would replace it by "can".
>
>> additionally described via IPFIX Options according to the
>> Information
>> Element Type Options Template defined in "Exporting Type
>> Information
>> for IPFIX Information Elements" [I-D.ietf-ipfix-exporting-type].
>
>> 5.4. Indexing and Searching
>
>> The IPFIX Message format has no direct support for indexing.
>
> Do you want so search within an IPFIX Message or within an IPFIX
> File (=
> sequence of IPFIX Messages)?
>
>> 5.5. Data Integrity
>
>> Note that more advanced error correction is almost certainly best
>
> omit "almost certainly"
>
>> handled at a layer below that addressed by this document. Error
>> correction is a topic well addressed by the storage industry in
>> general (e.g. by RAID and other technologies), and by specifying a
>> flow storage format based upon files, we can leverage these
>> features
>> to meet this requirement.
>
>
>> 6.2. Storage of NetFlow V9-collected Flow Data
>>
>> Although the IPFIX protocol is based on the Cisco Netflow Services,
>> Version 9 (NetFlow V9) protocol [RFC3954], the two have diverged
>
> I would not say that IPFIX is based on NetFlow V9. It's rather the
> successor.
>
>> since work began on IPFIX. However, since the NetFlow V9
>> information
>> model is a compatible subset of the IPFIX information model, it is
>> possible to use IPFIX files to store collected NetFlow V9 flow
>> data.
>> This approach may be particularly useful in multi-vendor, multi-
>> protocol collection infrastructures using both NetFlow V9 and IPFIX
>> to export flow data.
>
>
>> 7. Detailed Description
>>
>> An IPFIX File, as introduced in Section 3 and elaborated below,
>> is at
>> its core simply an IPFIX Message stream serialized to some
>> filesystem. Any valid serialized IPFIX Message stream MUST be
>> accepted by a File Reader as a valid IPFIX file. In this way, the
>
> IPFIX File
>
>> filesystem is simply treated as another IPFIX transport alongside
>> SCTP, TCP, and UDP. In contrast to normal IPFIX operation, the
>> time
>> between a File Writer writing an IPFIX Message stream to a File
>> and a
>> File Reader reading it can be extremely variable. In other words,
>> this notional file transport has unusually high latency, as the
>> File
>> Reader and File Writer do not necessarily run at the same time.
>>
>> This section specifies the detailed actions of File Readers and
>> File
>> Writers in handling IPFIX Files, and further specifies actions of
>> File Writers in specific use cases. Unless otherwise specified
>> herein, where appropriate IPFIX File Writers MUST behave as IPFIX
>
> remove "where"?
>
>> Exporting Processes, and IPFIX File Readers MUST behave as IPFIX
>> Collecting Processes.
>
>
>> 7.2. File Writer Specification
>>
>> While any valid serialized IPFIX Message stream is a valid IPFIX
>> File, the following recommendations will improve representation
>> simplicity and read performance in the general case, where
>> possible.
>>
>> File Writers SHOULD emit each Template Set or Options Template
>> Set to
>> appear in the file before any Data Set described by the Templates
>> within that Set, to ensure the File Reader can decode every Data
>> Set
>> without waiting to process subsequent Templates or Options
>> Templates.
>
> Is the File Reader allowed to wait for a Template anyway?
> In 7.1, you wrote that Templates can be redefined without withdrawal
> as
> it is the case in UDP sessions. In this case, decoding errors may
> occur
> if the Collector decodes previous Data Sets with the new Template.
>
>> File Writers SHOULD emit Data Records described by Options
>> Templates
>> to appear in the file before any Data Records which depend on the
>> scopes defined by those options.
>>
>> File Writers SHOULD use Template Withdrawals to withdraw
>> Templates if
>> template IDs need to be reused. In this case, the new Templates
>> reusing those IDs SHOULD appear directly in the file after the
>> Template Withdrawals making the IDs available for reuse. Template
>> Withdrawals SHOULD NOT be used unless necessary to reuse template
>
> Template
>
>> IDs.
>
> I do not understand the restrictions "SHOULD appear directly after"
> and
> "SHOULD NOT be used unless necessary". If there is no good reason, I
> would remove these restrictions.
>
>> Each IPFIX File is generally synonymous with a single Transport
>> Session. File Writers SHOULD store the Templates and Options
>> required to decode the data within the File in the File itself, and
>> File Readers SHOULD NOT use Templates or Options defined in one
>> file
>> to decode or interpret Data Sets in another.
>
> MUST the File Writer repeat Templates and Options in each File if the
> Transport Session is split into several files?
>
>>
>> File Writers SHOULD write IPFIX Messages within an IPFIX File in
>> ascending Export Time order.
>>
>> File Writers MAY write Data Records to an IPFIX File in any order.
>> However, File Writers that write flow records to an IPFIX File in
>> flowStartTime or flowEndTime order SHOULD be consistent in this
>> ordering within each File.
>
> The last sentence does not make sense to me since it is always
> true: If
> the IPFIX File Writer writes record ordered by time, then, of course,
> the order is consistent.
>
>> 7.3.1. Collocating a File Writer with a Collecting Process
>>
>> When collocating a File Writer with an IPFIX Collecting Process for
>> archival storage of collected data in IPFIX Files as described in
>> Section 6.1, the following recommendations may improve the
>> usefulness
>> of the stored data.
>>
>> The simplest way for a to store the data collected in a single
>
> "for a" what?
>
>> Transport Session is to simply write the incoming IPFIX Messages to
>> an IPFIX File as they are collected. However, the resulting files
>> will lack information about the IPFIX Transport Session used to
>> export them, such as the network addresses of the Exporting and
>> Collecting Processes and the protocols used to transport them. In
>> this case, if information about the Transport Session is required,
>> the File Writer SHOULD store a single IPFIX Transport Session in an
>> IPFIX File and SHOULD record information about the Transport
>> Session
>> using the Export Session Details Options Template described in
>> Section 8.1.3.
>
>> 7.3.2. Collocating a File Writer with a Metering Process
>>
>> Note that File Writers may also be collocated directly with IPFIX
>
> Remove "note that"
>
>> Metering Processes, for writing measured information directly to
>> disk
>> without intermediate IPFIX Exporting or Collecting Processes. This
>> arrangement may be particularly useful when providing data to an
>> analysis environment with an IPFIX File based workflow, or when
>> testing Metering Processes during development.
>>
>> When collocating a File Writer with a Metering Process, note that
>
> Remove "note that"
>
>> Information Elements associated with Exporting or Collecting
>> Processes are meaningless, and SHOULD NOT appear in the Export
>> Session Details Options Template described in Section 8.1.3 or the
>> Message Details Options Template described in Section 8.1.4.
>>
>> When collocating a File Writer with an Exporting Process, the
>> Export
>
> Collocating File Writer and Exporting Process? Do you mean Metering
> Process?
>
>> Time of each Message SHOULD be the time at which the first Data
>> Record in the Message was received from the Exporting Process.
>
> Hm, I would rather say that the Export Time should be the time of
> writing the Message into the File. This is typically the time of the
> LAST Data Record in the Message.
> When exported by an Exporting Process, the Export Time is never before
> the time of the last Data Record either.
>
>> 7.3.3. Using IPFIX Files for Archival Storage
>>
>> While in the general case File Writers should store one Transport
>> Session per IPFIX File, some applications storing large collections
>> of data over long periods of time may benefit from the ability to
>> treat a collection of IPFIX Files as a single Transport Session. A
>> File Writer MAY be configurable to write data from a single
>> Transport
>> Session into multiple IPFIX Files; however, File Writers supporting
>> such a configuration option MUST provide a configuration option to
>> support one-file-per-session behavior for interoperability
>> purposes.
>
> I do not think that a mandatory one-file-per-session option makes
> sense.
> Sometimes, it is not possible to enforce one-file-per-session, for
> example if the disk is full. What do you do then?
>
>> File Writers compressing or encrypting archival data and File
>> Readers
>> reading compressed or encrypted archival data SHOULD follow the
>> recommendations in Section 9.
>>
>> 7.3.4. Using IPFIX Files as Documents
>>
>> When IPFIX Files are used as documents, to store a set of flows
>> relevant to query, investigation, or other common context, or for
>> the
>> publication of flow data sets relevant to network research, each
>> File
>> MUST be readable as a single Transport Session, self-contained and
>> making no reference to metadata stored in separate Files, in
>> order to
>
> Maybe you should explain what you mean by metadata.
>
>> ensure interoperability.
>>
>> When writing Files to be used as documents, File Writers may emit
>> the
>> special Data Records described by Options Templates before any
>> other
>> Data Records in the File, in the following order, to ease the
>> inspection and use of documents by File Readers:
>>
>> o Time Window records described by the File Time Window Options
>> Template as defined in Section 8.1.2 below; followed by
>>
>> o commonPropertiesId definitions as described in "Reducing
>> Redundancy in IPFIX and PSAMP Reports"
>> [I-D.ietf-ipfix-reducing-redundancy]; followed by
>>
>> o Information Element Type Records as described in "Exporting Type
>> Information for IPFIX Information Elements"
>> [I-D.ietf-ipfix-exporting-type]; followed by
>
> Exporting type information should be before reducing redundancy since
> common properties may contain enterprise specific IEs.
>
>> o Export Session details records described by the Export Session
>> Details Options Template as defined in Section 8.1.3 below.
>
> Do we really need to specify the order?
> I think it suffices to specify that the Time Window should appear at
> the
> beginning of the file.
>
>> The Export Time of each Message within a File used as a document
>> SHOULD be the time at which the Message was written by the File
>> Writer.
>
> Agreed, but this contradicts what is written in 7.3.2.
>
>> If an IPFIX File used as a document uses the technique described in
>> "Reducing Redundancy in IPFIX and PSAMP Reports"
>> [I-D.ietf-ipfix-reducing-redundancy] AND all of the non-Options
>> Templates in the File contain the commonPropertiesId Information
>> Element, a File Reader MAY assume the set of commonPropertiesId
>> definitions provides a complete table of contents for the File for
>> searching purposes.
>
> There is a problem: Templates may be redefined in the middle of the
> file. How does the File Reader now that all Templates ever defined in
> the middle of the File use the same set of common properties?
>
>> 7.3.5. Using IPFIX Files for Testing
>>
>> IPFIX Files can be used for testing IPFIX Collecting Processes in
>> two
>> ways. First, IPFIX Files can be used to store specific flow data
>> for
>> regression and stress testing of Collectors; there are no special
>> considerations for IPFIX Files used in this way.
>>
>> Second, IPFIX Files are useful for storing reference messages which
>> do not comply to the IPFIX Protocol in order to test the error
>> handling and recovery behavior of Collectors. Of course, IPFIX
>> Files
>> intended to be used in this application necessarily MAY violate any
>> of of the specifications in this document or in [RFC5101], and such
>
> duplicated "of"
>
>> files MUST NOT be transmitted to Collecting Processes or given as
>> input File Readers not under test.
>>
>> Note that an extremely simple IPFIX Exporting Process may be
>> crafted
>> for testing purposes by simply reading an IPFIX File and
>> transmitting
>> it directly to a Collecting Process. Similarly, an extremely
>> simple
>> Collecting Process may be crafted for testing purposes by simply
>> accepting connections and/or IPFIX Messages from Exporting
>> Processes
>> and writing the session's message stream to an IPFIX File.
>
> What is the reason for testing such a simple Collecting Process? It
> will
> write the invalid messages into a file.
>
>> 8.1. Recommended Options Templates for IPFIX Files
>>
>> The following Options Templates allow IPFIX Message streams to meet
>> the requirements outlined above without extension to the message
>> format or protocol. They are defined in terms of existing
>> Information Elements defined in [RFC5102], the Information Elements
>> defined in "Exporting Type Information for IPFIX Information
>> Elements" [I-D.ietf-ipfix-exporting-type], as well as Information
>> Elements defined in Section 8.2. IPFIX File Readers and Writers
>> SHOULD support these options templates as defined below.
>
> Options Templates
>
>> In addition, IPFIX File Readers and Writers SHOULD support the
>> Options Templates defined in "Exporting Type Information for IPFIX
>> Information Elements" [I-D.ietf-ipfix-exporting-type] in order to
>> support self-description of enterprise-specific Information
>> Elements.
>
> I understand that a File Writer should add exporting type
> information if
> necessary and available. So, a writer should support this feature.
> But what about the File Reader? I do not think that supporting the
> reading of exporting type information options requires special
> capabilities that need to be supported.
>
>> 8.1.1. Message Checksum Options Template
>>
>> The Message Checksum Options Template specifies the structure of a
>> Data Record for attaching an MD5 message checksum to an IPFIX
>> Message. An MD5 message checksum as described MAY be used if long-
>> term data integrity is important to the application. The described
>
> why only "long-term" data integrity?
>
>> Data Record MUST appear only once per IPFIX Message, but MAY appear
>> anywhere within the Message.
>
>> 8.1.2. File Time Window Options Template
>>
>> The File Time Window Options Template specifies the structure of a
>> Data Record for attaching a time window to an IPFIX File; this Data
>> Record is referred to as a time window record. A time window
>> record
>> defines the earliest flow start time and the latest flow end time
>> of
>> the flow records within a File. One and only one time window
>> record
>> MAY appear within an IPFIX File if the time window information is
>> available; a File Writer MUST NOT write more than one time window
>> record to an IPFIX File. A File Writer that writes a time window
>> record to a File MUST NOT write any Flow with a start time before
>> the
>> beginning of the window or an end time after the end of the
>> window to
>> that File.
>
> Shouldn't the time window record be located at the beginning of the
> file? Yet this requires that we are able to prepend data to the file.
> Anyway, a SHOULD would be ok, in my opinion.
>
>> The template SHOULD contain the following Information Elements:
>>
>> +---------------------
>> +---------------------------------------------+
>> | IE |
>> Description |
>> +---------------------
>> +---------------------------------------------+
>> | sessionScope | A marker denoting this Option applies
>> to |
>> | [scope] | the whole IPFIX Transport Session
>> (i.e., |
>> | | IPFIX File); content is ignored.
>> This |
>> | | Information Element MUST be defined as
>> a |
>> | | Scope
>> Field. |
>> | minFlowStartSeconds | The start time of the earliest flow in
>> the |
>> | | Transport Session (i.e., File) in
>> epoch |
>> | |
>> seconds. |
>> | maxFlowEndSeconds | The end time of the latest flow in
>> the |
>> | | Transport Session (i.e., File) in
>> epoch |
>> | |
>> seconds. |
>> +---------------------
>> +---------------------------------------------+
>
> sessionScope is ambiguous if the session is split into multiple
> files. I
> assume that in this case, the time window record describes the
> limits of
> a single file?
>
> Is there a reason for having seconds and not milliseconds as
> alternative?
>
> What can we do if the flow start or end timestamp is missing in the
> records?
>
> In the case of PSAMP, we have observationTimeSeconds etc. and it makes
> sense to use this timestamp instead.
>
>> 8.1.3. Export Session Details Options Template
>>
>> The Export Session Details Options Template specifies the structure
>> of a Data Record for recording the details of an IPFIX Transport
>> Session in an IPFIX File. It is intended for use in storing a
>> single
>> complete IPFIX Transport Session in a single IPFIX File. The
>
> Why not use it when a session is stored in multiple files as well?
>
>> described Data Record SHOULD appear only once in a given IPFIX
>> File.
>>
>
>> 8.2. Recommended Information Elements for IPFIX Files
>>
>> The following Information Elements are used by the options
>> templates
>> in Section 8.1 to allow IPFIX Message streams to meet the
>> requirements outlined above without extension of the message format
>> or protocol. IPFIX File Readers and Writers SHOULD support these
>> Information Elements as defined below.
>>
>> In addition, IPFIX File Readers and Writers SHOULD support the
>> Information Elements defined in "Exporting Type Information for
>> IPFIX
>> Information Elements" [I-D.ietf-ipfix-exporting-type] in order to
>> support full self-description of Information Elements.
>
> I would remove this paragraph. You have already stated above that the
> writer should support exporting type information options. This implies
> the support of the corresponding IEs.
> For other purposes, the exporting type information IEs are not
> required.
>
>> 8.2.3. maxExportSeconds
>>
>> Description: The absolute Export Time of the latest IPFIX Message
>> within the scope containing this Information Element. This
>> Information Element SHOULD be bound to its containing IPFIX
>> Transport Session (i.e., File) via an options record and the
>> sessionScope Information Element, as defined below, and SHOULD
>> appear only once in a given IPFIX File.
>
> session != file in the case of multiple files per session
>
>> 10.1. Encapsulation of Non-IPFIX Data in IPFIX Files
>>
>> At times it may be useful to export or store non-IPFIX data
>> inline in
>> an IPFIX File or Message stream. To do this cleanly, this data
>> must
>> be encapsulated into IPFIX Messages so that an IPFIX File Reader or
>> Collecting Process can handle it without any need to interpret it.
>> At the same time, this data must not be changed during transmission
>> or storage. The opaqueOctets Information Element as defined in
>> Section 8.2.9 is provided for this encapsulation.
>
> I wonder if the standardization of opaqueOctets makes sense at all.
> If a vendor wants to add such information, he can specify an
> enterprise-specific IE for this purpose with optional type
> information.
> With opaqueOctets, we never know which kind of data is contained in
> the
> field. And we lose one important advantage of IE which is the implicit
> semantics of any IE.
>
>> Processing the encapsulated non-IPFIX data is left to a separate
>> processing mechanisms that can identify encapsulated non-IPFIX data
>> in an IPFIX message stream, but need not have any other IPFIX
>> handling capability, except the ability to skip over all IPFIX
>> messages that do not encapsulate non-IPFIX data.
>>
>> The Message Checksum Options Template, described in Section 8.1.1
>> may
>> be used as a uniform mechanism to identify errors within
>> encapsulated
>> data.
>>
>> Note that this mechanism can only encapsulate data objects up to
>> 65,515 octets in length. If the space available in one IPFIX
>> Message
>> is not enough for the amount of data to be encapsulated, then the
>> data must be broken into smaller segments that are encapsulated
>> into
>> consecutive IPFIX Messages. Any additional structuring or
>> semantics
>> of the raw data is outside the scope of IPFIX and must be
>> implemented
>> within the encapsulated binary data itself. Furthermore, the raw
>> encapsulated data cannot be assumed by an IPFIX File Reader to have
>> any specific format.
>>
>> 10.2. Encapsulation of IPFIX Files within Other File Formats
>>
>> Consequently, it may also be useful to reverse the encapsulation,
>> that is, to export or store IPFIX data inline within a non-IPFIX
>> file
>> or data stream. This makes sense when the other file format is not
>> compatible with the encapsulation described above in Section 10.1.
>> Generally speaking, the encapsulation here will be specific to the
>> format of the containing file. For example, IPFIX files may be
>> embedded in XML elements using hex or Base64 encoding, or in raw
>> binary files using start and end delimiters or some form of run-
>> length encoding. As there are as many potential encapsulations
>> here
>> as there are potential file formats, the specifics of each are
>> out of
>> scope for this specification.
>
> I would remove the whole section. It is completely out of scope.
>
>> B.1.3. Template Format
>>
>> Template FlowSets in NetFlow V9 support a subset of functionality
>> of
>> those in IPFIX. Specifically, NetFlow V9 does not have any support
>> for vendor-specific Information Elements as IPFIX does, so there is
>> no enterprise bit or facility for associating a private enterprise
>> number with an information element.
>
> NetFlow V9 does not support variable length fields.
>
>> Options Template FlowSets in NetFlow V9 are similar to Options
>> Template Sets in IPFIX in the same way.
>
>> B.1.4. Information Model
>>
>> The NetFlow V9 field type definitions are a compatible subset of,
>> and
>> have evolved in concert with, the IPFIX Information Model. IPFIX
>> Information Element numbers in the range 1-127 are defined by the
>
> RFC 5102 talks about IE IDs, not numbers.
>
>> IPFIX Information Model [RFC5102] to be compatible with the
>> corresponding NetFlow V9 field types.
>
>> B.2. A Method for Transforming NetFlow V9 messages to IPFIX
>>
>> This appendix describes a method for transforming NetFlow V9
>> Packets
>> into IPFIX Messages, which can be used to store NetFlow V9 data in
>> IPFIX Files. A process transforming NetFlow V9 Packets into IPFIX
>> Messages must handle the fact that NetFlow V9 Packets and IPFIX
>> Messages are framed differently, that sequence numbering works
>> differently, and that the NetFlow V9 field type definitions are
>> only
>> compatible with the IPFIX Information Model field and/or
>> information
>> element numbers below Information Element number 128.
>
> what is an Information Model field?
>
> Consider: ...are only compatible with the IPFIX Information Element
> IDs
> below 128.
>
>> For each incoming NetFlow V9 packet, the transformation process
>> must:
>>
>> 1. Verify that the Version field of the packet header is 9.
>>
>> 2. Verify that the Sequence Number field of the packet header is
>> valid.
>>
>> 3. Scan the packet to:
>>
>> 1. verify that it contains no Templates with field numbers
>
> field types
>
>> outside the range 1-127;
>>
>> 2. verify that it contains no FlowSets with Set IDs between 2
>> and 255 inclusive;
>>
>> 3. verify that it contains the number of records in FlowSets,
>> Template FlowSets, and Options Template FlowSets declared
>> in
>> the Count field of the packet header; and
>>
>> 4. count the number of records in FlowSets for calculating the
>
> only records of Data FlowSets must be counted!!!
>
>> IPFIX Sequence number.
>
> Number
>
>> 4. Calculate a Sequence Number for each IPFIX Observation Domain
>> by
>> storing the last Sequence Number sent for each Observation
>> Domain
>> plus the count of records in FlowSets in the previous step to
>> be
>
> Data FlowSets!
>
>> sent as the Sequence Number for the IPFIX Message within that
>> Observation Domain following this one.
>
> "following this one" is confusing.
> I would move step 4 behind step 6 and write that the Sequence Number
> is
> increased by the number of records in Data FlowSets contained in the
> written Message.
>
>> 5. Generate a new IPFIX Message Header with:
>>
>> 1. a Version field of 10;
>>
>> 2. a Length field with the number of octets in the IPFIX
>> Message, generally available by subtracting 4 from the
>> length
>> of the NetFlow V9 packet as returned from the transport
>> layer
>> (accounting for the difference in message header lengths);
>>
>> 3. the Sequence Number calculated for this message by the
>> Sequence Number calculation step; and
>>
>> 4. Export Time and Observation Domain ID taken from the UNIX
>> secs and Source ID fields of the NetFlow V9 packet header,
>> respectively.
>>
>> 6. Copy each FlowSet from the Netflow V9 packet to the IPFIX
>> Message
>> after the header. Replace Set ID 0 with Set ID 2 for Template
>> Sets, and Set ID 1 with Set ID 3 for Options Template Sets.
>>
>> Note that this process loses system uptime information; if such
>> information is required, the transformation process will have to
>> export that information using IPFIX Options. This may require a
>> more
>> sophisticated transformation process structure.
>>
>
>
>
> --
> Dipl.-Ing. Gerhard Münz
> Computer Networks and Internet
> Wilhelm Schickard Institute for Computer Science
> University of Tuebingen
> Sand 13 (Room B309), D-72076 Tuebingen, Germany
> Phone: +49 7071 29-70534 Fax: +49 7071 29-5220
> E-mail: muenz at informatik.uni-tuebingen.de
> WWW: http://net.informatik.uni-tuebingen.de/~muenz
>
> _______________________________________________
> IPFIX mailing list
> IPFIX at ietf.org
> https://www.ietf.org/mailman/listinfo/ipfix
_______________________________________________
IPFIX mailing list
IPFIX at ietf.org
https://www.ietf.org/mailman/listinfo/ipfix