-
Notifications
You must be signed in to change notification settings - Fork 0
/
draft-gray-sampled-streaming-03.txt
2184 lines (1753 loc) · 80.6 KB
/
draft-gray-sampled-streaming-03.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
OPSAWG A. Gray
Internet-Draft Charter Communications
Intended status: Informational L.J. Wobker
Expires: 2 July 2020 Cisco Systems
30 December 2019
Sampled Traffic Streaming
draft-gray-sampled-streaming-03
Abstract
This document standardizes both 1) a means of requesting a stream of
packet samples from any device generating, routing, or forwarding
traffic, and 2) receiving metadata information from the network
element about these packet samples, and the structure of said stream
metadata. A main design requirement is to provide network elements
with widely varying capabilities (e.g., ASICs, NPUs, NICs, vSwitches,
CPUs) a mechanism to sample and export packets at high rates, by
allowing communication of the specific bit formats of internal data
headers applied to the packet flow, in a way that enhances
interoperability between traffic sources and sinks. Historically,
Netflow and similar mechanisms have been used for these use cases;
however, the increasing packet rates of very high-speed devices and
increasing variance in the information available to data planes lends
itself to both a less-prescriptive set of packet formats as well as a
decoupling of the sampling action from the collection and analysis
mechanisms.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 2 July 2020.
Gray & Wobker Expires 2 July 2020 [Page 1]
Internet-Draft Sampled Traffic Streaming December 2019
Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Simplified BSD License text
as described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
1.3. Motivation for Disaggregation of Telemetry . . . . . . . 3
1.4. Comparisons with PSAMP . . . . . . . . . . . . . . . . . 4
2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Use Case 1: Traffic Analytics . . . . . . . . . . . . . . 5
2.2. Use Case 2: Network Behavior Verification . . . . . . . . 6
2.3. Use Case 3: Standardization . . . . . . . . . . . . . . . 6
2.4. Use Case 4: Security Automation . . . . . . . . . . . . . 6
3. Stream Setup . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Client queries Replicator for Points . . . . . . . . . . 7
3.2. Client submits a request to the Replicator . . . . . . . 8
3.2.1. Filtering Details . . . . . . . . . . . . . . . . . . 9
3.3. Replicator offers Proposals . . . . . . . . . . . . . . . 9
3.4. Client selects a Proposal . . . . . . . . . . . . . . . . 10
3.5. Ending sampling and cleanup . . . . . . . . . . . . . . . 11
4. Data Stream Format . . . . . . . . . . . . . . . . . . . . . 11
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15
6. Security Considerations . . . . . . . . . . . . . . . . . . . 15
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 16
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 16
8.1. Normative References . . . . . . . . . . . . . . . . . . 16
8.2. Informative References . . . . . . . . . . . . . . . . . 16
Appendix A. Yang Model Tree Reference . . . . . . . . . . . . . 17
Appendix B. Yang Model . . . . . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39
1. Introduction
Gray & Wobker Expires 2 July 2020 [Page 2]
Internet-Draft Sampled Traffic Streaming December 2019
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
1.2. Terminology
The following terms are used within this document:
Client: The device configuring the Replicator.
Receiver: The device receiving the packet stream.
Replicator: The device performing the actual packet replication,
as requested by a Client, and sending the resulting replicated
packet stream to a Receiver.
Point: The location inside the Replicator (e.g., a forwarding
ASIC) that performs the actual packet replication. There may be
multiple physical interfaces serviced by one Point, or one
interface may be serviced by multiple Points, that may have
different capabilities.
1.3. Motivation for Disaggregation of Telemetry
A key concept for this proposal is to enable very high rate sample
generation for network elements, while at the same time separating
the sampling mechanism itself from specific analysis or transport
protocols. If we separate the component functions of how these
problems have been traditionally solved, these functions lend
themselves to being viewed as a layered stack such as the one in the
figure:
Figure: Packet sampling and analysis viewed as a layered stack
Gray & Wobker Expires 2 July 2020 [Page 3]
Internet-Draft Sampled Traffic Streaming December 2019
+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Analysis | Higher level applications perform
+-+-+-+-+-+-+-+-+-+-+-+-+-+ further analysis on aggregated samples
^^
+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Collection / Decoding | Samples arrives at Receiver, decoded,
+-+-+-+-+-+-+-+-+-+-+-+-+-+ optionally stored/aggregated
^^
+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Export / Transport | Encapsulate packet sample and metadata,
+-+-+-+-+-+-+-+-+-+-+-+-+-+ send via configured transport protocol
^^
+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sampling / Metadata | Samples filters packets at a fixed
+-+-+-+-+-+-+-+-+-+-+-+-+-+ ratio from stream, appends metadata
^^
+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Plane Forwarding | Stream of data packets moving through
+-+-+-+-+-+-+-+-+-+-+-+-+-+ element data plane
Figure 1
The primary advantage of the stack model is the ability to
disaggregate functions from each other. For example, providing a
self-describing, flexible format for the metadata abstracts the data
plane -- in other words the upper layers do not need to know how many
bits wide a metadata field is, they only need to know that it is
present and the semantics. Separating the transport function allows
for multiple use cases: a router wishing to sample packets for
internal consumption within the same system might use a locally
defined (perhaps even proprietary) transport header, while putting
the sampled metadata and packet into a UDP packet allows for it to be
transported to any IP-reachable collector, regardless of the
geographic or topological distance from the Replicator itself.
This document standardizes the "Sampling / Metadata" and "Export /
Transport" components of the above stack.
1.4. Comparisons with PSAMP
Packet Sampling (PSAMP) from RFC 5476 [RFC5476] shares some of the
characteristics of Sampled Streaming, and parts of its YANG model as
documented in RFC 6728 [RFC6728] are in fact imported into this one
to share concepts where possible. However, Sampled Streaming differs
primarily in the ability to include information that is normally
internal to device that provides information about the packet's
handling through the device, and to have the Replicator specify the
Gray & Wobker Expires 2 July 2020 [Page 4]
Internet-Draft Sampled Traffic Streaming December 2019
outgoing packet format in a very dynamic fashion that suits itself as
best as possible. This is done to allow this replication to be done
natively on relatively low feature set forwarding hardware and to
ensure the only usage of high-capability CPU resources on the
Replicator is in the initial setup and negotiation. All other
aspects have been made to allow the Replicator to do the least amount
of work as possible, to extract as much information as possible, and
get it sent to the Receiver who is presumed to have orders of
magnitudes greater compute capability available. Other changes to
the setup and configuration are wrapped around this primary goal.
2. Use Cases
This document is designed around the following current and
foreseeable use cases that operators have today.
2.1. Use Case 1: Traffic Analytics
Operators typically use a mix of NetFlow, IPFIX, and in-line traffic
samplers spread throughout the network to gather data for analytics.
With the next generation of hardware, 400Gb/s interfaces are becoming
available, with higher data rates under development in their
respective standards bodies. This will require at least an
augmentation of any in-line traffic samplers, which are quite
expensive. Additionally, the pace of growth in the data plane is
outgrowing the pace of growth of the control plane. This is
especially visible with relatively control plane or CPU-heavy
protocols such as NetFlow, where current sampling rates are simply
not going to be sustainable long-term, primarily due to on-box
control plane hardware limitations. Being able to capture a
filtered, sampled collection of actual packets throughout the network
is very valuable for understanding how the network is being used, to
provide hard data to justify network topology augments and/or
technology changes.
This proposal addresses this use case by: 1) making the data
replication mechanism as simple as possible, reducing the need for
high levels of complexity in the data plane; 2) decoupling the
sampling/collection of packets from the analysis, which in turn
allows for the analysis to be performed on distributed, horizontally-
scalable platforms rather than being constrained to the compute and
storage capabilities of a local network element.
Gray & Wobker Expires 2 July 2020 [Page 5]
Internet-Draft Sampled Traffic Streaming December 2019
2.2. Use Case 2: Network Behavior Verification
This use case focuses on the potential ability to have the ASICs
stream discarded packets, along with an indication as to the reason
for the drop. With fields denoting the reason for dropped packets
such as QoS policies, buffer contention, ACLs, etc., such discarded
traffic could be streamed (potentially at a sampling rate of 1:1,
i.e. every packet) off-box for analysis to determine if the observed
behavior was expected, or trigger alerts that QoS policies may be
having adverse effects on the network. The ability to include the
packet payload provides additional context, allowing examination of
the platform behavior and affected policies.
This proposal addresses this use case by allowing samplers which have
such capabilities to communicate to the receiver: 1) drop
codes(reasons) that are known, 2) the semantics of those codes, and
3) the specific bit formats for the receiver to use when decoding.
2.3. Use Case 3: Standardization
Standardizing the way these data streams are formed and communicated
between the Replicators, Clients, and Receivers in a fashion that
allows vendors flexibility in what work the ASIC has to do to support
sampled streaming (by allowing communicating of an extremely dynamic
header in a manner than control planes can manage) allows systems to
be used between all platforms in an interoperable fashion. The
alternative is to build independent systems for each packet
replication solution that may end up being developed, resulting in
much higher costs for an overall solution.
This proposal addresses this use case by allowing the sampled packet
header to provide varying metadata fields, without mandating specific
positions or widths. This arrangement of fields and their format is
a function of the Replicator, and information about how to handle
this data is exchanged between the Replicator, Client, and Receiver
at the initialization of the session. The motivation for such
latitude in encoding and sizing is quite intentional, as it permits
widely varying capabilities within the Replicators.
2.4. Use Case 4: Security Automation
An automated security platform can utilize this proposal to set up a
"normal security analysis" stream at a very low sampling rate (for
example, 1 in 20,000) for constant monitoring at various points
throughout the network. Upon seeing something it deems
'interesting', or by manual input, it can add in an additional,
targeted, stream, at a very high sampling rate (potentially 1:1) for
detailed analysis and mitigation efforts.
Gray & Wobker Expires 2 July 2020 [Page 6]
Internet-Draft Sampled Traffic Streaming December 2019
Examples of past incidents where this may have been useful are the
NTP MONLIST attacks, DNS attacks, or DDoS attacks (although 1:1 would
most likely not be used in a DDoS case, unless performing the initial
data collection).
The security platform could potentially then use the collected
packets to generate an auto-mitigation plan based on heuristics
(i.e., 99% of this sudden burst of traffic has something in common,
deploy mitigation targeting that.)
3. Stream Setup
The configuration and setup between the Client and the Replicator
utilizes the YANG model as listed in Appendix B and any supported
configuration method (NETCONF, RESTCONF, gRPC, etc.). The tree
output of this model, as provided in Appendix A is provided as an aid
to understanding the interactions and tree structure as described in
this document.
3.1. Client queries Replicator for Points
A Client MUST first request from the Replicator the available
configurations via the 'points' branch, which provides the following
information:
* 'name' - The name of the Point. This serves as a key, and SHOULD
NOT be interpreted by software as anything other than a possibly-
human-readable uniquely identifying value. A Replicator MAY
choose to use an internal path, an encoded address, or any other
value of its choosing.
* 'interfaces' - The physical interfaces this Point is servicing. A
Replicator MAY offer the same interfaces under different Points,
with a different set of options. A Replicator MAY not offer a
Point for every interface available on the system.
* 'filters' - What filters can be applied (for example, against
certain IP fields, against parts of the frame, etc.). A
Replicator MAY not be able to honor every combination of filters
submitted in a request, or MAY not offer any filtering capability
at all. A Replicator MAY only be able to support a limited number
of filters, which MAY be returned in in the 'max-filters' branch.
* 'min-ratio' and 'max-ratio' - Minimum and maximum sampling rates
possible at this point. These are provided as a number N,
denoting one sample will be returned for every N. A Replicator
MAY not be able to offer a 'min-ratio' of 1 (i.e. every packet).
Gray & Wobker Expires 2 July 2020 [Page 7]
Internet-Draft Sampled Traffic Streaming December 2019
* 'samplers' - A list of any current samplers already active on this
Point as requested by this Client, and the branch manipulated in
the next section. A Replicator SHOULD NOT inform a Client about
the sampling sessions from other Clients.
* Optionally, the maximum frame length the Point can replicate into
the sample in 'max-frame-length-copy'.
* Optionally, the maximum offset into a frame the Point can inspect
in 'max-frame-depth-inspect'.
* Optionally, the maximum number of samplers that this Point can
accommodate in 'max-samplers'. A Client MUST still check for
success, as highly complex filters may reduce the amount of
replication the Point can do from this stated maximum.
3.2. Client submits a request to the Replicator
The Client then can request one or more streams to be set up on the
Replicator, taking into consideration the provided information. This
is performed by sending a request via adding an entry to the
'samplers' list in the 'points' branch and filling in the parameters
listed below:
* 'name' MUST be unique in the list, and MAY be any valid string
value up to 255 characters. The Replicator MUST isolate
namespaces between Clients (as one Client SHOULD NOT be able to
see other Clients' entries).
* 'destination' sets the transport mechanism and Receiver address.
It should be noted that the Client and Receiver MAY be separate
devices. The mechanism of exchanging information between the
Client and Receiver about this setup process is outside the scope
of this document. At present, the only supported transport
mechanism is a UDP tunnel, as detailed below in Section 4.
* 'client-heartbeat' MUST be set to 0.
* The desired sampling rate ('ratio'), along with what degree of
variance the Client can accept ('min-ratio' and 'max-ratio'). For
example, the client may request a 1 in 2000 rate, but specify a
range in the variance of 1900-2100. A proposal may come back with
the sampling rate offered of 1 in 2048, due to restrictions on the
Replicator.
* Optionally, one, or more filters in the 'filters' container, as
seen in the 'filter-type' typedef in the Yang model. Generally, a
Gray & Wobker Expires 2 July 2020 [Page 8]
Internet-Draft Sampled Traffic Streaming December 2019
Client would filter at least on a specific interface and
direction, but many other filter options are possible.
When the client is done with its configuration, it MUST set 'status'
to the 'client-request-complete' value, and the 'request' branch MUST
be read-only from this point forward.
3.2.1. Filtering Details
The filtering discussed above is designed to be as flexible as the
Replicator can realistically support. There are a few cases worth
discussing in detail, which are covered here.
3.2.1.1. Interfaces
All of the use cases focus on filtering to specific interface(s) to
filter on. A Replicator MAY, at its discretion, offer some or all of
its possible physical interfaces, offer logical interfaces (i.e.
routed interfaces on a port or VLAN, or subscriber interfaces), or
LAG interfaces. LAGs may be especially tricky, as the member ports
of the LAG may span line cards of different capturing capabilities.
Replicators SHOULD make an attempt to offer LAGs if all ports are of
identical capability, and MAY offer them in the case where they are
not, with a lowest-common capability set. Clients SHOULD NOT expect
LAG functionality to be present, and SHOULD be prepared to set up
separate sessions on each of the individual member ports if the
Replicator does not offer the LAG, or offers it with an insufficient
set.
3.3. Replicator offers Proposals
Upon receiving the 'status' change to 'client-request-complete', the
Replicator updates the 'proposals' branch. This branch details zero,
one, or more ways the Replicator can fulfill the sampling request.
While generally there will only be zero or one proposals, a
Replicator MAY offer more. For example, matching a sampling rate
exactly would result in performance loss but a 'close enough' option
can be offered that does not, or offers of what headers can be
captured in the resulting stream. Each proposal includes a unique ID
number, allowing the Client to select one, as detailed below.
If the Replicator is unable to provide any Proposals, the 'proposals'
list MUST be empty, a human-readable error message MAY be returned in
the 'proposal-error' field, then the 'status' field MUST be set to
'replicator-proposal-error'.
If the Replicator was able to provide Proposals, it MUST set the
'status' field to 'replicator-proposals-available' when it is
Gray & Wobker Expires 2 July 2020 [Page 9]
Internet-Draft Sampled Traffic Streaming December 2019
finished, and the 'proposals' branch MUST be read-only until the
Client finishes the Proposal selection step below.
Part of each Proposal is a 'stream-format' branch, which informs the
Client of the packet format the Receiver will be receiving. This
format completely defines the entirety of the resulting data flow
format besides the outer UDP wrapper - there is no normative format.
A couple non-normative examples of what may result are provided in
Section 4.
To adequately addresses the use cases stated above, a Replicator
SHOULD support as a minimum set of capabilities:
* An action field that denotes a pass or drop (ideally with drop
reason)
* Capturing at least 128 octets of payload
* The original frame length
* Sampling rates up to 1:1 (i.e. every packet is replicated), and
down to 1:20000 or smaller.
* Having different sampling sessions having different sampling rates
(to allow a "general" session to be watching a broad selection of
traffic, and more specific sessions targeting exact flows or
situations)
* At least two sessions per physical interface
* Filtering on ingress port
* Filtering on action
* Filtering on direction of traffic
3.4. Client selects a Proposal
Upon either a notification or detection that the 'status' field has
been updated, the Client then may then set the 'proposal-selected'
entry to the value of the desired ID offered in 'proposals', and then
set 'status' to 'client-proposal selected'. At this point, the
Replicator:
* MAY remove unnecessary branches in the 'proposals' list, but MUST
retain the selected one.
Gray & Wobker Expires 2 July 2020 [Page 10]
Internet-Draft Sampled Traffic Streaming December 2019
* MUST either install the requested sampling stream if possible,
then MUST set 'status' to 'replicator-install-success'. If it
cannot, it MAY set 'install-error' to a human-readable error
message and MUST set 'status' to 'replicator-install-error'.
* If the Proposal selected includes any of the 'dropped-' action-
types as a filter, or does not specify an action-type filter at
all, a Replicator MUST install the requested sampling before any
filtering actions occur to the stream, as the sampling session is
explicitly interested in pre-drop traffic.
* If the Proposal selected does not include any of the 'dropped-'
action-types as a filter, a Replicator MUST install the requested
sampling after any filtering actions occur to the stream, to
ensure the sampling ratio remains correct.
3.5. Ending sampling and cleanup
When a Client is finished with a sampling session, it deletes its
entry in the 'samplers' tree to terminate a sampling session.
Otherwise, a Client MUST refresh its entry by setting 'client-
heartbeat' to 0 at least every 3600 seconds. The 'client-heartbeat'
is then incremented by the Replicator. If 'client-heartbeat' exceeds
3600, the Replicator SHOULD consider the sampling configuration and
any associated sampling session no longer necessary, terminate the
sampling, and delete the entry. A Replicator MAY allow configuration
to increase this timeout.
4. Data Stream Format
After the stream setup has been completed, the Receiver MUST use the
stream-format data that the Replicator has calculated in its
proposal. The Client and Receiver MUST NOT assume that the stream-
format data is consistent between one stream setup and any other
(there may be different versions of ASICs, different capabilities,
different versions of operating systems, or different filters may
yield a different format), or that the payload is always at the end
(it could appear at the beginning or in the middle, and sufficient
data is provided by the other fields to extract the data correctly).
The stream-format data provides the Client with what information is
provided at what location in the resulting packet. The Replicator
MUST follow the expectation that is provided in these fields.
There is one captured packet per encapsulated packet, and thus the
outer encapsulation length can be used to deduce the length of one
variable-length field (designated by a field length of 0) contained
within. If there is more than one variable-length field, a matching
"-size"; field type MUST be provided for all but one of the variable-
Gray & Wobker Expires 2 July 2020 [Page 11]
Internet-Draft Sampled Traffic Streaming December 2019
length fields (as a single variable length can be deduced from the
wrapper length).
This means there is no normative packet format or data layout - a
large point of this specification is to allow that packet format to
be negotiated and decided between the Client and Replicator, with the
information passed back via the stream-format data.
One example of what the resulting packet may look like (but not a
normative listing of what it is - the actual format can be any
combination of fields, of any size, in any order), the data inside
the resulting data stream after the UDP tunnel header may look like
the following:
Example 1: Packet layout
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Incoming Port | Timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Act| Frame Length | Internal Data 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Payload |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2
This non-normative example may be associated with a stream-format as
per the following table:
+-----------+-------+------------------------+----------------------+
| Field | Field | Field Type | Field Type-Data |
| Name | Size | | |
+===========+=======+========================+======================+
| Incoming | 8 | port-ingress | A listing of |
| port | | | values that may be |
| | | | seen in this |
| | | | field, mapped to |
| | | | interface-refs |
| | | | from [RFC8343]. |
+-----------+-------+------------------------+----------------------+
| Timestamp | 24 | timestamp-nsec-ingress | Two 32-bit numbers |
| | | | giving when the |
| | | | "0" of this field |
Gray & Wobker Expires 2 July 2020 [Page 12]
Internet-Draft Sampled Traffic Streaming December 2019
| | | | is based off of, |
| | | | using the PTP |
| | | | Truncated |
| | | | Timestamp format. |
+-----------+-------+------------------------+----------------------+
| Act | 2 | action | A listing of |
| | | | values that may be |
| | | | seen in this |
| | | | field, mapped to |
| | | | action types |
| | | | (accepted, |
| | | | dropped, etc.) |
+-----------+-------+------------------------+----------------------+
| Frame | 17 | frame-length-ingress | Note that this |
| Length | | | denotes the |
| | | | original frame |
| | | | length - the |
| | | | payload field MAY |
| | | | not include the |
| | | | entire payload. |
+-----------+-------+------------------------+----------------------+
| Internal | 13 | padding | Note that this may |
| Data 1 | | | be ASIC-internal- |
| | | | only data, or some |
| | | | other information |
| | | | that would be |
| | | | expensive to prune |
| | | | out. 'padding' |
| | | | fields MUST have |
| | | | all content |
| | | | ignored. |
+-----------+-------+------------------------+----------------------+
| Payload | 0 | frame-payload-ingress | |
+-----------+-------+------------------------+----------------------+
Table 1: Example 1: Stream-format data
Another non-normative example, which is similar to the
[I-D.tuexen-opsawg-pcapng] enhanced packet block (EPB) format (and
thus, this Replicator may in fact be a server offering a tcpdump-
based backend using this frontend):
Gray & Wobker Expires 2 July 2020 [Page 13]
Internet-Draft Sampled Traffic Streaming December 2019
+-----------+-------+--------------------+---------------------+
| Field | Field | Field Type | Field Type-Data |
| Name | Size | | |
+===========+=======+====================+=====================+
| Interface | 32 | port | A listing of values |
| ID | | | that may be seen in |
| | | | this field, mapped |
| | | | to interface-refs |
| | | | from [RFC8343]. |
+-----------+-------+--------------------+---------------------+
| Timestamp | 64 | timestamp-msec | Two 32-bit numbers |
| | | | giving when the "0" |
| | | | of this field is |
| | | | based off of, using |
| | | | the PTP Truncated |
| | | | Timestamp format. |
+-----------+-------+--------------------+---------------------+
| Captured | 32 | frame-payload-size | Note: This allows |
| Packet | | | us to have the |
| Length | | | Options field as |
| | | | our real variable |
| | | | length field. |
+-----------+-------+--------------------+---------------------+
| Original | 32 | frame-length | |
| Packet | | | |
| Length | | | |
+-----------+-------+--------------------+---------------------+
| Packet | 0 | frame-payload | |
| Data | | | |
+-----------+-------+--------------------+---------------------+
| Options | 0 | padding | |
+-----------+-------+--------------------+---------------------+
Table 2: Packet-format response example 2
To restate the prior note, the above is purely an example of what the
format could be - the actual format used is negotiated between the
Client and Replicator, and can have practically any layout, with any
additional fields.
A Client SHOULD take efforts to be notified when a change has
occurred on the Replicator (e.g., port or line card changes, device
reboot, etc.), and re-verify and re-apply as needed its sampled
streaming configurations when such a change is detected.
Gray & Wobker Expires 2 July 2020 [Page 14]
Internet-Draft Sampled Traffic Streaming December 2019
5. IANA Considerations
This document defines a new UDP port number, entitled "Sampled
Streaming", and assigns a value of TBD1 from the Service Name and
Transport Protocol Port Number Registry
https://www.iana.org/assignments/service-names-port-numbers/service-
names-port-numbers.xhtml:
+------+-------------------+
| Tag | Description |
+======+===================+
| TBD1 | Sampled Streaming |
+------+-------------------+
Table 3
This document requests registration of a URI in the "IETF XML
Registry" RFC 3688 [RFC3688]. Following the format in RFC 3688, the
following registration is suggested:
URI: urn:ietf:params:xml:ns:yang:ietf-sampled-streaming
Registrant Contact: The IESG.
XML: N/A, the requested URI is an XML namespace.
This document registers a YANG module in the "YANG Module Names"
registry RFC 6020 [RFC6020]:
name: ietf-sampled-streaming
namespace: urn:ietf:params:xml:ns:yang:ietf-sampled-streaming
prefix: ss
reference: This document
6. Security Considerations
Vendors and deployments must take into consideration that this
functionality allows a mirroring of traffic, with configurable
destinations and filters. Similar functionality already exists in
various remote packet mirroring systems, and similar considerations
should be taken. Filters utilizing the source port of TBD1 SHOULD be
applied at the edges of a provider's network to provide an additional
layer of security.
A Replicator SHOULD ensure that Clients can only see their own
entries in the 'samplers', and MUST ensure that once a Client has
created an entry in the samplers list, only that same Client may re-
query or make changes to it.
Gray & Wobker Expires 2 July 2020 [Page 15]
Internet-Draft Sampled Traffic Streaming December 2019
7. Acknowledgments
The authors would like to thank Joe Clarke, Marek Hajduczenia, Brian
Harber, Paolo Lucente, Jim Rampley, and Dmytro Shytyi for their
reviews and providing helpful suggestions and feedback of this draft.
8. References
8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3688] Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688,
DOI 10.17487/RFC3688, January 2004,
<https://www.rfc-editor.org/info/rfc3688>.
[RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet
Sampling (PSAMP) Protocol Specifications", RFC 5476,
DOI 10.17487/RFC5476, March 2009,
<https://www.rfc-editor.org/info/rfc5476>.
[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for
the Network Configuration Protocol (NETCONF)", RFC 6020,
DOI 10.17487/RFC6020, October 2010,
<https://www.rfc-editor.org/info/rfc6020>.
[RFC6728] Muenz, G., Claise, B., and P. Aitken, "Configuration Data
Model for the IP Flow Information Export (IPFIX) and
Packet Sampling (PSAMP) Protocols", RFC 6728,
DOI 10.17487/RFC6728, October 2012,
<https://www.rfc-editor.org/info/rfc6728>.
[RFC8343] Bjorklund, M., "A YANG Data Model for Interface
Management", RFC 8343, DOI 10.17487/RFC8343, March 2018,
<https://www.rfc-editor.org/info/rfc8343>.
8.2. Informative References
[I-D.tuexen-opsawg-pcapng]
Tuexen, M., Risso, F., Bongertz, J., and G. Harris, "PCAP
Next Generation (PCAPNG) Dump File Format", Work in
Progress, Internet-Draft, draft-tuexen-opsawg-pcapng-00,
26 June 2014, <http://www.ietf.org/internet-drafts/draft-
tuexen-opsawg-pcapng-00.txt>.
Gray & Wobker Expires 2 July 2020 [Page 16]
Internet-Draft Sampled Traffic Streaming December 2019
Appendix A. Yang Model Tree Reference
module: ietf-sampled-streaming
+--rw points* [name]
+--rw name psamp:nameType
+--rw observationPoints* [name]
| +--rw name psamp:nameType
| +--ro observationPointId? uint32
| +--rw observationDomainId uint32
| +--rw ifName* ifNameType
| +--rw ifIndex* uint32
| +--rw entPhysicalName* string
| +--rw entPhysicalIndex* uint32
| +--rw direction? direction
+--rw selectors* [name]
| +--rw name psamp:nameType
| +--rw (Method)
| | +--:(selectAll)
| | | +--rw selectAll? empty
| | +--:(sampCountBased)
| | | +--rw sampCountBased {psampSampCountBased}?
| | | +--rw packetInterval uint32
| | | +--rw packetSpace uint32
| | +--:(sampTimeBased)
| | | +--rw sampTimeBased {psampSampTimeBased}?
| | | +--rw timeInterval uint32
| | | +--rw timeSpace uint32
| | +--:(sampRandOutOfN)
| | | +--rw sampRandOutOfN {psampSampRandOutOfN}?
| | | +--rw size uint32
| | | +--rw population uint32
| | +--:(sampUniProb)
| | | +--rw sampUniProb {psampSampUniProb}?
| | | +--rw probability decimal64
| | +--:(filterMatch)
| | | +--rw filterMatch {psampFilterMatch}?
| | | +--rw (nameOrId)
| | | | +--:(ieName)
| | | | | +--rw ieName? ieNameType
| | | | +--:(ieId)
| | | | +--rw ieId? ieIdType
| | | +--rw ieEnterpriseNumber? uint32
| | | +--rw value string
| | +--:(filterHash)
| | +--rw filterHash {psampFilterHash}?
| | +--rw hashFunction? identityref
| | +--rw initializerValue? uint64
| | +--rw ipPayloadOffset? uint64
Gray & Wobker Expires 2 July 2020 [Page 17]
Internet-Draft Sampled Traffic Streaming December 2019
| | +--rw ipPayloadSize? uint64
| | +--rw digestOutput? boolean
| | +--ro outputRangeMin? uint64
| | +--ro outputRangeMax? uint64
| | +--rw selectedRange* [name]
| | +--rw name nameType
| | +--rw min? uint64
| | +--rw max? uint64
| +--ro packetsObserved? yang:counter64
| +--ro packetsDropped? yang:counter64
| +--ro selectorDiscontinuityTime? yang:date-and-time
+--ro filters* []
| +--ro filter filter-type
+--ro max-samplers? uint32
+--ro max-filters? uint32
+--ro max-frame-length-copy? uint16
+--ro max-frame-depth-inspect? uint16
+--rw samplers* [name]
+--rw name string
+--rw status status-type
+--rw client-heartbeat uint32
+--rw destination
| +--rw type destination-type
| +--rw udp-parameters
| +--rw destination-ip inet:ip-address-no-zone
| +--rw destination-port inet:port-number
+--rw request
| +--rw filters
| | +--rw name? string
| | +--rw interfaces* [int]
| | | +--rw int if:interface-ref
| | +--rw actions* [action]
| | | +--rw action action-type
| | +--rw direction? psamp:direction
| | +--rw type filter-type
| | +--rw ipv4-address? inet:ipv4-address-no-zone
| | +--rw ipv6-address? inet:ipv6-address-no-zone
| | +--rw version? inet:ip-version
| | +--rw frame-payload
| | | +--rw offset? uint16
| | | +--rw match? binary
| | +--rw frame-length? uint16
| +--rw selector
| | +--rw (Method)