Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limited number of VRFs in VPNv4 #16744

Open
2 tasks done
mnaiem81 opened this issue Sep 4, 2024 · 4 comments
Open
2 tasks done

Limited number of VRFs in VPNv4 #16744

mnaiem81 opened this issue Sep 4, 2024 · 4 comments
Labels
triage Needs further investigation

Comments

@mnaiem81
Copy link

mnaiem81 commented Sep 4, 2024 •
edited
Loading

Description

I see an issue with v9.1 where I deploy +255 VRFs per server with dynamically assigned VPN label using 'label vpn export auto'

Two below scenarios happen here:

  1. If it's just booted with BGP config with at least one VRF, it assigns 128 labels pool. usually [16-143] or [80-207]
FRR1# show debugging label-table
Proto ldp: [16/79]
Proto bgp: [80/207]

FRR2# show debugging label-table
Proto bgp: [16/143]
  1. If its booted with BGP config with +255 VRFs, it assigns 255 labels pool
FRR1# show debugging label-table
Proto ldp: [16/79]
Proto ldp: [80/143]
Proto ldp: [144/207]
Proto ldp: [208/271]
Proto ldp: [272/335]
Proto ldp: [336/399]
Proto ldp: [400/463]
Proto bgp: [464/719]

What is the problem?

  • Any new configured VRF above the assigned pool, will never assigned a label which breaks VPNv4. For example, VRF#256 is broken.
  • If any VRF is deleted, its assigned label is freed up but never used for new VRFs until a reboot to re-order assigned labels.

Output of show mpls table below from one FRR with 260 VRFs, the last configured 15 VRFs not working because no labels for them although from table label#466,469,471,472, etc are free from previously deleted VRFs.

464            BGP   013f10d46      -
465            BGP   0144d3d78      -
467            BGP   85405a33d      -
468            BGP   04480589a      -
470            BGP   088c7620d      -
473            BGP   0cd47cb93      -
476            BGP   137929dcf      -
477            BGP   13d51117f      -
478            BGP   1682af3f8      -
479            BGP   17cf5b54b      -
482            BGP   1d311c183      -
485            BGP   221dd6776      -
486            BGP   2276e52f7      -
487            BGP   234790c7a      -
488            BGP   24637169a      -
489            BGP   24d50840a      -
490            BGP   25fcc4268      -
491            BGP   264e04eb4      -
492            BGP   270339265      -
494            BGP   2b01c4086      -
496            BGP   2dbadccb9      -
497            BGP   2dd12c61f      -
499            BGP   2fd1ea814      -
500            BGP   304010694      -
501            BGP   31a9a8394      -
502            BGP   325003920      -
503            BGP   36aecdc61      -
504            BGP   38480e887      -
506            BGP   395f6d4ef      -
508            BGP   3be3063f0      -
510            BGP   3f423b026      -
515            BGP   468f463d0      -
517            BGP   4a0f75177      -
518            BGP   4d5938d5c      -
520            BGP   4de8b43f8      -
522            BGP   4eb560bb1      -
523            BGP   4f5d2dc82      -
524            BGP   4f7f5845b      -
526            BGP   52876563c      -
527            BGP   53af66025      -
529            BGP   56c567d50      -
531            BGP   5b7fe590c      -
533            BGP   60bec47fd      -
534            BGP   62a844ac2      -
538            BGP   6849d06c0      -
539            BGP   68bc70d71      -
540            BGP   68f31bb39      -
541            BGP   6a2cbaf78      -
542            BGP   6bd6806ad      -
543            BGP   6c197e6a1      -
544            BGP   70194d54c      -
547            BGP   730a9c284      -
549            BGP   7376d07d6      -
550            BGP   74b8906f7      -
551            BGP   794feaf76      -
553            BGP   7c177a060      -
554            BGP   7d332668a      -
555            BGP   7e1f0d776      -
557            BGP   7fc1a3acb      -
558            BGP   8091152e8      -
559            BGP   87c4c6969      -
563            BGP   89bd5c304      -
564            BGP   8a0b5b6eb      -
565            BGP   8a73b4ac1      -
566            BGP   8abb59e9b      -
567            BGP   8b4dee69b      -
568            BGP   8ba361a06      -
569            BGP   8bb3bd80d      -
570            BGP   8bb61157f      -
571            BGP   8cf2520f7      -
572            BGP   90755d43d      -
573            BGP   90cf89179      -
574            BGP   921accf27      -
575            BGP   925743e39      -
581            BGP   9b6e3bbc9      -
583            BGP   9dbb83dba      -
584            BGP   9fc94b62f      -
586            BGP   a0f0db387      -
587            BGP   a2e7ddf3f      -
588            BGP   a8dc7065b      -
589            BGP   aa1a61a66      -
590            BGP   aaf51722f      -
591            BGP   ab7a3b36b      -
592            BGP   ab9e59768      -
593            BGP   acfff2092      -
596            BGP   b4eb05d0f      -
597            BGP   b4fdfcb24      -
598            BGP   bb2671448      -
599            BGP   bcb4d8515      -
602            BGP   c01409859      -
603            BGP   c0828ef85      -
607            BGP   c7fecf00d      -
608            BGP   ca2b1c849      -
609            BGP   caa319c8f      -
610            BGP   cb1b2a13a      -
611            BGP   d56284b7a      -
613            BGP   d79f758af      -
614            BGP   dba992c58      -
615            BGP   dccb5e998      -
616            BGP   de1f18d11      -
617            BGP   de5e82878      -
620            BGP   e76d836c0      -
621            BGP   ebb90d64d      -
624            BGP   f0193db13      -
627            BGP   f86e9f55b      -
629            BGP   fc7d503bc      -
630            BGP   fd558b91d      -
631            BGP   fff044d15      -
632            BGP   484d8660a      -
633            BGP   12a2acde5      -
634            BGP   49a0b1757      -
636            BGP   34a640b4a      -
637            BGP   fa186ad90      -
638            BGP   42df6832c      -
640            BGP   c80539e3c      -
643            BGP   cecda5258      -
645            BGP   261e40e7e      -
647            BGP   6c759f0e5      -
650            BGP   fba9691ac      -
653            BGP   596961922      -
655            BGP   f179584d9      -
656            BGP   566072ab3      -
657            BGP   c1a140bd1      -
661            BGP   d3b10c25e      -
664            BGP   7c3645892      -
665            BGP   171671901      -
668            BGP   ea6dabba2      -
669            BGP   e57d4fe6c      -
670            BGP   d1d2236b0      -
671            BGP   5d3f8986f      -
672            BGP   2550d01a3      -
675            BGP   1b01c73c4      -
676            BGP   7d4f521f6      -
678            BGP   696591593      -
684            BGP   828bb533c      -
685            BGP   28bb20ad3      -
688            BGP   92d9fb9d7      -
692            BGP   041cb7e24      -
698            BGP   9d967434c      -
702            BGP   4d60f1282      -
703            BGP   11a12983f      -
710            BGP   a896470ec      -
711            BGP   410b4b96d      -
713            BGP   d906abfdc      -
715            BGP   2af87759f      -
717            BGP   fc4db85bd      -
719            BGP   060b0a466      -

VRF config (nothing special)

 router bgp 65444 vrf 088c7620d
 no bgp ebgp-requires-policy
 no bgp suppress-duplicates
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 no bgp network import-check
 !
 address-family ipv4 unicast
  redistribute connected
  label vpn export auto
  rd vpn export 5655:7036
  rt vpn both 65000:7036
  export vpn
  import vpn
 exit-address-family
exit

Version

FRR1# show version
FRRouting 9.1_git (sevrer) on Linux(4.15.0-172-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--prefix=/usr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--libdir=/usr/lib' '--localstatedir=/var/run/frr' '--enable-rpki' '--enable-vtysh' '--enable-multipath=64' '--enable-vty-group=frrvty' '--enable-user=frr' '--enable-group=frr' '--enable-pcre2posix' '--enable-scripting' 'CC=gcc' 'CXX=g++'

How to reproduce

  • To reproduce, run 'show debugg label-table' to check assigned pool chunk.
    If it is 128, you need to create more than 128 VRFs and verify from show mpls table.
    if it is 256, you need to create more than 256 VRFs and verify from show mpls table.

  • Delete any VRF to free-up its assigned label and try to get it dynamically assigned again.

Expected behavior

Expected behaviors here:

  • Dynamic label pool could be anything between 16 - 1048575 and makes no sense to be limited to 128 or 256 BGP labels.
  • Pool should be dynamic and can assign new labels pool when needed without need to reboot or be limited to specific number.
  • Released labels should be available for re-assignment without need to reboot.

Actual behavior

Limitation of supported VRFs per server limits its usage

Additional context

No response

Checklist

  • I have searched the open issues for this bug.
  • I have not included sensitive information in this report.
@mnaiem81 mnaiem81 added the triage Needs further investigation label Sep 4, 2024
@pguibert6WIND
Copy link
Member

Hi @mnaiem81 , this looks like a label manager issue in BGP, where BGP does not know:

  • how to reuse freed labels
  • how to enlarge the label pool
    more information should be given:
show bgp labelpool chunks
show bgp labelpool inuse
show bgp labelpool ledger
show bgp labelpool requests
show bgp labelpool summary

@mnaiem81
Copy link
Author

mnaiem81 commented Sep 5, 2024

Hi @pguibert6WIND,
Please find output below from FRR that currently has 227 VRFs configured some of them has no labels assigned because of old VRFs were deleted and new ones added after a point where more than 256 were configured.

FRR1# show bgp labelpool chunks
     First       Last       Size      nfree
-------------------------------------------
       464        719        256          0
       
FRR1# show bgp labelpool inuse
Prefix                Label
---------------------------
VRF                        464
VRF                        465
VRF                        466
VRF                        467
VRF                        468
VRF                        469
VRF                        470
VRF                        471
VRF                        472
VRF                        473
VRF                        474
VRF                        475
VRF                        476
VRF                        477
VRF                        478
VRF                        479
VRF                        480
VRF                        481
VRF                        482
VRF                        483
VRF                        484
VRF                        485
VRF                        486
VRF                        487
VRF                        488
VRF                        489
VRF                        490
VRF                        491
VRF                        492
VRF                        493
VRF                        494
VRF                        495
VRF                        496
VRF                        497
VRF                        498
VRF                        499
VRF                        500
VRF                        501
VRF                        502
VRF                        503
VRF                        504
VRF                        505
VRF                        506
VRF                        507
VRF                        508
VRF                        509
VRF                        510
VRF                        511
VRF                        512
VRF                        513
VRF                        514
VRF                        515
VRF                        516
VRF                        517
VRF                        518
VRF                        519
VRF                        520
VRF                        521
VRF                        522
VRF                        523
VRF                        524
VRF                        525
VRF                        526
VRF                        527
VRF                        528
VRF                        529
VRF                        530
VRF                        531
VRF                        532
VRF                        533
VRF                        534
VRF                        535
VRF                        536
VRF                        537
VRF                        538
VRF                        539
VRF                        540
VRF                        541
VRF                        542
VRF                        543
VRF                        544
VRF                        545
VRF                        546
VRF                        547
VRF                        548
VRF                        549
VRF                        550
VRF                        551
VRF                        552
VRF                        553
VRF                        554
VRF                        555
VRF                        556
VRF                        557
VRF                        558
VRF                        559
VRF                        560
VRF                        561
VRF                        562
VRF                        563
VRF                        564
VRF                        565
VRF                        566
VRF                        567
VRF                        568
VRF                        569
VRF                        570
VRF                        571
VRF                        572
VRF                        573
VRF                        574
VRF                        575
VRF                        576
VRF                        577
VRF                        578
VRF                        579
VRF                        580
VRF                        581
VRF                        582
VRF                        583
VRF                        584
VRF                        585
VRF                        586
VRF                        587
VRF                        588
VRF                        589
VRF                        590
VRF                        591
VRF                        592
VRF                        593
VRF                        594
VRF                        595
VRF                        596
VRF                        597
VRF                        598
VRF                        599
VRF                        600
VRF                        601
VRF                        602
VRF                        603
VRF                        604
VRF                        605
VRF                        606
VRF                        607
VRF                        608
VRF                        609
VRF                        610
VRF                        611
VRF                        612
VRF                        613
VRF                        614
VRF                        615
VRF                        616
VRF                        617
VRF                        618
VRF                        619
VRF                        620
VRF                        621
VRF                        622
VRF                        623
VRF                        624
VRF                        625
VRF                        626
VRF                        627
VRF                        628
VRF                        629
VRF                        630
VRF                        631
VRF                        632
VRF                        633
VRF                        634
VRF                        635
VRF                        636
VRF                        637
VRF                        638
VRF                        639
VRF                        640
VRF                        641
VRF                        642
VRF                        643
VRF                        644
VRF                        645
VRF                        646
VRF                        647
VRF                        648
VRF                        649
VRF                        650
VRF                        651
VRF                        652
VRF                        653
VRF                        654
VRF                        655
VRF                        656
VRF                        657
VRF                        658
VRF                        659
VRF                        660
VRF                        661
VRF                        662
VRF                        663
VRF                        664
VRF                        665
VRF                        666
VRF                        667
VRF                        668
VRF                        669
VRF                        670
VRF                        671
VRF                        672
VRF                        673
VRF                        674
VRF                        675
VRF                        676
VRF                        677
VRF                        678
VRF                        679
VRF                        680
VRF                        681
VRF                        682
VRF                        683
VRF                        684
VRF                        685
VRF                        686
VRF                        687
VRF                        688
VRF                        689
VRF                        690
VRF                        691
VRF                        692
VRF                        693
VRF                        694
VRF                        695
VRF                        696
VRF                        697
VRF                        698
VRF                        699
VRF                        700
VRF                        701
VRF                        702
VRF                        703
VRF                        704
VRF                        705
VRF                        706
VRF                        707
VRF                        708
VRF                        709
VRF                        710
VRF                        711
VRF                        712
VRF                        713
VRF                        714
VRF                        715
VRF                        716
VRF                        717
VRF                        718
VRF                        719
FRR1# show bgp labelpool ledger
Prefix                Label
---------------------------
VRF                        675
VRF                        676
VRF                        677
VRF                        678
VRF                        679
VRF                        681
VRF                        684
VRF                        686
VRF                        687
VRF                        697
VRF                        703
VRF                        689
VRF                        4294967295
VRF                        709
VRF                        4294967295
VRF                        710
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        654
VRF                        690
VRF                        717
VRF                        655
VRF                        656
VRF                        657
VRF                        658
VRF                        683
VRF                        706
VRF                        707
VRF                        708
VRF                        718
VRF                        4294967295
VRF                        4294967295
VRF                        659
VRF                        660
VRF                        699
VRF                        704
VRF                        719
VRF                        661
VRF                        662
VRF                        700
VRF                        705
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        663
VRF                        664
VRF                        665
VRF                        666
VRF                        701
VRF                        4294967295
VRF                        4294967295
VRF                        667
VRF                        691
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        668
VRF                        669
VRF                        670
VRF                        671
VRF                        672
VRF                        673
VRF                        692
VRF                        674
VRF                        630
VRF                        631
VRF                        632
VRF                        633
VRF                        634
VRF                        635
VRF                        636
VRF                        637
VRF                        638
VRF                        639
VRF                        712
VRF                        4294967295
VRF                        640
VRF                        641
VRF                        642
VRF                        680
VRF                        713
VRF                        643
VRF                        644
VRF                        645
VRF                        646
VRF                        647
VRF                        648
VRF                        695
VRF                        714
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        649
VRF                        696
VRF                        715
VRF                        650
VRF                        651
VRF                        682
VRF                        685
VRF                        652
VRF                        716
VRF                        4294967295
VRF                        653
VRF                        606
VRF                        607
VRF                        608
VRF                        609
VRF                        610
VRF                        611
VRF                        612
VRF                        4294967295
VRF                        613
VRF                        614
VRF                        615
VRF                        616
VRF                        617
VRF                        618
VRF                        619
VRF                        688
VRF                        620
VRF                        621
VRF                        622
VRF                        623
VRF                        624
VRF                        625
VRF                        694
VRF                        4294967295
VRF                        626
VRF                        627
VRF                        628
VRF                        4294967295
VRF                        629
VRF                        582
VRF                        693
VRF                        4294967295
VRF                        583
VRF                        584
VRF                        585
VRF                        586
VRF                        587
VRF                        588
VRF                        589
VRF                        590
VRF                        591
VRF                        592
VRF                        593
VRF                        594
VRF                        595
VRF                        596
VRF                        597
VRF                        598
VRF                        599
VRF                        600
VRF                        601
VRF                        602
VRF                        603
VRF                        604
VRF                        605
VRF                        4294967295
VRF                        4294967295
VRF                        4294967295
VRF                        558
VRF                        559
VRF                        560
VRF                        561
VRF                        562
VRF                        563
VRF                        564
VRF                        565
VRF                        566
VRF                        567
VRF                        568
VRF                        569
VRF                        570
VRF                        571
VRF                        572
VRF                        573
VRF                        574
VRF                        575
VRF                        576
VRF                        4294967295
VRF                        577
VRF                        578
VRF                        579
VRF                        580
VRF                        581
VRF                        546
VRF                        4294967295
VRF                        547
VRF                        548
VRF                        549
VRF                        550
VRF                        551
VRF                        552
VRF                        553
VRF                        554
VRF                        555
VRF                        556
VRF                        557
VRF                        534
VRF                        535
VRF                        536
VRF                        702
VRF                        537
VRF                        4294967295
VRF                        538
VRF                        539
VRF                        540
VRF                        541
VRF                        542
VRF                        543
VRF                        544
VRF                        545
VRF                        522
VRF                        523
VRF                        524
VRF                        525
VRF                        526
VRF                        527
VRF                        528
VRF                        529
VRF                        530
VRF                        4294967295
VRF                        531
VRF                        532
VRF                        533
VRF                        510
VRF                        511
VRF                        512
VRF                        513
VRF                        514
VRF                        515
VRF                        516
VRF                        517
VRF                        518
VRF                        519
VRF                        711
VRF                        520
VRF                        521
VRF                        504
VRF                        505
VRF                        506
VRF                        507
VRF                        508
VRF                        509
VRF                        498
VRF                        499
VRF                        500
VRF                        501
VRF                        502
VRF                        503
VRF                        492
VRF                        493
VRF                        494
VRF                        495
VRF                        496
VRF                        497
VRF                        486
VRF                        487
VRF                        488
VRF                        489
VRF                        490
VRF                        491
VRF                        483
VRF                        484
VRF                        485
VRF                        480
VRF                        481
VRF                        4294967295
VRF                        482
VRF                        477
VRF                        478
VRF                        479
VRF                        474
VRF                        4294967295
VRF                        475
VRF                        4294967295
VRF                        4294967295
VRF                        476
VRF                        471
VRF                        472
VRF                        473
VRF                        468
VRF                        469
VRF                        4294967295
VRF                        470
VRF                        465
VRF                        466
VRF                        4294967295
VRF                        467
VRF                        698
VRF                        464
FRR1#  show bgp labelpool requests
Prefix
----------------
FRR1# show bgp labelpool summary
Labelpool Summary
-----------------
Ledger:       302
InUse:        256
Requests:     0
LabelChunks:  1
Pending:      256
Reconnects:   1

@mnaiem81
Copy link
Author

Is this the same for FRR v10 ? or a plan to fix?

@pguibert6WIND
Copy link
Member

pguibert6WIND commented Sep 26, 2024 •
edited
Loading

Is this the same for FRR v10 ? or a plan to fix?

I think the issue still remains, and a fix needs to be done.
BGP label manager issue.

No plan to fix as of today.
you should contact your support team:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage Needs further investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pguibert6WIND @mnaiem81