forked from svn2github/pcre2
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
1523 lines (1108 loc) · 69.1 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Change Log for PCRE2
--------------------
Version 10.30-RC1 18-July-2017
------------------------------
1. The main interpreter, pcre2_match(), has been refactored into a new version
that does not use recursive function calls (and therefore the stack) for
remembering backtracking positions. This makes --disable-stack-for-recursion a
NOOP. The new implementation allows backtracking into recursive group calls in
patterns, making it more compatible with Perl, and also fixes some other
hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
the old code had a number of fudges to try to reduce stack usage. It seems to
run no slower than the old code.
A number of bugs in the refactored code were subsequently fixed during testing
before release, but after the code was made available in the repository. These
bugs were never in fully released code, but are noted here for the record.
(a) If a pattern had fewer capturing parentheses than the ovector supplied in
the match data block, a memory error (detectable by ASAN) occurred after
a match, because the external block was being set from non-existent
internal ovector fields. Fixes oss-fuzz issue 781.
(b) A pattern with very many capturing parentheses (when the internal frame
size was greater than the initial frame vector on the stack) caused a
crash. A vector on the heap is now set up at the start of matching if the
vector on the stack is not big enough to handle at least 10 frames.
Fixes oss-fuzz issue 783.
(c) Handling of (*VERB)s in recursions was wrong in some cases.
(d) Captures in negative assertions that were used as conditions were not
happening if the assertion matched via (*ACCEPT).
(e) Mark values were not being passed out of recursions.
(f) Refactor some code in do_callout() to avoid picky compiler warnings about
negative indices. Fixes oss-fuzz issue 1454.
(g) Similarly refactor the way the variable length ovector is addressed for
similar reasons. Fixes oss-fuzz issue 1465.
2. Now that pcre2_match() no longer uses recursive function calls (see above),
the "match limit recursion" value seems misnamed. It still exists, and limits
the depth of tree that is searched. To avoid future confusion, it has been
renamed as "depth limit" in all relevant places (--with-depth-limit,
(*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
available for backwards compatibility.
3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
(a) Check for malloc failures when getting memory for the ovector (POSIX) or
the match data block (non-POSIX).
4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
for a character with a code point greater than 0x10ffff (the Unicode maximum)
caused a crash.
5. If a lookbehind assertion that contained a back reference to a group
appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
undefined actions (often a segmentation fault) could occur, depending on what
other options were set. An example assertion is (?<!\1(abc)) where the
reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
pcre2test to use it to output the frame size when the "framesize" modifier is
given.
7. Reworked the recursive pattern matching in the JIT compiler to follow the
interpreter changes.
8. When the zero_terminate modifier was specified on a pcre2test subject line
for global matching, unpredictable things could happen. For example, in UTF-8
mode, the pattern //g,zero_terminate read random memory when matched against an
empty string with zero_terminate. This was a bug in pcre2test, not the library.
9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
of the section that is compiled when Unix-style directory scanning is
available, and into a new section that is always compiled for Windows.
10. In pcre2test, explicitly close the file after an error during serialization
or deserialization (the "load" or "save" commands).
11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
a NULL pattern pointer when Unicode support is available.
13. When the 32-bit library was being tested by pcre2test, error messages that
were longer than 64 code units could cause a buffer overflow. This was a bug in
pcre2test.
14. The alternative matching function, pcre2_dfa_match() misbehaved if it
encountered a character class with a possessive repeat, for example [a-f]{3}+.
15. The depth (formerly recursion) limit now applies to DFA matching (as
of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
matching to find the minimum value for this limit.
16. Since 10.21, if pcre2_match() was called with a null context, default
memory allocation functions were used instead of whatever was used when the
pattern was compiled.
17. Changes to the pcre2test "memory" modifier on a subject line. These apply
only to pcre2_match():
(a) Warn if null_context is set on both pattern and subject, because the
memory details cannot then be shown.
(b) Remember (up to a certain number of) memory allocations and their
lengths, and list only the lengths, so as to be system-independent.
(In practice, the new interpreter never has more than 2 blocks allocated
simultaneously.)
18. Make pcre2test detect an error return from pcre2_get_error_message(), give
a message, and abandon the run (this would have detected #13 above).
19. Implemented PCRE2_ENDANCHORED.
20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
the --output=text (-O) option and the inbuilt callout echo.
21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
single-branch conditions with a false condition (e.g. DEFINE) at the start of a
branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
anchored.
22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
heap limit along with other pattern information, and to find the minimum when
the find_limits modifier is set.
23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
pattern is set up so as to initialize any padding the compiler might have
included. This avoids valgrind warnings when a compiled pattern is copied, in
particular when it is serialized.
24. Remove a redundant line of code left in accidentally a long time ago.
25. Remove a duplication typo in pcre2_tables.c
26. Correct an incorrect cast in pcre2_valid_utf.c
27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
tests to improve coverage.
28. Some fixes/tidies as a result of looking at Coverity Scan output:
(a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
(b) Added some casts to avoid "suspicious implicit sign extension".
(c) Resource leaks in pcre2test in rare error cases.
(d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
for checking at compile time that tables are the right size.
(e) Add missing "fall through" comment.
29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.
30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.
31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
pcre2test, a crash could occur.
32. Make -bigstack in RunTest allocate a 64Mb stack (instead of 16 MB) so that
all the tests can run with clang's sanitizing options.
33. Implement extra compile options in the compile context and add the first
one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.
34. Implement newline type PCRE2_NEWLINE_NUL.
35. A lookbehind assertion that had a zero-length branch caused undefined
behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.
36. The match limit value now also applies to pcre2_dfa_match() as there are
patterns that can use up a lot of resources without necessarily recursing very
deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.
37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
38. Fix returned offsets from regexec() when REG_STARTEND is used with a
starting offset greater than zero.
39. Implement REG_PEND (GNU extension) for the POSIX wrapper.
40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
pattern lines.
41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.
42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
of pcre2grep.
43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:
(a) The -F option did not work for fixed strings containing \E.
(b) The -w option did not work for patterns with multiple branches.
44. Added configuration options for the SELinux compatible execmem allocator in
JIT.
45. Increased the limit for searching for a "must be present" code unit in
subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
much faster.
46. Arrange for anchored patterns to record and use "first code unit" data,
because this can give a fast "no match" without searching for a "required code
unit". Previously only non-anchored patterns did this.
47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.
48. Add the callout_no_where modifier to pcre2test.
49. Update extended grapheme breaking rules to the latest set that are in
Unicode Standard Annex #29.
50. Added experimental foreign pattern conversion facilities
(pcre2_pattern_convert() and friends).
51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
is defined in a system header in cygwin. Also modified some of the #ifdefs in
pcre2grep related to Windows and Cygwin support.
52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
character class is the last character in the class, Perl does not give a
warning. PCRE2 now also treats this as a literal.
53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
not doing so for [\d-X] (and similar escapes), as is documented.
Version 10.23 14-February-2017
------------------------------
1. Extended pcre2test with the utf8_input modifier so that it is able to
generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
PCRE2_UCP set, a negative character type such as \D in a positive class should
cause all characters greater than 255 to match, whatever else is in the class.
There was a bug that caused this not to happen if a Unicode property item was
added to such a class, for example [\D\P{Nd}] or [\W\pL].
3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
checking is now done in the pre-pass that identifies capturing groups. This has
reduced the amount of duplication and made the code tidier. While doing this,
some minor bugs and Perl incompatibilities were fixed, including:
(a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
of giving an invalid quantifier error.
(b) {0} can now be used after a group in a lookbehind assertion; previously
this caused an "assertion is not fixed length" error.
(c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
the name "DEFINE" exists. PCRE2 now does likewise.
(d) A recursion condition test such as (?(R2)...) must now refer to an
existing subpattern.
(e) A conditional recursion test such as (?(R)...) misbehaved if there was a
group whose name began with "R".
(f) When testing zero-terminated patterns under valgrind, the terminating
zero is now marked "no access". This catches bugs that would otherwise
show up only with non-zero-terminated patterns.
(g) A hyphen appearing immediately after a POSIX character class (for example
/[[:ascii:]-z]/) now generates an error. Perl does accept this as a
literal, but gives a warning, so it seems best to fail it in PCRE.
(h) An empty \Q\E sequence may appear after a callout that precedes an
assertion condition (it is, of course, ignored).
One effect of the refactoring is that some error numbers and messages have
changed, and the pattern offset given for compiling errors is not always the
right-most character that has been read. In particular, for a variable-length
lookbehind assertion it now points to the start of the assertion. Another
change is that when a callout appears before a group, the "length of next
pattern item" that is passed now just gives the length of the opening
parenthesis item, not the length of the whole group. A length of zero is now
given only for a callout at the end of the pattern. Automatic callouts are no
longer inserted before and after explicit callouts in the pattern.
A number of bugs in the refactored code were subsequently fixed during testing
before release, but after the code was made available in the repository. Many
of the bugs were discovered by fuzzing testing. Several of them were related to
the change from assuming a zero-terminated pattern (which previously had
required non-zero terminated strings to be copied). These bugs were never in
fully released code, but are noted here for the record.
(a) An overall recursion such as (?0) inside a lookbehind assertion was not
being diagnosed as an error.
(b) In utf mode, the length of a *MARK (or other verb) name was being checked
in characters instead of code units, which could lead to bad code being
compiled, leading to unpredictable behaviour.
(c) In extended /x mode, characters whose code was greater than 255 caused
a lookup outside one of the global tables. A similar bug existed for wide
characters in *VERB names.
(d) The amount of memory needed for a compiled pattern was miscalculated if a
lookbehind contained more than one toplevel branch and the first branch
was of length zero.
(e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
terminated pattern, if a # comment ran on to the end of the pattern, one
or more code units past the end were being read.
(f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
"{2,2") could cause reading beyond the pattern.
(g) When reading a callout string, if the end delimiter was at the end of the
pattern one further code unit was read.
(h) An unterminated number after \g' could cause reading beyond the pattern.
(i) An insufficient memory size was being computed for compiling with
PCRE2_AUTO_CALLOUT.
(j) A conditional group with an assertion condition used more memory than was
allowed for it during parsing, so too many of them could therefore
overrun a buffer.
(k) If parsing a pattern exactly filled the buffer, the internal test for
overrun did not check when the final META_END item was added.
(l) If a lookbehind contained a subroutine call, and the called group
contained an option setting such as (?s), and the PCRE2_ANCHORED option
was set, unpredictable behaviour could occur. The underlying bug was
incorrect code and insufficient checking while searching for the end of
the called subroutine in the parsed pattern.
(m) Quantifiers following (*VERB)s were not being diagnosed as errors.
(n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
(o) If \Q was preceded by a quantified item, and the following \E was
followed by '?' or '+', and there was at least one literal character
between them, an internal error "unexpected repeat" occurred (example:
/.+\QX\E+/).
(p) A buffer overflow could occur while sorting the names in the group name
list (depending on the order in which the names were seen).
(q) A conditional group that started with a callout was not doing the right
check for a following assertion, leading to compiling bad code. Example:
/(?(C'XX))?!XX/
(r) If a character whose code point was greater than 0xffff appeared within
a lookbehind that was within another lookbehind, the calculation of the
lookbehind length went wrong and could provoke an internal error.
(t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
an internal error. Now the hyphen is treated as a literal.
4. Back references are now permitted in lookbehind assertions when there are
no duplicated group numbers (that is, (?| has not been used), and, if the
reference is by name, there is only one group of that name. The referenced
group must, of course be of fixed length.
5. pcre2test has been upgraded so that, when run under valgrind with valgrind
support enabled, reading past the end of the pattern is detected, both when
compiling and during callout processing.
6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
not recognize this syntax.
7. Automatic callouts are no longer generated before and after callouts in the
pattern.
8. When pcre2test was outputing information from a callout, the caret indicator
for the current position in the subject line was incorrect if it was after an
escape sequence for a character whose code point was greater than \x{ff}.
9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
expansion when long lines are encountered. Original patch by Dmitry
Cherniachenko.
11. If pcre2grep was compiled with JIT support, but the library was compiled
without it (something that neither ./configure nor CMake allow, but it can be
done by editing config.h), pcre2grep was giving a JIT error. Now it detects
this situation and does not try to use JIT.
12. Added some "const" qualifiers to variables in pcre2grep.
13. Added Dmitry Cherniachenko's patch for colouring output in Windows
(untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
14. Add the -t (grand total) option to pcre2grep.
15. A number of bugs have been mended relating to match start-up optimizations
when the first thing in a pattern is a positive lookahead. These all applied
only when PCRE2_NO_START_OPTIMIZE was *not* set:
(a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
both an initial 'X' and a following 'X'.
(b) Some patterns starting with an assertion that started with .* were
incorrectly optimized as having to match at the start of the subject or
after a newline. There are cases where this is not true, for example,
(?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
start with spaces. Starting .* in an assertion is no longer taken as an
indication of matching at the start (or after a newline).
16. The "offset" modifier in pcre2test was not being ignored (as documented)
when the POSIX API was in use.
17. Added --enable-fuzz-support to "configure", causing an non-installed
library containing a test function that can be called by fuzzers to be
compiled. A non-installed binary to run the test function locally, called
pcre2fuzzcheck is also compiled.
18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
which started with .* inside a positive lookahead was incorrectly being
compiled as implicitly anchored.
19. Removed all instances of "register" declarations, as they are considered
obsolete these days and in any case had become very haphazard.
20. Add strerror() to pcre2test for failed file opening.
21. Make pcre2test -C list valgrind support when it is enabled.
22. Add the use_length modifier to pcre2test.
23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
'copy' modifiers.
24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
is apparently needed there as well as in the function definitions. (Why did
nobody ask for this in PCRE1?)
25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
compliant and unique.
26. pcre2-config --libs-posix was listing -lpcre2posix instead of
-lpcre2-posix. Also, the CMake build process was building the library with the
wrong name.
27. In pcre2test, give some offset information for errors in hex patterns.
This uses the C99 formatting sequence %td, except for MSVC which doesn't
support it - %lu is used instead.
28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
pcre2test for testing it.
29. Fix small memory leak in pcre2test.
30. Fix out-of-bounds read for partial matching of /./ against an empty string
when the newline type is CRLF.
31. Fix a bug in pcre2test that caused a crash when a locale was set either in
the current pattern or a previous one and a wide character was matched.
32. The appearance of \p, \P, or \X in a substitution string when
PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
dereference).
33. If the starting offset was specified as greater than the subject length in
a call to pcre2_substitute() an out-of-bounds memory reference could occur.
34. When PCRE2 was compiled to use the heap instead of the stack for recursive
calls to match(), a repeated minimizing caseless back reference, or a
maximizing one where the two cases had different numbers of code units,
followed by a caseful back reference, could lose the caselessness of the first
repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
but didn't).
35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
matching length and just records zero. Typically this happens when there are
too many nested or recursive back references. If the limit was reached in
certain recursive cases it failed to be triggered and an internal error could
be the result.
36. The pcre2_dfa_match() function now takes note of the recursion limit for
the internal recursive calls that are used for lookrounds and recursions within
the pattern.
37. More refactoring has got rid of the internal could_be_empty_branch()
function (around 400 lines of code, including comments) by keeping track of
could-be-emptiness as the pattern is compiled instead of scanning compiled
groups. (This would have been much harder before the refactoring of #3 above.)
This lifts a restriction on the number of branches in a group (more than about
1100 would give "pattern is too complicated").
38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
auto_callout".
39. In a library with Unicode support, incorrect data was compiled for a
pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
characters to match (for example, /[\s[:^ascii:]]/).
40. The callout_error modifier has been added to pcre2test to make it possible
to return PCRE2_ERROR_CALLOUT from a callout.
41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
"<esc>[00m".
42. The limit in the auto-possessification code that was intended to catch
overly-complicated patterns and not spend too much time auto-possessifying was
being reset too often, resulting in very long compile times for some patterns.
Now such patterns are no longer completely auto-possessified.
43. Applied Jason Hood's revised patch for RunTest.bat.
44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
45. Minor cosmetic fix to pcre2test: move a variable that is not used under
Windows into the "not Windows" code.
46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
some of the code:
* normalised the Windows condition by ensuring WIN32 is defined;
* enables the callout feature under Windows;
* adds globbing (Microsoft's implementation expands quoted args),
using a tweaked opendirectory;
* implements the is_*_tty functions for Windows;
* --color=always will write the ANSI sequences to file;
* add sequences 4 (underline works on Win10) and 5 (blink as bright
background, relatively standard on DOS/Win);
* remove the (char *) casts for the now-const strings;
* remove GREP_COLOUR (grep's command line allowed the 'u', but not
the environment), parsing GREP_COLORS instead;
* uses the current colour if not set, rather than black;
* add print_match for the undefined case;
* fixes a typo.
In addition, colour settings containing anything other than digits and
semicolon are ignored, and the colour controls are no longer output for empty
strings.
47. Detecting patterns that are too large inside the length-measuring loop
saves processing ridiculously long patterns to their end.
48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
just wastes time. In the UTF case it can also produce redundant entries in
XCLASS lists caused by characters with multiple other cases and pairs of
characters in the same "not-x" sublists.
49. A pattern such as /(?=(a\K))/ can report the end of the match being before
its start; pcre2test was not handling this correctly when using the POSIX
interface (it was OK with the native interface).
50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
continue to work, falling back to interpretation if anything goes wrong with
JIT.
51. Applied patches from Christian Persch to configure.ac to make use of the
AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
modules.
52. Minor fixes to pcre2grep from Jason Hood:
* fixed some spacing;
* Windows doesn't usually use single quotes, so I've added a define
to use appropriate quotes [in an example];
* LC_ALL was displayed as "LCC_ALL";
* numbers 11, 12 & 13 should end in "th";
* use double quotes in usage message.
53. When autopossessifying, skip empty branches without recursion, to reduce
stack usage for the benefit of clang with -fsanitize-address, which uses huge
stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
54. A pattern with very many explicit back references to a group that is a long
way from the start of the pattern could take a long time to compile because
searching for the referenced group in order to find the minimum length was
being done repeatedly. Now up to 128 group minimum lengths are cached and the
attempt to find a minimum length is abandoned if there is a back reference to a
group whose number is greater than 128. (In that case, the pattern is so
complicated that this optimization probably isn't worth it.) This fixes
oss-fuzz issue 557.
55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
mode with --only-matching matched several lines, it restarted scanning at the
next line instead of moving on to the end of the matched string, which can be
several lines after the start.
56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
with updates to the non-Windows version.
Version 10.22 29-July-2016
--------------------------
1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
to fix problems with running the tests under Windows.
2. Implemented a facility for quoting literal characters within hexadecimal
patterns in pcre2test, to make it easier to create patterns with just a few
non-printing characters.
3. Binary zeros are not supported in pcre2test input files. It now detects them
and gives an error.
4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
that it matches only unknown objects.
5. Updated the maintenance script maint/ManyConfigTests to make it easier to
select individual groups of tests.
6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
disables the use of back references (and subroutine calls), which are supported
by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
and pmatch when regexec() is called.
7. Because of 6 above, pcre2test has been modified with a new modifier called
posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
modifier had this effect. That option is now ignored when the POSIX API is in
use.
8. Minor tidies to the pcre2demo.c sample program, including more comments
about its 8-bit-ness.
9. Detect unmatched closing parentheses and give the error in the pre-scan
instead of later. Previously the pre-scan carried on and could give a
misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
message about invalid duplicate group names.
10. It has happened that pcre2test was accidentally linked with another POSIX
regex library instead of libpcre2-posix. In this situation, a call to regcomp()
(in the other library) may succeed, returning zero, but of course putting its
own data into the regex_t block. In one example the re_pcre2_code field was
left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
so it treated the next line as another pattern line, resulting in a confusing
error message. A check has been added to pcre2test to see if the data returned
from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
are not, an error message is output and the pcre2test run is abandoned. The
message points out the possibility of a mis-linking. Hopefully this will avoid
some head-scratching the next time this happens.
11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
assertion, caused pcre2test to output a very large number of spaces when the
callout was taken, making the program appearing to loop.
12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
nested set of parentheses of sufficient size caused an overflow of the
compiling workspace (which was diagnosed, but of course is not desirable).
13. Detect missing closing parentheses during the pre-pass for group
identification.
14. Changed some integer variable types and put in a number of casts, following
a report of compiler warnings from Visual Studio 2013 and a few tests with
gcc's -Wconversion (which still throws up a lot).
15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
for testing it.
16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
regerror(). When the error buffer is too small, my version of snprintf() puts a
binary zero in the final byte. Bug #1801 seems to show that other versions do
not do this, leading to bad output from pcre2test when it was checking for
buffer overflow. It no longer assumes a binary zero at the end of a too-small
regerror() buffer.
17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
actually affect anything, by sheer luck.
18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
"const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
older MSVC compilers. This has been done both in src/pcre2_internal.h for most
of the library, and also in src/pcre2posix.c, which no longer includes
pcre2_internal.h (see 24 below).
19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
static compilation. Subsequently applied Chris Wilson's second patch, putting
the first patch under a new option instead of being unconditional when
PCRE_STATIC is set.
20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
to convert \r\n at the ends of reflected lines into \r\r\n. This required
ensuring that other output that is written to stdout (e.g. file names) uses the
appropriate line terminator: \r\n for Windows, \n otherwise.
21. When a line is too long for pcre2grep's internal buffer, show the maximum
length in the error message.
22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
additions).
23. RunTest.bat was missing a "set type" line for test 22.
24. The pcre2posix.c file was including pcre2_internal.h, and using some
"private" knowledge of the data structures. This is unnecessary; the code has
been re-factored and no longer includes pcre2_internal.h.
25. A racing condition is fixed in JIT reported by Mozilla.
26. Minor code refactor to avoid "array subscript is below array bounds"
compiler warning.
27. Minor code refactor to avoid "left shift of negative number" warning.
28. Add a bit more sanity checking to pcre2_serialize_decode() and document
that it expects trusted data.
29. Fix typo in pcre2_jit_test.c
30. Due to an oversight, pcre2grep was not making use of JIT when available.
This is now fixed.
31. The RunGrepTest script is updated to use the valgrind suppressions file
when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
file is updated so that is now the same as for PCRE1: it suppresses the
Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
code). Also changed smc-check=all to smc-check=all-non-file as was done for
RunTest (see 4 above).
32. Implemented the PCRE2_NO_JIT option for pcre2_match().
33. Fix typo that gave a compiler error when JIT not supported.
34. Fix comment describing the returns from find_fixedlength().
35. Fix potential negative index in pcre2test.
36. Calls to pcre2_get_error_message() with error numbers that are never
returned by PCRE2 functions were returning empty strings. Now the error code
PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
show the texts for given error numbers (i.e. to call pcre2_get_error_message()
and display what it returns) and a few representative error codes are now
checked in RunTest.
37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
pcre2_match.c, in anticipation that this is needed for the same reason it was
recently added to pcrecpp.cc in PCRE1.
38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
the match extended over a line boundary, as it tried to find more matches "on
the same line" - but it was already over the end.
39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
to the same code as '.' when PCRE2_DOTALL is set).
40. Fix two clang compiler warnings in pcre2test when only one code unit width
is supported.
41. Upgrade RunTest to automatically re-run test 2 with a large (64M) stack if
it fails when running the interpreter with a 16M stack (and if changing the
stack size via pcre2test is possible). This avoids having to manually set a
large stack size when testing with clang.
42. Fix register overwite in JIT when SSE2 acceleration is enabled.
43. Detect integer overflow in pcre2test pattern and data repetition counts.
44. In pcre2test, ignore "allcaptures" after DFA matching.
45. Fix unaligned accesses on x86. Patch by Marc Mutz.
46. Fix some more clang compiler warnings.
Version 10.21 12-January-2016
-----------------------------
1. Improve matching speed of patterns starting with + or * in JIT.
2. Use memchr() to find the first character in an unanchored match in 8-bit
mode in the interpreter. This gives a significant speed improvement.
3. Removed a redundant copy of the opcode_possessify table in the
pcre2_auto_possessify.c source.
4. Fix typos in dftables.c for z/OS.
5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
processing them could involve a buffer overflow if the following character was
an opening parenthesis.
6. Change 36 for 10.20 also introduced a bug in processing this pattern:
/((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
setting (which (*:0) is), then (?x) did not get unset at the end of its group
during the scan for named groups, and hence the external # was incorrectly
treated as a comment and the invalid (?' at the end of the pattern was not
diagnosed. This caused a buffer overflow during the real compile. This bug was
discovered by Karl Skomski with the LLVM fuzzer.
7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
own source module to avoid a circular dependency between src/pcre2_compile.c
and src/pcre2_study.c
8. A callout with a string argument containing an opening square bracket, for
example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
9. The handling of callouts during the pre-pass for named group identification
has been tightened up.
10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
possessive. This is a very minor optimization.
11. A possessively repeated conditional group that could match an empty string,
for example, /(?(R))*+/, was incorrectly compiled.
12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
Persch).
13. An empty comment (?#) in a pattern was incorrectly processed and could
provoke a buffer overflow. This bug was discovered by Karl Skomski with the
LLVM fuzzer.
14. Fix infinite recursion in the JIT compiler when certain patterns such as
/(?:|a|){100}x/ are analysed.
15. Some patterns with character classes involving [: and \\ were incorrectly
compiled and could cause reading from uninitialized memory or an incorrect
error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
16. Pathological patterns containing many nested occurrences of [: caused
pcre2_compile() to run for a very long time. This bug was found by the LLVM
fuzzer.
17. A missing closing parenthesis for a callout with a string argument was not
being diagnosed, possibly leading to a buffer overflow. This bug was found by
the LLVM fuzzer.
18. A conditional group with only one branch has an implicit empty alternative
branch and must therefore be treated as potentially matching an empty string.
19. If (?R was followed by - or + incorrect behaviour happened instead of a
diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
20. Another bug that was introduced by change 36 for 10.20: conditional groups
whose condition was an assertion preceded by an explicit callout with a string
argument might be incorrectly processed, especially if the string contained \Q.
This bug was discovered by Karl Skomski with the LLVM fuzzer.
21. Compiling PCRE2 with the sanitize options of clang showed up a number of
very pedantic coding infelicities and a buffer overflow while checking a UTF-8
string if the final multi-byte UTF-8 character was truncated.
22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
class, where both values are literal letters in the same case, omit the
non-letter EBCDIC code points within the range.
23. Finding the minimum matching length of complex patterns with back
references and/or recursions can take a long time. There is now a cut-off that
gives up trying to find a minimum length when things get too complex.
24. An optimization has been added that speeds up finding the minimum matching
length for patterns containing repeated capturing groups or recursions.
25. If a pattern contained a back reference to a group whose number was
duplicated as a result of appearing in a (?|...) group, the computation of the
minimum matching length gave a wrong result, which could cause incorrect "no
match" errors. For such patterns, a minimum matching length cannot at present
be computed.
26. Added a check for integer overflow in conditions (?(<digits>) and
(?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
fuzzer.
27. Fixed an issue when \p{Any} inside an xclass did not read the current
character.
28. If pcre2grep was given the -q option with -c or -l, or when handling a
binary file, it incorrectly wrote output to stdout.
29. The JIT compiler did not restore the control verb head in case of *THEN
control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
30. The way recursive references such as (?3) are compiled has been re-written
because the old way was the cause of many issues. Now, conversion of the group
number into a pattern offset does not happen until the pattern has been
completely compiled. This does mean that detection of all infinitely looping
recursions is postponed till match time. In the past, some easy ones were
detected at compile time. This re-writing was done in response to yet another
bug found by the LLVM fuzzer.
31. A test for a back reference to a non-existent group was missing for items
such as \987. This caused incorrect code to be compiled. This issue was found
by Karl Skomski with a custom LLVM fuzzer.
32. Error messages for syntax errors following \g and \k were giving inaccurate
offsets in the pattern.
33. Improve the performance of starting single character repetitions in JIT.
34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
give the right offset instead of zero.
36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
This issue was found by Karl Skomski with a custom LLVM fuzzer.
37. The JIT compiler should restore the control chain for empty possessive
repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
38. A bug which was introduced by the single character repetition optimization
was fixed.
39. Match limit check added to recursion. This issue was found by Karl Skomski
with a custom LLVM fuzzer.
40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
only at the part of the subject that is relevant when the starting offset is
non-zero.
41. Improve first character match in JIT with SSE2 on x86.
42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
with a custom LLVM fuzzer.
43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
III).
44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
test (there are now 20 in total).
45. Fixed a corner case of range optimization in JIT.
46. Add the ${*MARK} facility to pcre2_substitute().
47. Modifier lists in pcre2test were splitting at spaces without the required
commas.
48. Implemented PCRE2_ALT_VERBNAMES.
49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
LLVM fuzzer.
50. The pcre2test program has been extended by adding the #newline_default
command. This has made it possible to run the standard tests when PCRE2 is
compiled with either CR or CRLF as the default newline convention. As part of
this work, the new command was added to several test files and the testing
scripts were modified. The pcre2grep tests can now also be run when there is no
LF in the default newline convention.
51. The RunTest script has been modified so that, when JIT is used and valgrind
is specified, a valgrind suppressions file is set up to ignore "Invalid read of
size 16" errors because these are false positives when the hardware supports
the SSE2 instruction set.
52. It is now possible to have comment lines amid the subject strings in
pcre2test (and perltest.sh) input.
53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
and the matching functions with NULL contexts can be tested.
55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
56. In a character class such as [\W\p{Any}] where both a negative-type escape
("not a word character") and a property escape were present, the property
escape was being ignored.
57. Fixed integer overflow for patterns whose minimum matching length is very,
very large.
58. Implemented --never-backslash-C.
59. Change 55 above introduced a bug by which certain patterns provoked the
erroneous error "\ at end of pattern".
60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
errors or other strange effects if compiled in UCP mode. Found with libFuzzer
and AddressSanitizer.
61. Whitespace at the end of a pcre2test pattern line caused a spurious error
message if there were only single-character modifiers. It should be ignored.
62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
or segmentation errors for some patterns. Found with libFuzzer and
AddressSanitizer.
63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
overflow.
64. Improve error message for overly-complicated patterns.
65. Implemented an optional replication feature for patterns in pcre2test, to
make it easier to test long repetitive patterns. The tests for 63 above are
converted to use the new feature.
66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
misbehave.
67. In pcre2_substitute() in UTF mode, the UTF validity check on the
replacement string was happening before the length setting when the replacement
string was zero-terminated.