Annotation of win32/pcre/pcre.txt, revision 1.3
1.1 misha 1: -----------------------------------------------------------------------------
2: This file contains a concatenation of the PCRE man pages, converted to plain
3: text format for ease of searching with a text editor, or for use on systems
4: that do not have a man page processor. The small individual files that give
5: synopses of each function in the library have not been included. There are
6: separate text files for the pcregrep and pcretest commands.
7: -----------------------------------------------------------------------------
8:
9:
10: PCRE(3) PCRE(3)
11:
12:
13: NAME
14: PCRE - Perl-compatible regular expressions
15:
16:
17: INTRODUCTION
18:
19: The PCRE library is a set of functions that implement regular expres-
20: sion pattern matching using the same syntax and semantics as Perl, with
21: just a few differences. Certain features that appeared in Python and
22: PCRE before they appeared in Perl are also available using the Python
23: syntax. There is also some support for certain .NET and Oniguruma syn-
24: tax items, and there is an option for requesting some minor changes
25: that give better JavaScript compatibility.
26:
27: The current implementation of PCRE (release 7.x) corresponds approxi-
28: mately with Perl 5.10, including support for UTF-8 encoded strings and
29: Unicode general category properties. However, UTF-8 and Unicode support
30: has to be explicitly enabled; it is not the default. The Unicode tables
1.3 ! misha 31: correspond to Unicode release 5.1.
1.1 misha 32:
33: In addition to the Perl-compatible matching function, PCRE contains an
34: alternative matching function that matches the same compiled patterns
35: in a different way. In certain circumstances, the alternative function
36: has some advantages. For a discussion of the two matching algorithms,
37: see the pcrematching page.
38:
39: PCRE is written in C and released as a C library. A number of people
40: have written wrappers and interfaces of various kinds. In particular,
41: Google Inc. have provided a comprehensive C++ wrapper. This is now
42: included as part of the PCRE distribution. The pcrecpp page has details
43: of this interface. Other people's contributions can be found in the
44: Contrib directory at the primary FTP site, which is:
45:
46: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
47:
48: Details of exactly which Perl regular expression features are and are
49: not supported by PCRE are given in separate documents. See the pcrepat-
50: tern and pcrecompat pages. There is a syntax summary in the pcresyntax
51: page.
52:
53: Some features of PCRE can be included, excluded, or changed when the
54: library is built. The pcre_config() function makes it possible for a
55: client to discover which features are available. The features them-
56: selves are described in the pcrebuild page. Documentation about build-
57: ing PCRE for various operating systems can be found in the README file
58: in the source distribution.
59:
60: The library contains a number of undocumented internal functions and
61: data tables that are used by more than one of the exported external
62: functions, but which are not intended for use by external callers.
63: Their names all begin with "_pcre_", which hopefully will not provoke
64: any name clashes. In some environments, it is possible to control which
65: external symbols are exported when a shared library is built, and in
66: these cases the undocumented symbols are not exported.
67:
68:
69: USER DOCUMENTATION
70:
71: The user documentation for PCRE comprises a number of different sec-
72: tions. In the "man" format, each of these is a separate "man page". In
73: the HTML format, each is a separate page, linked from the index page.
74: In the plain text format, all the sections are concatenated, for ease
75: of searching. The sections are as follows:
76:
77: pcre this document
78: pcre-config show PCRE installation configuration information
79: pcreapi details of PCRE's native C API
80: pcrebuild options for building PCRE
81: pcrecallout details of the callout feature
82: pcrecompat discussion of Perl compatibility
83: pcrecpp details of the C++ wrapper
84: pcregrep description of the pcregrep command
85: pcrematching discussion of the two matching algorithms
86: pcrepartial details of the partial matching facility
87: pcrepattern syntax and semantics of supported
88: regular expressions
89: pcresyntax quick syntax reference
90: pcreperform discussion of performance issues
91: pcreposix the POSIX-compatible C API
92: pcreprecompile details of saving and re-using precompiled patterns
93: pcresample discussion of the sample program
94: pcrestack discussion of stack usage
95: pcretest description of the pcretest testing command
96:
1.3 ! misha 97: In addition, in the "man" and HTML formats, there is a short page for
1.1 misha 98: each C library function, listing its arguments and results.
99:
100:
101: LIMITATIONS
102:
1.3 ! misha 103: There are some size limitations in PCRE but it is hoped that they will
1.1 misha 104: never in practice be relevant.
105:
1.3 ! misha 106: The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
1.1 misha 107: is compiled with the default internal linkage size of 2. If you want to
1.3 ! misha 108: process regular expressions that are truly enormous, you can compile
! 109: PCRE with an internal linkage size of 3 or 4 (see the README file in
! 110: the source distribution and the pcrebuild documentation for details).
! 111: In these cases the limit is substantially larger. However, the speed
1.1 misha 112: of execution is slower.
113:
114: All values in repeating quantifiers must be less than 65536.
115:
116: There is no limit to the number of parenthesized subpatterns, but there
117: can be no more than 65535 capturing subpatterns.
118:
119: The maximum length of name for a named subpattern is 32 characters, and
120: the maximum number of named subpatterns is 10000.
121:
1.3 ! misha 122: The maximum length of a subject string is the largest positive number
! 123: that an integer variable can hold. However, when using the traditional
1.1 misha 124: matching function, PCRE uses recursion to handle subpatterns and indef-
1.3 ! misha 125: inite repetition. This means that the available stack space may limit
1.1 misha 126: the size of a subject string that can be processed by certain patterns.
127: For a discussion of stack issues, see the pcrestack documentation.
128:
129:
130: UTF-8 AND UNICODE PROPERTY SUPPORT
131:
1.3 ! misha 132: From release 3.3, PCRE has had some support for character strings
! 133: encoded in the UTF-8 format. For release 4.0 this was greatly extended
! 134: to cover most common requirements, and in release 5.0 additional sup-
1.1 misha 135: port for Unicode general category properties was added.
136:
1.3 ! misha 137: In order process UTF-8 strings, you must build PCRE to include UTF-8
! 138: support in the code, and, in addition, you must call pcre_compile()
! 139: with the PCRE_UTF8 option flag, or the pattern must start with the
! 140: sequence (*UTF8). When either of these is the case, both the pattern
! 141: and any subject strings that are matched against it are treated as
! 142: UTF-8 strings instead of just strings of bytes.
1.1 misha 143:
144: If you compile PCRE with UTF-8 support, but do not use it at run time,
145: the library will be a bit bigger, but the additional run time overhead
146: is limited to testing the PCRE_UTF8 flag occasionally, so should not be
147: very big.
148:
149: If PCRE is built with Unicode character property support (which implies
150: UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
151: ported. The available properties that can be tested are limited to the
152: general category properties such as Lu for an upper case letter or Nd
153: for a decimal number, the Unicode script names such as Arabic or Han,
154: and the derived properties Any and L&. A full list is given in the
155: pcrepattern documentation. Only the short names for properties are sup-
156: ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let-
157: ter}, is not supported. Furthermore, in Perl, many properties may
158: optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE
159: does not support this.
160:
161: Validity of UTF-8 strings
162:
163: When you set the PCRE_UTF8 flag, the strings passed as patterns and
164: subjects are (by default) checked for validity on entry to the relevant
165: functions. From release 7.3 of PCRE, the check is according the rules
166: of RFC 3629, which are themselves derived from the Unicode specifica-
167: tion. Earlier releases of PCRE followed the rules of RFC 2279, which
168: allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current
169: check allows only values in the range U+0 to U+10FFFF, excluding U+D800
170: to U+DFFF.
171:
172: The excluded code points are the "Low Surrogate Area" of Unicode, of
173: which the Unicode Standard says this: "The Low Surrogate Area does not
174: contain any character assignments, consequently no character code
175: charts or namelists are provided for this area. Surrogates are reserved
176: for use with UTF-16 and then must be used in pairs." The code points
177: that are encoded by UTF-16 pairs are available as independent code
178: points in the UTF-8 encoding. (In other words, the whole surrogate
179: thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
180:
181: If an invalid UTF-8 string is passed to PCRE, an error return
182: (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know
183: that your strings are valid, and therefore want to skip these checks in
184: order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at
185: compile time or at run time, PCRE assumes that the pattern or subject
186: it is given (respectively) contains only valid UTF-8 codes. In this
187: case, it does not diagnose an invalid UTF-8 string.
188:
189: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set,
190: what happens depends on why the string is invalid. If the string con-
191: forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a
192: string of characters in the range 0 to 0x7FFFFFFF. In other words,
193: apart from the initial validity test, PCRE (when in UTF-8 mode) handles
194: strings according to the more liberal rules of RFC 2279. However, if
195: the string does not even conform to RFC 2279, the result is undefined.
196: Your program may crash.
197:
198: If you want to process strings of values in the full range 0 to
199: 0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can
200: set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in
201: this situation, you will have to apply your own validity check.
202:
203: General comments about UTF-8 mode
204:
205: 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a
206: two-byte UTF-8 character if the value is greater than 127.
207:
208: 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
209: characters for values greater than \177.
210:
211: 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
212: vidual bytes, for example: \x{100}{3}.
213:
214: 4. The dot metacharacter matches one UTF-8 character instead of a sin-
215: gle byte.
216:
217: 5. The escape sequence \C can be used to match a single byte in UTF-8
218: mode, but its use can lead to some strange effects. This facility is
219: not available in the alternative matching function, pcre_dfa_exec().
220:
221: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
222: test characters of any code value, but the characters that PCRE recog-
223: nizes as digits, spaces, or word characters remain the same set as
224: before, all with values less than 256. This remains true even when PCRE
225: includes Unicode property support, because to do otherwise would slow
226: down PCRE in many common cases. If you really want to test for a wider
227: sense of, say, "digit", you must use Unicode property tests such as
1.3 ! misha 228: \p{Nd}. Note that this also applies to \b, because it is defined in
! 229: terms of \w and \W.
1.1 misha 230:
1.3 ! misha 231: 7. Similarly, characters that match the POSIX named character classes
1.1 misha 232: are all low-valued characters.
233:
1.3 ! misha 234: 8. However, the Perl 5.10 horizontal and vertical whitespace matching
1.1 misha 235: escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char-
236: acters.
237:
1.3 ! misha 238: 9. Case-insensitive matching applies only to characters whose values
! 239: are less than 128, unless PCRE is built with Unicode property support.
! 240: Even when Unicode property support is available, PCRE still uses its
! 241: own character tables when checking the case of low-valued characters,
! 242: so as not to degrade performance. The Unicode property information is
1.1 misha 243: used only for characters with higher values. Even when Unicode property
244: support is available, PCRE supports case-insensitive matching only when
1.3 ! misha 245: there is a one-to-one mapping between a letter's cases. There are a
! 246: small number of many-to-one mappings in Unicode; these are not sup-
1.1 misha 247: ported by PCRE.
248:
249:
250: AUTHOR
251:
252: Philip Hazel
253: University Computing Service
254: Cambridge CB2 3QH, England.
255:
1.3 ! misha 256: Putting an actual email address here seems to have been a spam magnet,
! 257: so I've taken it away. If you want to email me, use my two initials,
1.1 misha 258: followed by the two digits 10, at the domain cam.ac.uk.
259:
260:
261: REVISION
262:
1.3 ! misha 263: Last updated: 11 April 2009
! 264: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 265: ------------------------------------------------------------------------------
266:
267:
268: PCREBUILD(3) PCREBUILD(3)
269:
270:
271: NAME
272: PCRE - Perl-compatible regular expressions
273:
274:
275: PCRE BUILD-TIME OPTIONS
276:
277: This document describes the optional features of PCRE that can be
278: selected when the library is compiled. It assumes use of the configure
279: script, where the optional features are selected or deselected by pro-
280: viding options to configure before running the make command. However,
281: the same options can be selected in both Unix-like and non-Unix-like
282: environments using the GUI facility of CMakeSetup if you are using
283: CMake instead of configure to build PCRE.
284:
285: The complete list of options for configure (which includes the standard
286: ones such as the selection of the installation directory) can be
287: obtained by running
288:
289: ./configure --help
290:
291: The following sections include descriptions of options whose names
292: begin with --enable or --disable. These settings specify changes to the
293: defaults for the configure command. Because of the way that configure
294: works, --enable and --disable always come in pairs, so the complemen-
295: tary option always exists as well, but as it specifies the default, it
296: is not described.
297:
298:
299: C++ SUPPORT
300:
301: By default, the configure script will search for a C++ compiler and C++
302: header files. If it finds them, it automatically builds the C++ wrapper
303: library for PCRE. You can disable this by adding
304:
305: --disable-cpp
306:
307: to the configure command.
308:
309:
310: UTF-8 SUPPORT
311:
1.3 ! misha 312: To build PCRE with support for UTF-8 Unicode character strings, add
1.1 misha 313:
314: --enable-utf8
315:
316: to the configure command. Of itself, this does not make PCRE treat
317: strings as UTF-8. As well as compiling PCRE with this option, you also
318: have have to set the PCRE_UTF8 option when you call the pcre_compile()
319: function.
320:
1.3 ! misha 321: If you set --enable-utf8 when compiling in an EBCDIC environment, PCRE
! 322: expects its input to be either ASCII or UTF-8 (depending on the runtime
! 323: option). It is not possible to support both EBCDIC and UTF-8 codes in
! 324: the same version of the library. Consequently, --enable-utf8 and
! 325: --enable-ebcdic are mutually exclusive.
! 326:
1.1 misha 327:
328: UNICODE CHARACTER PROPERTY SUPPORT
329:
330: UTF-8 support allows PCRE to process character values greater than 255
331: in the strings that it handles. On its own, however, it does not pro-
332: vide any facilities for accessing the properties of such characters. If
333: you want to be able to use the pattern escapes \P, \p, and \X, which
334: refer to Unicode character properties, you must add
335:
336: --enable-unicode-properties
337:
338: to the configure command. This implies UTF-8 support, even if you have
339: not explicitly requested it.
340:
341: Including Unicode property support adds around 30K of tables to the
342: PCRE library. Only the general category properties such as Lu and Nd
343: are supported. Details are given in the pcrepattern documentation.
344:
345:
346: CODE VALUE OF NEWLINE
347:
1.3 ! misha 348: By default, PCRE interprets the linefeed (LF) character as indicating
1.1 misha 349: the end of a line. This is the normal newline character on Unix-like
1.3 ! misha 350: systems. You can compile PCRE to use carriage return (CR) instead, by
! 351: adding
1.1 misha 352:
353: --enable-newline-is-cr
354:
355: to the configure command. There is also a --enable-newline-is-lf
356: option, which explicitly specifies linefeed as the newline character.
357:
358: Alternatively, you can specify that line endings are to be indicated by
359: the two character sequence CRLF. If you want this, add
360:
361: --enable-newline-is-crlf
362:
363: to the configure command. There is a fourth option, specified by
364:
365: --enable-newline-is-anycrlf
366:
367: which causes PCRE to recognize any of the three sequences CR, LF, or
368: CRLF as indicating a line ending. Finally, a fifth option, specified by
369:
370: --enable-newline-is-any
371:
372: causes PCRE to recognize any Unicode newline sequence.
373:
1.3 ! misha 374: Whatever line ending convention is selected when PCRE is built can be
! 375: overridden when the library functions are called. At build time it is
1.1 misha 376: conventional to use the standard for your operating system.
377:
378:
379: WHAT \R MATCHES
380:
1.3 ! misha 381: By default, the sequence \R in a pattern matches any Unicode newline
! 382: sequence, whatever has been selected as the line ending sequence. If
1.1 misha 383: you specify
384:
385: --enable-bsr-anycrlf
386:
1.3 ! misha 387: the default is changed so that \R matches only CR, LF, or CRLF. What-
! 388: ever is selected when PCRE is built can be overridden when the library
1.1 misha 389: functions are called.
390:
391:
392: BUILDING SHARED AND STATIC LIBRARIES
393:
1.3 ! misha 394: The PCRE building process uses libtool to build both shared and static
! 395: Unix libraries by default. You can suppress one of these by adding one
1.1 misha 396: of
397:
398: --disable-shared
399: --disable-static
400:
401: to the configure command, as required.
402:
403:
404: POSIX MALLOC USAGE
405:
406: When PCRE is called through the POSIX interface (see the pcreposix doc-
1.3 ! misha 407: umentation), additional working storage is required for holding the
! 408: pointers to capturing substrings, because PCRE requires three integers
! 409: per substring, whereas the POSIX interface provides only two. If the
1.1 misha 410: number of expected substrings is small, the wrapper function uses space
411: on the stack, because this is faster than using malloc() for each call.
412: The default threshold above which the stack is no longer used is 10; it
413: can be changed by adding a setting such as
414:
415: --with-posix-malloc-threshold=20
416:
417: to the configure command.
418:
419:
420: HANDLING VERY LARGE PATTERNS
421:
1.3 ! misha 422: Within a compiled pattern, offset values are used to point from one
! 423: part to another (for example, from an opening parenthesis to an alter-
! 424: nation metacharacter). By default, two-byte values are used for these
! 425: offsets, leading to a maximum size for a compiled pattern of around
! 426: 64K. This is sufficient to handle all but the most gigantic patterns.
! 427: Nevertheless, some people do want to process enormous patterns, so it
! 428: is possible to compile PCRE to use three-byte or four-byte offsets by
1.1 misha 429: adding a setting such as
430:
431: --with-link-size=3
432:
1.3 ! misha 433: to the configure command. The value given must be 2, 3, or 4. Using
! 434: longer offsets slows down the operation of PCRE because it has to load
1.1 misha 435: additional bytes when handling them.
436:
437:
438: AVOIDING EXCESSIVE STACK USAGE
439:
440: When matching with the pcre_exec() function, PCRE implements backtrack-
1.3 ! misha 441: ing by making recursive calls to an internal function called match().
! 442: In environments where the size of the stack is limited, this can se-
! 443: verely limit PCRE's operation. (The Unix environment does not usually
1.1 misha 444: suffer from this problem, but it may sometimes be necessary to increase
1.3 ! misha 445: the maximum stack size. There is a discussion in the pcrestack docu-
! 446: mentation.) An alternative approach to recursion that uses memory from
! 447: the heap to remember data, instead of using recursive function calls,
! 448: has been implemented to work round the problem of limited stack size.
1.1 misha 449: If you want to build a version of PCRE that works this way, add
450:
451: --disable-stack-for-recursion
452:
1.3 ! misha 453: to the configure command. With this configuration, PCRE will use the
! 454: pcre_stack_malloc and pcre_stack_free variables to call memory manage-
! 455: ment functions. By default these point to malloc() and free(), but you
1.1 misha 456: can replace the pointers so that your own functions are used.
457:
1.3 ! misha 458: Separate functions are provided rather than using pcre_malloc and
! 459: pcre_free because the usage is very predictable: the block sizes
! 460: requested are always the same, and the blocks are always freed in
! 461: reverse order. A calling program might be able to implement optimized
! 462: functions that perform better than malloc() and free(). PCRE runs
1.1 misha 463: noticeably more slowly when built in this way. This option affects only
1.3 ! misha 464: the pcre_exec() function; it is not relevant for the the
1.1 misha 465: pcre_dfa_exec() function.
466:
467:
468: LIMITING PCRE RESOURCE USAGE
469:
1.3 ! misha 470: Internally, PCRE has a function called match(), which it calls repeat-
! 471: edly (sometimes recursively) when matching a pattern with the
! 472: pcre_exec() function. By controlling the maximum number of times this
! 473: function may be called during a single matching operation, a limit can
! 474: be placed on the resources used by a single call to pcre_exec(). The
! 475: limit can be changed at run time, as described in the pcreapi documen-
! 476: tation. The default is 10 million, but this can be changed by adding a
1.1 misha 477: setting such as
478:
479: --with-match-limit=500000
480:
1.3 ! misha 481: to the configure command. This setting has no effect on the
1.1 misha 482: pcre_dfa_exec() matching function.
483:
1.3 ! misha 484: In some environments it is desirable to limit the depth of recursive
1.1 misha 485: calls of match() more strictly than the total number of calls, in order
1.3 ! misha 486: to restrict the maximum amount of stack (or heap, if --disable-stack-
1.1 misha 487: for-recursion is specified) that is used. A second limit controls this;
1.3 ! misha 488: it defaults to the value that is set for --with-match-limit, which
! 489: imposes no additional constraints. However, you can set a lower limit
1.1 misha 490: by adding, for example,
491:
492: --with-match-limit-recursion=10000
493:
1.3 ! misha 494: to the configure command. This value can also be overridden at run
1.1 misha 495: time.
496:
497:
498: CREATING CHARACTER TABLES AT BUILD TIME
499:
1.3 ! misha 500: PCRE uses fixed tables for processing characters whose code values are
! 501: less than 256. By default, PCRE is built with a set of tables that are
! 502: distributed in the file pcre_chartables.c.dist. These tables are for
1.1 misha 503: ASCII codes only. If you add
504:
505: --enable-rebuild-chartables
506:
1.3 ! misha 507: to the configure command, the distributed tables are no longer used.
! 508: Instead, a program called dftables is compiled and run. This outputs
1.1 misha 509: the source for new set of tables, created in the default locale of your
510: C runtime system. (This method of replacing the tables does not work if
1.3 ! misha 511: you are cross compiling, because dftables is run on the local host. If
! 512: you need to create alternative tables when cross compiling, you will
1.1 misha 513: have to do so "by hand".)
514:
515:
516: USING EBCDIC CODE
517:
1.3 ! misha 518: PCRE assumes by default that it will run in an environment where the
! 519: character code is ASCII (or Unicode, which is a superset of ASCII).
! 520: This is the case for most computer operating systems. PCRE can, how-
1.1 misha 521: ever, be compiled to run in an EBCDIC environment by adding
522:
523: --enable-ebcdic
524:
525: to the configure command. This setting implies --enable-rebuild-charta-
1.3 ! misha 526: bles. You should only use it if you know that you are in an EBCDIC
! 527: environment (for example, an IBM mainframe operating system). The
! 528: --enable-ebcdic option is incompatible with --enable-utf8.
1.1 misha 529:
530:
531: PCREGREP OPTIONS FOR COMPRESSED FILE SUPPORT
532:
533: By default, pcregrep reads all files as plain text. You can build it so
534: that it recognizes files whose names end in .gz or .bz2, and reads them
535: with libz or libbz2, respectively, by adding one or both of
536:
537: --enable-pcregrep-libz
538: --enable-pcregrep-libbz2
539:
540: to the configure command. These options naturally require that the rel-
541: evant libraries are installed on your system. Configuration will fail
542: if they are not.
543:
544:
545: PCRETEST OPTION FOR LIBREADLINE SUPPORT
546:
547: If you add
548:
549: --enable-pcretest-libreadline
550:
551: to the configure command, pcretest is linked with the libreadline
552: library, and when its input is from a terminal, it reads it using the
553: readline() function. This provides line-editing and history facilities.
554: Note that libreadline is GPL-licenced, so if you distribute a binary of
555: pcretest linked in this way, there may be licensing issues.
556:
557: Setting this option causes the -lreadline option to be added to the
558: pcretest build. In many operating environments with a sytem-installed
559: libreadline this is sufficient. However, in some environments (e.g. if
560: an unmodified distribution version of readline is in use), some extra
561: configuration may be necessary. The INSTALL file for libreadline says
562: this:
563:
564: "Readline uses the termcap functions, but does not link with the
565: termcap or curses library itself, allowing applications which link
566: with readline the to choose an appropriate library."
567:
568: If your environment has not been set up so that an appropriate library
569: is automatically included, you may need to add something like
570:
571: LIBS="-ncurses"
572:
573: immediately before the configure command.
574:
575:
576: SEE ALSO
577:
578: pcreapi(3), pcre_config(3).
579:
580:
581: AUTHOR
582:
583: Philip Hazel
584: University Computing Service
585: Cambridge CB2 3QH, England.
586:
587:
588: REVISION
589:
1.3 ! misha 590: Last updated: 17 March 2009
! 591: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 592: ------------------------------------------------------------------------------
593:
594:
595: PCREMATCHING(3) PCREMATCHING(3)
596:
597:
598: NAME
599: PCRE - Perl-compatible regular expressions
600:
601:
602: PCRE MATCHING ALGORITHMS
603:
604: This document describes the two different algorithms that are available
605: in PCRE for matching a compiled regular expression against a given sub-
606: ject string. The "standard" algorithm is the one provided by the
607: pcre_exec() function. This works in the same was as Perl's matching
608: function, and provides a Perl-compatible matching operation.
609:
610: An alternative algorithm is provided by the pcre_dfa_exec() function;
611: this operates in a different way, and is not Perl-compatible. It has
612: advantages and disadvantages compared with the standard algorithm, and
613: these are described below.
614:
615: When there is only one possible way in which a given subject string can
616: match a pattern, the two algorithms give the same answer. A difference
617: arises, however, when there are multiple possibilities. For example, if
618: the pattern
619:
620: ^<.*>
621:
622: is matched against the string
623:
624: <something> <something else> <something further>
625:
626: there are three possible answers. The standard algorithm finds only one
627: of them, whereas the alternative algorithm finds all three.
628:
629:
630: REGULAR EXPRESSIONS AS TREES
631:
632: The set of strings that are matched by a regular expression can be rep-
633: resented as a tree structure. An unlimited repetition in the pattern
634: makes the tree of infinite size, but it is still a tree. Matching the
635: pattern to a given subject string (from a given starting point) can be
636: thought of as a search of the tree. There are two ways to search a
637: tree: depth-first and breadth-first, and these correspond to the two
638: matching algorithms provided by PCRE.
639:
640:
641: THE STANDARD MATCHING ALGORITHM
642:
643: In the terminology of Jeffrey Friedl's book "Mastering Regular Expres-
644: sions", the standard algorithm is an "NFA algorithm". It conducts a
645: depth-first search of the pattern tree. That is, it proceeds along a
646: single path through the tree, checking that the subject matches what is
647: required. When there is a mismatch, the algorithm tries any alterna-
648: tives at the current point, and if they all fail, it backs up to the
649: previous branch point in the tree, and tries the next alternative
650: branch at that level. This often involves backing up (moving to the
651: left) in the subject string as well. The order in which repetition
652: branches are tried is controlled by the greedy or ungreedy nature of
653: the quantifier.
654:
655: If a leaf node is reached, a matching string has been found, and at
656: that point the algorithm stops. Thus, if there is more than one possi-
657: ble match, this algorithm returns the first one that it finds. Whether
658: this is the shortest, the longest, or some intermediate length depends
659: on the way the greedy and ungreedy repetition quantifiers are specified
660: in the pattern.
661:
662: Because it ends up with a single path through the tree, it is rela-
663: tively straightforward for this algorithm to keep track of the sub-
664: strings that are matched by portions of the pattern in parentheses.
665: This provides support for capturing parentheses and back references.
666:
667:
668: THE ALTERNATIVE MATCHING ALGORITHM
669:
670: This algorithm conducts a breadth-first search of the tree. Starting
671: from the first matching point in the subject, it scans the subject
672: string from left to right, once, character by character, and as it does
673: this, it remembers all the paths through the tree that represent valid
674: matches. In Friedl's terminology, this is a kind of "DFA algorithm",
675: though it is not implemented as a traditional finite state machine (it
676: keeps multiple states active simultaneously).
677:
678: The scan continues until either the end of the subject is reached, or
679: there are no more unterminated paths. At this point, terminated paths
680: represent the different matching possibilities (if there are none, the
681: match has failed). Thus, if there is more than one possible match,
682: this algorithm finds all of them, and in particular, it finds the long-
683: est. In PCRE, there is an option to stop the algorithm after the first
684: match (which is necessarily the shortest) has been found.
685:
686: Note that all the matches that are found start at the same point in the
687: subject. If the pattern
688:
689: cat(er(pillar)?)
690:
691: is matched against the string "the caterpillar catchment", the result
692: will be the three strings "cat", "cater", and "caterpillar" that start
693: at the fourth character of the subject. The algorithm does not automat-
694: ically move on to find matches that start at later positions.
695:
696: There are a number of features of PCRE regular expressions that are not
697: supported by the alternative matching algorithm. They are as follows:
698:
699: 1. Because the algorithm finds all possible matches, the greedy or
700: ungreedy nature of repetition quantifiers is not relevant. Greedy and
701: ungreedy quantifiers are treated in exactly the same way. However, pos-
702: sessive quantifiers can make a difference when what follows could also
703: match what is quantified, for example in a pattern like this:
704:
705: ^a++\w!
706:
707: This pattern matches "aaab!" but not "aaa!", which would be matched by
708: a non-possessive quantifier. Similarly, if an atomic group is present,
709: it is matched as if it were a standalone pattern at the current point,
710: and the longest match is then "locked in" for the rest of the overall
711: pattern.
712:
713: 2. When dealing with multiple paths through the tree simultaneously, it
714: is not straightforward to keep track of captured substrings for the
715: different matching possibilities, and PCRE's implementation of this
716: algorithm does not attempt to do this. This means that no captured sub-
717: strings are available.
718:
719: 3. Because no substrings are captured, back references within the pat-
720: tern are not supported, and cause errors if encountered.
721:
722: 4. For the same reason, conditional expressions that use a backrefer-
723: ence as the condition or test for a specific group recursion are not
724: supported.
725:
726: 5. Because many paths through the tree may be active, the \K escape
727: sequence, which resets the start of the match when encountered (but may
728: be on some paths and not on others), is not supported. It causes an
729: error if encountered.
730:
731: 6. Callouts are supported, but the value of the capture_top field is
732: always 1, and the value of the capture_last field is always -1.
733:
734: 7. The \C escape sequence, which (in the standard algorithm) matches a
735: single byte, even in UTF-8 mode, is not supported because the alterna-
736: tive algorithm moves through the subject string one character at a
737: time, for all active paths through the tree.
738:
739: 8. Except for (*FAIL), the backtracking control verbs such as (*PRUNE)
740: are not supported. (*FAIL) is supported, and behaves like a failing
741: negative assertion.
742:
743:
744: ADVANTAGES OF THE ALTERNATIVE ALGORITHM
745:
746: Using the alternative matching algorithm provides the following advan-
747: tages:
748:
749: 1. All possible matches (at a single point in the subject) are automat-
750: ically found, and in particular, the longest match is found. To find
751: more than one match using the standard algorithm, you have to do kludgy
752: things with callouts.
753:
754: 2. There is much better support for partial matching. The restrictions
755: on the content of the pattern that apply when using the standard algo-
756: rithm for partial matching do not apply to the alternative algorithm.
757: For non-anchored patterns, the starting position of a partial match is
758: available.
759:
760: 3. Because the alternative algorithm scans the subject string just
761: once, and never needs to backtrack, it is possible to pass very long
762: subject strings to the matching function in several pieces, checking
763: for partial matching each time.
764:
765:
766: DISADVANTAGES OF THE ALTERNATIVE ALGORITHM
767:
768: The alternative algorithm suffers from a number of disadvantages:
769:
770: 1. It is substantially slower than the standard algorithm. This is
771: partly because it has to search for all possible matches, but is also
772: because it is less susceptible to optimization.
773:
774: 2. Capturing parentheses and back references are not supported.
775:
776: 3. Although atomic groups are supported, their use does not provide the
777: performance advantage that it does for the standard algorithm.
778:
779:
780: AUTHOR
781:
782: Philip Hazel
783: University Computing Service
784: Cambridge CB2 3QH, England.
785:
786:
787: REVISION
788:
789: Last updated: 19 April 2008
790: Copyright (c) 1997-2008 University of Cambridge.
791: ------------------------------------------------------------------------------
792:
793:
794: PCREAPI(3) PCREAPI(3)
795:
796:
797: NAME
798: PCRE - Perl-compatible regular expressions
799:
800:
801: PCRE NATIVE API
802:
803: #include <pcre.h>
804:
805: pcre *pcre_compile(const char *pattern, int options,
806: const char **errptr, int *erroffset,
807: const unsigned char *tableptr);
808:
809: pcre *pcre_compile2(const char *pattern, int options,
810: int *errorcodeptr,
811: const char **errptr, int *erroffset,
812: const unsigned char *tableptr);
813:
814: pcre_extra *pcre_study(const pcre *code, int options,
815: const char **errptr);
816:
817: int pcre_exec(const pcre *code, const pcre_extra *extra,
818: const char *subject, int length, int startoffset,
819: int options, int *ovector, int ovecsize);
820:
821: int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
822: const char *subject, int length, int startoffset,
823: int options, int *ovector, int ovecsize,
824: int *workspace, int wscount);
825:
826: int pcre_copy_named_substring(const pcre *code,
827: const char *subject, int *ovector,
828: int stringcount, const char *stringname,
829: char *buffer, int buffersize);
830:
831: int pcre_copy_substring(const char *subject, int *ovector,
832: int stringcount, int stringnumber, char *buffer,
833: int buffersize);
834:
835: int pcre_get_named_substring(const pcre *code,
836: const char *subject, int *ovector,
837: int stringcount, const char *stringname,
838: const char **stringptr);
839:
840: int pcre_get_stringnumber(const pcre *code,
841: const char *name);
842:
843: int pcre_get_stringtable_entries(const pcre *code,
844: const char *name, char **first, char **last);
845:
846: int pcre_get_substring(const char *subject, int *ovector,
847: int stringcount, int stringnumber,
848: const char **stringptr);
849:
850: int pcre_get_substring_list(const char *subject,
851: int *ovector, int stringcount, const char ***listptr);
852:
853: void pcre_free_substring(const char *stringptr);
854:
855: void pcre_free_substring_list(const char **stringptr);
856:
857: const unsigned char *pcre_maketables(void);
858:
859: int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
860: int what, void *where);
861:
862: int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
863:
864: int pcre_refcount(pcre *code, int adjust);
865:
866: int pcre_config(int what, void *where);
867:
868: char *pcre_version(void);
869:
870: void *(*pcre_malloc)(size_t);
871:
872: void (*pcre_free)(void *);
873:
874: void *(*pcre_stack_malloc)(size_t);
875:
876: void (*pcre_stack_free)(void *);
877:
878: int (*pcre_callout)(pcre_callout_block *);
879:
880:
881: PCRE API OVERVIEW
882:
883: PCRE has its own native API, which is described in this document. There
884: are also some wrapper functions that correspond to the POSIX regular
885: expression API. These are described in the pcreposix documentation.
886: Both of these APIs define a set of C function calls. A C++ wrapper is
887: distributed with PCRE. It is documented in the pcrecpp page.
888:
889: The native API C function prototypes are defined in the header file
890: pcre.h, and on Unix systems the library itself is called libpcre. It
891: can normally be accessed by adding -lpcre to the command for linking an
892: application that uses PCRE. The header file defines the macros
893: PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
894: bers for the library. Applications can use these to include support
895: for different releases of PCRE.
896:
897: The functions pcre_compile(), pcre_compile2(), pcre_study(), and
898: pcre_exec() are used for compiling and matching regular expressions in
899: a Perl-compatible manner. A sample program that demonstrates the sim-
900: plest way of using them is provided in the file called pcredemo.c in
901: the source distribution. The pcresample documentation describes how to
902: compile and run it.
903:
904: A second matching function, pcre_dfa_exec(), which is not Perl-compati-
905: ble, is also provided. This uses a different algorithm for the match-
906: ing. The alternative algorithm finds all possible matches (at a given
907: point in the subject), and scans the subject just once. However, this
908: algorithm does not return captured substrings. A description of the two
909: matching algorithms and their advantages and disadvantages is given in
910: the pcrematching documentation.
911:
912: In addition to the main compiling and matching functions, there are
913: convenience functions for extracting captured substrings from a subject
914: string that is matched by pcre_exec(). They are:
915:
916: pcre_copy_substring()
917: pcre_copy_named_substring()
918: pcre_get_substring()
919: pcre_get_named_substring()
920: pcre_get_substring_list()
921: pcre_get_stringnumber()
922: pcre_get_stringtable_entries()
923:
924: pcre_free_substring() and pcre_free_substring_list() are also provided,
925: to free the memory used for extracted strings.
926:
927: The function pcre_maketables() is used to build a set of character
928: tables in the current locale for passing to pcre_compile(),
929: pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
930: provided for specialist use. Most commonly, no special tables are
931: passed, in which case internal tables that are generated when PCRE is
932: built are used.
933:
934: The function pcre_fullinfo() is used to find out information about a
935: compiled pattern; pcre_info() is an obsolete version that returns only
936: some of the available information, but is retained for backwards com-
937: patibility. The function pcre_version() returns a pointer to a string
938: containing the version of PCRE and its date of release.
939:
940: The function pcre_refcount() maintains a reference count in a data
941: block containing a compiled pattern. This is provided for the benefit
942: of object-oriented applications.
943:
944: The global variables pcre_malloc and pcre_free initially contain the
945: entry points of the standard malloc() and free() functions, respec-
946: tively. PCRE calls the memory management functions via these variables,
947: so a calling program can replace them if it wishes to intercept the
948: calls. This should be done before calling any PCRE functions.
949:
950: The global variables pcre_stack_malloc and pcre_stack_free are also
951: indirections to memory management functions. These special functions
952: are used only when PCRE is compiled to use the heap for remembering
953: data, instead of recursive function calls, when running the pcre_exec()
954: function. See the pcrebuild documentation for details of how to do
955: this. It is a non-standard way of building PCRE, for use in environ-
956: ments that have limited stacks. Because of the greater use of memory
957: management, it runs more slowly. Separate functions are provided so
958: that special-purpose external code can be used for this case. When
959: used, these functions are always called in a stack-like manner (last
960: obtained, first freed), and always for memory blocks of the same size.
961: There is a discussion about PCRE's stack usage in the pcrestack docu-
962: mentation.
963:
964: The global variable pcre_callout initially contains NULL. It can be set
965: by the caller to a "callout" function, which PCRE will then call at
966: specified points during a matching operation. Details are given in the
967: pcrecallout documentation.
968:
969:
970: NEWLINES
971:
972: PCRE supports five different conventions for indicating line breaks in
973: strings: a single CR (carriage return) character, a single LF (line-
974: feed) character, the two-character sequence CRLF, any of the three pre-
975: ceding, or any Unicode newline sequence. The Unicode newline sequences
976: are the three just mentioned, plus the single characters VT (vertical
977: tab, U+000B), FF (formfeed, U+000C), NEL (next line, U+0085), LS (line
978: separator, U+2028), and PS (paragraph separator, U+2029).
979:
980: Each of the first three conventions is used by at least one operating
981: system as its standard newline sequence. When PCRE is built, a default
982: can be specified. The default default is LF, which is the Unix stan-
983: dard. When PCRE is run, the default can be overridden, either when a
984: pattern is compiled, or when it is matched.
985:
986: At compile time, the newline convention can be specified by the options
987: argument of pcre_compile(), or it can be specified by special text at
988: the start of the pattern itself; this overrides any other settings. See
989: the pcrepattern page for details of the special character sequences.
990:
991: In the PCRE documentation the word "newline" is used to mean "the char-
992: acter or pair of characters that indicate a line break". The choice of
993: newline convention affects the handling of the dot, circumflex, and
994: dollar metacharacters, the handling of #-comments in /x mode, and, when
995: CRLF is a recognized line ending sequence, the match position advance-
996: ment for a non-anchored pattern. There is more detail about this in the
997: section on pcre_exec() options below.
998:
999: The choice of newline convention does not affect the interpretation of
1000: the \n or \r escape sequences, nor does it affect what \R matches,
1001: which is controlled in a similar way, but by separate options.
1002:
1003:
1004: MULTITHREADING
1005:
1006: The PCRE functions can be used in multi-threading applications, with
1007: the proviso that the memory management functions pointed to by
1008: pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
1009: callout function pointed to by pcre_callout, are shared by all threads.
1010:
1.3 ! misha 1011: The compiled form of a regular expression is not altered during match-
1.1 misha 1012: ing, so the same compiled pattern can safely be used by several threads
1013: at once.
1014:
1015:
1016: SAVING PRECOMPILED PATTERNS FOR LATER USE
1017:
1018: The compiled form of a regular expression can be saved and re-used at a
1.3 ! misha 1019: later time, possibly by a different program, and even on a host other
! 1020: than the one on which it was compiled. Details are given in the
! 1021: pcreprecompile documentation. However, compiling a regular expression
! 1022: with one version of PCRE for use with a different version is not guar-
1.1 misha 1023: anteed to work and may cause crashes.
1024:
1025:
1026: CHECKING BUILD-TIME OPTIONS
1027:
1028: int pcre_config(int what, void *where);
1029:
1.3 ! misha 1030: The function pcre_config() makes it possible for a PCRE client to dis-
1.1 misha 1031: cover which optional features have been compiled into the PCRE library.
1.3 ! misha 1032: The pcrebuild documentation has more details about these optional fea-
1.1 misha 1033: tures.
1034:
1.3 ! misha 1035: The first argument for pcre_config() is an integer, specifying which
1.1 misha 1036: information is required; the second argument is a pointer to a variable
1.3 ! misha 1037: into which the information is placed. The following information is
1.1 misha 1038: available:
1039:
1040: PCRE_CONFIG_UTF8
1041:
1.3 ! misha 1042: The output is an integer that is set to one if UTF-8 support is avail-
1.1 misha 1043: able; otherwise it is set to zero.
1044:
1045: PCRE_CONFIG_UNICODE_PROPERTIES
1046:
1.3 ! misha 1047: The output is an integer that is set to one if support for Unicode
1.1 misha 1048: character properties is available; otherwise it is set to zero.
1049:
1050: PCRE_CONFIG_NEWLINE
1051:
1.3 ! misha 1052: The output is an integer whose value specifies the default character
! 1053: sequence that is recognized as meaning "newline". The four values that
1.1 misha 1054: are supported are: 10 for LF, 13 for CR, 3338 for CRLF, -2 for ANYCRLF,
1.3 ! misha 1055: and -1 for ANY. Though they are derived from ASCII, the same values
! 1056: are returned in EBCDIC environments. The default should normally corre-
! 1057: spond to the standard sequence for your operating system.
1.1 misha 1058:
1059: PCRE_CONFIG_BSR
1060:
1061: The output is an integer whose value indicates what character sequences
1062: the \R escape sequence matches by default. A value of 0 means that \R
1063: matches any Unicode line ending sequence; a value of 1 means that \R
1064: matches only CR, LF, or CRLF. The default can be overridden when a pat-
1065: tern is compiled or matched.
1066:
1067: PCRE_CONFIG_LINK_SIZE
1068:
1069: The output is an integer that contains the number of bytes used for
1070: internal linkage in compiled regular expressions. The value is 2, 3, or
1071: 4. Larger values allow larger regular expressions to be compiled, at
1072: the expense of slower matching. The default value of 2 is sufficient
1073: for all but the most massive patterns, since it allows the compiled
1074: pattern to be up to 64K in size.
1075:
1076: PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
1077:
1078: The output is an integer that contains the threshold above which the
1079: POSIX interface uses malloc() for output vectors. Further details are
1080: given in the pcreposix documentation.
1081:
1082: PCRE_CONFIG_MATCH_LIMIT
1083:
1.3 ! misha 1084: The output is a long integer that gives the default limit for the num-
! 1085: ber of internal matching function calls in a pcre_exec() execution.
! 1086: Further details are given with pcre_exec() below.
1.1 misha 1087:
1088: PCRE_CONFIG_MATCH_LIMIT_RECURSION
1089:
1.3 ! misha 1090: The output is a long integer that gives the default limit for the depth
! 1091: of recursion when calling the internal matching function in a
! 1092: pcre_exec() execution. Further details are given with pcre_exec()
! 1093: below.
1.1 misha 1094:
1095: PCRE_CONFIG_STACKRECURSE
1096:
1.3 ! misha 1097: The output is an integer that is set to one if internal recursion when
1.1 misha 1098: running pcre_exec() is implemented by recursive function calls that use
1.3 ! misha 1099: the stack to remember their state. This is the usual way that PCRE is
1.1 misha 1100: compiled. The output is zero if PCRE was compiled to use blocks of data
1.3 ! misha 1101: on the heap instead of recursive function calls. In this case,
! 1102: pcre_stack_malloc and pcre_stack_free are called to manage memory
1.1 misha 1103: blocks on the heap, thus avoiding the use of the stack.
1104:
1105:
1106: COMPILING A PATTERN
1107:
1108: pcre *pcre_compile(const char *pattern, int options,
1109: const char **errptr, int *erroffset,
1110: const unsigned char *tableptr);
1111:
1112: pcre *pcre_compile2(const char *pattern, int options,
1113: int *errorcodeptr,
1114: const char **errptr, int *erroffset,
1115: const unsigned char *tableptr);
1116:
1117: Either of the functions pcre_compile() or pcre_compile2() can be called
1118: to compile a pattern into an internal form. The only difference between
1.3 ! misha 1119: the two interfaces is that pcre_compile2() has an additional argument,
1.1 misha 1120: errorcodeptr, via which a numerical error code can be returned.
1121:
1122: The pattern is a C string terminated by a binary zero, and is passed in
1.3 ! misha 1123: the pattern argument. A pointer to a single block of memory that is
! 1124: obtained via pcre_malloc is returned. This contains the compiled code
1.1 misha 1125: and related data. The pcre type is defined for the returned block; this
1126: is a typedef for a structure whose contents are not externally defined.
1127: It is up to the caller to free the memory (via pcre_free) when it is no
1128: longer required.
1129:
1.3 ! misha 1130: Although the compiled code of a PCRE regex is relocatable, that is, it
1.1 misha 1131: does not depend on memory location, the complete pcre data block is not
1.3 ! misha 1132: fully relocatable, because it may contain a copy of the tableptr argu-
1.1 misha 1133: ment, which is an address (see below).
1134:
1135: The options argument contains various bit settings that affect the com-
1.3 ! misha 1136: pilation. It should be zero if no options are required. The available
! 1137: options are described below. Some of them (in particular, those that
! 1138: are compatible with Perl, but also some others) can also be set and
! 1139: unset from within the pattern (see the detailed description in the
! 1140: pcrepattern documentation). For those options that can be different in
! 1141: different parts of the pattern, the contents of the options argument
! 1142: specifies their initial settings at the start of compilation and execu-
! 1143: tion. The PCRE_ANCHORED and PCRE_NEWLINE_xxx options can be set at the
! 1144: time of matching as well as at compile time.
1.1 misha 1145:
1146: If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
1147: if compilation of a pattern fails, pcre_compile() returns NULL, and
1148: sets the variable pointed to by errptr to point to a textual error mes-
1149: sage. This is a static string that is part of the library. You must not
1150: try to free it. The offset from the start of the pattern to the charac-
1151: ter where the error was discovered is placed in the variable pointed to
1152: by erroffset, which must not be NULL. If it is, an immediate error is
1153: given.
1154:
1155: If pcre_compile2() is used instead of pcre_compile(), and the error-
1156: codeptr argument is not NULL, a non-zero error code number is returned
1157: via this argument in the event of an error. This is in addition to the
1158: textual error message. Error codes and messages are listed below.
1159:
1160: If the final argument, tableptr, is NULL, PCRE uses a default set of
1161: character tables that are built when PCRE is compiled, using the
1162: default C locale. Otherwise, tableptr must be an address that is the
1163: result of a call to pcre_maketables(). This value is stored with the
1164: compiled pattern, and used again by pcre_exec(), unless another table
1165: pointer is passed to it. For more discussion, see the section on locale
1166: support below.
1167:
1168: This code fragment shows a typical straightforward call to pcre_com-
1169: pile():
1170:
1171: pcre *re;
1172: const char *error;
1173: int erroffset;
1174: re = pcre_compile(
1175: "^A.*Z", /* the pattern */
1176: 0, /* default options */
1177: &error, /* for error message */
1178: &erroffset, /* for error offset */
1179: NULL); /* use default character tables */
1180:
1181: The following names for option bits are defined in the pcre.h header
1182: file:
1183:
1184: PCRE_ANCHORED
1185:
1186: If this bit is set, the pattern is forced to be "anchored", that is, it
1187: is constrained to match only at the first matching point in the string
1188: that is being searched (the "subject string"). This effect can also be
1189: achieved by appropriate constructs in the pattern itself, which is the
1190: only way to do it in Perl.
1191:
1192: PCRE_AUTO_CALLOUT
1193:
1194: If this bit is set, pcre_compile() automatically inserts callout items,
1195: all with number 255, before each pattern item. For discussion of the
1196: callout facility, see the pcrecallout documentation.
1197:
1198: PCRE_BSR_ANYCRLF
1199: PCRE_BSR_UNICODE
1200:
1201: These options (which are mutually exclusive) control what the \R escape
1202: sequence matches. The choice is either to match only CR, LF, or CRLF,
1203: or to match any Unicode newline sequence. The default is specified when
1204: PCRE is built. It can be overridden from within the pattern, or by set-
1205: ting an option when a compiled pattern is matched.
1206:
1207: PCRE_CASELESS
1208:
1209: If this bit is set, letters in the pattern match both upper and lower
1210: case letters. It is equivalent to Perl's /i option, and it can be
1211: changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
1212: always understands the concept of case for characters whose values are
1213: less than 128, so caseless matching is always possible. For characters
1214: with higher values, the concept of case is supported if PCRE is com-
1215: piled with Unicode property support, but not otherwise. If you want to
1216: use caseless matching for characters 128 and above, you must ensure
1217: that PCRE is compiled with Unicode property support as well as with
1218: UTF-8 support.
1219:
1220: PCRE_DOLLAR_ENDONLY
1221:
1222: If this bit is set, a dollar metacharacter in the pattern matches only
1223: at the end of the subject string. Without this option, a dollar also
1224: matches immediately before a newline at the end of the string (but not
1225: before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored
1226: if PCRE_MULTILINE is set. There is no equivalent to this option in
1227: Perl, and no way to set it within a pattern.
1228:
1229: PCRE_DOTALL
1230:
1231: If this bit is set, a dot metacharater in the pattern matches all char-
1232: acters, including those that indicate newline. Without it, a dot does
1233: not match when the current position is at a newline. This option is
1234: equivalent to Perl's /s option, and it can be changed within a pattern
1235: by a (?s) option setting. A negative class such as [^a] always matches
1236: newline characters, independent of the setting of this option.
1237:
1238: PCRE_DUPNAMES
1239:
1240: If this bit is set, names used to identify capturing subpatterns need
1241: not be unique. This can be helpful for certain types of pattern when it
1242: is known that only one instance of the named subpattern can ever be
1243: matched. There are more details of named subpatterns below; see also
1244: the pcrepattern documentation.
1245:
1246: PCRE_EXTENDED
1247:
1248: If this bit is set, whitespace data characters in the pattern are
1249: totally ignored except when escaped or inside a character class. White-
1250: space does not include the VT character (code 11). In addition, charac-
1251: ters between an unescaped # outside a character class and the next new-
1252: line, inclusive, are also ignored. This is equivalent to Perl's /x
1253: option, and it can be changed within a pattern by a (?x) option set-
1254: ting.
1255:
1256: This option makes it possible to include comments inside complicated
1257: patterns. Note, however, that this applies only to data characters.
1258: Whitespace characters may never appear within special character
1259: sequences in a pattern, for example within the sequence (?( which
1260: introduces a conditional subpattern.
1261:
1262: PCRE_EXTRA
1263:
1264: This option was invented in order to turn on additional functionality
1265: of PCRE that is incompatible with Perl, but it is currently of very
1266: little use. When set, any backslash in a pattern that is followed by a
1267: letter that has no special meaning causes an error, thus reserving
1268: these combinations for future expansion. By default, as in Perl, a
1269: backslash followed by a letter with no special meaning is treated as a
1270: literal. (Perl can, however, be persuaded to give a warning for this.)
1271: There are at present no other features controlled by this option. It
1272: can also be set by a (?X) option setting within a pattern.
1273:
1274: PCRE_FIRSTLINE
1275:
1276: If this option is set, an unanchored pattern is required to match
1277: before or at the first newline in the subject string, though the
1278: matched text may continue over the newline.
1279:
1280: PCRE_JAVASCRIPT_COMPAT
1281:
1282: If this option is set, PCRE's behaviour is changed in some ways so that
1283: it is compatible with JavaScript rather than Perl. The changes are as
1284: follows:
1285:
1286: (1) A lone closing square bracket in a pattern causes a compile-time
1287: error, because this is illegal in JavaScript (by default it is treated
1288: as a data character). Thus, the pattern AB]CD becomes illegal when this
1289: option is set.
1290:
1291: (2) At run time, a back reference to an unset subpattern group matches
1292: an empty string (by default this causes the current matching alterna-
1293: tive to fail). A pattern such as (\1)(a) succeeds when this option is
1294: set (assuming it can find an "a" in the subject), whereas it fails by
1295: default, for Perl compatibility.
1296:
1297: PCRE_MULTILINE
1298:
1299: By default, PCRE treats the subject string as consisting of a single
1300: line of characters (even if it actually contains newlines). The "start
1301: of line" metacharacter (^) matches only at the start of the string,
1302: while the "end of line" metacharacter ($) matches only at the end of
1303: the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
1304: is set). This is the same as Perl.
1305:
1306: When PCRE_MULTILINE it is set, the "start of line" and "end of line"
1307: constructs match immediately following or immediately before internal
1308: newlines in the subject string, respectively, as well as at the very
1309: start and end. This is equivalent to Perl's /m option, and it can be
1310: changed within a pattern by a (?m) option setting. If there are no new-
1311: lines in a subject string, or no occurrences of ^ or $ in a pattern,
1312: setting PCRE_MULTILINE has no effect.
1313:
1314: PCRE_NEWLINE_CR
1315: PCRE_NEWLINE_LF
1316: PCRE_NEWLINE_CRLF
1317: PCRE_NEWLINE_ANYCRLF
1318: PCRE_NEWLINE_ANY
1319:
1320: These options override the default newline definition that was chosen
1321: when PCRE was built. Setting the first or the second specifies that a
1322: newline is indicated by a single character (CR or LF, respectively).
1323: Setting PCRE_NEWLINE_CRLF specifies that a newline is indicated by the
1324: two-character CRLF sequence. Setting PCRE_NEWLINE_ANYCRLF specifies
1325: that any of the three preceding sequences should be recognized. Setting
1326: PCRE_NEWLINE_ANY specifies that any Unicode newline sequence should be
1327: recognized. The Unicode newline sequences are the three just mentioned,
1328: plus the single characters VT (vertical tab, U+000B), FF (formfeed,
1329: U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS
1330: (paragraph separator, U+2029). The last two are recognized only in
1331: UTF-8 mode.
1332:
1333: The newline setting in the options word uses three bits that are
1334: treated as a number, giving eight possibilities. Currently only six are
1335: used (default plus the five values above). This means that if you set
1336: more than one newline option, the combination may or may not be sensi-
1337: ble. For example, PCRE_NEWLINE_CR with PCRE_NEWLINE_LF is equivalent to
1338: PCRE_NEWLINE_CRLF, but other combinations may yield unused numbers and
1339: cause an error.
1340:
1341: The only time that a line break is specially recognized when compiling
1342: a pattern is if PCRE_EXTENDED is set, and an unescaped # outside a
1343: character class is encountered. This indicates a comment that lasts
1344: until after the next line break sequence. In other circumstances, line
1345: break sequences are treated as literal data, except that in
1346: PCRE_EXTENDED mode, both CR and LF are treated as whitespace characters
1347: and are therefore ignored.
1348:
1349: The newline option that is set at compile time becomes the default that
1.3 ! misha 1350: is used for pcre_exec() and pcre_dfa_exec(), but it can be overridden.
1.1 misha 1351:
1352: PCRE_NO_AUTO_CAPTURE
1353:
1354: If this option is set, it disables the use of numbered capturing paren-
1.3 ! misha 1355: theses in the pattern. Any opening parenthesis that is not followed by
! 1356: ? behaves as if it were followed by ?: but named parentheses can still
! 1357: be used for capturing (and they acquire numbers in the usual way).
1.1 misha 1358: There is no equivalent of this option in Perl.
1359:
1360: PCRE_UNGREEDY
1361:
1.3 ! misha 1362: This option inverts the "greediness" of the quantifiers so that they
! 1363: are not greedy by default, but become greedy if followed by "?". It is
! 1364: not compatible with Perl. It can also be set by a (?U) option setting
1.1 misha 1365: within the pattern.
1366:
1367: PCRE_UTF8
1368:
1.3 ! misha 1369: This option causes PCRE to regard both the pattern and the subject as
! 1370: strings of UTF-8 characters instead of single-byte character strings.
! 1371: However, it is available only when PCRE is built to include UTF-8 sup-
! 1372: port. If not, the use of this option provokes an error. Details of how
! 1373: this option changes the behaviour of PCRE are given in the section on
1.1 misha 1374: UTF-8 support in the main pcre page.
1375:
1376: PCRE_NO_UTF8_CHECK
1377:
1378: When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1.3 ! misha 1379: automatically checked. There is a discussion about the validity of
! 1380: UTF-8 strings in the main pcre page. If an invalid UTF-8 sequence of
! 1381: bytes is found, pcre_compile() returns an error. If you already know
1.1 misha 1382: that your pattern is valid, and you want to skip this check for perfor-
1.3 ! misha 1383: mance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is
! 1384: set, the effect of passing an invalid UTF-8 string as a pattern is
! 1385: undefined. It may cause your program to crash. Note that this option
! 1386: can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the
1.1 misha 1387: UTF-8 validity checking of subject strings.
1388:
1389:
1390: COMPILATION ERROR CODES
1391:
1.3 ! misha 1392: The following table lists the error codes than may be returned by
! 1393: pcre_compile2(), along with the error messages that may be returned by
! 1394: both compiling functions. As PCRE has developed, some error codes have
1.1 misha 1395: fallen out of use. To avoid confusion, they have not been re-used.
1396:
1397: 0 no error
1398: 1 \ at end of pattern
1399: 2 \c at end of pattern
1400: 3 unrecognized character follows \
1401: 4 numbers out of order in {} quantifier
1402: 5 number too big in {} quantifier
1403: 6 missing terminating ] for character class
1404: 7 invalid escape sequence in character class
1405: 8 range out of order in character class
1406: 9 nothing to repeat
1407: 10 [this code is not in use]
1408: 11 internal error: unexpected repeat
1409: 12 unrecognized character after (? or (?-
1410: 13 POSIX named classes are supported only within a class
1411: 14 missing )
1412: 15 reference to non-existent subpattern
1413: 16 erroffset passed as NULL
1414: 17 unknown option bit(s) set
1415: 18 missing ) after comment
1416: 19 [this code is not in use]
1417: 20 regular expression is too large
1418: 21 failed to get memory
1419: 22 unmatched parentheses
1420: 23 internal error: code overflow
1421: 24 unrecognized character after (?<
1422: 25 lookbehind assertion is not fixed length
1423: 26 malformed number or name after (?(
1424: 27 conditional group contains more than two branches
1425: 28 assertion expected after (?(
1426: 29 (?R or (?[+-]digits must be followed by )
1427: 30 unknown POSIX class name
1428: 31 POSIX collating elements are not supported
1429: 32 this version of PCRE is not compiled with PCRE_UTF8 support
1430: 33 [this code is not in use]
1431: 34 character value in \x{...} sequence is too large
1432: 35 invalid condition (?(0)
1433: 36 \C not allowed in lookbehind assertion
1434: 37 PCRE does not support \L, \l, \N, \U, or \u
1435: 38 number after (?C is > 255
1436: 39 closing ) for (?C expected
1437: 40 recursive call could loop indefinitely
1438: 41 unrecognized character after (?P
1439: 42 syntax error in subpattern name (missing terminator)
1440: 43 two named subpatterns have the same name
1441: 44 invalid UTF-8 string
1442: 45 support for \P, \p, and \X has not been compiled
1443: 46 malformed \P or \p sequence
1444: 47 unknown property name after \P or \p
1445: 48 subpattern name is too long (maximum 32 characters)
1446: 49 too many named subpatterns (maximum 10000)
1447: 50 [this code is not in use]
1448: 51 octal value is greater than \377 (not in UTF-8 mode)
1449: 52 internal error: overran compiling workspace
1.3 ! misha 1450: 53 internal error: previously-checked referenced subpattern not
1.1 misha 1451: found
1452: 54 DEFINE group contains more than one branch
1453: 55 repeating a DEFINE group is not allowed
1454: 56 inconsistent NEWLINE options
1455: 57 \g is not followed by a braced, angle-bracketed, or quoted
1456: name/number or by a plain number
1457: 58 a numbered reference must not be zero
1458: 59 (*VERB) with an argument is not supported
1459: 60 (*VERB) not recognized
1460: 61 number is too big
1461: 62 subpattern name expected
1462: 63 digit expected after (?+
1463: 64 ] is an invalid data character in JavaScript compatibility mode
1464:
1.3 ! misha 1465: The numbers 32 and 10000 in errors 48 and 49 are defaults; different
1.1 misha 1466: values may be used if the limits were changed when PCRE was built.
1467:
1468:
1469: STUDYING A PATTERN
1470:
1471: pcre_extra *pcre_study(const pcre *code, int options
1472: const char **errptr);
1473:
1.3 ! misha 1474: If a compiled pattern is going to be used several times, it is worth
1.1 misha 1475: spending more time analyzing it in order to speed up the time taken for
1.3 ! misha 1476: matching. The function pcre_study() takes a pointer to a compiled pat-
1.1 misha 1477: tern as its first argument. If studying the pattern produces additional
1.3 ! misha 1478: information that will help speed up matching, pcre_study() returns a
! 1479: pointer to a pcre_extra block, in which the study_data field points to
1.1 misha 1480: the results of the study.
1481:
1482: The returned value from pcre_study() can be passed directly to
1.3 ! misha 1483: pcre_exec(). However, a pcre_extra block also contains other fields
! 1484: that can be set by the caller before the block is passed; these are
1.1 misha 1485: described below in the section on matching a pattern.
1486:
1.3 ! misha 1487: If studying the pattern does not produce any additional information
1.1 misha 1488: pcre_study() returns NULL. In that circumstance, if the calling program
1.3 ! misha 1489: wants to pass any of the other fields to pcre_exec(), it must set up
1.1 misha 1490: its own pcre_extra block.
1491:
1.3 ! misha 1492: The second argument of pcre_study() contains option bits. At present,
1.1 misha 1493: no options are defined, and this argument should always be zero.
1494:
1.3 ! misha 1495: The third argument for pcre_study() is a pointer for an error message.
! 1496: If studying succeeds (even if no data is returned), the variable it
! 1497: points to is set to NULL. Otherwise it is set to point to a textual
1.1 misha 1498: error message. This is a static string that is part of the library. You
1.3 ! misha 1499: must not try to free it. You should test the error pointer for NULL
1.1 misha 1500: after calling pcre_study(), to be sure that it has run successfully.
1501:
1502: This is a typical call to pcre_study():
1503:
1504: pcre_extra *pe;
1505: pe = pcre_study(
1506: re, /* result of pcre_compile() */
1507: 0, /* no options exist */
1508: &error); /* set to NULL or points to a message */
1509:
1510: At present, studying a pattern is useful only for non-anchored patterns
1.3 ! misha 1511: that do not have a single fixed starting character. A bitmap of possi-
1.1 misha 1512: ble starting bytes is created.
1513:
1514:
1515: LOCALE SUPPORT
1516:
1.3 ! misha 1517: PCRE handles caseless matching, and determines whether characters are
! 1518: letters, digits, or whatever, by reference to a set of tables, indexed
! 1519: by character value. When running in UTF-8 mode, this applies only to
! 1520: characters with codes less than 128. Higher-valued codes never match
! 1521: escapes such as \w or \d, but can be tested with \p if PCRE is built
! 1522: with Unicode character property support. The use of locales with Uni-
! 1523: code is discouraged. If you are handling characters with codes greater
! 1524: than 128, you should either use UTF-8 and Unicode, or use locales, but
1.1 misha 1525: not try to mix the two.
1526:
1.3 ! misha 1527: PCRE contains an internal set of tables that are used when the final
! 1528: argument of pcre_compile() is NULL. These are sufficient for many
1.1 misha 1529: applications. Normally, the internal tables recognize only ASCII char-
1530: acters. However, when PCRE is built, it is possible to cause the inter-
1531: nal tables to be rebuilt in the default "C" locale of the local system,
1532: which may cause them to be different.
1533:
1.3 ! misha 1534: The internal tables can always be overridden by tables supplied by the
1.1 misha 1535: application that calls PCRE. These may be created in a different locale
1.3 ! misha 1536: from the default. As more and more applications change to using Uni-
1.1 misha 1537: code, the need for this locale support is expected to die away.
1538:
1.3 ! misha 1539: External tables are built by calling the pcre_maketables() function,
! 1540: which has no arguments, in the relevant locale. The result can then be
! 1541: passed to pcre_compile() or pcre_exec() as often as necessary. For
! 1542: example, to build and use tables that are appropriate for the French
! 1543: locale (where accented characters with values greater than 128 are
1.1 misha 1544: treated as letters), the following code could be used:
1545:
1546: setlocale(LC_CTYPE, "fr_FR");
1547: tables = pcre_maketables();
1548: re = pcre_compile(..., tables);
1549:
1.3 ! misha 1550: The locale name "fr_FR" is used on Linux and other Unix-like systems;
1.1 misha 1551: if you are using Windows, the name for the French locale is "french".
1552:
1.3 ! misha 1553: When pcre_maketables() runs, the tables are built in memory that is
! 1554: obtained via pcre_malloc. It is the caller's responsibility to ensure
! 1555: that the memory containing the tables remains available for as long as
1.1 misha 1556: it is needed.
1557:
1558: The pointer that is passed to pcre_compile() is saved with the compiled
1.3 ! misha 1559: pattern, and the same tables are used via this pointer by pcre_study()
1.1 misha 1560: and normally also by pcre_exec(). Thus, by default, for any single pat-
1561: tern, compilation, studying and matching all happen in the same locale,
1562: but different patterns can be compiled in different locales.
1563:
1.3 ! misha 1564: It is possible to pass a table pointer or NULL (indicating the use of
! 1565: the internal tables) to pcre_exec(). Although not intended for this
! 1566: purpose, this facility could be used to match a pattern in a different
1.1 misha 1567: locale from the one in which it was compiled. Passing table pointers at
1568: run time is discussed below in the section on matching a pattern.
1569:
1570:
1571: INFORMATION ABOUT A PATTERN
1572:
1573: int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1574: int what, void *where);
1575:
1.3 ! misha 1576: The pcre_fullinfo() function returns information about a compiled pat-
1.1 misha 1577: tern. It replaces the obsolete pcre_info() function, which is neverthe-
1578: less retained for backwards compability (and is documented below).
1579:
1.3 ! misha 1580: The first argument for pcre_fullinfo() is a pointer to the compiled
! 1581: pattern. The second argument is the result of pcre_study(), or NULL if
! 1582: the pattern was not studied. The third argument specifies which piece
! 1583: of information is required, and the fourth argument is a pointer to a
! 1584: variable to receive the data. The yield of the function is zero for
1.1 misha 1585: success, or one of the following negative numbers:
1586:
1587: PCRE_ERROR_NULL the argument code was NULL
1588: the argument where was NULL
1589: PCRE_ERROR_BADMAGIC the "magic number" was not found
1590: PCRE_ERROR_BADOPTION the value of what was invalid
1591:
1.3 ! misha 1592: The "magic number" is placed at the start of each compiled pattern as
! 1593: an simple check against passing an arbitrary memory pointer. Here is a
! 1594: typical call of pcre_fullinfo(), to obtain the length of the compiled
1.1 misha 1595: pattern:
1596:
1597: int rc;
1598: size_t length;
1599: rc = pcre_fullinfo(
1600: re, /* result of pcre_compile() */
1601: pe, /* result of pcre_study(), or NULL */
1602: PCRE_INFO_SIZE, /* what is required */
1603: &length); /* where to put the data */
1604:
1.3 ! misha 1605: The possible values for the third argument are defined in pcre.h, and
1.1 misha 1606: are as follows:
1607:
1608: PCRE_INFO_BACKREFMAX
1609:
1.3 ! misha 1610: Return the number of the highest back reference in the pattern. The
! 1611: fourth argument should point to an int variable. Zero is returned if
1.1 misha 1612: there are no back references.
1613:
1614: PCRE_INFO_CAPTURECOUNT
1615:
1.3 ! misha 1616: Return the number of capturing subpatterns in the pattern. The fourth
1.1 misha 1617: argument should point to an int variable.
1618:
1619: PCRE_INFO_DEFAULT_TABLES
1620:
1.3 ! misha 1621: Return a pointer to the internal default character tables within PCRE.
! 1622: The fourth argument should point to an unsigned char * variable. This
1.1 misha 1623: information call is provided for internal use by the pcre_study() func-
1.3 ! misha 1624: tion. External callers can cause PCRE to use its internal tables by
1.1 misha 1625: passing a NULL table pointer.
1626:
1627: PCRE_INFO_FIRSTBYTE
1628:
1.3 ! misha 1629: Return information about the first byte of any matched string, for a
! 1630: non-anchored pattern. The fourth argument should point to an int vari-
! 1631: able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name
1.1 misha 1632: is still recognized for backwards compatibility.)
1633:
1.3 ! misha 1634: If there is a fixed first byte, for example, from a pattern such as
1.1 misha 1635: (cat|cow|coyote), its value is returned. Otherwise, if either
1636:
1.3 ! misha 1637: (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1.1 misha 1638: branch starts with "^", or
1639:
1640: (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1641: set (if it were set, the pattern would be anchored),
1642:
1.3 ! misha 1643: -1 is returned, indicating that the pattern matches only at the start
! 1644: of a subject string or after any newline within the string. Otherwise
1.1 misha 1645: -2 is returned. For anchored patterns, -2 is returned.
1646:
1647: PCRE_INFO_FIRSTTABLE
1648:
1.3 ! misha 1649: If the pattern was studied, and this resulted in the construction of a
1.1 misha 1650: 256-bit table indicating a fixed set of bytes for the first byte in any
1.3 ! misha 1651: matching string, a pointer to the table is returned. Otherwise NULL is
! 1652: returned. The fourth argument should point to an unsigned char * vari-
1.1 misha 1653: able.
1654:
1655: PCRE_INFO_HASCRORLF
1656:
1.3 ! misha 1657: Return 1 if the pattern contains any explicit matches for CR or LF
! 1658: characters, otherwise 0. The fourth argument should point to an int
! 1659: variable. An explicit match is either a literal CR or LF character, or
1.1 misha 1660: \r or \n.
1661:
1662: PCRE_INFO_JCHANGED
1663:
1.3 ! misha 1664: Return 1 if the (?J) or (?-J) option setting is used in the pattern,
! 1665: otherwise 0. The fourth argument should point to an int variable. (?J)
1.1 misha 1666: and (?-J) set and unset the local PCRE_DUPNAMES option, respectively.
1667:
1668: PCRE_INFO_LASTLITERAL
1669:
1.3 ! misha 1670: Return the value of the rightmost literal byte that must exist in any
! 1671: matched string, other than at its start, if such a byte has been
1.1 misha 1672: recorded. The fourth argument should point to an int variable. If there
1.3 ! misha 1673: is no such byte, -1 is returned. For anchored patterns, a last literal
! 1674: byte is recorded only if it follows something of variable length. For
1.1 misha 1675: example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1676: /^a\dz\d/ the returned value is -1.
1677:
1678: PCRE_INFO_NAMECOUNT
1679: PCRE_INFO_NAMEENTRYSIZE
1680: PCRE_INFO_NAMETABLE
1681:
1.3 ! misha 1682: PCRE supports the use of named as well as numbered capturing parenthe-
! 1683: ses. The names are just an additional way of identifying the parenthe-
1.1 misha 1684: ses, which still acquire numbers. Several convenience functions such as
1.3 ! misha 1685: pcre_get_named_substring() are provided for extracting captured sub-
! 1686: strings by name. It is also possible to extract the data directly, by
! 1687: first converting the name to a number in order to access the correct
1.1 misha 1688: pointers in the output vector (described with pcre_exec() below). To do
1.3 ! misha 1689: the conversion, you need to use the name-to-number map, which is
1.1 misha 1690: described by these three values.
1691:
1692: The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1693: gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1.3 ! misha 1694: of each entry; both of these return an int value. The entry size
! 1695: depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
! 1696: a pointer to the first entry of the table (a pointer to char). The
1.1 misha 1697: first two bytes of each entry are the number of the capturing parenthe-
1.3 ! misha 1698: sis, most significant byte first. The rest of the entry is the corre-
! 1699: sponding name, zero terminated. The names are in alphabetical order.
1.1 misha 1700: When PCRE_DUPNAMES is set, duplicate names are in order of their paren-
1.3 ! misha 1701: theses numbers. For example, consider the following pattern (assume
! 1702: PCRE_EXTENDED is set, so white space - including newlines - is
1.1 misha 1703: ignored):
1704:
1705: (?<date> (?<year>(\d\d)?\d\d) -
1706: (?<month>\d\d) - (?<day>\d\d) )
1707:
1.3 ! misha 1708: There are four named subpatterns, so the table has four entries, and
! 1709: each entry in the table is eight bytes long. The table is as follows,
1.1 misha 1710: with non-printing bytes shows in hexadecimal, and undefined bytes shown
1711: as ??:
1712:
1713: 00 01 d a t e 00 ??
1714: 00 05 d a y 00 ?? ??
1715: 00 04 m o n t h 00
1716: 00 02 y e a r 00 ??
1717:
1.3 ! misha 1718: When writing code to extract data from named subpatterns using the
! 1719: name-to-number map, remember that the length of the entries is likely
1.1 misha 1720: to be different for each compiled pattern.
1721:
1722: PCRE_INFO_OKPARTIAL
1723:
1.3 ! misha 1724: Return 1 if the pattern can be used for partial matching, otherwise 0.
! 1725: The fourth argument should point to an int variable. The pcrepartial
! 1726: documentation lists the restrictions that apply to patterns when par-
1.1 misha 1727: tial matching is used.
1728:
1729: PCRE_INFO_OPTIONS
1730:
1.3 ! misha 1731: Return a copy of the options with which the pattern was compiled. The
! 1732: fourth argument should point to an unsigned long int variable. These
1.1 misha 1733: option bits are those specified in the call to pcre_compile(), modified
1734: by any top-level option settings at the start of the pattern itself. In
1.3 ! misha 1735: other words, they are the options that will be in force when matching
! 1736: starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with
! 1737: the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE,
1.1 misha 1738: and PCRE_EXTENDED.
1739:
1.3 ! misha 1740: A pattern is automatically anchored by PCRE if all of its top-level
1.1 misha 1741: alternatives begin with one of the following:
1742:
1743: ^ unless PCRE_MULTILINE is set
1744: \A always
1745: \G always
1746: .* if PCRE_DOTALL is set and there are no back
1747: references to the subpattern in which .* appears
1748:
1749: For such patterns, the PCRE_ANCHORED bit is set in the options returned
1750: by pcre_fullinfo().
1751:
1752: PCRE_INFO_SIZE
1753:
1.3 ! misha 1754: Return the size of the compiled pattern, that is, the value that was
1.1 misha 1755: passed as the argument to pcre_malloc() when PCRE was getting memory in
1756: which to place the compiled data. The fourth argument should point to a
1757: size_t variable.
1758:
1759: PCRE_INFO_STUDYSIZE
1760:
1761: Return the size of the data block pointed to by the study_data field in
1.3 ! misha 1762: a pcre_extra block. That is, it is the value that was passed to
1.1 misha 1763: pcre_malloc() when PCRE was getting memory into which to place the data
1.3 ! misha 1764: created by pcre_study(). The fourth argument should point to a size_t
1.1 misha 1765: variable.
1766:
1767:
1768: OBSOLETE INFO FUNCTION
1769:
1770: int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1771:
1.3 ! misha 1772: The pcre_info() function is now obsolete because its interface is too
! 1773: restrictive to return all the available data about a compiled pattern.
! 1774: New programs should use pcre_fullinfo() instead. The yield of
! 1775: pcre_info() is the number of capturing subpatterns, or one of the fol-
1.1 misha 1776: lowing negative numbers:
1777:
1778: PCRE_ERROR_NULL the argument code was NULL
1779: PCRE_ERROR_BADMAGIC the "magic number" was not found
1780:
1.3 ! misha 1781: If the optptr argument is not NULL, a copy of the options with which
! 1782: the pattern was compiled is placed in the integer it points to (see
1.1 misha 1783: PCRE_INFO_OPTIONS above).
1784:
1.3 ! misha 1785: If the pattern is not anchored and the firstcharptr argument is not
! 1786: NULL, it is used to pass back information about the first character of
1.1 misha 1787: any matched string (see PCRE_INFO_FIRSTBYTE above).
1788:
1789:
1790: REFERENCE COUNTS
1791:
1792: int pcre_refcount(pcre *code, int adjust);
1793:
1.3 ! misha 1794: The pcre_refcount() function is used to maintain a reference count in
1.1 misha 1795: the data block that contains a compiled pattern. It is provided for the
1.3 ! misha 1796: benefit of applications that operate in an object-oriented manner,
1.1 misha 1797: where different parts of the application may be using the same compiled
1798: pattern, but you want to free the block when they are all done.
1799:
1800: When a pattern is compiled, the reference count field is initialized to
1.3 ! misha 1801: zero. It is changed only by calling this function, whose action is to
! 1802: add the adjust value (which may be positive or negative) to it. The
1.1 misha 1803: yield of the function is the new value. However, the value of the count
1.3 ! misha 1804: is constrained to lie between 0 and 65535, inclusive. If the new value
1.1 misha 1805: is outside these limits, it is forced to the appropriate limit value.
1806:
1.3 ! misha 1807: Except when it is zero, the reference count is not correctly preserved
! 1808: if a pattern is compiled on one host and then transferred to a host
1.1 misha 1809: whose byte-order is different. (This seems a highly unlikely scenario.)
1810:
1811:
1812: MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1813:
1814: int pcre_exec(const pcre *code, const pcre_extra *extra,
1815: const char *subject, int length, int startoffset,
1816: int options, int *ovector, int ovecsize);
1817:
1818: The function pcre_exec() is called to match a subject string against a
1819: compiled pattern, which is passed in the code argument. If the pattern
1820: has been studied, the result of the study should be passed in the extra
1821: argument. This function is the main matching facility of the library,
1822: and it operates in a Perl-like manner. For specialist use there is also
1823: an alternative matching function, which is described below in the sec-
1824: tion about the pcre_dfa_exec() function.
1825:
1826: In most applications, the pattern will have been compiled (and option-
1827: ally studied) in the same process that calls pcre_exec(). However, it
1828: is possible to save compiled patterns and study data, and then use them
1829: later in different processes, possibly even on different hosts. For a
1830: discussion about this, see the pcreprecompile documentation.
1831:
1832: Here is an example of a simple call to pcre_exec():
1833:
1834: int rc;
1835: int ovector[30];
1836: rc = pcre_exec(
1837: re, /* result of pcre_compile() */
1838: NULL, /* we didn't study the pattern */
1839: "some string", /* the subject string */
1840: 11, /* the length of the subject string */
1841: 0, /* start at offset 0 in the subject */
1842: 0, /* default options */
1843: ovector, /* vector of integers for substring information */
1844: 30); /* number of elements (NOT size in bytes) */
1845:
1846: Extra data for pcre_exec()
1847:
1848: If the extra argument is not NULL, it must point to a pcre_extra data
1849: block. The pcre_study() function returns such a block (when it doesn't
1850: return NULL), but you can also create one for yourself, and pass addi-
1851: tional information in it. The pcre_extra block contains the following
1852: fields (not necessarily in this order):
1853:
1854: unsigned long int flags;
1855: void *study_data;
1856: unsigned long int match_limit;
1857: unsigned long int match_limit_recursion;
1858: void *callout_data;
1859: const unsigned char *tables;
1860:
1861: The flags field is a bitmap that specifies which of the other fields
1862: are set. The flag bits are:
1863:
1864: PCRE_EXTRA_STUDY_DATA
1865: PCRE_EXTRA_MATCH_LIMIT
1866: PCRE_EXTRA_MATCH_LIMIT_RECURSION
1867: PCRE_EXTRA_CALLOUT_DATA
1868: PCRE_EXTRA_TABLES
1869:
1870: Other flag bits should be set to zero. The study_data field is set in
1871: the pcre_extra block that is returned by pcre_study(), together with
1872: the appropriate flag bit. You should not set this yourself, but you may
1873: add to the block by setting the other fields and their corresponding
1874: flag bits.
1875:
1876: The match_limit field provides a means of preventing PCRE from using up
1877: a vast amount of resources when running patterns that are not going to
1878: match, but which have a very large number of possibilities in their
1879: search trees. The classic example is the use of nested unlimited
1880: repeats.
1881:
1882: Internally, PCRE uses a function called match() which it calls repeat-
1883: edly (sometimes recursively). The limit set by match_limit is imposed
1884: on the number of times this function is called during a match, which
1885: has the effect of limiting the amount of backtracking that can take
1886: place. For patterns that are not anchored, the count restarts from zero
1887: for each position in the subject string.
1888:
1889: The default value for the limit can be set when PCRE is built; the
1890: default default is 10 million, which handles all but the most extreme
1891: cases. You can override the default by suppling pcre_exec() with a
1892: pcre_extra block in which match_limit is set, and
1893: PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1894: exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1895:
1896: The match_limit_recursion field is similar to match_limit, but instead
1897: of limiting the total number of times that match() is called, it limits
1898: the depth of recursion. The recursion depth is a smaller number than
1899: the total number of calls, because not all calls to match() are recur-
1900: sive. This limit is of use only if it is set smaller than match_limit.
1901:
1.3 ! misha 1902: Limiting the recursion depth limits the amount of stack that can be
1.1 misha 1903: used, or, when PCRE has been compiled to use memory on the heap instead
1904: of the stack, the amount of heap memory that can be used.
1905:
1.3 ! misha 1906: The default value for match_limit_recursion can be set when PCRE is
! 1907: built; the default default is the same value as the default for
! 1908: match_limit. You can override the default by suppling pcre_exec() with
! 1909: a pcre_extra block in which match_limit_recursion is set, and
! 1910: PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the
1.1 misha 1911: limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
1912:
1.3 ! misha 1913: The pcre_callout field is used in conjunction with the "callout" fea-
1.1 misha 1914: ture, which is described in the pcrecallout documentation.
1915:
1.3 ! misha 1916: The tables field is used to pass a character tables pointer to
! 1917: pcre_exec(); this overrides the value that is stored with the compiled
! 1918: pattern. A non-NULL value is stored with the compiled pattern only if
! 1919: custom tables were supplied to pcre_compile() via its tableptr argu-
1.1 misha 1920: ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1.3 ! misha 1921: PCRE's internal tables to be used. This facility is helpful when re-
! 1922: using patterns that have been saved after compiling with an external
! 1923: set of tables, because the external tables might be at a different
! 1924: address when pcre_exec() is called. See the pcreprecompile documenta-
1.1 misha 1925: tion for a discussion of saving compiled patterns for later use.
1926:
1927: Option bits for pcre_exec()
1928:
1.3 ! misha 1929: The unused bits of the options argument for pcre_exec() must be zero.
! 1930: The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx,
! 1931: PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_START_OPTIMIZE,
! 1932: PCRE_NO_UTF8_CHECK and PCRE_PARTIAL.
1.1 misha 1933:
1934: PCRE_ANCHORED
1935:
1.3 ! misha 1936: The PCRE_ANCHORED option limits pcre_exec() to matching at the first
! 1937: matching position. If a pattern was compiled with PCRE_ANCHORED, or
! 1938: turned out to be anchored by virtue of its contents, it cannot be made
1.1 misha 1939: unachored at matching time.
1940:
1941: PCRE_BSR_ANYCRLF
1942: PCRE_BSR_UNICODE
1943:
1944: These options (which are mutually exclusive) control what the \R escape
1.3 ! misha 1945: sequence matches. The choice is either to match only CR, LF, or CRLF,
! 1946: or to match any Unicode newline sequence. These options override the
1.1 misha 1947: choice that was made or defaulted when the pattern was compiled.
1948:
1949: PCRE_NEWLINE_CR
1950: PCRE_NEWLINE_LF
1951: PCRE_NEWLINE_CRLF
1952: PCRE_NEWLINE_ANYCRLF
1953: PCRE_NEWLINE_ANY
1954:
1.3 ! misha 1955: These options override the newline definition that was chosen or
! 1956: defaulted when the pattern was compiled. For details, see the descrip-
! 1957: tion of pcre_compile() above. During matching, the newline choice
! 1958: affects the behaviour of the dot, circumflex, and dollar metacharac-
! 1959: ters. It may also alter the way the match position is advanced after a
1.1 misha 1960: match failure for an unanchored pattern.
1961:
1.3 ! misha 1962: When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is
! 1963: set, and a match attempt for an unanchored pattern fails when the cur-
! 1964: rent position is at a CRLF sequence, and the pattern contains no
! 1965: explicit matches for CR or LF characters, the match position is
1.1 misha 1966: advanced by two characters instead of one, in other words, to after the
1967: CRLF.
1968:
1969: The above rule is a compromise that makes the most common cases work as
1.3 ! misha 1970: expected. For example, if the pattern is .+A (and the PCRE_DOTALL
1.1 misha 1971: option is not set), it does not match the string "\r\nA" because, after
1.3 ! misha 1972: failing at the start, it skips both the CR and the LF before retrying.
! 1973: However, the pattern [\r\n]A does match that string, because it con-
1.1 misha 1974: tains an explicit CR or LF reference, and so advances only by one char-
1975: acter after the first failure.
1976:
1977: An explicit match for CR of LF is either a literal appearance of one of
1.3 ! misha 1978: those characters, or one of the \r or \n escape sequences. Implicit
! 1979: matches such as [^X] do not count, nor does \s (which includes CR and
1.1 misha 1980: LF in the characters that it matches).
1981:
1.3 ! misha 1982: Notwithstanding the above, anomalous effects may still occur when CRLF
1.1 misha 1983: is a valid newline sequence and explicit \r or \n escapes appear in the
1984: pattern.
1985:
1986: PCRE_NOTBOL
1987:
1988: This option specifies that first character of the subject string is not
1.3 ! misha 1989: the beginning of a line, so the circumflex metacharacter should not
! 1990: match before it. Setting this without PCRE_MULTILINE (at compile time)
! 1991: causes circumflex never to match. This option affects only the behav-
1.1 misha 1992: iour of the circumflex metacharacter. It does not affect \A.
1993:
1994: PCRE_NOTEOL
1995:
1996: This option specifies that the end of the subject string is not the end
1.3 ! misha 1997: of a line, so the dollar metacharacter should not match it nor (except
! 1998: in multiline mode) a newline immediately before it. Setting this with-
1.1 misha 1999: out PCRE_MULTILINE (at compile time) causes dollar never to match. This
1.3 ! misha 2000: option affects only the behaviour of the dollar metacharacter. It does
1.1 misha 2001: not affect \Z or \z.
2002:
2003: PCRE_NOTEMPTY
2004:
2005: An empty string is not considered to be a valid match if this option is
1.3 ! misha 2006: set. If there are alternatives in the pattern, they are tried. If all
! 2007: the alternatives match the empty string, the entire match fails. For
1.1 misha 2008: example, if the pattern
2009:
2010: a?b?
2011:
1.3 ! misha 2012: is applied to a string not beginning with "a" or "b", it matches the
! 2013: empty string at the start of the subject. With PCRE_NOTEMPTY set, this
1.1 misha 2014: match is not valid, so PCRE searches further into the string for occur-
2015: rences of "a" or "b".
2016:
2017: Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
1.3 ! misha 2018: cial case of a pattern match of the empty string within its split()
! 2019: function, and when using the /g modifier. It is possible to emulate
1.1 misha 2020: Perl's behaviour after matching a null string by first trying the match
2021: again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
1.3 ! misha 2022: if that fails by advancing the starting offset (see below) and trying
1.1 misha 2023: an ordinary match again. There is some code that demonstrates how to do
2024: this in the pcredemo.c sample program.
2025:
1.3 ! misha 2026: PCRE_NO_START_OPTIMIZE
! 2027:
! 2028: There are a number of optimizations that pcre_exec() uses at the start
! 2029: of a match, in order to speed up the process. For example, if it is
! 2030: known that a match must start with a specific character, it searches
! 2031: the subject for that character, and fails immediately if it cannot find
! 2032: it, without actually running the main matching function. When callouts
! 2033: are in use, these optimizations can cause them to be skipped. This
! 2034: option disables the "start-up" optimizations, causing performance to
! 2035: suffer, but ensuring that the callouts do occur.
! 2036:
1.1 misha 2037: PCRE_NO_UTF8_CHECK
2038:
2039: When PCRE_UTF8 is set at compile time, the validity of the subject as a
2040: UTF-8 string is automatically checked when pcre_exec() is subsequently
2041: called. The value of startoffset is also checked to ensure that it
2042: points to the start of a UTF-8 character. There is a discussion about
2043: the validity of UTF-8 strings in the section on UTF-8 support in the
2044: main pcre page. If an invalid UTF-8 sequence of bytes is found,
2045: pcre_exec() returns the error PCRE_ERROR_BADUTF8. If startoffset con-
2046: tains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned.
2047:
2048: If you already know that your subject is valid, and you want to skip
2049: these checks for performance reasons, you can set the
2050: PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
2051: do this for the second and subsequent calls to pcre_exec() if you are
2052: making repeated calls to find all the matches in a single subject
2053: string. However, you should be sure that the value of startoffset
2054: points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
2055: set, the effect of passing an invalid UTF-8 string as a subject, or a
2056: value of startoffset that does not point to the start of a UTF-8 char-
2057: acter, is undefined. Your program may crash.
2058:
2059: PCRE_PARTIAL
2060:
2061: This option turns on the partial matching feature. If the subject
2062: string fails to match the pattern, but at some point during the match-
2063: ing process the end of the subject was reached (that is, the subject
2064: partially matches the pattern and the failure to match occurred only
2065: because there were not enough subject characters), pcre_exec() returns
2066: PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
2067: used, there are restrictions on what may appear in the pattern. These
2068: are discussed in the pcrepartial documentation.
2069:
2070: The string to be matched by pcre_exec()
2071:
2072: The subject string is passed to pcre_exec() as a pointer in subject, a
1.2 misha 2073: length (in bytes) in length, and a starting byte offset in startoffset.
2074: In UTF-8 mode, the byte offset must point to the start of a UTF-8 char-
2075: acter. Unlike the pattern string, the subject may contain binary zero
2076: bytes. When the starting offset is zero, the search for a match starts
2077: at the beginning of the subject, and this is by far the most common
2078: case.
2079:
2080: A non-zero starting offset is useful when searching for another match
2081: in the same subject by calling pcre_exec() again after a previous suc-
2082: cess. Setting startoffset differs from just passing over a shortened
2083: string and setting PCRE_NOTBOL in the case of a pattern that begins
1.1 misha 2084: with any kind of lookbehind. For example, consider the pattern
2085:
2086: \Biss\B
2087:
1.2 misha 2088: which finds occurrences of "iss" in the middle of words. (\B matches
2089: only if the current position in the subject is not a word boundary.)
2090: When applied to the string "Mississipi" the first call to pcre_exec()
2091: finds the first occurrence. If pcre_exec() is called again with just
2092: the remainder of the subject, namely "issipi", it does not match,
1.1 misha 2093: because \B is always false at the start of the subject, which is deemed
1.2 misha 2094: to be a word boundary. However, if pcre_exec() is passed the entire
1.1 misha 2095: string again, but with startoffset set to 4, it finds the second occur-
1.2 misha 2096: rence of "iss" because it is able to look behind the starting point to
1.1 misha 2097: discover that it is preceded by a letter.
2098:
1.2 misha 2099: If a non-zero starting offset is passed when the pattern is anchored,
1.1 misha 2100: one attempt to match at the given offset is made. This can only succeed
1.2 misha 2101: if the pattern does not require the match to be at the start of the
1.1 misha 2102: subject.
2103:
2104: How pcre_exec() returns captured substrings
2105:
1.2 misha 2106: In general, a pattern matches a certain portion of the subject, and in
2107: addition, further substrings from the subject may be picked out by
2108: parts of the pattern. Following the usage in Jeffrey Friedl's book,
2109: this is called "capturing" in what follows, and the phrase "capturing
2110: subpattern" is used for a fragment of a pattern that picks out a sub-
2111: string. PCRE supports several other kinds of parenthesized subpattern
1.1 misha 2112: that do not cause substrings to be captured.
2113:
1.2 misha 2114: Captured substrings are returned to the caller via a vector of integers
2115: whose address is passed in ovector. The number of elements in the vec-
2116: tor is passed in ovecsize, which must be a non-negative number. Note:
2117: this argument is NOT the size of ovector in bytes.
2118:
2119: The first two-thirds of the vector is used to pass back captured sub-
2120: strings, each substring using a pair of integers. The remaining third
2121: of the vector is used as workspace by pcre_exec() while matching cap-
2122: turing subpatterns, and is not available for passing back information.
2123: The number passed in ovecsize should always be a multiple of three. If
1.1 misha 2124: it is not, it is rounded down.
2125:
1.2 misha 2126: When a match is successful, information about captured substrings is
2127: returned in pairs of integers, starting at the beginning of ovector,
2128: and continuing up to two-thirds of its length at the most. The first
2129: element of each pair is set to the byte offset of the first character
2130: in a substring, and the second is set to the byte offset of the first
2131: character after the end of a substring. Note: these values are always
2132: byte offsets, even in UTF-8 mode. They are not character counts.
2133:
2134: The first pair of integers, ovector[0] and ovector[1], identify the
2135: portion of the subject string matched by the entire pattern. The next
2136: pair is used for the first capturing subpattern, and so on. The value
2137: returned by pcre_exec() is one more than the highest numbered pair that
2138: has been set. For example, if two substrings have been captured, the
2139: returned value is 3. If there are no capturing subpatterns, the return
2140: value from a successful match is 1, indicating that just the first pair
2141: of offsets has been set.
1.1 misha 2142:
2143: If a capturing subpattern is matched repeatedly, it is the last portion
2144: of the string that it matched that is returned.
2145:
2146: If the vector is too small to hold all the captured substring offsets,
2147: it is used as far as possible (up to two-thirds of its length), and the
1.2 misha 2148: function returns a value of zero. If the substring offsets are not of
2149: interest, pcre_exec() may be called with ovector passed as NULL and
2150: ovecsize as zero. However, if the pattern contains back references and
2151: the ovector is not big enough to remember the related substrings, PCRE
2152: has to get additional memory for use during matching. Thus it is usu-
2153: ally advisable to supply an ovector.
1.1 misha 2154:
2155: The pcre_info() function can be used to find out how many capturing
2156: subpatterns there are in a compiled pattern. The smallest size for
2157: ovector that will allow for n captured substrings, in addition to the
2158: offsets of the substring matched by the whole pattern, is (n+1)*3.
2159:
2160: It is possible for capturing subpattern number n+1 to match some part
2161: of the subject when subpattern n has not been used at all. For example,
2162: if the string "abc" is matched against the pattern (a|(z))(bc) the
2163: return from the function is 4, and subpatterns 1 and 3 are matched, but
2164: 2 is not. When this happens, both values in the offset pairs corre-
2165: sponding to unused subpatterns are set to -1.
2166:
2167: Offset values that correspond to unused subpatterns at the end of the
2168: expression are also set to -1. For example, if the string "abc" is
2169: matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not
2170: matched. The return from the function is 2, because the highest used
2171: capturing subpattern number is 1. However, you can refer to the offsets
2172: for the second and third capturing subpatterns if you wish (assuming
2173: the vector is large enough, of course).
2174:
2175: Some convenience functions are provided for extracting the captured
2176: substrings as separate strings. These are described below.
2177:
2178: Error return values from pcre_exec()
2179:
2180: If pcre_exec() fails, it returns a negative number. The following are
2181: defined in the header file:
2182:
2183: PCRE_ERROR_NOMATCH (-1)
2184:
2185: The subject string did not match the pattern.
2186:
2187: PCRE_ERROR_NULL (-2)
2188:
2189: Either code or subject was passed as NULL, or ovector was NULL and
2190: ovecsize was not zero.
2191:
2192: PCRE_ERROR_BADOPTION (-3)
2193:
2194: An unrecognized bit was set in the options argument.
2195:
2196: PCRE_ERROR_BADMAGIC (-4)
2197:
2198: PCRE stores a 4-byte "magic number" at the start of the compiled code,
2199: to catch the case when it is passed a junk pointer and to detect when a
2200: pattern that was compiled in an environment of one endianness is run in
2201: an environment with the other endianness. This is the error that PCRE
2202: gives when the magic number is not present.
2203:
2204: PCRE_ERROR_UNKNOWN_OPCODE (-5)
2205:
2206: While running the pattern match, an unknown item was encountered in the
2207: compiled pattern. This error could be caused by a bug in PCRE or by
2208: overwriting of the compiled pattern.
2209:
2210: PCRE_ERROR_NOMEMORY (-6)
2211:
2212: If a pattern contains back references, but the ovector that is passed
2213: to pcre_exec() is not big enough to remember the referenced substrings,
2214: PCRE gets a block of memory at the start of matching to use for this
2215: purpose. If the call via pcre_malloc() fails, this error is given. The
2216: memory is automatically freed at the end of matching.
2217:
2218: PCRE_ERROR_NOSUBSTRING (-7)
2219:
2220: This error is used by the pcre_copy_substring(), pcre_get_substring(),
2221: and pcre_get_substring_list() functions (see below). It is never
2222: returned by pcre_exec().
2223:
2224: PCRE_ERROR_MATCHLIMIT (-8)
2225:
2226: The backtracking limit, as specified by the match_limit field in a
2227: pcre_extra structure (or defaulted) was reached. See the description
2228: above.
2229:
2230: PCRE_ERROR_CALLOUT (-9)
2231:
2232: This error is never generated by pcre_exec() itself. It is provided for
2233: use by callout functions that want to yield a distinctive error code.
2234: See the pcrecallout documentation for details.
2235:
2236: PCRE_ERROR_BADUTF8 (-10)
2237:
2238: A string that contains an invalid UTF-8 byte sequence was passed as a
2239: subject.
2240:
2241: PCRE_ERROR_BADUTF8_OFFSET (-11)
2242:
2243: The UTF-8 byte sequence that was passed as a subject was valid, but the
2244: value of startoffset did not point to the beginning of a UTF-8 charac-
2245: ter.
2246:
2247: PCRE_ERROR_PARTIAL (-12)
2248:
2249: The subject string did not match, but it did match partially. See the
2250: pcrepartial documentation for details of partial matching.
2251:
2252: PCRE_ERROR_BADPARTIAL (-13)
2253:
2254: The PCRE_PARTIAL option was used with a compiled pattern containing
2255: items that are not supported for partial matching. See the pcrepartial
2256: documentation for details of partial matching.
2257:
2258: PCRE_ERROR_INTERNAL (-14)
2259:
2260: An unexpected internal error has occurred. This error could be caused
2261: by a bug in PCRE or by overwriting of the compiled pattern.
2262:
2263: PCRE_ERROR_BADCOUNT (-15)
2264:
1.3 ! misha 2265: This error is given if the value of the ovecsize argument is negative.
1.1 misha 2266:
2267: PCRE_ERROR_RECURSIONLIMIT (-21)
2268:
2269: The internal recursion limit, as specified by the match_limit_recursion
1.3 ! misha 2270: field in a pcre_extra structure (or defaulted) was reached. See the
1.1 misha 2271: description above.
2272:
2273: PCRE_ERROR_BADNEWLINE (-23)
2274:
2275: An invalid combination of PCRE_NEWLINE_xxx options was given.
2276:
2277: Error numbers -16 to -20 and -22 are not used by pcre_exec().
2278:
2279:
2280: EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
2281:
2282: int pcre_copy_substring(const char *subject, int *ovector,
2283: int stringcount, int stringnumber, char *buffer,
2284: int buffersize);
2285:
2286: int pcre_get_substring(const char *subject, int *ovector,
2287: int stringcount, int stringnumber,
2288: const char **stringptr);
2289:
2290: int pcre_get_substring_list(const char *subject,
2291: int *ovector, int stringcount, const char ***listptr);
2292:
1.3 ! misha 2293: Captured substrings can be accessed directly by using the offsets
! 2294: returned by pcre_exec() in ovector. For convenience, the functions
1.1 misha 2295: pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
1.3 ! misha 2296: string_list() are provided for extracting captured substrings as new,
! 2297: separate, zero-terminated strings. These functions identify substrings
! 2298: by number. The next section describes functions for extracting named
1.1 misha 2299: substrings.
2300:
1.3 ! misha 2301: A substring that contains a binary zero is correctly extracted and has
! 2302: a further zero added on the end, but the result is not, of course, a C
! 2303: string. However, you can process such a string by referring to the
! 2304: length that is returned by pcre_copy_substring() and pcre_get_sub-
1.1 misha 2305: string(). Unfortunately, the interface to pcre_get_substring_list() is
1.3 ! misha 2306: not adequate for handling strings containing binary zeros, because the
1.1 misha 2307: end of the final string is not independently indicated.
2308:
1.3 ! misha 2309: The first three arguments are the same for all three of these func-
! 2310: tions: subject is the subject string that has just been successfully
1.1 misha 2311: matched, ovector is a pointer to the vector of integer offsets that was
2312: passed to pcre_exec(), and stringcount is the number of substrings that
1.3 ! misha 2313: were captured by the match, including the substring that matched the
1.1 misha 2314: entire regular expression. This is the value returned by pcre_exec() if
1.3 ! misha 2315: it is greater than zero. If pcre_exec() returned zero, indicating that
! 2316: it ran out of space in ovector, the value passed as stringcount should
1.1 misha 2317: be the number of elements in the vector divided by three.
2318:
1.3 ! misha 2319: The functions pcre_copy_substring() and pcre_get_substring() extract a
! 2320: single substring, whose number is given as stringnumber. A value of
! 2321: zero extracts the substring that matched the entire pattern, whereas
! 2322: higher values extract the captured substrings. For pcre_copy_sub-
! 2323: string(), the string is placed in buffer, whose length is given by
! 2324: buffersize, while for pcre_get_substring() a new block of memory is
! 2325: obtained via pcre_malloc, and its address is returned via stringptr.
! 2326: The yield of the function is the length of the string, not including
1.1 misha 2327: the terminating zero, or one of these error codes:
2328:
2329: PCRE_ERROR_NOMEMORY (-6)
2330:
1.3 ! misha 2331: The buffer was too small for pcre_copy_substring(), or the attempt to
1.1 misha 2332: get memory failed for pcre_get_substring().
2333:
2334: PCRE_ERROR_NOSUBSTRING (-7)
2335:
2336: There is no substring whose number is stringnumber.
2337:
1.3 ! misha 2338: The pcre_get_substring_list() function extracts all available sub-
! 2339: strings and builds a list of pointers to them. All this is done in a
1.1 misha 2340: single block of memory that is obtained via pcre_malloc. The address of
1.3 ! misha 2341: the memory block is returned via listptr, which is also the start of
! 2342: the list of string pointers. The end of the list is marked by a NULL
! 2343: pointer. The yield of the function is zero if all went well, or the
1.1 misha 2344: error code
2345:
2346: PCRE_ERROR_NOMEMORY (-6)
2347:
2348: if the attempt to get the memory block failed.
2349:
1.3 ! misha 2350: When any of these functions encounter a substring that is unset, which
! 2351: can happen when capturing subpattern number n+1 matches some part of
! 2352: the subject, but subpattern n has not been used at all, they return an
1.1 misha 2353: empty string. This can be distinguished from a genuine zero-length sub-
1.3 ! misha 2354: string by inspecting the appropriate offset in ovector, which is nega-
1.1 misha 2355: tive for unset substrings.
2356:
1.3 ! misha 2357: The two convenience functions pcre_free_substring() and pcre_free_sub-
! 2358: string_list() can be used to free the memory returned by a previous
1.1 misha 2359: call of pcre_get_substring() or pcre_get_substring_list(), respec-
1.3 ! misha 2360: tively. They do nothing more than call the function pointed to by
! 2361: pcre_free, which of course could be called directly from a C program.
! 2362: However, PCRE is used in some situations where it is linked via a spe-
! 2363: cial interface to another programming language that cannot use
! 2364: pcre_free directly; it is for these cases that the functions are pro-
1.1 misha 2365: vided.
2366:
2367:
2368: EXTRACTING CAPTURED SUBSTRINGS BY NAME
2369:
2370: int pcre_get_stringnumber(const pcre *code,
2371: const char *name);
2372:
2373: int pcre_copy_named_substring(const pcre *code,
2374: const char *subject, int *ovector,
2375: int stringcount, const char *stringname,
2376: char *buffer, int buffersize);
2377:
2378: int pcre_get_named_substring(const pcre *code,
2379: const char *subject, int *ovector,
2380: int stringcount, const char *stringname,
2381: const char **stringptr);
2382:
1.3 ! misha 2383: To extract a substring by name, you first have to find associated num-
1.1 misha 2384: ber. For example, for this pattern
2385:
2386: (a+)b(?<xxx>\d+)...
2387:
2388: the number of the subpattern called "xxx" is 2. If the name is known to
2389: be unique (PCRE_DUPNAMES was not set), you can find the number from the
2390: name by calling pcre_get_stringnumber(). The first argument is the com-
2391: piled pattern, and the second is the name. The yield of the function is
1.3 ! misha 2392: the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no
1.1 misha 2393: subpattern of that name.
2394:
2395: Given the number, you can extract the substring directly, or use one of
2396: the functions described in the previous section. For convenience, there
2397: are also two functions that do the whole job.
2398:
1.3 ! misha 2399: Most of the arguments of pcre_copy_named_substring() and
! 2400: pcre_get_named_substring() are the same as those for the similarly
! 2401: named functions that extract by number. As these are described in the
! 2402: previous section, they are not re-described here. There are just two
1.1 misha 2403: differences:
2404:
1.3 ! misha 2405: First, instead of a substring number, a substring name is given. Sec-
1.1 misha 2406: ond, there is an extra argument, given at the start, which is a pointer
1.3 ! misha 2407: to the compiled pattern. This is needed in order to gain access to the
1.1 misha 2408: name-to-number translation table.
2409:
1.3 ! misha 2410: These functions call pcre_get_stringnumber(), and if it succeeds, they
! 2411: then call pcre_copy_substring() or pcre_get_substring(), as appropri-
! 2412: ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the
1.1 misha 2413: behaviour may not be what you want (see the next section).
2414:
1.3 ! misha 2415: Warning: If the pattern uses the "(?|" feature to set up multiple sub-
! 2416: patterns with the same number, you cannot use names to distinguish
! 2417: them, because names are not included in the compiled code. The matching
! 2418: process uses only numbers.
! 2419:
1.1 misha 2420:
2421: DUPLICATE SUBPATTERN NAMES
2422:
2423: int pcre_get_stringtable_entries(const pcre *code,
2424: const char *name, char **first, char **last);
2425:
2426: When a pattern is compiled with the PCRE_DUPNAMES option, names for
2427: subpatterns are not required to be unique. Normally, patterns with
2428: duplicate names are such that in any one match, only one of the named
2429: subpatterns participates. An example is shown in the pcrepattern docu-
2430: mentation.
2431:
2432: When duplicates are present, pcre_copy_named_substring() and
2433: pcre_get_named_substring() return the first substring corresponding to
2434: the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING
2435: (-7) is returned; no data is returned. The pcre_get_stringnumber()
2436: function returns one of the numbers that are associated with the name,
2437: but it is not defined which it is.
2438:
2439: If you want to get full details of all captured substrings for a given
2440: name, you must use the pcre_get_stringtable_entries() function. The
2441: first argument is the compiled pattern, and the second is the name. The
2442: third and fourth are pointers to variables which are updated by the
2443: function. After it has run, they point to the first and last entries in
2444: the name-to-number table for the given name. The function itself
2445: returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if
2446: there are none. The format of the table is described above in the sec-
2447: tion entitled Information about a pattern. Given all the relevant
2448: entries for the name, you can extract each of their numbers, and hence
2449: the captured data, if any.
2450:
2451:
2452: FINDING ALL POSSIBLE MATCHES
2453:
2454: The traditional matching function uses a similar algorithm to Perl,
2455: which stops when it finds the first match, starting at a given point in
2456: the subject. If you want to find all possible matches, or the longest
2457: possible match, consider using the alternative matching function (see
2458: below) instead. If you cannot use the alternative function, but still
2459: need to find all possible matches, you can kludge it up by making use
2460: of the callout facility, which is described in the pcrecallout documen-
2461: tation.
2462:
2463: What you have to do is to insert a callout right at the end of the pat-
2464: tern. When your callout function is called, extract and save the cur-
2465: rent matched substring. Then return 1, which forces pcre_exec() to
2466: backtrack and try other alternatives. Ultimately, when it runs out of
2467: matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
2468:
2469:
2470: MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
2471:
2472: int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
2473: const char *subject, int length, int startoffset,
2474: int options, int *ovector, int ovecsize,
2475: int *workspace, int wscount);
2476:
2477: The function pcre_dfa_exec() is called to match a subject string
2478: against a compiled pattern, using a matching algorithm that scans the
2479: subject string just once, and does not backtrack. This has different
2480: characteristics to the normal algorithm, and is not compatible with
2481: Perl. Some of the features of PCRE patterns are not supported. Never-
2482: theless, there are times when this kind of matching can be useful. For
2483: a discussion of the two matching algorithms, see the pcrematching docu-
2484: mentation.
2485:
2486: The arguments for the pcre_dfa_exec() function are the same as for
2487: pcre_exec(), plus two extras. The ovector argument is used in a differ-
2488: ent way, and this is described below. The other common arguments are
2489: used in the same way as for pcre_exec(), so their description is not
2490: repeated here.
2491:
2492: The two additional arguments provide workspace for the function. The
2493: workspace vector should contain at least 20 elements. It is used for
2494: keeping track of multiple paths through the pattern tree. More
2495: workspace will be needed for patterns and subjects where there are a
2496: lot of potential matches.
2497:
2498: Here is an example of a simple call to pcre_dfa_exec():
2499:
2500: int rc;
2501: int ovector[10];
2502: int wspace[20];
2503: rc = pcre_dfa_exec(
2504: re, /* result of pcre_compile() */
2505: NULL, /* we didn't study the pattern */
2506: "some string", /* the subject string */
2507: 11, /* the length of the subject string */
2508: 0, /* start at offset 0 in the subject */
2509: 0, /* default options */
2510: ovector, /* vector of integers for substring information */
2511: 10, /* number of elements (NOT size in bytes) */
2512: wspace, /* working space vector */
2513: 20); /* number of elements (NOT size in bytes) */
2514:
2515: Option bits for pcre_dfa_exec()
2516:
2517: The unused bits of the options argument for pcre_dfa_exec() must be
2518: zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW-
2519: LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK,
2520: PCRE_PARTIAL, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last
2521: three of these are the same as for pcre_exec(), so their description is
2522: not repeated here.
2523:
2524: PCRE_PARTIAL
2525:
2526: This has the same general effect as it does for pcre_exec(), but the
2527: details are slightly different. When PCRE_PARTIAL is set for
2528: pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
2529: PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
2530: been no complete matches, but there is still at least one matching pos-
2531: sibility. The portion of the string that provided the partial match is
2532: set as the first matching string.
2533:
2534: PCRE_DFA_SHORTEST
2535:
2536: Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
2537: stop as soon as it has found one match. Because of the way the alterna-
2538: tive algorithm works, this is necessarily the shortest possible match
2539: at the first possible matching point in the subject string.
2540:
2541: PCRE_DFA_RESTART
2542:
2543: When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
2544: returns a partial match, it is possible to call it again, with addi-
2545: tional subject characters, and have it continue with the same match.
2546: The PCRE_DFA_RESTART option requests this action; when it is set, the
2547: workspace and wscount options must reference the same vector as before
2548: because data about the match so far is left in them after a partial
2549: match. There is more discussion of this facility in the pcrepartial
2550: documentation.
2551:
2552: Successful returns from pcre_dfa_exec()
2553:
2554: When pcre_dfa_exec() succeeds, it may have matched more than one sub-
2555: string in the subject. Note, however, that all the matches from one run
2556: of the function start at the same point in the subject. The shorter
2557: matches are all initial substrings of the longer matches. For example,
2558: if the pattern
2559:
2560: <.*>
2561:
2562: is matched against the string
2563:
2564: This is <something> <something else> <something further> no more
2565:
2566: the three matched strings are
2567:
2568: <something>
2569: <something> <something else>
2570: <something> <something else> <something further>
2571:
2572: On success, the yield of the function is a number greater than zero,
2573: which is the number of matched substrings. The substrings themselves
2574: are returned in ovector. Each string uses two elements; the first is
2575: the offset to the start, and the second is the offset to the end. In
2576: fact, all the strings have the same start offset. (Space could have
2577: been saved by giving this only once, but it was decided to retain some
2578: compatibility with the way pcre_exec() returns data, even though the
2579: meaning of the strings is different.)
2580:
2581: The strings are returned in reverse order of length; that is, the long-
2582: est matching string is given first. If there were too many matches to
2583: fit into ovector, the yield of the function is zero, and the vector is
2584: filled with the longest matches.
2585:
2586: Error returns from pcre_dfa_exec()
2587:
2588: The pcre_dfa_exec() function returns a negative number when it fails.
2589: Many of the errors are the same as for pcre_exec(), and these are
2590: described above. There are in addition the following errors that are
2591: specific to pcre_dfa_exec():
2592:
2593: PCRE_ERROR_DFA_UITEM (-16)
2594:
2595: This return is given if pcre_dfa_exec() encounters an item in the pat-
2596: tern that it does not support, for instance, the use of \C or a back
2597: reference.
2598:
2599: PCRE_ERROR_DFA_UCOND (-17)
2600:
2601: This return is given if pcre_dfa_exec() encounters a condition item
2602: that uses a back reference for the condition, or a test for recursion
2603: in a specific group. These are not supported.
2604:
2605: PCRE_ERROR_DFA_UMLIMIT (-18)
2606:
2607: This return is given if pcre_dfa_exec() is called with an extra block
2608: that contains a setting of the match_limit field. This is not supported
2609: (it is meaningless).
2610:
2611: PCRE_ERROR_DFA_WSSIZE (-19)
2612:
2613: This return is given if pcre_dfa_exec() runs out of space in the
2614: workspace vector.
2615:
2616: PCRE_ERROR_DFA_RECURSE (-20)
2617:
2618: When a recursive subpattern is processed, the matching function calls
2619: itself recursively, using private vectors for ovector and workspace.
2620: This error is given if the output vector is not large enough. This
2621: should be extremely rare, as a vector of size 1000 is used.
2622:
2623:
2624: SEE ALSO
2625:
2626: pcrebuild(3), pcrecallout(3), pcrecpp(3)(3), pcrematching(3), pcrepar-
1.3 ! misha 2627: tial(3), pcreposix(3), pcreprecompile(3), pcresample(3), pcrestack(3).
1.1 misha 2628:
2629:
2630: AUTHOR
2631:
2632: Philip Hazel
2633: University Computing Service
2634: Cambridge CB2 3QH, England.
2635:
2636:
2637: REVISION
2638:
1.3 ! misha 2639: Last updated: 11 April 2009
! 2640: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 2641: ------------------------------------------------------------------------------
2642:
2643:
2644: PCRECALLOUT(3) PCRECALLOUT(3)
2645:
2646:
2647: NAME
2648: PCRE - Perl-compatible regular expressions
2649:
2650:
2651: PCRE CALLOUTS
2652:
2653: int (*pcre_callout)(pcre_callout_block *);
2654:
2655: PCRE provides a feature called "callout", which is a means of temporar-
2656: ily passing control to the caller of PCRE in the middle of pattern
2657: matching. The caller of PCRE provides an external function by putting
2658: its entry point in the global variable pcre_callout. By default, this
2659: variable contains NULL, which disables all calling out.
2660:
2661: Within a regular expression, (?C) indicates the points at which the
2662: external function is to be called. Different callout points can be
2663: identified by putting a number less than 256 after the letter C. The
2664: default value is zero. For example, this pattern has two callout
2665: points:
2666:
2667: (?C1)abc(?C2)def
2668:
2669: If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
2670: called, PCRE automatically inserts callouts, all with number 255,
2671: before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
2672: used with the pattern
2673:
2674: A(\d{2}|--)
2675:
2676: it is processed as if it were
2677:
2678: (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2679:
2680: Notice that there is a callout before and after each parenthesis and
2681: alternation bar. Automatic callouts can be used for tracking the
2682: progress of pattern matching. The pcretest command has an option that
2683: sets automatic callouts; when it is used, the output indicates how the
2684: pattern is matched. This is useful information when you are trying to
2685: optimize the performance of a particular pattern.
2686:
2687:
2688: MISSING CALLOUTS
2689:
2690: You should be aware that, because of optimizations in the way PCRE
1.3 ! misha 2691: matches patterns by default, callouts sometimes do not happen. For
! 2692: example, if the pattern is
1.1 misha 2693:
2694: ab(?C4)cd
2695:
2696: PCRE knows that any matching string must contain the letter "d". If the
2697: subject string is "abyz", the lack of "d" means that matching doesn't
2698: ever start, and the callout is never reached. However, with "abyd",
2699: though the result is still no match, the callout is obeyed.
2700:
1.3 ! misha 2701: You can disable these optimizations by passing the PCRE_NO_START_OPTI-
! 2702: MIZE option to pcre_exec() or pcre_dfa_exec(). This slows down the
! 2703: matching process, but does ensure that callouts such as the example
! 2704: above are obeyed.
! 2705:
1.1 misha 2706:
2707: THE CALLOUT INTERFACE
2708:
1.3 ! misha 2709: During matching, when PCRE reaches a callout point, the external func-
! 2710: tion defined by pcre_callout is called (if it is set). This applies to
! 2711: both the pcre_exec() and the pcre_dfa_exec() matching functions. The
! 2712: only argument to the callout function is a pointer to a pcre_callout
1.1 misha 2713: block. This structure contains the following fields:
2714:
2715: int version;
2716: int callout_number;
2717: int *offset_vector;
2718: const char *subject;
2719: int subject_length;
2720: int start_match;
2721: int current_position;
2722: int capture_top;
2723: int capture_last;
2724: void *callout_data;
2725: int pattern_position;
2726: int next_item_length;
2727:
1.3 ! misha 2728: The version field is an integer containing the version number of the
! 2729: block format. The initial version was 0; the current version is 1. The
! 2730: version number will change again in future if additional fields are
1.1 misha 2731: added, but the intention is never to remove any of the existing fields.
2732:
2733: The callout_number field contains the number of the callout, as com-
2734: piled into the pattern (that is, the number after ?C for manual call-
2735: outs, and 255 for automatically generated callouts).
2736:
2737: The offset_vector field is a pointer to the vector of offsets that was
2738: passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2739: pcre_exec() is used, the contents can be inspected in order to extract
2740: substrings that have been matched so far, in the same way as for
2741: extracting substrings after a match has completed. For pcre_dfa_exec()
2742: this field is not useful.
2743:
2744: The subject and subject_length fields contain copies of the values that
2745: were passed to pcre_exec().
2746:
2747: The start_match field normally contains the offset within the subject
2748: at which the current match attempt started. However, if the escape
2749: sequence \K has been encountered, this value is changed to reflect the
2750: modified starting point. If the pattern is not anchored, the callout
2751: function may be called several times from the same point in the pattern
2752: for different starting points in the subject.
2753:
2754: The current_position field contains the offset within the subject of
2755: the current match pointer.
2756:
2757: When the pcre_exec() function is used, the capture_top field contains
2758: one more than the number of the highest numbered captured substring so
2759: far. If no substrings have been captured, the value of capture_top is
2760: one. This is always the case when pcre_dfa_exec() is used, because it
2761: does not support captured substrings.
2762:
2763: The capture_last field contains the number of the most recently cap-
2764: tured substring. If no substrings have been captured, its value is -1.
2765: This is always the case when pcre_dfa_exec() is used.
2766:
2767: The callout_data field contains a value that is passed to pcre_exec()
2768: or pcre_dfa_exec() specifically so that it can be passed back in call-
2769: outs. It is passed in the pcre_callout field of the pcre_extra data
2770: structure. If no such data was passed, the value of callout_data in a
2771: pcre_callout block is NULL. There is a description of the pcre_extra
2772: structure in the pcreapi documentation.
2773:
2774: The pattern_position field is present from version 1 of the pcre_call-
2775: out structure. It contains the offset to the next item to be matched in
2776: the pattern string.
2777:
2778: The next_item_length field is present from version 1 of the pcre_call-
2779: out structure. It contains the length of the next item to be matched in
2780: the pattern string. When the callout immediately precedes an alterna-
2781: tion bar, a closing parenthesis, or the end of the pattern, the length
2782: is zero. When the callout precedes an opening parenthesis, the length
2783: is that of the entire subpattern.
2784:
2785: The pattern_position and next_item_length fields are intended to help
2786: in distinguishing between different automatic callouts, which all have
2787: the same callout number. However, they are set for all callouts.
2788:
2789:
2790: RETURN VALUES
2791:
2792: The external callout function returns an integer to PCRE. If the value
2793: is zero, matching proceeds as normal. If the value is greater than
2794: zero, matching fails at the current point, but the testing of other
2795: matching possibilities goes ahead, just as if a lookahead assertion had
2796: failed. If the value is less than zero, the match is abandoned, and
2797: pcre_exec() (or pcre_dfa_exec()) returns the negative value.
2798:
2799: Negative values should normally be chosen from the set of
2800: PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2801: dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2802: reserved for use by callout functions; it will never be used by PCRE
2803: itself.
2804:
2805:
2806: AUTHOR
2807:
2808: Philip Hazel
2809: University Computing Service
2810: Cambridge CB2 3QH, England.
2811:
2812:
2813: REVISION
2814:
1.3 ! misha 2815: Last updated: 15 March 2009
! 2816: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 2817: ------------------------------------------------------------------------------
2818:
2819:
2820: PCRECOMPAT(3) PCRECOMPAT(3)
2821:
2822:
2823: NAME
2824: PCRE - Perl-compatible regular expressions
2825:
2826:
2827: DIFFERENCES BETWEEN PCRE AND PERL
2828:
2829: This document describes the differences in the ways that PCRE and Perl
2830: handle regular expressions. The differences described here are mainly
2831: with respect to Perl 5.8, though PCRE versions 7.0 and later contain
2832: some features that are expected to be in the forthcoming Perl 5.10.
2833:
2834: 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details
2835: of what it does have are given in the section on UTF-8 support in the
2836: main pcre page.
2837:
2838: 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2839: permits them, but they do not mean what you might think. For example,
2840: (?!a){3} does not assert that the next three characters are not "a". It
2841: just asserts that the next character is not "a" three times.
2842:
2843: 3. Capturing subpatterns that occur inside negative lookahead asser-
2844: tions are counted, but their entries in the offsets vector are never
2845: set. Perl sets its numerical variables from any such patterns that are
2846: matched before the assertion fails to match something (thereby succeed-
2847: ing), but only if the negative lookahead assertion contains just one
2848: branch.
2849:
2850: 4. Though binary zero characters are supported in the subject string,
2851: they are not allowed in a pattern string because it is passed as a nor-
2852: mal C string, terminated by zero. The escape sequence \0 can be used in
2853: the pattern to represent a binary zero.
2854:
2855: 5. The following Perl escape sequences are not supported: \l, \u, \L,
2856: \U, and \N. In fact these are implemented by Perl's general string-han-
2857: dling and are not part of its pattern matching engine. If any of these
2858: are encountered by PCRE, an error is generated.
2859:
2860: 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2861: is built with Unicode character property support. The properties that
2862: can be tested with \p and \P are limited to the general category prop-
2863: erties such as Lu and Nd, script names such as Greek or Han, and the
2864: derived properties Any and L&.
2865:
2866: 7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2867: ters in between are treated as literals. This is slightly different
2868: from Perl in that $ and @ are also handled as literals inside the
2869: quotes. In Perl, they cause variable interpolation (but of course PCRE
2870: does not have variables). Note the following examples:
2871:
2872: Pattern PCRE matches Perl matches
2873:
2874: \Qabc$xyz\E abc$xyz abc followed by the
2875: contents of $xyz
2876: \Qabc\$xyz\E abc\$xyz abc\$xyz
2877: \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2878:
2879: The \Q...\E sequence is recognized both inside and outside character
2880: classes.
2881:
2882: 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code})
2883: constructions. However, there is support for recursive patterns. This
2884: is not available in Perl 5.8, but will be in Perl 5.10. Also, the PCRE
2885: "callout" feature allows an external function to be called during pat-
2886: tern matching. See the pcrecallout documentation for details.
2887:
2888: 9. Subpatterns that are called recursively or as "subroutines" are
2889: always treated as atomic groups in PCRE. This is like Python, but
2890: unlike Perl.
2891:
2892: 10. There are some differences that are concerned with the settings of
2893: captured strings when part of a pattern is repeated. For example,
2894: matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
2895: unset, but in PCRE it is set to "b".
2896:
2897: 11. PCRE does support Perl 5.10's backtracking verbs (*ACCEPT),
2898: (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in
2899: the forms without an argument. PCRE does not support (*MARK). If
2900: (*ACCEPT) is within capturing parentheses, PCRE does not set that cap-
2901: ture group; this is different to Perl.
2902:
2903: 12. PCRE provides some extensions to the Perl regular expression facil-
2904: ities. Perl 5.10 will include new features that are not in earlier
2905: versions, some of which (such as named parentheses) have been in PCRE
2906: for some time. This list is with respect to Perl 5.10:
2907:
2908: (a) Although lookbehind assertions must match fixed length strings,
2909: each alternative branch of a lookbehind assertion can match a different
2910: length of string. Perl requires them all to have the same length.
2911:
2912: (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
2913: meta-character matches only at the very end of the string.
2914:
2915: (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
2916: cial meaning is faulted. Otherwise, like Perl, the backslash is quietly
2917: ignored. (Perl can be made to issue a warning.)
2918:
2919: (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
2920: fiers is inverted, that is, by default they are not greedy, but if fol-
2921: lowed by a question mark they are.
2922:
2923: (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
2924: tried only at the first matching position in the subject string.
2925:
2926: (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
2927: TURE options for pcre_exec() have no Perl equivalents.
2928:
2929: (g) The \R escape sequence can be restricted to match only CR, LF, or
2930: CRLF by the PCRE_BSR_ANYCRLF option.
2931:
2932: (h) The callout facility is PCRE-specific.
2933:
2934: (i) The partial matching facility is PCRE-specific.
2935:
2936: (j) Patterns compiled by PCRE can be saved and re-used at a later time,
2937: even on different hosts that have the other endianness.
2938:
2939: (k) The alternative matching function (pcre_dfa_exec()) matches in a
2940: different way and is not Perl-compatible.
2941:
2942: (l) PCRE recognizes some special sequences such as (*CR) at the start
2943: of a pattern that set overall options that cannot be changed within the
2944: pattern.
2945:
2946:
2947: AUTHOR
2948:
2949: Philip Hazel
2950: University Computing Service
2951: Cambridge CB2 3QH, England.
2952:
2953:
2954: REVISION
2955:
2956: Last updated: 11 September 2007
2957: Copyright (c) 1997-2007 University of Cambridge.
2958: ------------------------------------------------------------------------------
2959:
2960:
2961: PCREPATTERN(3) PCREPATTERN(3)
2962:
2963:
2964: NAME
2965: PCRE - Perl-compatible regular expressions
2966:
2967:
2968: PCRE REGULAR EXPRESSION DETAILS
2969:
2970: The syntax and semantics of the regular expressions that are supported
2971: by PCRE are described in detail below. There is a quick-reference syn-
2972: tax summary in the pcresyntax page. PCRE tries to match Perl syntax and
2973: semantics as closely as it can. PCRE also supports some alternative
2974: regular expression syntax (which does not conflict with the Perl syn-
2975: tax) in order to provide some compatibility with regular expressions in
2976: Python, .NET, and Oniguruma.
2977:
2978: Perl's regular expressions are described in its own documentation, and
2979: regular expressions in general are covered in a number of books, some
2980: of which have copious examples. Jeffrey Friedl's "Mastering Regular
2981: Expressions", published by O'Reilly, covers regular expressions in
2982: great detail. This description of PCRE's regular expressions is
2983: intended as reference material.
2984:
2985: The original operation of PCRE was on strings of one-byte characters.
2986: However, there is now also support for UTF-8 character strings. To use
2987: this, you must build PCRE to include UTF-8 support, and then call
1.3 ! misha 2988: pcre_compile() with the PCRE_UTF8 option. There is also a special
! 2989: sequence that can be given at the start of a pattern:
! 2990:
! 2991: (*UTF8)
! 2992:
! 2993: Starting a pattern with this sequence is equivalent to setting the
! 2994: PCRE_UTF8 option. This feature is not Perl-compatible. How setting
! 2995: UTF-8 mode affects pattern matching is mentioned in several places
! 2996: below. There is also a summary of UTF-8 features in the section on
! 2997: UTF-8 support in the main pcre page.
1.1 misha 2998:
2999: The remainder of this document discusses the patterns that are sup-
3000: ported by PCRE when its main matching function, pcre_exec(), is used.
3001: From release 6.0, PCRE offers a second matching function,
3002: pcre_dfa_exec(), which matches using a different algorithm that is not
3003: Perl-compatible. Some of the features discussed below are not available
3004: when pcre_dfa_exec() is used. The advantages and disadvantages of the
3005: alternative function, and how it differs from the normal function, are
3006: discussed in the pcrematching page.
3007:
3008:
3009: NEWLINE CONVENTIONS
3010:
3011: PCRE supports five different conventions for indicating line breaks in
3012: strings: a single CR (carriage return) character, a single LF (line-
3013: feed) character, the two-character sequence CRLF, any of the three pre-
3014: ceding, or any Unicode newline sequence. The pcreapi page has further
3015: discussion about newlines, and shows how to set the newline convention
3016: in the options arguments for the compiling and matching functions.
3017:
3018: It is also possible to specify a newline convention by starting a pat-
3019: tern string with one of the following five sequences:
3020:
3021: (*CR) carriage return
3022: (*LF) linefeed
3023: (*CRLF) carriage return, followed by linefeed
3024: (*ANYCRLF) any of the three above
3025: (*ANY) all Unicode newline sequences
3026:
3027: These override the default and the options given to pcre_compile(). For
3028: example, on a Unix system where LF is the default newline sequence, the
3029: pattern
3030:
3031: (*CR)a.b
3032:
3033: changes the convention to CR. That pattern matches "a\nb" because LF is
3034: no longer a newline. Note that these special settings, which are not
3035: Perl-compatible, are recognized only at the very start of a pattern,
3036: and that they must be in upper case. If more than one of them is
3037: present, the last one is used.
3038:
3039: The newline convention does not affect what the \R escape sequence
3040: matches. By default, this is any Unicode newline sequence, for Perl
3041: compatibility. However, this can be changed; see the description of \R
3042: in the section entitled "Newline sequences" below. A change of \R set-
3043: ting can be combined with a change of newline convention.
3044:
3045:
3046: CHARACTERS AND METACHARACTERS
3047:
3048: A regular expression is a pattern that is matched against a subject
3049: string from left to right. Most characters stand for themselves in a
3050: pattern, and match the corresponding characters in the subject. As a
3051: trivial example, the pattern
3052:
3053: The quick brown fox
3054:
3055: matches a portion of a subject string that is identical to itself. When
3056: caseless matching is specified (the PCRE_CASELESS option), letters are
3057: matched independently of case. In UTF-8 mode, PCRE always understands
3058: the concept of case for characters whose values are less than 128, so
3059: caseless matching is always possible. For characters with higher val-
3060: ues, the concept of case is supported if PCRE is compiled with Unicode
3061: property support, but not otherwise. If you want to use caseless
3062: matching for characters 128 and above, you must ensure that PCRE is
3063: compiled with Unicode property support as well as with UTF-8 support.
3064:
3065: The power of regular expressions comes from the ability to include
3066: alternatives and repetitions in the pattern. These are encoded in the
3067: pattern by the use of metacharacters, which do not stand for themselves
3068: but instead are interpreted in some special way.
3069:
3070: There are two different sets of metacharacters: those that are recog-
3071: nized anywhere in the pattern except within square brackets, and those
3072: that are recognized within square brackets. Outside square brackets,
3073: the metacharacters are as follows:
3074:
3075: \ general escape character with several uses
3076: ^ assert start of string (or line, in multiline mode)
3077: $ assert end of string (or line, in multiline mode)
3078: . match any character except newline (by default)
3079: [ start character class definition
3080: | start of alternative branch
3081: ( start subpattern
3082: ) end subpattern
3083: ? extends the meaning of (
3084: also 0 or 1 quantifier
3085: also quantifier minimizer
3086: * 0 or more quantifier
3087: + 1 or more quantifier
3088: also "possessive quantifier"
3089: { start min/max quantifier
3090:
3091: Part of a pattern that is in square brackets is called a "character
3092: class". In a character class the only metacharacters are:
3093:
3094: \ general escape character
3095: ^ negate the class, but only if the first character
3096: - indicates character range
3097: [ POSIX character class (only if followed by POSIX
3098: syntax)
3099: ] terminates the character class
3100:
1.3 ! misha 3101: The following sections describe the use of each of the metacharacters.
1.1 misha 3102:
3103:
3104: BACKSLASH
3105:
3106: The backslash character has several uses. Firstly, if it is followed by
1.3 ! misha 3107: a non-alphanumeric character, it takes away any special meaning that
! 3108: character may have. This use of backslash as an escape character
1.1 misha 3109: applies both inside and outside character classes.
3110:
1.3 ! misha 3111: For example, if you want to match a * character, you write \* in the
! 3112: pattern. This escaping action applies whether or not the following
! 3113: character would otherwise be interpreted as a metacharacter, so it is
! 3114: always safe to precede a non-alphanumeric with backslash to specify
! 3115: that it stands for itself. In particular, if you want to match a back-
1.1 misha 3116: slash, you write \\.
3117:
1.3 ! misha 3118: If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
! 3119: the pattern (other than in a character class) and characters between a
1.1 misha 3120: # outside a character class and the next newline are ignored. An escap-
1.3 ! misha 3121: ing backslash can be used to include a whitespace or # character as
1.1 misha 3122: part of the pattern.
3123:
1.3 ! misha 3124: If you want to remove the special meaning from a sequence of charac-
! 3125: ters, you can do so by putting them between \Q and \E. This is differ-
! 3126: ent from Perl in that $ and @ are handled as literals in \Q...\E
! 3127: sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
1.1 misha 3128: tion. Note the following examples:
3129:
3130: Pattern PCRE matches Perl matches
3131:
3132: \Qabc$xyz\E abc$xyz abc followed by the
3133: contents of $xyz
3134: \Qabc\$xyz\E abc\$xyz abc\$xyz
3135: \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
3136:
1.3 ! misha 3137: The \Q...\E sequence is recognized both inside and outside character
1.1 misha 3138: classes.
3139:
3140: Non-printing characters
3141:
3142: A second use of backslash provides a way of encoding non-printing char-
1.3 ! misha 3143: acters in patterns in a visible manner. There is no restriction on the
! 3144: appearance of non-printing characters, apart from the binary zero that
! 3145: terminates a pattern, but when a pattern is being prepared by text
! 3146: editing, it is usually easier to use one of the following escape
1.1 misha 3147: sequences than the binary character it represents:
3148:
3149: \a alarm, that is, the BEL character (hex 07)
3150: \cx "control-x", where x is any character
3151: \e escape (hex 1B)
3152: \f formfeed (hex 0C)
3153: \n linefeed (hex 0A)
3154: \r carriage return (hex 0D)
3155: \t tab (hex 09)
3156: \ddd character with octal code ddd, or backreference
3157: \xhh character with hex code hh
3158: \x{hhh..} character with hex code hhh..
3159:
1.3 ! misha 3160: The precise effect of \cx is as follows: if x is a lower case letter,
! 3161: it is converted to upper case. Then bit 6 of the character (hex 40) is
! 3162: inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
1.1 misha 3163: becomes hex 7B.
3164:
1.3 ! misha 3165: After \x, from zero to two hexadecimal digits are read (letters can be
! 3166: in upper or lower case). Any number of hexadecimal digits may appear
! 3167: between \x{ and }, but the value of the character code must be less
1.1 misha 3168: than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is,
1.3 ! misha 3169: the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
1.1 misha 3170: than the largest Unicode code point, which is 10FFFF.
3171:
1.3 ! misha 3172: If characters other than hexadecimal digits appear between \x{ and },
1.1 misha 3173: or if there is no terminating }, this form of escape is not recognized.
1.3 ! misha 3174: Instead, the initial \x will be interpreted as a basic hexadecimal
! 3175: escape, with no following digits, giving a character whose value is
1.1 misha 3176: zero.
3177:
3178: Characters whose value is less than 256 can be defined by either of the
1.3 ! misha 3179: two syntaxes for \x. There is no difference in the way they are han-
1.1 misha 3180: dled. For example, \xdc is exactly the same as \x{dc}.
3181:
1.3 ! misha 3182: After \0 up to two further octal digits are read. If there are fewer
! 3183: than two digits, just those that are present are used. Thus the
1.1 misha 3184: sequence \0\x\07 specifies two binary zeros followed by a BEL character
1.3 ! misha 3185: (code value 7). Make sure you supply two digits after the initial zero
1.1 misha 3186: if the pattern character that follows is itself an octal digit.
3187:
3188: The handling of a backslash followed by a digit other than 0 is compli-
3189: cated. Outside a character class, PCRE reads it and any following dig-
1.3 ! misha 3190: its as a decimal number. If the number is less than 10, or if there
1.1 misha 3191: have been at least that many previous capturing left parentheses in the
1.3 ! misha 3192: expression, the entire sequence is taken as a back reference. A
! 3193: description of how this works is given later, following the discussion
1.1 misha 3194: of parenthesized subpatterns.
3195:
1.3 ! misha 3196: Inside a character class, or if the decimal number is greater than 9
! 3197: and there have not been that many capturing subpatterns, PCRE re-reads
1.1 misha 3198: up to three octal digits following the backslash, and uses them to gen-
1.3 ! misha 3199: erate a data character. Any subsequent digits stand for themselves. In
! 3200: non-UTF-8 mode, the value of a character specified in octal must be
! 3201: less than \400. In UTF-8 mode, values up to \777 are permitted. For
1.1 misha 3202: example:
3203:
3204: \040 is another way of writing a space
3205: \40 is the same, provided there are fewer than 40
3206: previous capturing subpatterns
3207: \7 is always a back reference
3208: \11 might be a back reference, or another way of
3209: writing a tab
3210: \011 is always a tab
3211: \0113 is a tab followed by the character "3"
3212: \113 might be a back reference, otherwise the
3213: character with octal code 113
3214: \377 might be a back reference, otherwise
3215: the byte consisting entirely of 1 bits
3216: \81 is either a back reference, or a binary zero
3217: followed by the two characters "8" and "1"
3218:
1.3 ! misha 3219: Note that octal values of 100 or greater must not be introduced by a
1.1 misha 3220: leading zero, because no more than three octal digits are ever read.
3221:
3222: All the sequences that define a single character value can be used both
1.3 ! misha 3223: inside and outside character classes. In addition, inside a character
! 3224: class, the sequence \b is interpreted as the backspace character (hex
! 3225: 08), and the sequences \R and \X are interpreted as the characters "R"
! 3226: and "X", respectively. Outside a character class, these sequences have
1.1 misha 3227: different meanings (see below).
3228:
3229: Absolute and relative back references
3230:
1.3 ! misha 3231: The sequence \g followed by an unsigned or a negative number, option-
! 3232: ally enclosed in braces, is an absolute or relative back reference. A
1.1 misha 3233: named back reference can be coded as \g{name}. Back references are dis-
3234: cussed later, following the discussion of parenthesized subpatterns.
3235:
3236: Absolute and relative subroutine calls
3237:
1.3 ! misha 3238: For compatibility with Oniguruma, the non-Perl syntax \g followed by a
1.1 misha 3239: name or a number enclosed either in angle brackets or single quotes, is
1.3 ! misha 3240: an alternative syntax for referencing a subpattern as a "subroutine".
! 3241: Details are discussed later. Note that \g{...} (Perl syntax) and
! 3242: \g<...> (Oniguruma syntax) are not synonymous. The former is a back
1.1 misha 3243: reference; the latter is a subroutine call.
3244:
3245: Generic character types
3246:
3247: Another use of backslash is for specifying generic character types. The
3248: following are always recognized:
3249:
3250: \d any decimal digit
3251: \D any character that is not a decimal digit
3252: \h any horizontal whitespace character
3253: \H any character that is not a horizontal whitespace character
3254: \s any whitespace character
3255: \S any character that is not a whitespace character
3256: \v any vertical whitespace character
3257: \V any character that is not a vertical whitespace character
3258: \w any "word" character
3259: \W any "non-word" character
3260:
3261: Each pair of escape sequences partitions the complete set of characters
1.3 ! misha 3262: into two disjoint sets. Any given character matches one, and only one,
1.1 misha 3263: of each pair.
3264:
3265: These character type sequences can appear both inside and outside char-
1.3 ! misha 3266: acter classes. They each match one character of the appropriate type.
! 3267: If the current matching point is at the end of the subject string, all
1.1 misha 3268: of them fail, since there is no character to match.
3269:
1.3 ! misha 3270: For compatibility with Perl, \s does not match the VT character (code
! 3271: 11). This makes it different from the the POSIX "space" class. The \s
! 3272: characters are HT (9), LF (10), FF (12), CR (13), and space (32). If
1.1 misha 3273: "use locale;" is included in a Perl script, \s may match the VT charac-
3274: ter. In PCRE, it never does.
3275:
1.3 ! misha 3276: In UTF-8 mode, characters with values greater than 128 never match \d,
1.1 misha 3277: \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
1.3 ! misha 3278: code character property support is available. These sequences retain
1.1 misha 3279: their original meanings from before UTF-8 support was available, mainly
1.3 ! misha 3280: for efficiency reasons. Note that this also affects \b, because it is
! 3281: defined in terms of \w and \W.
1.1 misha 3282:
3283: The sequences \h, \H, \v, and \V are Perl 5.10 features. In contrast to
3284: the other sequences, these do match certain high-valued codepoints in
3285: UTF-8 mode. The horizontal space characters are:
3286:
3287: U+0009 Horizontal tab
3288: U+0020 Space
3289: U+00A0 Non-break space
3290: U+1680 Ogham space mark
3291: U+180E Mongolian vowel separator
3292: U+2000 En quad
3293: U+2001 Em quad
3294: U+2002 En space
3295: U+2003 Em space
3296: U+2004 Three-per-em space
3297: U+2005 Four-per-em space
3298: U+2006 Six-per-em space
3299: U+2007 Figure space
3300: U+2008 Punctuation space
3301: U+2009 Thin space
3302: U+200A Hair space
3303: U+202F Narrow no-break space
3304: U+205F Medium mathematical space
3305: U+3000 Ideographic space
3306:
3307: The vertical space characters are:
3308:
3309: U+000A Linefeed
3310: U+000B Vertical tab
3311: U+000C Formfeed
3312: U+000D Carriage return
3313: U+0085 Next line
3314: U+2028 Line separator
3315: U+2029 Paragraph separator
3316:
3317: A "word" character is an underscore or any character less than 256 that
3318: is a letter or digit. The definition of letters and digits is con-
3319: trolled by PCRE's low-valued character tables, and may vary if locale-
3320: specific matching is taking place (see "Locale support" in the pcreapi
3321: page). For example, in a French locale such as "fr_FR" in Unix-like
3322: systems, or "french" in Windows, some character codes greater than 128
3323: are used for accented letters, and these are matched by \w. The use of
3324: locales with Unicode is discouraged.
3325:
3326: Newline sequences
3327:
3328: Outside a character class, by default, the escape sequence \R matches
3329: any Unicode newline sequence. This is a Perl 5.10 feature. In non-UTF-8
3330: mode \R is equivalent to the following:
3331:
3332: (?>\r\n|\n|\x0b|\f|\r|\x85)
3333:
3334: This is an example of an "atomic group", details of which are given
3335: below. This particular group matches either the two-character sequence
3336: CR followed by LF, or one of the single characters LF (linefeed,
3337: U+000A), VT (vertical tab, U+000B), FF (formfeed, U+000C), CR (carriage
3338: return, U+000D), or NEL (next line, U+0085). The two-character sequence
3339: is treated as a single unit that cannot be split.
3340:
3341: In UTF-8 mode, two additional characters whose codepoints are greater
3342: than 255 are added: LS (line separator, U+2028) and PS (paragraph sepa-
3343: rator, U+2029). Unicode character property support is not needed for
3344: these characters to be recognized.
3345:
3346: It is possible to restrict \R to match only CR, LF, or CRLF (instead of
3347: the complete set of Unicode line endings) by setting the option
3348: PCRE_BSR_ANYCRLF either at compile time or when the pattern is matched.
3349: (BSR is an abbrevation for "backslash R".) This can be made the default
3350: when PCRE is built; if this is the case, the other behaviour can be
3351: requested via the PCRE_BSR_UNICODE option. It is also possible to
3352: specify these settings by starting a pattern string with one of the
3353: following sequences:
3354:
3355: (*BSR_ANYCRLF) CR, LF, or CRLF only
3356: (*BSR_UNICODE) any Unicode newline sequence
3357:
3358: These override the default and the options given to pcre_compile(), but
3359: they can be overridden by options given to pcre_exec(). Note that these
3360: special settings, which are not Perl-compatible, are recognized only at
3361: the very start of a pattern, and that they must be in upper case. If
3362: more than one of them is present, the last one is used. They can be
3363: combined with a change of newline convention, for example, a pattern
3364: can start with:
3365:
3366: (*ANY)(*BSR_ANYCRLF)
3367:
3368: Inside a character class, \R matches the letter "R".
3369:
3370: Unicode character properties
3371:
3372: When PCRE is built with Unicode character property support, three addi-
3373: tional escape sequences that match characters with specific properties
3374: are available. When not in UTF-8 mode, these sequences are of course
3375: limited to testing characters whose codepoints are less than 256, but
3376: they do work in this mode. The extra escape sequences are:
3377:
3378: \p{xx} a character with the xx property
3379: \P{xx} a character without the xx property
3380: \X an extended Unicode sequence
3381:
3382: The property names represented by xx above are limited to the Unicode
3383: script names, the general category properties, and "Any", which matches
3384: any character (including newline). Other properties such as "InMusical-
3385: Symbols" are not currently supported by PCRE. Note that \P{Any} does
3386: not match any characters, so always causes a match failure.
3387:
3388: Sets of Unicode characters are defined as belonging to certain scripts.
3389: A character from one of these sets can be matched using a script name.
3390: For example:
3391:
3392: \p{Greek}
3393: \P{Han}
3394:
3395: Those that are not part of an identified script are lumped together as
3396: "Common". The current list of scripts is:
3397:
3398: Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
3399: Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
3400: Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
3401: Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hira-
3402: gana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
3403: Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko,
3404: Ogham, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician,
3405: Runic, Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa,
3406: Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.
3407:
3408: Each character has exactly one general category property, specified by
3409: a two-letter abbreviation. For compatibility with Perl, negation can be
3410: specified by including a circumflex between the opening brace and the
3411: property name. For example, \p{^Lu} is the same as \P{Lu}.
3412:
3413: If only one letter is specified with \p or \P, it includes all the gen-
3414: eral category properties that start with that letter. In this case, in
3415: the absence of negation, the curly brackets in the escape sequence are
3416: optional; these two examples have the same effect:
3417:
3418: \p{L}
3419: \pL
3420:
3421: The following general category property codes are supported:
3422:
3423: C Other
3424: Cc Control
3425: Cf Format
3426: Cn Unassigned
3427: Co Private use
3428: Cs Surrogate
3429:
3430: L Letter
3431: Ll Lower case letter
3432: Lm Modifier letter
3433: Lo Other letter
3434: Lt Title case letter
3435: Lu Upper case letter
3436:
3437: M Mark
3438: Mc Spacing mark
3439: Me Enclosing mark
3440: Mn Non-spacing mark
3441:
3442: N Number
3443: Nd Decimal number
3444: Nl Letter number
3445: No Other number
3446:
3447: P Punctuation
3448: Pc Connector punctuation
3449: Pd Dash punctuation
3450: Pe Close punctuation
3451: Pf Final punctuation
3452: Pi Initial punctuation
3453: Po Other punctuation
3454: Ps Open punctuation
3455:
3456: S Symbol
3457: Sc Currency symbol
3458: Sk Modifier symbol
3459: Sm Mathematical symbol
3460: So Other symbol
3461:
3462: Z Separator
3463: Zl Line separator
3464: Zp Paragraph separator
3465: Zs Space separator
3466:
3467: The special property L& is also supported: it matches a character that
3468: has the Lu, Ll, or Lt property, in other words, a letter that is not
3469: classified as a modifier or "other".
3470:
3471: The Cs (Surrogate) property applies only to characters in the range
3472: U+D800 to U+DFFF. Such characters are not valid in UTF-8 strings (see
3473: RFC 3629) and so cannot be tested by PCRE, unless UTF-8 validity check-
3474: ing has been turned off (see the discussion of PCRE_NO_UTF8_CHECK in
3475: the pcreapi page).
3476:
3477: The long synonyms for these properties that Perl supports (such as
3478: \p{Letter}) are not supported by PCRE, nor is it permitted to prefix
3479: any of these properties with "Is".
3480:
3481: No character that is in the Unicode table has the Cn (unassigned) prop-
3482: erty. Instead, this property is assumed for any code point that is not
3483: in the Unicode table.
3484:
3485: Specifying caseless matching does not affect these escape sequences.
3486: For example, \p{Lu} always matches only upper case letters.
3487:
3488: The \X escape matches any number of Unicode characters that form an
3489: extended Unicode sequence. \X is equivalent to
3490:
3491: (?>\PM\pM*)
3492:
3493: That is, it matches a character without the "mark" property, followed
3494: by zero or more characters with the "mark" property, and treats the
3495: sequence as an atomic group (see below). Characters with the "mark"
3496: property are typically accents that affect the preceding character.
3497: None of them have codepoints less than 256, so in non-UTF-8 mode \X
3498: matches any one character.
3499:
3500: Matching characters by Unicode property is not fast, because PCRE has
3501: to search a structure that contains data for over fifteen thousand
3502: characters. That is why the traditional escape sequences such as \d and
3503: \w do not use Unicode properties in PCRE.
3504:
3505: Resetting the match start
3506:
3507: The escape sequence \K, which is a Perl 5.10 feature, causes any previ-
3508: ously matched characters not to be included in the final matched
3509: sequence. For example, the pattern:
3510:
3511: foo\Kbar
3512:
3513: matches "foobar", but reports that it has matched "bar". This feature
3514: is similar to a lookbehind assertion (described below). However, in
3515: this case, the part of the subject before the real match does not have
3516: to be of fixed length, as lookbehind assertions do. The use of \K does
3517: not interfere with the setting of captured substrings. For example,
3518: when the pattern
3519:
3520: (foo)\Kbar
3521:
3522: matches "foobar", the first substring is still set to "foo".
3523:
3524: Simple assertions
3525:
3526: The final use of backslash is for certain simple assertions. An asser-
3527: tion specifies a condition that has to be met at a particular point in
3528: a match, without consuming any characters from the subject string. The
3529: use of subpatterns for more complicated assertions is described below.
3530: The backslashed assertions are:
3531:
3532: \b matches at a word boundary
3533: \B matches when not at a word boundary
3534: \A matches at the start of the subject
3535: \Z matches at the end of the subject
3536: also matches before a newline at the end of the subject
3537: \z matches only at the end of the subject
3538: \G matches at the first matching position in the subject
3539:
3540: These assertions may not appear in character classes (but note that \b
3541: has a different meaning, namely the backspace character, inside a char-
3542: acter class).
3543:
3544: A word boundary is a position in the subject string where the current
3545: character and the previous character do not both match \w or \W (i.e.
3546: one matches \w and the other matches \W), or the start or end of the
3547: string if the first or last character matches \w, respectively.
3548:
3549: The \A, \Z, and \z assertions differ from the traditional circumflex
3550: and dollar (described in the next section) in that they only ever match
3551: at the very start and end of the subject string, whatever options are
3552: set. Thus, they are independent of multiline mode. These three asser-
3553: tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which
3554: affect only the behaviour of the circumflex and dollar metacharacters.
3555: However, if the startoffset argument of pcre_exec() is non-zero, indi-
3556: cating that matching is to start at a point other than the beginning of
3557: the subject, \A can never match. The difference between \Z and \z is
3558: that \Z matches before a newline at the end of the string as well as at
3559: the very end, whereas \z matches only at the end.
3560:
3561: The \G assertion is true only when the current matching position is at
3562: the start point of the match, as specified by the startoffset argument
3563: of pcre_exec(). It differs from \A when the value of startoffset is
3564: non-zero. By calling pcre_exec() multiple times with appropriate argu-
3565: ments, you can mimic Perl's /g option, and it is in this kind of imple-
3566: mentation where \G can be useful.
3567:
3568: Note, however, that PCRE's interpretation of \G, as the start of the
3569: current match, is subtly different from Perl's, which defines it as the
3570: end of the previous match. In Perl, these can be different when the
3571: previously matched string was empty. Because PCRE does just one match
3572: at a time, it cannot reproduce this behaviour.
3573:
3574: If all the alternatives of a pattern begin with \G, the expression is
3575: anchored to the starting match position, and the "anchored" flag is set
3576: in the compiled regular expression.
3577:
3578:
3579: CIRCUMFLEX AND DOLLAR
3580:
3581: Outside a character class, in the default matching mode, the circumflex
3582: character is an assertion that is true only if the current matching
3583: point is at the start of the subject string. If the startoffset argu-
3584: ment of pcre_exec() is non-zero, circumflex can never match if the
3585: PCRE_MULTILINE option is unset. Inside a character class, circumflex
3586: has an entirely different meaning (see below).
3587:
3588: Circumflex need not be the first character of the pattern if a number
3589: of alternatives are involved, but it should be the first thing in each
3590: alternative in which it appears if the pattern is ever to match that
3591: branch. If all possible alternatives start with a circumflex, that is,
3592: if the pattern is constrained to match only at the start of the sub-
3593: ject, it is said to be an "anchored" pattern. (There are also other
3594: constructs that can cause a pattern to be anchored.)
3595:
3596: A dollar character is an assertion that is true only if the current
3597: matching point is at the end of the subject string, or immediately
3598: before a newline at the end of the string (by default). Dollar need not
3599: be the last character of the pattern if a number of alternatives are
3600: involved, but it should be the last item in any branch in which it
3601: appears. Dollar has no special meaning in a character class.
3602:
3603: The meaning of dollar can be changed so that it matches only at the
3604: very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at
3605: compile time. This does not affect the \Z assertion.
3606:
3607: The meanings of the circumflex and dollar characters are changed if the
3608: PCRE_MULTILINE option is set. When this is the case, a circumflex
3609: matches immediately after internal newlines as well as at the start of
3610: the subject string. It does not match after a newline that ends the
3611: string. A dollar matches before any newlines in the string, as well as
3612: at the very end, when PCRE_MULTILINE is set. When newline is specified
3613: as the two-character sequence CRLF, isolated CR and LF characters do
3614: not indicate newlines.
3615:
3616: For example, the pattern /^abc$/ matches the subject string "def\nabc"
3617: (where \n represents a newline) in multiline mode, but not otherwise.
3618: Consequently, patterns that are anchored in single line mode because
3619: all branches start with ^ are not anchored in multiline mode, and a
3620: match for circumflex is possible when the startoffset argument of
3621: pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if
3622: PCRE_MULTILINE is set.
3623:
3624: Note that the sequences \A, \Z, and \z can be used to match the start
3625: and end of the subject in both modes, and if all branches of a pattern
3626: start with \A it is always anchored, whether or not PCRE_MULTILINE is
3627: set.
3628:
3629:
3630: FULL STOP (PERIOD, DOT)
3631:
3632: Outside a character class, a dot in the pattern matches any one charac-
3633: ter in the subject string except (by default) a character that signi-
3634: fies the end of a line. In UTF-8 mode, the matched character may be
3635: more than one byte long.
3636:
3637: When a line ending is defined as a single character, dot never matches
3638: that character; when the two-character sequence CRLF is used, dot does
3639: not match CR if it is immediately followed by LF, but otherwise it
3640: matches all characters (including isolated CRs and LFs). When any Uni-
3641: code line endings are being recognized, dot does not match CR or LF or
3642: any of the other line ending characters.
3643:
3644: The behaviour of dot with regard to newlines can be changed. If the
3645: PCRE_DOTALL option is set, a dot matches any one character, without
3646: exception. If the two-character sequence CRLF is present in the subject
3647: string, it takes two dots to match it.
3648:
3649: The handling of dot is entirely independent of the handling of circum-
3650: flex and dollar, the only relationship being that they both involve
3651: newlines. Dot has no special meaning in a character class.
3652:
3653:
3654: MATCHING A SINGLE BYTE
3655:
3656: Outside a character class, the escape sequence \C matches any one byte,
3657: both in and out of UTF-8 mode. Unlike a dot, it always matches any
3658: line-ending characters. The feature is provided in Perl in order to
3659: match individual bytes in UTF-8 mode. Because it breaks up UTF-8 char-
3660: acters into individual bytes, what remains in the string may be a mal-
3661: formed UTF-8 string. For this reason, the \C escape sequence is best
3662: avoided.
3663:
3664: PCRE does not allow \C to appear in lookbehind assertions (described
3665: below), because in UTF-8 mode this would make it impossible to calcu-
3666: late the length of the lookbehind.
3667:
3668:
3669: SQUARE BRACKETS AND CHARACTER CLASSES
3670:
3671: An opening square bracket introduces a character class, terminated by a
3672: closing square bracket. A closing square bracket on its own is not spe-
3673: cial. If a closing square bracket is required as a member of the class,
3674: it should be the first data character in the class (after an initial
3675: circumflex, if present) or escaped with a backslash.
3676:
3677: A character class matches a single character in the subject. In UTF-8
3678: mode, the character may occupy more than one byte. A matched character
3679: must be in the set of characters defined by the class, unless the first
3680: character in the class definition is a circumflex, in which case the
3681: subject character must not be in the set defined by the class. If a
3682: circumflex is actually required as a member of the class, ensure it is
3683: not the first character, or escape it with a backslash.
3684:
3685: For example, the character class [aeiou] matches any lower case vowel,
3686: while [^aeiou] matches any character that is not a lower case vowel.
3687: Note that a circumflex is just a convenient notation for specifying the
3688: characters that are in the class by enumerating those that are not. A
3689: class that starts with a circumflex is not an assertion: it still con-
3690: sumes a character from the subject string, and therefore it fails if
3691: the current pointer is at the end of the string.
3692:
3693: In UTF-8 mode, characters with values greater than 255 can be included
3694: in a class as a literal string of bytes, or by using the \x{ escaping
3695: mechanism.
3696:
3697: When caseless matching is set, any letters in a class represent both
3698: their upper case and lower case versions, so for example, a caseless
3699: [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not
3700: match "A", whereas a caseful version would. In UTF-8 mode, PCRE always
3701: understands the concept of case for characters whose values are less
3702: than 128, so caseless matching is always possible. For characters with
3703: higher values, the concept of case is supported if PCRE is compiled
3704: with Unicode property support, but not otherwise. If you want to use
3705: caseless matching for characters 128 and above, you must ensure that
3706: PCRE is compiled with Unicode property support as well as with UTF-8
3707: support.
3708:
3709: Characters that might indicate line breaks are never treated in any
3710: special way when matching character classes, whatever line-ending
3711: sequence is in use, and whatever setting of the PCRE_DOTALL and
3712: PCRE_MULTILINE options is used. A class such as [^a] always matches one
3713: of these characters.
3714:
3715: The minus (hyphen) character can be used to specify a range of charac-
3716: ters in a character class. For example, [d-m] matches any letter
3717: between d and m, inclusive. If a minus character is required in a
3718: class, it must be escaped with a backslash or appear in a position
3719: where it cannot be interpreted as indicating a range, typically as the
3720: first or last character in the class.
3721:
3722: It is not possible to have the literal character "]" as the end charac-
3723: ter of a range. A pattern such as [W-]46] is interpreted as a class of
3724: two characters ("W" and "-") followed by a literal string "46]", so it
3725: would match "W46]" or "-46]". However, if the "]" is escaped with a
3726: backslash it is interpreted as the end of range, so [W-\]46] is inter-
3727: preted as a class containing a range followed by two other characters.
3728: The octal or hexadecimal representation of "]" can also be used to end
3729: a range.
3730:
3731: Ranges operate in the collating sequence of character values. They can
3732: also be used for characters specified numerically, for example
3733: [\000-\037]. In UTF-8 mode, ranges can include characters whose values
3734: are greater than 255, for example [\x{100}-\x{2ff}].
3735:
3736: If a range that includes letters is used when caseless matching is set,
3737: it matches the letters in either case. For example, [W-c] is equivalent
3738: to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if
3739: character tables for a French locale are in use, [\xc8-\xcb] matches
3740: accented E characters in both cases. In UTF-8 mode, PCRE supports the
3741: concept of case for characters with values greater than 128 only when
3742: it is compiled with Unicode property support.
3743:
3744: The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear
3745: in a character class, and add the characters that they match to the
3746: class. For example, [\dABCDEF] matches any hexadecimal digit. A circum-
3747: flex can conveniently be used with the upper case character types to
3748: specify a more restricted set of characters than the matching lower
3749: case type. For example, the class [^\W_] matches any letter or digit,
3750: but not underscore.
3751:
3752: The only metacharacters that are recognized in character classes are
3753: backslash, hyphen (only where it can be interpreted as specifying a
3754: range), circumflex (only at the start), opening square bracket (only
3755: when it can be interpreted as introducing a POSIX class name - see the
3756: next section), and the terminating closing square bracket. However,
3757: escaping other non-alphanumeric characters does no harm.
3758:
3759:
3760: POSIX CHARACTER CLASSES
3761:
3762: Perl supports the POSIX notation for character classes. This uses names
3763: enclosed by [: and :] within the enclosing square brackets. PCRE also
3764: supports this notation. For example,
3765:
3766: [01[:alpha:]%]
3767:
3768: matches "0", "1", any alphabetic character, or "%". The supported class
3769: names are
3770:
3771: alnum letters and digits
3772: alpha letters
3773: ascii character codes 0 - 127
3774: blank space or tab only
3775: cntrl control characters
3776: digit decimal digits (same as \d)
3777: graph printing characters, excluding space
3778: lower lower case letters
3779: print printing characters, including space
3780: punct printing characters, excluding letters and digits
3781: space white space (not quite the same as \s)
3782: upper upper case letters
3783: word "word" characters (same as \w)
3784: xdigit hexadecimal digits
3785:
3786: The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13),
3787: and space (32). Notice that this list includes the VT character (code
3788: 11). This makes "space" different to \s, which does not include VT (for
3789: Perl compatibility).
3790:
3791: The name "word" is a Perl extension, and "blank" is a GNU extension
3792: from Perl 5.8. Another Perl extension is negation, which is indicated
3793: by a ^ character after the colon. For example,
3794:
3795: [12[:^digit:]]
3796:
3797: matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the
3798: POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but
3799: these are not supported, and an error is given if they are encountered.
3800:
3801: In UTF-8 mode, characters with values greater than 128 do not match any
3802: of the POSIX character classes.
3803:
3804:
3805: VERTICAL BAR
3806:
1.3 ! misha 3807: Vertical bar characters are used to separate alternative patterns. For
1.1 misha 3808: example, the pattern
3809:
3810: gilbert|sullivan
3811:
1.3 ! misha 3812: matches either "gilbert" or "sullivan". Any number of alternatives may
! 3813: appear, and an empty alternative is permitted (matching the empty
1.1 misha 3814: string). The matching process tries each alternative in turn, from left
1.3 ! misha 3815: to right, and the first one that succeeds is used. If the alternatives
! 3816: are within a subpattern (defined below), "succeeds" means matching the
! 3817: rest of the main pattern as well as the alternative in the subpattern.
1.1 misha 3818:
3819:
3820: INTERNAL OPTION SETTING
3821:
3822: The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and
3823: PCRE_EXTENDED options (which are Perl-compatible) can be changed from
3824: within the pattern by a sequence of Perl option letters enclosed
3825: between "(?" and ")". The option letters are
3826:
3827: i for PCRE_CASELESS
3828: m for PCRE_MULTILINE
3829: s for PCRE_DOTALL
3830: x for PCRE_EXTENDED
3831:
3832: For example, (?im) sets caseless, multiline matching. It is also possi-
3833: ble to unset these options by preceding the letter with a hyphen, and a
3834: combined setting and unsetting such as (?im-sx), which sets PCRE_CASE-
3835: LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED,
3836: is also permitted. If a letter appears both before and after the
3837: hyphen, the option is unset.
3838:
3839: The PCRE-specific options PCRE_DUPNAMES, PCRE_UNGREEDY, and PCRE_EXTRA
3840: can be changed in the same way as the Perl-compatible options by using
3841: the characters J, U and X respectively.
3842:
1.3 ! misha 3843: When one of these option changes occurs at top level (that is, not
! 3844: inside subpattern parentheses), the change applies to the remainder of
! 3845: the pattern that follows. If the change is placed right at the start of
! 3846: a pattern, PCRE extracts it into the global options (and it will there-
! 3847: fore show up in data extracted by the pcre_fullinfo() function).
1.1 misha 3848:
3849: An option change within a subpattern (see below for a description of
3850: subpatterns) affects only that part of the current pattern that follows
3851: it, so
3852:
3853: (a(?i)b)c
3854:
3855: matches abc and aBc and no other strings (assuming PCRE_CASELESS is not
3856: used). By this means, options can be made to have different settings
3857: in different parts of the pattern. Any changes made in one alternative
3858: do carry on into subsequent branches within the same subpattern. For
3859: example,
3860:
3861: (a(?i)b|c)
3862:
3863: matches "ab", "aB", "c", and "C", even though when matching "C" the
3864: first branch is abandoned before the option setting. This is because
3865: the effects of option settings happen at compile time. There would be
3866: some very weird behaviour otherwise.
3867:
3868: Note: There are other PCRE-specific options that can be set by the
3869: application when the compile or match functions are called. In some
1.3 ! misha 3870: cases the pattern can contain special leading sequences such as (*CRLF)
! 3871: to override what the application has set or what has been defaulted.
! 3872: Details are given in the section entitled "Newline sequences" above.
! 3873: There is also the (*UTF8) leading sequence that can be used to set
! 3874: UTF-8 mode; this is equivalent to setting the PCRE_UTF8 option.
1.1 misha 3875:
3876:
3877: SUBPATTERNS
3878:
3879: Subpatterns are delimited by parentheses (round brackets), which can be
3880: nested. Turning part of a pattern into a subpattern does two things:
3881:
3882: 1. It localizes a set of alternatives. For example, the pattern
3883:
3884: cat(aract|erpillar|)
3885:
3886: matches one of the words "cat", "cataract", or "caterpillar". Without
3887: the parentheses, it would match "cataract", "erpillar" or an empty
3888: string.
3889:
3890: 2. It sets up the subpattern as a capturing subpattern. This means
3891: that, when the whole pattern matches, that portion of the subject
3892: string that matched the subpattern is passed back to the caller via the
3893: ovector argument of pcre_exec(). Opening parentheses are counted from
3894: left to right (starting from 1) to obtain numbers for the capturing
3895: subpatterns.
3896:
3897: For example, if the string "the red king" is matched against the pat-
3898: tern
3899:
3900: the ((red|white) (king|queen))
3901:
3902: the captured substrings are "red king", "red", and "king", and are num-
3903: bered 1, 2, and 3, respectively.
3904:
3905: The fact that plain parentheses fulfil two functions is not always
3906: helpful. There are often times when a grouping subpattern is required
3907: without a capturing requirement. If an opening parenthesis is followed
3908: by a question mark and a colon, the subpattern does not do any captur-
3909: ing, and is not counted when computing the number of any subsequent
3910: capturing subpatterns. For example, if the string "the white queen" is
3911: matched against the pattern
3912:
3913: the ((?:red|white) (king|queen))
3914:
3915: the captured substrings are "white queen" and "queen", and are numbered
3916: 1 and 2. The maximum number of capturing subpatterns is 65535.
3917:
3918: As a convenient shorthand, if any option settings are required at the
3919: start of a non-capturing subpattern, the option letters may appear
3920: between the "?" and the ":". Thus the two patterns
3921:
3922: (?i:saturday|sunday)
3923: (?:(?i)saturday|sunday)
3924:
3925: match exactly the same set of strings. Because alternative branches are
3926: tried from left to right, and options are not reset until the end of
3927: the subpattern is reached, an option setting in one branch does affect
3928: subsequent branches, so the above patterns match "SUNDAY" as well as
3929: "Saturday".
3930:
3931:
3932: DUPLICATE SUBPATTERN NUMBERS
3933:
3934: Perl 5.10 introduced a feature whereby each alternative in a subpattern
3935: uses the same numbers for its capturing parentheses. Such a subpattern
3936: starts with (?| and is itself a non-capturing subpattern. For example,
3937: consider this pattern:
3938:
3939: (?|(Sat)ur|(Sun))day
3940:
3941: Because the two alternatives are inside a (?| group, both sets of cap-
3942: turing parentheses are numbered one. Thus, when the pattern matches,
3943: you can look at captured substring number one, whichever alternative
3944: matched. This construct is useful when you want to capture part, but
3945: not all, of one of a number of alternatives. Inside a (?| group, paren-
3946: theses are numbered as usual, but the number is reset at the start of
3947: each branch. The numbers of any capturing buffers that follow the sub-
3948: pattern start after the highest number used in any branch. The follow-
3949: ing example is taken from the Perl documentation. The numbers under-
3950: neath show in which buffer the captured content will be stored.
3951:
3952: # before ---------------branch-reset----------- after
3953: / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
3954: # 1 2 2 3 2 3 4
3955:
3956: A backreference or a recursive call to a numbered subpattern always
3957: refers to the first one in the pattern with the given number.
3958:
3959: An alternative approach to using this "branch reset" feature is to use
3960: duplicate named subpatterns, as described in the next section.
3961:
3962:
3963: NAMED SUBPATTERNS
3964:
3965: Identifying capturing parentheses by number is simple, but it can be
3966: very hard to keep track of the numbers in complicated regular expres-
3967: sions. Furthermore, if an expression is modified, the numbers may
3968: change. To help with this difficulty, PCRE supports the naming of sub-
3969: patterns. This feature was not added to Perl until release 5.10. Python
3970: had the feature earlier, and PCRE introduced it at release 4.0, using
3971: the Python syntax. PCRE now supports both the Perl and the Python syn-
3972: tax.
3973:
3974: In PCRE, a subpattern can be named in one of three ways: (?<name>...)
3975: or (?'name'...) as in Perl, or (?P<name>...) as in Python. References
3976: to capturing parentheses from other parts of the pattern, such as back-
3977: references, recursion, and conditions, can be made by name as well as
3978: by number.
3979:
3980: Names consist of up to 32 alphanumeric characters and underscores.
3981: Named capturing parentheses are still allocated numbers as well as
3982: names, exactly as if the names were not present. The PCRE API provides
3983: function calls for extracting the name-to-number translation table from
3984: a compiled pattern. There is also a convenience function for extracting
3985: a captured substring by name.
3986:
3987: By default, a name must be unique within a pattern, but it is possible
3988: to relax this constraint by setting the PCRE_DUPNAMES option at compile
3989: time. This can be useful for patterns where only one instance of the
3990: named parentheses can match. Suppose you want to match the name of a
3991: weekday, either as a 3-letter abbreviation or as the full name, and in
3992: both cases you want to extract the abbreviation. This pattern (ignoring
3993: the line breaks) does the job:
3994:
3995: (?<DN>Mon|Fri|Sun)(?:day)?|
3996: (?<DN>Tue)(?:sday)?|
3997: (?<DN>Wed)(?:nesday)?|
3998: (?<DN>Thu)(?:rsday)?|
3999: (?<DN>Sat)(?:urday)?
4000:
4001: There are five capturing substrings, but only one is ever set after a
4002: match. (An alternative way of solving this problem is to use a "branch
4003: reset" subpattern, as described in the previous section.)
4004:
4005: The convenience function for extracting the data by name returns the
4006: substring for the first (and in this example, the only) subpattern of
4007: that name that matched. This saves searching to find which numbered
4008: subpattern it was. If you make a reference to a non-unique named sub-
4009: pattern from elsewhere in the pattern, the one that corresponds to the
4010: lowest number is used. For further details of the interfaces for han-
4011: dling named subpatterns, see the pcreapi documentation.
4012:
1.3 ! misha 4013: Warning: You cannot use different names to distinguish between two sub-
! 4014: patterns with the same number (see the previous section) because PCRE
! 4015: uses only the numbers when matching.
! 4016:
1.1 misha 4017:
4018: REPETITION
4019:
4020: Repetition is specified by quantifiers, which can follow any of the
4021: following items:
4022:
4023: a literal data character
4024: the dot metacharacter
4025: the \C escape sequence
4026: the \X escape sequence (in UTF-8 mode with Unicode properties)
4027: the \R escape sequence
4028: an escape such as \d that matches a single character
4029: a character class
4030: a back reference (see next section)
4031: a parenthesized subpattern (unless it is an assertion)
4032:
4033: The general repetition quantifier specifies a minimum and maximum num-
4034: ber of permitted matches, by giving the two numbers in curly brackets
4035: (braces), separated by a comma. The numbers must be less than 65536,
4036: and the first must be less than or equal to the second. For example:
4037:
4038: z{2,4}
4039:
4040: matches "zz", "zzz", or "zzzz". A closing brace on its own is not a
4041: special character. If the second number is omitted, but the comma is
4042: present, there is no upper limit; if the second number and the comma
4043: are both omitted, the quantifier specifies an exact number of required
4044: matches. Thus
4045:
4046: [aeiou]{3,}
4047:
4048: matches at least 3 successive vowels, but may match many more, while
4049:
4050: \d{8}
4051:
4052: matches exactly 8 digits. An opening curly bracket that appears in a
4053: position where a quantifier is not allowed, or one that does not match
4054: the syntax of a quantifier, is taken as a literal character. For exam-
4055: ple, {,6} is not a quantifier, but a literal string of four characters.
4056:
1.3 ! misha 4057: In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to
1.1 misha 4058: individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char-
4059: acters, each of which is represented by a two-byte sequence. Similarly,
4060: when Unicode property support is available, \X{3} matches three Unicode
1.3 ! misha 4061: extended sequences, each of which may be several bytes long (and they
1.1 misha 4062: may be of different lengths).
4063:
4064: The quantifier {0} is permitted, causing the expression to behave as if
4065: the previous item and the quantifier were not present. This may be use-
1.3 ! misha 4066: ful for subpatterns that are referenced as subroutines from elsewhere
1.1 misha 4067: in the pattern. Items other than subpatterns that have a {0} quantifier
4068: are omitted from the compiled pattern.
4069:
1.3 ! misha 4070: For convenience, the three most common quantifiers have single-charac-
1.1 misha 4071: ter abbreviations:
4072:
4073: * is equivalent to {0,}
4074: + is equivalent to {1,}
4075: ? is equivalent to {0,1}
4076:
1.3 ! misha 4077: It is possible to construct infinite loops by following a subpattern
1.1 misha 4078: that can match no characters with a quantifier that has no upper limit,
4079: for example:
4080:
4081: (a?)*
4082:
4083: Earlier versions of Perl and PCRE used to give an error at compile time
1.3 ! misha 4084: for such patterns. However, because there are cases where this can be
! 4085: useful, such patterns are now accepted, but if any repetition of the
! 4086: subpattern does in fact match no characters, the loop is forcibly bro-
1.1 misha 4087: ken.
4088:
1.3 ! misha 4089: By default, the quantifiers are "greedy", that is, they match as much
! 4090: as possible (up to the maximum number of permitted times), without
! 4091: causing the rest of the pattern to fail. The classic example of where
1.1 misha 4092: this gives problems is in trying to match comments in C programs. These
1.3 ! misha 4093: appear between /* and */ and within the comment, individual * and /
! 4094: characters may appear. An attempt to match C comments by applying the
1.1 misha 4095: pattern
4096:
4097: /\*.*\*/
4098:
4099: to the string
4100:
4101: /* first comment */ not comment /* second comment */
4102:
1.3 ! misha 4103: fails, because it matches the entire string owing to the greediness of
1.1 misha 4104: the .* item.
4105:
1.3 ! misha 4106: However, if a quantifier is followed by a question mark, it ceases to
1.1 misha 4107: be greedy, and instead matches the minimum number of times possible, so
4108: the pattern
4109:
4110: /\*.*?\*/
4111:
1.3 ! misha 4112: does the right thing with the C comments. The meaning of the various
! 4113: quantifiers is not otherwise changed, just the preferred number of
! 4114: matches. Do not confuse this use of question mark with its use as a
! 4115: quantifier in its own right. Because it has two uses, it can sometimes
1.1 misha 4116: appear doubled, as in
4117:
4118: \d??\d
4119:
4120: which matches one digit by preference, but can match two if that is the
4121: only way the rest of the pattern matches.
4122:
1.3 ! misha 4123: If the PCRE_UNGREEDY option is set (an option that is not available in
! 4124: Perl), the quantifiers are not greedy by default, but individual ones
! 4125: can be made greedy by following them with a question mark. In other
1.1 misha 4126: words, it inverts the default behaviour.
4127:
1.3 ! misha 4128: When a parenthesized subpattern is quantified with a minimum repeat
! 4129: count that is greater than 1 or with a limited maximum, more memory is
! 4130: required for the compiled pattern, in proportion to the size of the
1.1 misha 4131: minimum or maximum.
4132:
4133: If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv-
1.3 ! misha 4134: alent to Perl's /s) is set, thus allowing the dot to match newlines,
! 4135: the pattern is implicitly anchored, because whatever follows will be
! 4136: tried against every character position in the subject string, so there
! 4137: is no point in retrying the overall match at any position after the
! 4138: first. PCRE normally treats such a pattern as though it were preceded
1.1 misha 4139: by \A.
4140:
1.3 ! misha 4141: In cases where it is known that the subject string contains no new-
! 4142: lines, it is worth setting PCRE_DOTALL in order to obtain this opti-
1.1 misha 4143: mization, or alternatively using ^ to indicate anchoring explicitly.
4144:
1.3 ! misha 4145: However, there is one situation where the optimization cannot be used.
! 4146: When .* is inside capturing parentheses that are the subject of a
! 4147: backreference elsewhere in the pattern, a match at the start may fail
1.1 misha 4148: where a later one succeeds. Consider, for example:
4149:
4150: (.*)abc\1
4151:
1.3 ! misha 4152: If the subject is "xyz123abc123" the match point is the fourth charac-
1.1 misha 4153: ter. For this reason, such a pattern is not implicitly anchored.
4154:
4155: When a capturing subpattern is repeated, the value captured is the sub-
4156: string that matched the final iteration. For example, after
4157:
4158: (tweedle[dume]{3}\s*)+
4159:
4160: has matched "tweedledum tweedledee" the value of the captured substring
1.3 ! misha 4161: is "tweedledee". However, if there are nested capturing subpatterns,
! 4162: the corresponding captured values may have been set in previous itera-
1.1 misha 4163: tions. For example, after
4164:
4165: /(a|(b))+/
4166:
4167: matches "aba" the value of the second captured substring is "b".
4168:
4169:
4170: ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
4171:
1.3 ! misha 4172: With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy")
! 4173: repetition, failure of what follows normally causes the repeated item
! 4174: to be re-evaluated to see if a different number of repeats allows the
! 4175: rest of the pattern to match. Sometimes it is useful to prevent this,
! 4176: either to change the nature of the match, or to cause it fail earlier
! 4177: than it otherwise might, when the author of the pattern knows there is
1.1 misha 4178: no point in carrying on.
4179:
1.3 ! misha 4180: Consider, for example, the pattern \d+foo when applied to the subject
1.1 misha 4181: line
4182:
4183: 123456bar
4184:
4185: After matching all 6 digits and then failing to match "foo", the normal
1.3 ! misha 4186: action of the matcher is to try again with only 5 digits matching the
! 4187: \d+ item, and then with 4, and so on, before ultimately failing.
! 4188: "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides
! 4189: the means for specifying that once a subpattern has matched, it is not
1.1 misha 4190: to be re-evaluated in this way.
4191:
1.3 ! misha 4192: If we use atomic grouping for the previous example, the matcher gives
! 4193: up immediately on failing to match "foo" the first time. The notation
1.1 misha 4194: is a kind of special parenthesis, starting with (?> as in this example:
4195:
4196: (?>\d+)foo
4197:
4198: This kind of parenthesis "locks up" the part of the pattern it con-
4199: tains once it has matched, and a failure further into the pattern is
4200: prevented from backtracking into it. Backtracking past it to previous
4201: items, however, works as normal.
4202:
4203: An alternative description is that a subpattern of this type matches
4204: the string of characters that an identical standalone pattern would
4205: match, if anchored at the current point in the subject string.
4206:
4207: Atomic grouping subpatterns are not capturing subpatterns. Simple cases
4208: such as the above example can be thought of as a maximizing repeat that
4209: must swallow everything it can. So, while both \d+ and \d+? are pre-
4210: pared to adjust the number of digits they match in order to make the
4211: rest of the pattern match, (?>\d+) can only match an entire sequence of
4212: digits.
4213:
4214: Atomic groups in general can of course contain arbitrarily complicated
4215: subpatterns, and can be nested. However, when the subpattern for an
4216: atomic group is just a single repeated item, as in the example above, a
4217: simpler notation, called a "possessive quantifier" can be used. This
4218: consists of an additional + character following a quantifier. Using
4219: this notation, the previous example can be rewritten as
4220:
4221: \d++foo
4222:
4223: Note that a possessive quantifier can be used with an entire group, for
4224: example:
4225:
4226: (abc|xyz){2,3}+
4227:
4228: Possessive quantifiers are always greedy; the setting of the
4229: PCRE_UNGREEDY option is ignored. They are a convenient notation for the
4230: simpler forms of atomic group. However, there is no difference in the
4231: meaning of a possessive quantifier and the equivalent atomic group,
4232: though there may be a performance difference; possessive quantifiers
4233: should be slightly faster.
4234:
4235: The possessive quantifier syntax is an extension to the Perl 5.8 syn-
4236: tax. Jeffrey Friedl originated the idea (and the name) in the first
4237: edition of his book. Mike McCloskey liked it, so implemented it when he
4238: built Sun's Java package, and PCRE copied it from there. It ultimately
4239: found its way into Perl at release 5.10.
4240:
4241: PCRE has an optimization that automatically "possessifies" certain sim-
4242: ple pattern constructs. For example, the sequence A+B is treated as
4243: A++B because there is no point in backtracking into a sequence of A's
4244: when B must follow.
4245:
4246: When a pattern contains an unlimited repeat inside a subpattern that
4247: can itself be repeated an unlimited number of times, the use of an
4248: atomic group is the only way to avoid some failing matches taking a
4249: very long time indeed. The pattern
4250:
4251: (\D+|<\d+>)*[!?]
4252:
4253: matches an unlimited number of substrings that either consist of non-
4254: digits, or digits enclosed in <>, followed by either ! or ?. When it
4255: matches, it runs quickly. However, if it is applied to
4256:
4257: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
4258:
4259: it takes a long time before reporting failure. This is because the
4260: string can be divided between the internal \D+ repeat and the external
4261: * repeat in a large number of ways, and all have to be tried. (The
4262: example uses [!?] rather than a single character at the end, because
4263: both PCRE and Perl have an optimization that allows for fast failure
4264: when a single character is used. They remember the last single charac-
4265: ter that is required for a match, and fail early if it is not present
4266: in the string.) If the pattern is changed so that it uses an atomic
4267: group, like this:
4268:
4269: ((?>\D+)|<\d+>)*[!?]
4270:
1.3 ! misha 4271: sequences of non-digits cannot be broken, and failure happens quickly.
1.1 misha 4272:
4273:
4274: BACK REFERENCES
4275:
4276: Outside a character class, a backslash followed by a digit greater than
4277: 0 (and possibly further digits) is a back reference to a capturing sub-
1.3 ! misha 4278: pattern earlier (that is, to its left) in the pattern, provided there
1.1 misha 4279: have been that many previous capturing left parentheses.
4280:
4281: However, if the decimal number following the backslash is less than 10,
1.3 ! misha 4282: it is always taken as a back reference, and causes an error only if
! 4283: there are not that many capturing left parentheses in the entire pat-
! 4284: tern. In other words, the parentheses that are referenced need not be
! 4285: to the left of the reference for numbers less than 10. A "forward back
! 4286: reference" of this type can make sense when a repetition is involved
! 4287: and the subpattern to the right has participated in an earlier itera-
1.1 misha 4288: tion.
4289:
1.3 ! misha 4290: It is not possible to have a numerical "forward back reference" to a
! 4291: subpattern whose number is 10 or more using this syntax because a
! 4292: sequence such as \50 is interpreted as a character defined in octal.
1.1 misha 4293: See the subsection entitled "Non-printing characters" above for further
1.3 ! misha 4294: details of the handling of digits following a backslash. There is no
! 4295: such problem when named parentheses are used. A back reference to any
1.1 misha 4296: subpattern is possible using named parentheses (see below).
4297:
1.3 ! misha 4298: Another way of avoiding the ambiguity inherent in the use of digits
1.1 misha 4299: following a backslash is to use the \g escape sequence, which is a fea-
1.3 ! misha 4300: ture introduced in Perl 5.10. This escape must be followed by an
! 4301: unsigned number or a negative number, optionally enclosed in braces.
1.1 misha 4302: These examples are all identical:
4303:
4304: (ring), \1
4305: (ring), \g1
4306: (ring), \g{1}
4307:
1.3 ! misha 4308: An unsigned number specifies an absolute reference without the ambigu-
1.1 misha 4309: ity that is present in the older syntax. It is also useful when literal
4310: digits follow the reference. A negative number is a relative reference.
4311: Consider this example:
4312:
4313: (abc(def)ghi)\g{-1}
4314:
4315: The sequence \g{-1} is a reference to the most recently started captur-
1.3 ! misha 4316: ing subpattern before \g, that is, is it equivalent to \2. Similarly,
1.1 misha 4317: \g{-2} would be equivalent to \1. The use of relative references can be
1.3 ! misha 4318: helpful in long patterns, and also in patterns that are created by
1.1 misha 4319: joining together fragments that contain references within themselves.
4320:
1.3 ! misha 4321: A back reference matches whatever actually matched the capturing sub-
! 4322: pattern in the current subject string, rather than anything matching
1.1 misha 4323: the subpattern itself (see "Subpatterns as subroutines" below for a way
4324: of doing that). So the pattern
4325:
4326: (sens|respons)e and \1ibility
4327:
1.3 ! misha 4328: matches "sense and sensibility" and "response and responsibility", but
! 4329: not "sense and responsibility". If caseful matching is in force at the
! 4330: time of the back reference, the case of letters is relevant. For exam-
1.1 misha 4331: ple,
4332:
4333: ((?i)rah)\s+\1
4334:
1.3 ! misha 4335: matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
1.1 misha 4336: original capturing subpattern is matched caselessly.
4337:
1.3 ! misha 4338: There are several different ways of writing back references to named
! 4339: subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or
! 4340: \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's
1.1 misha 4341: unified back reference syntax, in which \g can be used for both numeric
1.3 ! misha 4342: and named references, is also supported. We could rewrite the above
1.1 misha 4343: example in any of the following ways:
4344:
4345: (?<p1>(?i)rah)\s+\k<p1>
4346: (?'p1'(?i)rah)\s+\k{p1}
4347: (?P<p1>(?i)rah)\s+(?P=p1)
4348: (?<p1>(?i)rah)\s+\g{p1}
4349:
1.3 ! misha 4350: A subpattern that is referenced by name may appear in the pattern
1.1 misha 4351: before or after the reference.
4352:
1.3 ! misha 4353: There may be more than one back reference to the same subpattern. If a
! 4354: subpattern has not actually been used in a particular match, any back
1.1 misha 4355: references to it always fail. For example, the pattern
4356:
4357: (a|(bc))\2
4358:
1.3 ! misha 4359: always fails if it starts to match "a" rather than "bc". Because there
! 4360: may be many capturing parentheses in a pattern, all digits following
! 4361: the backslash are taken as part of a potential back reference number.
1.1 misha 4362: If the pattern continues with a digit character, some delimiter must be
1.3 ! misha 4363: used to terminate the back reference. If the PCRE_EXTENDED option is
! 4364: set, this can be whitespace. Otherwise an empty comment (see "Com-
1.1 misha 4365: ments" below) can be used.
4366:
1.3 ! misha 4367: A back reference that occurs inside the parentheses to which it refers
! 4368: fails when the subpattern is first used, so, for example, (a\1) never
! 4369: matches. However, such references can be useful inside repeated sub-
1.1 misha 4370: patterns. For example, the pattern
4371:
4372: (a|b\1)+
4373:
4374: matches any number of "a"s and also "aba", "ababbaa" etc. At each iter-
1.3 ! misha 4375: ation of the subpattern, the back reference matches the character
! 4376: string corresponding to the previous iteration. In order for this to
! 4377: work, the pattern must be such that the first iteration does not need
! 4378: to match the back reference. This can be done using alternation, as in
1.1 misha 4379: the example above, or by a quantifier with a minimum of zero.
4380:
4381:
4382: ASSERTIONS
4383:
1.3 ! misha 4384: An assertion is a test on the characters following or preceding the
! 4385: current matching point that does not actually consume any characters.
! 4386: The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are
1.1 misha 4387: described above.
4388:
1.3 ! misha 4389: More complicated assertions are coded as subpatterns. There are two
! 4390: kinds: those that look ahead of the current position in the subject
! 4391: string, and those that look behind it. An assertion subpattern is
! 4392: matched in the normal way, except that it does not cause the current
1.1 misha 4393: matching position to be changed.
4394:
1.3 ! misha 4395: Assertion subpatterns are not capturing subpatterns, and may not be
! 4396: repeated, because it makes no sense to assert the same thing several
! 4397: times. If any kind of assertion contains capturing subpatterns within
! 4398: it, these are counted for the purposes of numbering the capturing sub-
1.1 misha 4399: patterns in the whole pattern. However, substring capturing is carried
1.3 ! misha 4400: out only for positive assertions, because it does not make sense for
1.1 misha 4401: negative assertions.
4402:
4403: Lookahead assertions
4404:
4405: Lookahead assertions start with (?= for positive assertions and (?! for
4406: negative assertions. For example,
4407:
4408: \w+(?=;)
4409:
1.3 ! misha 4410: matches a word followed by a semicolon, but does not include the semi-
1.1 misha 4411: colon in the match, and
4412:
4413: foo(?!bar)
4414:
1.3 ! misha 4415: matches any occurrence of "foo" that is not followed by "bar". Note
1.1 misha 4416: that the apparently similar pattern
4417:
4418: (?!foo)bar
4419:
1.3 ! misha 4420: does not find an occurrence of "bar" that is preceded by something
! 4421: other than "foo"; it finds any occurrence of "bar" whatsoever, because
1.1 misha 4422: the assertion (?!foo) is always true when the next three characters are
4423: "bar". A lookbehind assertion is needed to achieve the other effect.
4424:
4425: If you want to force a matching failure at some point in a pattern, the
1.3 ! misha 4426: most convenient way to do it is with (?!) because an empty string
! 4427: always matches, so an assertion that requires there not to be an empty
1.1 misha 4428: string must always fail.
4429:
4430: Lookbehind assertions
4431:
1.3 ! misha 4432: Lookbehind assertions start with (?<= for positive assertions and (?<!
1.1 misha 4433: for negative assertions. For example,
4434:
4435: (?<!foo)bar
4436:
1.3 ! misha 4437: does find an occurrence of "bar" that is not preceded by "foo". The
! 4438: contents of a lookbehind assertion are restricted such that all the
1.1 misha 4439: strings it matches must have a fixed length. However, if there are sev-
1.3 ! misha 4440: eral top-level alternatives, they do not all have to have the same
1.1 misha 4441: fixed length. Thus
4442:
4443: (?<=bullock|donkey)
4444:
4445: is permitted, but
4446:
4447: (?<!dogs?|cats?)
4448:
1.3 ! misha 4449: causes an error at compile time. Branches that match different length
! 4450: strings are permitted only at the top level of a lookbehind assertion.
! 4451: This is an extension compared with Perl (at least for 5.8), which
! 4452: requires all branches to match the same length of string. An assertion
1.1 misha 4453: such as
4454:
4455: (?<=ab(c|de))
4456:
1.3 ! misha 4457: is not permitted, because its single top-level branch can match two
! 4458: different lengths, but it is acceptable if rewritten to use two top-
1.1 misha 4459: level branches:
4460:
4461: (?<=abc|abde)
4462:
4463: In some cases, the Perl 5.10 escape sequence \K (see above) can be used
1.3 ! misha 4464: instead of a lookbehind assertion; this is not restricted to a fixed-
1.1 misha 4465: length.
4466:
1.3 ! misha 4467: The implementation of lookbehind assertions is, for each alternative,
! 4468: to temporarily move the current position back by the fixed length and
1.1 misha 4469: then try to match. If there are insufficient characters before the cur-
4470: rent position, the assertion fails.
4471:
4472: PCRE does not allow the \C escape (which matches a single byte in UTF-8
1.3 ! misha 4473: mode) to appear in lookbehind assertions, because it makes it impossi-
! 4474: ble to calculate the length of the lookbehind. The \X and \R escapes,
1.1 misha 4475: which can match different numbers of bytes, are also not permitted.
4476:
1.3 ! misha 4477: Possessive quantifiers can be used in conjunction with lookbehind
! 4478: assertions to specify efficient matching at the end of the subject
1.1 misha 4479: string. Consider a simple pattern such as
4480:
4481: abcd$
4482:
1.3 ! misha 4483: when applied to a long string that does not match. Because matching
1.1 misha 4484: proceeds from left to right, PCRE will look for each "a" in the subject
1.3 ! misha 4485: and then see if what follows matches the rest of the pattern. If the
1.1 misha 4486: pattern is specified as
4487:
4488: ^.*abcd$
4489:
1.3 ! misha 4490: the initial .* matches the entire string at first, but when this fails
1.1 misha 4491: (because there is no following "a"), it backtracks to match all but the
1.3 ! misha 4492: last character, then all but the last two characters, and so on. Once
! 4493: again the search for "a" covers the entire string, from right to left,
1.1 misha 4494: so we are no better off. However, if the pattern is written as
4495:
4496: ^.*+(?<=abcd)
4497:
1.3 ! misha 4498: there can be no backtracking for the .*+ item; it can match only the
! 4499: entire string. The subsequent lookbehind assertion does a single test
! 4500: on the last four characters. If it fails, the match fails immediately.
! 4501: For long strings, this approach makes a significant difference to the
1.1 misha 4502: processing time.
4503:
4504: Using multiple assertions
4505:
4506: Several assertions (of any sort) may occur in succession. For example,
4507:
4508: (?<=\d{3})(?<!999)foo
4509:
1.3 ! misha 4510: matches "foo" preceded by three digits that are not "999". Notice that
! 4511: each of the assertions is applied independently at the same point in
! 4512: the subject string. First there is a check that the previous three
! 4513: characters are all digits, and then there is a check that the same
1.1 misha 4514: three characters are not "999". This pattern does not match "foo" pre-
1.3 ! misha 4515: ceded by six characters, the first of which are digits and the last
! 4516: three of which are not "999". For example, it doesn't match "123abc-
1.1 misha 4517: foo". A pattern to do that is
4518:
4519: (?<=\d{3}...)(?<!999)foo
4520:
1.3 ! misha 4521: This time the first assertion looks at the preceding six characters,
1.1 misha 4522: checking that the first three are digits, and then the second assertion
4523: checks that the preceding three characters are not "999".
4524:
4525: Assertions can be nested in any combination. For example,
4526:
4527: (?<=(?<!foo)bar)baz
4528:
1.3 ! misha 4529: matches an occurrence of "baz" that is preceded by "bar" which in turn
1.1 misha 4530: is not preceded by "foo", while
4531:
4532: (?<=\d{3}(?!999)...)foo
4533:
1.3 ! misha 4534: is another pattern that matches "foo" preceded by three digits and any
1.1 misha 4535: three characters that are not "999".
4536:
4537:
4538: CONDITIONAL SUBPATTERNS
4539:
1.3 ! misha 4540: It is possible to cause the matching process to obey a subpattern con-
! 4541: ditionally or to choose between two alternative subpatterns, depending
! 4542: on the result of an assertion, or whether a previous capturing subpat-
! 4543: tern matched or not. The two possible forms of conditional subpattern
1.1 misha 4544: are
4545:
4546: (?(condition)yes-pattern)
4547: (?(condition)yes-pattern|no-pattern)
4548:
1.3 ! misha 4549: If the condition is satisfied, the yes-pattern is used; otherwise the
! 4550: no-pattern (if present) is used. If there are more than two alterna-
1.1 misha 4551: tives in the subpattern, a compile-time error occurs.
4552:
1.3 ! misha 4553: There are four kinds of condition: references to subpatterns, refer-
1.1 misha 4554: ences to recursion, a pseudo-condition called DEFINE, and assertions.
4555:
4556: Checking for a used subpattern by number
4557:
1.3 ! misha 4558: If the text between the parentheses consists of a sequence of digits,
! 4559: the condition is true if the capturing subpattern of that number has
! 4560: previously matched. An alternative notation is to precede the digits
1.1 misha 4561: with a plus or minus sign. In this case, the subpattern number is rela-
4562: tive rather than absolute. The most recently opened parentheses can be
1.3 ! misha 4563: referenced by (?(-1), the next most recent by (?(-2), and so on. In
1.1 misha 4564: looping constructs it can also make sense to refer to subsequent groups
4565: with constructs such as (?(+2).
4566:
1.3 ! misha 4567: Consider the following pattern, which contains non-significant white
1.1 misha 4568: space to make it more readable (assume the PCRE_EXTENDED option) and to
4569: divide it into three parts for ease of discussion:
4570:
4571: ( \( )? [^()]+ (?(1) \) )
4572:
1.3 ! misha 4573: The first part matches an optional opening parenthesis, and if that
1.1 misha 4574: character is present, sets it as the first captured substring. The sec-
1.3 ! misha 4575: ond part matches one or more characters that are not parentheses. The
1.1 misha 4576: third part is a conditional subpattern that tests whether the first set
4577: of parentheses matched or not. If they did, that is, if subject started
4578: with an opening parenthesis, the condition is true, and so the yes-pat-
1.3 ! misha 4579: tern is executed and a closing parenthesis is required. Otherwise,
! 4580: since no-pattern is not present, the subpattern matches nothing. In
! 4581: other words, this pattern matches a sequence of non-parentheses,
1.1 misha 4582: optionally enclosed in parentheses.
4583:
1.3 ! misha 4584: If you were embedding this pattern in a larger one, you could use a
1.1 misha 4585: relative reference:
4586:
4587: ...other stuff... ( \( )? [^()]+ (?(-1) \) ) ...
4588:
1.3 ! misha 4589: This makes the fragment independent of the parentheses in the larger
1.1 misha 4590: pattern.
4591:
4592: Checking for a used subpattern by name
4593:
1.3 ! misha 4594: Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a
! 4595: used subpattern by name. For compatibility with earlier versions of
! 4596: PCRE, which had this facility before Perl, the syntax (?(name)...) is
! 4597: also recognized. However, there is a possible ambiguity with this syn-
! 4598: tax, because subpattern names may consist entirely of digits. PCRE
! 4599: looks first for a named subpattern; if it cannot find one and the name
! 4600: consists entirely of digits, PCRE looks for a subpattern of that num-
! 4601: ber, which must be greater than zero. Using subpattern names that con-
1.1 misha 4602: sist entirely of digits is not recommended.
4603:
4604: Rewriting the above example to use a named subpattern gives this:
4605:
4606: (?<OPEN> \( )? [^()]+ (?(<OPEN>) \) )
4607:
4608:
4609: Checking for pattern recursion
4610:
4611: If the condition is the string (R), and there is no subpattern with the
1.3 ! misha 4612: name R, the condition is true if a recursive call to the whole pattern
1.1 misha 4613: or any subpattern has been made. If digits or a name preceded by amper-
4614: sand follow the letter R, for example:
4615:
4616: (?(R3)...) or (?(R&name)...)
4617:
1.3 ! misha 4618: the condition is true if the most recent recursion is into the subpat-
! 4619: tern whose number or name is given. This condition does not check the
1.1 misha 4620: entire recursion stack.
4621:
1.3 ! misha 4622: At "top level", all these recursion test conditions are false. Recur-
1.1 misha 4623: sive patterns are described below.
4624:
4625: Defining subpatterns for use by reference only
4626:
1.3 ! misha 4627: If the condition is the string (DEFINE), and there is no subpattern
! 4628: with the name DEFINE, the condition is always false. In this case,
! 4629: there may be only one alternative in the subpattern. It is always
! 4630: skipped if control reaches this point in the pattern; the idea of
! 4631: DEFINE is that it can be used to define "subroutines" that can be ref-
! 4632: erenced from elsewhere. (The use of "subroutines" is described below.)
! 4633: For example, a pattern to match an IPv4 address could be written like
1.1 misha 4634: this (ignore whitespace and line breaks):
4635:
4636: (?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) )
4637: \b (?&byte) (\.(?&byte)){3} \b
4638:
1.3 ! misha 4639: The first part of the pattern is a DEFINE group inside which a another
! 4640: group named "byte" is defined. This matches an individual component of
! 4641: an IPv4 address (a number less than 256). When matching takes place,
! 4642: this part of the pattern is skipped because DEFINE acts like a false
1.1 misha 4643: condition.
4644:
4645: The rest of the pattern uses references to the named group to match the
1.3 ! misha 4646: four dot-separated components of an IPv4 address, insisting on a word
1.1 misha 4647: boundary at each end.
4648:
4649: Assertion conditions
4650:
1.3 ! misha 4651: If the condition is not in any of the above formats, it must be an
! 4652: assertion. This may be a positive or negative lookahead or lookbehind
! 4653: assertion. Consider this pattern, again containing non-significant
1.1 misha 4654: white space, and with the two alternatives on the second line:
4655:
4656: (?(?=[^a-z]*[a-z])
4657: \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} )
4658:
1.3 ! misha 4659: The condition is a positive lookahead assertion that matches an
! 4660: optional sequence of non-letters followed by a letter. In other words,
! 4661: it tests for the presence of at least one letter in the subject. If a
! 4662: letter is found, the subject is matched against the first alternative;
! 4663: otherwise it is matched against the second. This pattern matches
! 4664: strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are
1.1 misha 4665: letters and dd are digits.
4666:
4667:
4668: COMMENTS
4669:
1.3 ! misha 4670: The sequence (?# marks the start of a comment that continues up to the
! 4671: next closing parenthesis. Nested parentheses are not permitted. The
! 4672: characters that make up a comment play no part in the pattern matching
1.1 misha 4673: at all.
4674:
1.3 ! misha 4675: If the PCRE_EXTENDED option is set, an unescaped # character outside a
! 4676: character class introduces a comment that continues to immediately
1.1 misha 4677: after the next newline in the pattern.
4678:
4679:
4680: RECURSIVE PATTERNS
4681:
1.3 ! misha 4682: Consider the problem of matching a string in parentheses, allowing for
! 4683: unlimited nested parentheses. Without the use of recursion, the best
! 4684: that can be done is to use a pattern that matches up to some fixed
! 4685: depth of nesting. It is not possible to handle an arbitrary nesting
1.1 misha 4686: depth.
4687:
4688: For some time, Perl has provided a facility that allows regular expres-
1.3 ! misha 4689: sions to recurse (amongst other things). It does this by interpolating
! 4690: Perl code in the expression at run time, and the code can refer to the
1.1 misha 4691: expression itself. A Perl pattern using code interpolation to solve the
4692: parentheses problem can be created like this:
4693:
4694: $re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x;
4695:
4696: The (?p{...}) item interpolates Perl code at run time, and in this case
4697: refers recursively to the pattern in which it appears.
4698:
4699: Obviously, PCRE cannot support the interpolation of Perl code. Instead,
1.3 ! misha 4700: it supports special syntax for recursion of the entire pattern, and
! 4701: also for individual subpattern recursion. After its introduction in
! 4702: PCRE and Python, this kind of recursion was introduced into Perl at
1.1 misha 4703: release 5.10.
4704:
1.3 ! misha 4705: A special item that consists of (? followed by a number greater than
1.1 misha 4706: zero and a closing parenthesis is a recursive call of the subpattern of
1.3 ! misha 4707: the given number, provided that it occurs inside that subpattern. (If
! 4708: not, it is a "subroutine" call, which is described in the next sec-
! 4709: tion.) The special item (?R) or (?0) is a recursive call of the entire
1.1 misha 4710: regular expression.
4711:
1.3 ! misha 4712: In PCRE (like Python, but unlike Perl), a recursive subpattern call is
1.1 misha 4713: always treated as an atomic group. That is, once it has matched some of
4714: the subject string, it is never re-entered, even if it contains untried
4715: alternatives and there is a subsequent matching failure.
4716:
1.3 ! misha 4717: This PCRE pattern solves the nested parentheses problem (assume the
1.1 misha 4718: PCRE_EXTENDED option is set so that white space is ignored):
4719:
4720: \( ( (?>[^()]+) | (?R) )* \)
4721:
1.3 ! misha 4722: First it matches an opening parenthesis. Then it matches any number of
! 4723: substrings which can either be a sequence of non-parentheses, or a
! 4724: recursive match of the pattern itself (that is, a correctly parenthe-
1.1 misha 4725: sized substring). Finally there is a closing parenthesis.
4726:
1.3 ! misha 4727: If this were part of a larger pattern, you would not want to recurse
1.1 misha 4728: the entire pattern, so instead you could use this:
4729:
4730: ( \( ( (?>[^()]+) | (?1) )* \) )
4731:
1.3 ! misha 4732: We have put the pattern into parentheses, and caused the recursion to
1.1 misha 4733: refer to them instead of the whole pattern.
4734:
1.3 ! misha 4735: In a larger pattern, keeping track of parenthesis numbers can be
! 4736: tricky. This is made easier by the use of relative references. (A Perl
! 4737: 5.10 feature.) Instead of (?1) in the pattern above you can write
1.1 misha 4738: (?-2) to refer to the second most recently opened parentheses preceding
1.3 ! misha 4739: the recursion. In other words, a negative number counts capturing
1.1 misha 4740: parentheses leftwards from the point at which it is encountered.
4741:
1.3 ! misha 4742: It is also possible to refer to subsequently opened parentheses, by
! 4743: writing references such as (?+2). However, these cannot be recursive
! 4744: because the reference is not inside the parentheses that are refer-
! 4745: enced. They are always "subroutine" calls, as described in the next
1.1 misha 4746: section.
4747:
1.3 ! misha 4748: An alternative approach is to use named parentheses instead. The Perl
! 4749: syntax for this is (?&name); PCRE's earlier syntax (?P>name) is also
1.1 misha 4750: supported. We could rewrite the above example as follows:
4751:
4752: (?<pn> \( ( (?>[^()]+) | (?&pn) )* \) )
4753:
1.3 ! misha 4754: If there is more than one subpattern with the same name, the earliest
1.1 misha 4755: one is used.
4756:
1.3 ! misha 4757: This particular example pattern that we have been looking at contains
! 4758: nested unlimited repeats, and so the use of atomic grouping for match-
! 4759: ing strings of non-parentheses is important when applying the pattern
1.1 misha 4760: to strings that do not match. For example, when this pattern is applied
4761: to
4762:
4763: (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
4764:
1.3 ! misha 4765: it yields "no match" quickly. However, if atomic grouping is not used,
! 4766: the match runs for a very long time indeed because there are so many
! 4767: different ways the + and * repeats can carve up the subject, and all
1.1 misha 4768: have to be tested before failure can be reported.
4769:
4770: At the end of a match, the values set for any capturing subpatterns are
4771: those from the outermost level of the recursion at which the subpattern
1.3 ! misha 4772: value is set. If you want to obtain intermediate values, a callout
! 4773: function can be used (see below and the pcrecallout documentation). If
1.1 misha 4774: the pattern above is matched against
4775:
4776: (ab(cd)ef)
4777:
1.3 ! misha 4778: the value for the capturing parentheses is "ef", which is the last
! 4779: value taken on at the top level. If additional parentheses are added,
1.1 misha 4780: giving
4781:
4782: \( ( ( (?>[^()]+) | (?R) )* ) \)
4783: ^ ^
4784: ^ ^
4785:
1.3 ! misha 4786: the string they capture is "ab(cd)ef", the contents of the top level
! 4787: parentheses. If there are more than 15 capturing parentheses in a pat-
1.1 misha 4788: tern, PCRE has to obtain extra memory to store data during a recursion,
1.3 ! misha 4789: which it does by using pcre_malloc, freeing it via pcre_free after-
! 4790: wards. If no memory can be obtained, the match fails with the
1.1 misha 4791: PCRE_ERROR_NOMEMORY error.
4792:
1.3 ! misha 4793: Do not confuse the (?R) item with the condition (R), which tests for
! 4794: recursion. Consider this pattern, which matches text in angle brack-
! 4795: ets, allowing for arbitrary nesting. Only digits are allowed in nested
! 4796: brackets (that is, when recursing), whereas any characters are permit-
1.1 misha 4797: ted at the outer level.
4798:
4799: < (?: (?(R) \d++ | [^<>]*+) | (?R)) * >
4800:
1.3 ! misha 4801: In this pattern, (?(R) is the start of a conditional subpattern, with
! 4802: two different alternatives for the recursive and non-recursive cases.
1.1 misha 4803: The (?R) item is the actual recursive call.
4804:
4805:
4806: SUBPATTERNS AS SUBROUTINES
4807:
4808: If the syntax for a recursive subpattern reference (either by number or
1.3 ! misha 4809: by name) is used outside the parentheses to which it refers, it oper-
! 4810: ates like a subroutine in a programming language. The "called" subpat-
1.1 misha 4811: tern may be defined before or after the reference. A numbered reference
4812: can be absolute or relative, as in these examples:
4813:
4814: (...(absolute)...)...(?2)...
4815: (...(relative)...)...(?-1)...
4816: (...(?+1)...(relative)...
4817:
4818: An earlier example pointed out that the pattern
4819:
4820: (sens|respons)e and \1ibility
4821:
1.3 ! misha 4822: matches "sense and sensibility" and "response and responsibility", but
1.1 misha 4823: not "sense and responsibility". If instead the pattern
4824:
4825: (sens|respons)e and (?1)ibility
4826:
1.3 ! misha 4827: is used, it does match "sense and responsibility" as well as the other
! 4828: two strings. Another example is given in the discussion of DEFINE
1.1 misha 4829: above.
4830:
4831: Like recursive subpatterns, a "subroutine" call is always treated as an
1.3 ! misha 4832: atomic group. That is, once it has matched some of the subject string,
! 4833: it is never re-entered, even if it contains untried alternatives and
1.1 misha 4834: there is a subsequent matching failure.
4835:
1.3 ! misha 4836: When a subpattern is used as a subroutine, processing options such as
1.1 misha 4837: case-independence are fixed when the subpattern is defined. They cannot
4838: be changed for different calls. For example, consider this pattern:
4839:
4840: (abc)(?i:(?-1))
4841:
1.3 ! misha 4842: It matches "abcabc". It does not match "abcABC" because the change of
1.1 misha 4843: processing option does not affect the called subpattern.
4844:
4845:
4846: ONIGURUMA SUBROUTINE SYNTAX
4847:
1.3 ! misha 4848: For compatibility with Oniguruma, the non-Perl syntax \g followed by a
1.1 misha 4849: name or a number enclosed either in angle brackets or single quotes, is
1.3 ! misha 4850: an alternative syntax for referencing a subpattern as a subroutine,
! 4851: possibly recursively. Here are two of the examples used above, rewrit-
1.1 misha 4852: ten using this syntax:
4853:
4854: (?<pn> \( ( (?>[^()]+) | \g<pn> )* \) )
4855: (sens|respons)e and \g'1'ibility
4856:
1.3 ! misha 4857: PCRE supports an extension to Oniguruma: if a number is preceded by a
1.1 misha 4858: plus or a minus sign it is taken as a relative reference. For example:
4859:
4860: (abc)(?i:\g<-1>)
4861:
1.3 ! misha 4862: Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not
! 4863: synonymous. The former is a back reference; the latter is a subroutine
1.1 misha 4864: call.
4865:
4866:
4867: CALLOUTS
4868:
4869: Perl has a feature whereby using the sequence (?{...}) causes arbitrary
1.3 ! misha 4870: Perl code to be obeyed in the middle of matching a regular expression.
1.1 misha 4871: This makes it possible, amongst other things, to extract different sub-
4872: strings that match the same pair of parentheses when there is a repeti-
4873: tion.
4874:
4875: PCRE provides a similar feature, but of course it cannot obey arbitrary
4876: Perl code. The feature is called "callout". The caller of PCRE provides
1.3 ! misha 4877: an external function by putting its entry point in the global variable
! 4878: pcre_callout. By default, this variable contains NULL, which disables
1.1 misha 4879: all calling out.
4880:
1.3 ! misha 4881: Within a regular expression, (?C) indicates the points at which the
! 4882: external function is to be called. If you want to identify different
! 4883: callout points, you can put a number less than 256 after the letter C.
! 4884: The default value is zero. For example, this pattern has two callout
1.1 misha 4885: points:
4886:
4887: (?C1)abc(?C2)def
4888:
4889: If the PCRE_AUTO_CALLOUT flag is passed to pcre_compile(), callouts are
1.3 ! misha 4890: automatically installed before each item in the pattern. They are all
1.1 misha 4891: numbered 255.
4892:
4893: During matching, when PCRE reaches a callout point (and pcre_callout is
1.3 ! misha 4894: set), the external function is called. It is provided with the number
! 4895: of the callout, the position in the pattern, and, optionally, one item
! 4896: of data originally supplied by the caller of pcre_exec(). The callout
! 4897: function may cause matching to proceed, to backtrack, or to fail alto-
1.1 misha 4898: gether. A complete description of the interface to the callout function
4899: is given in the pcrecallout documentation.
4900:
4901:
4902: BACKTRACKING CONTROL
4903:
1.3 ! misha 4904: Perl 5.10 introduced a number of "Special Backtracking Control Verbs",
1.1 misha 4905: which are described in the Perl documentation as "experimental and sub-
1.3 ! misha 4906: ject to change or removal in a future version of Perl". It goes on to
! 4907: say: "Their usage in production code should be noted to avoid problems
1.1 misha 4908: during upgrades." The same remarks apply to the PCRE features described
4909: in this section.
4910:
1.3 ! misha 4911: Since these verbs are specifically related to backtracking, most of
! 4912: them can be used only when the pattern is to be matched using
1.1 misha 4913: pcre_exec(), which uses a backtracking algorithm. With the exception of
4914: (*FAIL), which behaves like a failing negative assertion, they cause an
4915: error if encountered by pcre_dfa_exec().
4916:
1.3 ! misha 4917: The new verbs make use of what was previously invalid syntax: an open-
1.1 misha 4918: ing parenthesis followed by an asterisk. In Perl, they are generally of
4919: the form (*VERB:ARG) but PCRE does not support the use of arguments, so
1.3 ! misha 4920: its general form is just (*VERB). Any number of these verbs may occur
1.1 misha 4921: in a pattern. There are two kinds:
4922:
4923: Verbs that act immediately
4924:
4925: The following verbs act as soon as they are encountered:
4926:
4927: (*ACCEPT)
4928:
1.3 ! misha 4929: This verb causes the match to end successfully, skipping the remainder
! 4930: of the pattern. When inside a recursion, only the innermost pattern is
! 4931: ended immediately. PCRE differs from Perl in what happens if the
! 4932: (*ACCEPT) is inside capturing parentheses. In Perl, the data so far is
1.1 misha 4933: captured: in PCRE no data is captured. For example:
4934:
4935: A(A|B(*ACCEPT)|C)D
4936:
1.3 ! misha 4937: This matches "AB", "AAD", or "ACD", but when it matches "AB", no data
1.1 misha 4938: is captured.
4939:
4940: (*FAIL) or (*F)
4941:
1.3 ! misha 4942: This verb causes the match to fail, forcing backtracking to occur. It
! 4943: is equivalent to (?!) but easier to read. The Perl documentation notes
! 4944: that it is probably useful only when combined with (?{}) or (??{}).
! 4945: Those are, of course, Perl features that are not present in PCRE. The
! 4946: nearest equivalent is the callout feature, as for example in this pat-
1.1 misha 4947: tern:
4948:
4949: a+(?C)(*FAIL)
4950:
1.3 ! misha 4951: A match with the string "aaaa" always fails, but the callout is taken
1.1 misha 4952: before each backtrack happens (in this example, 10 times).
4953:
4954: Verbs that act after backtracking
4955:
4956: The following verbs do nothing when they are encountered. Matching con-
1.3 ! misha 4957: tinues with what follows, but if there is no subsequent match, a fail-
! 4958: ure is forced. The verbs differ in exactly what kind of failure
1.1 misha 4959: occurs.
4960:
4961: (*COMMIT)
4962:
1.3 ! misha 4963: This verb causes the whole match to fail outright if the rest of the
! 4964: pattern does not match. Even if the pattern is unanchored, no further
! 4965: attempts to find a match by advancing the start point take place. Once
! 4966: (*COMMIT) has been passed, pcre_exec() is committed to finding a match
1.1 misha 4967: at the current starting point, or not at all. For example:
4968:
4969: a+(*COMMIT)b
4970:
1.3 ! misha 4971: This matches "xxaab" but not "aacaab". It can be thought of as a kind
1.1 misha 4972: of dynamic anchor, or "I've started, so I must finish."
4973:
4974: (*PRUNE)
4975:
1.3 ! misha 4976: This verb causes the match to fail at the current position if the rest
1.1 misha 4977: of the pattern does not match. If the pattern is unanchored, the normal
1.3 ! misha 4978: "bumpalong" advance to the next starting character then happens. Back-
! 4979: tracking can occur as usual to the left of (*PRUNE), or when matching
! 4980: to the right of (*PRUNE), but if there is no match to the right, back-
! 4981: tracking cannot cross (*PRUNE). In simple cases, the use of (*PRUNE)
1.1 misha 4982: is just an alternative to an atomic group or possessive quantifier, but
1.3 ! misha 4983: there are some uses of (*PRUNE) that cannot be expressed in any other
1.1 misha 4984: way.
4985:
4986: (*SKIP)
4987:
1.3 ! misha 4988: This verb is like (*PRUNE), except that if the pattern is unanchored,
! 4989: the "bumpalong" advance is not to the next character, but to the posi-
! 4990: tion in the subject where (*SKIP) was encountered. (*SKIP) signifies
! 4991: that whatever text was matched leading up to it cannot be part of a
1.1 misha 4992: successful match. Consider:
4993:
4994: a+(*SKIP)b
4995:
1.3 ! misha 4996: If the subject is "aaaac...", after the first match attempt fails
! 4997: (starting at the first character in the string), the starting point
1.1 misha 4998: skips on to start the next attempt at "c". Note that a possessive quan-
1.3 ! misha 4999: tifer does not have the same effect in this example; although it would
! 5000: suppress backtracking during the first match attempt, the second
! 5001: attempt would start at the second character instead of skipping on to
1.1 misha 5002: "c".
5003:
5004: (*THEN)
5005:
5006: This verb causes a skip to the next alternation if the rest of the pat-
5007: tern does not match. That is, it cancels pending backtracking, but only
1.3 ! misha 5008: within the current alternation. Its name comes from the observation
1.1 misha 5009: that it can be used for a pattern-based if-then-else block:
5010:
5011: ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ...
5012:
1.3 ! misha 5013: If the COND1 pattern matches, FOO is tried (and possibly further items
! 5014: after the end of the group if FOO succeeds); on failure the matcher
! 5015: skips to the second alternative and tries COND2, without backtracking
! 5016: into COND1. If (*THEN) is used outside of any alternation, it acts
1.1 misha 5017: exactly like (*PRUNE).
5018:
5019:
5020: SEE ALSO
5021:
5022: pcreapi(3), pcrecallout(3), pcrematching(3), pcre(3).
5023:
5024:
5025: AUTHOR
5026:
5027: Philip Hazel
5028: University Computing Service
5029: Cambridge CB2 3QH, England.
5030:
5031:
5032: REVISION
5033:
1.3 ! misha 5034: Last updated: 11 April 2009
! 5035: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 5036: ------------------------------------------------------------------------------
5037:
5038:
5039: PCRESYNTAX(3) PCRESYNTAX(3)
5040:
5041:
5042: NAME
5043: PCRE - Perl-compatible regular expressions
5044:
5045:
5046: PCRE REGULAR EXPRESSION SYNTAX SUMMARY
5047:
5048: The full syntax and semantics of the regular expressions that are sup-
5049: ported by PCRE are described in the pcrepattern documentation. This
5050: document contains just a quick-reference summary of the syntax.
5051:
5052:
5053: QUOTING
5054:
5055: \x where x is non-alphanumeric is a literal x
5056: \Q...\E treat enclosed characters as literal
5057:
5058:
5059: CHARACTERS
5060:
5061: \a alarm, that is, the BEL character (hex 07)
5062: \cx "control-x", where x is any character
5063: \e escape (hex 1B)
5064: \f formfeed (hex 0C)
5065: \n newline (hex 0A)
5066: \r carriage return (hex 0D)
5067: \t tab (hex 09)
5068: \ddd character with octal code ddd, or backreference
5069: \xhh character with hex code hh
5070: \x{hhh..} character with hex code hhh..
5071:
5072:
5073: CHARACTER TYPES
5074:
5075: . any character except newline;
5076: in dotall mode, any character whatsoever
5077: \C one byte, even in UTF-8 mode (best avoided)
5078: \d a decimal digit
5079: \D a character that is not a decimal digit
5080: \h a horizontal whitespace character
5081: \H a character that is not a horizontal whitespace character
5082: \p{xx} a character with the xx property
5083: \P{xx} a character without the xx property
5084: \R a newline sequence
5085: \s a whitespace character
5086: \S a character that is not a whitespace character
5087: \v a vertical whitespace character
5088: \V a character that is not a vertical whitespace character
5089: \w a "word" character
5090: \W a "non-word" character
5091: \X an extended Unicode sequence
5092:
5093: In PCRE, \d, \D, \s, \S, \w, and \W recognize only ASCII characters.
5094:
5095:
5096: GENERAL CATEGORY PROPERTY CODES FOR \p and \P
5097:
5098: C Other
5099: Cc Control
5100: Cf Format
5101: Cn Unassigned
5102: Co Private use
5103: Cs Surrogate
5104:
5105: L Letter
5106: Ll Lower case letter
5107: Lm Modifier letter
5108: Lo Other letter
5109: Lt Title case letter
5110: Lu Upper case letter
5111: L& Ll, Lu, or Lt
5112:
5113: M Mark
5114: Mc Spacing mark
5115: Me Enclosing mark
5116: Mn Non-spacing mark
5117:
5118: N Number
5119: Nd Decimal number
5120: Nl Letter number
5121: No Other number
5122:
5123: P Punctuation
5124: Pc Connector punctuation
5125: Pd Dash punctuation
5126: Pe Close punctuation
5127: Pf Final punctuation
5128: Pi Initial punctuation
5129: Po Other punctuation
5130: Ps Open punctuation
5131:
5132: S Symbol
5133: Sc Currency symbol
5134: Sk Modifier symbol
5135: Sm Mathematical symbol
5136: So Other symbol
5137:
5138: Z Separator
5139: Zl Line separator
5140: Zp Paragraph separator
5141: Zs Space separator
5142:
5143:
5144: SCRIPT NAMES FOR \p AND \P
5145:
5146: Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
1.3 ! misha 5147: Buhid, Canadian_Aboriginal, Carian, Cham, Cherokee, Common, Coptic, Cu-
! 5148: neiform, Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian,
! 5149: Glagolitic, Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo,
! 5150: Hebrew, Hiragana, Inherited, Kannada, Katakana, Kayah_Li, Kharoshthi,
! 5151: Khmer, Lao, Latin, Lepcha, Limbu, Linear_B, Lycian, Lydian, Malayalam,
! 5152: Mongolian, Myanmar, New_Tai_Lue, Nko, Ogham, Old_Italic, Old_Persian,
! 5153: Ol_Chiki, Oriya, Osmanya, Phags_Pa, Phoenician, Rejang, Runic, Saurash-
! 5154: tra, Shavian, Sinhala, Sudanese, Syloti_Nagri, Syriac, Tagalog, Tag-
! 5155: banwa, Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh,
! 5156: Ugaritic, Vai, Yi.
1.1 misha 5157:
5158:
5159: CHARACTER CLASSES
5160:
5161: [...] positive character class
5162: [^...] negative character class
5163: [x-y] range (can be used for hex characters)
5164: [[:xxx:]] positive POSIX named set
5165: [[:^xxx:]] negative POSIX named set
5166:
5167: alnum alphanumeric
5168: alpha alphabetic
5169: ascii 0-127
5170: blank space or tab
5171: cntrl control character
5172: digit decimal digit
5173: graph printing, excluding space
5174: lower lower case letter
5175: print printing, including space
5176: punct printing, excluding alphanumeric
5177: space whitespace
5178: upper upper case letter
5179: word same as \w
5180: xdigit hexadecimal digit
5181:
5182: In PCRE, POSIX character set names recognize only ASCII characters. You
5183: can use \Q...\E inside a character class.
5184:
5185:
5186: QUANTIFIERS
5187:
5188: ? 0 or 1, greedy
5189: ?+ 0 or 1, possessive
5190: ?? 0 or 1, lazy
5191: * 0 or more, greedy
5192: *+ 0 or more, possessive
5193: *? 0 or more, lazy
5194: + 1 or more, greedy
5195: ++ 1 or more, possessive
5196: +? 1 or more, lazy
5197: {n} exactly n
5198: {n,m} at least n, no more than m, greedy
5199: {n,m}+ at least n, no more than m, possessive
5200: {n,m}? at least n, no more than m, lazy
5201: {n,} n or more, greedy
5202: {n,}+ n or more, possessive
5203: {n,}? n or more, lazy
5204:
5205:
5206: ANCHORS AND SIMPLE ASSERTIONS
5207:
1.3 ! misha 5208: \b word boundary (only ASCII letters recognized)
1.1 misha 5209: \B not a word boundary
5210: ^ start of subject
5211: also after internal newline in multiline mode
5212: \A start of subject
5213: $ end of subject
5214: also before newline at end of subject
5215: also before internal newline in multiline mode
5216: \Z end of subject
5217: also before newline at end of subject
5218: \z end of subject
5219: \G first matching position in subject
5220:
5221:
5222: MATCH POINT RESET
5223:
5224: \K reset start of match
5225:
5226:
5227: ALTERNATION
5228:
5229: expr|expr|expr...
5230:
5231:
5232: CAPTURING
5233:
1.3 ! misha 5234: (...) capturing group
! 5235: (?<name>...) named capturing group (Perl)
! 5236: (?'name'...) named capturing group (Perl)
! 5237: (?P<name>...) named capturing group (Python)
! 5238: (?:...) non-capturing group
! 5239: (?|...) non-capturing group; reset group numbers for
! 5240: capturing groups in each alternative
1.1 misha 5241:
5242:
5243: ATOMIC GROUPS
5244:
1.3 ! misha 5245: (?>...) atomic, non-capturing group
1.1 misha 5246:
5247:
5248: COMMENT
5249:
1.3 ! misha 5250: (?#....) comment (not nestable)
1.1 misha 5251:
5252:
5253: OPTION SETTING
5254:
1.3 ! misha 5255: (?i) caseless
! 5256: (?J) allow duplicate names
! 5257: (?m) multiline
! 5258: (?s) single line (dotall)
! 5259: (?U) default ungreedy (lazy)
! 5260: (?x) extended (ignore white space)
! 5261: (?-...) unset option(s)
! 5262:
! 5263: The following is recognized only at the start of a pattern or after one
! 5264: of the newline-setting options with similar syntax:
! 5265:
! 5266: (*UTF8) set UTF-8 mode
1.1 misha 5267:
5268:
5269: LOOKAHEAD AND LOOKBEHIND ASSERTIONS
5270:
1.3 ! misha 5271: (?=...) positive look ahead
! 5272: (?!...) negative look ahead
! 5273: (?<=...) positive look behind
! 5274: (?<!...) negative look behind
1.1 misha 5275:
5276: Each top-level branch of a look behind must be of a fixed length.
5277:
5278:
5279: BACKREFERENCES
5280:
1.3 ! misha 5281: \n reference by number (can be ambiguous)
! 5282: \gn reference by number
! 5283: \g{n} reference by number
! 5284: \g{-n} relative reference by number
! 5285: \k<name> reference by name (Perl)
! 5286: \k'name' reference by name (Perl)
! 5287: \g{name} reference by name (Perl)
! 5288: \k{name} reference by name (.NET)
! 5289: (?P=name) reference by name (Python)
1.1 misha 5290:
5291:
5292: SUBROUTINE REFERENCES (POSSIBLY RECURSIVE)
5293:
1.3 ! misha 5294: (?R) recurse whole pattern
! 5295: (?n) call subpattern by absolute number
! 5296: (?+n) call subpattern by relative number
! 5297: (?-n) call subpattern by relative number
! 5298: (?&name) call subpattern by name (Perl)
! 5299: (?P>name) call subpattern by name (Python)
! 5300: \g<name> call subpattern by name (Oniguruma)
! 5301: \g'name' call subpattern by name (Oniguruma)
! 5302: \g<n> call subpattern by absolute number (Oniguruma)
! 5303: \g'n' call subpattern by absolute number (Oniguruma)
! 5304: \g<+n> call subpattern by relative number (PCRE extension)
! 5305: \g'+n' call subpattern by relative number (PCRE extension)
! 5306: \g<-n> call subpattern by relative number (PCRE extension)
! 5307: \g'-n' call subpattern by relative number (PCRE extension)
1.1 misha 5308:
5309:
5310: CONDITIONAL PATTERNS
5311:
5312: (?(condition)yes-pattern)
5313: (?(condition)yes-pattern|no-pattern)
5314:
1.3 ! misha 5315: (?(n)... absolute reference condition
! 5316: (?(+n)... relative reference condition
! 5317: (?(-n)... relative reference condition
! 5318: (?(<name>)... named reference condition (Perl)
! 5319: (?('name')... named reference condition (Perl)
! 5320: (?(name)... named reference condition (PCRE)
! 5321: (?(R)... overall recursion condition
! 5322: (?(Rn)... specific group recursion condition
! 5323: (?(R&name)... specific recursion condition
! 5324: (?(DEFINE)... define subpattern for reference
! 5325: (?(assert)... assertion condition
1.1 misha 5326:
5327:
5328: BACKTRACKING CONTROL
5329:
5330: The following act immediately they are reached:
5331:
1.3 ! misha 5332: (*ACCEPT) force successful match
! 5333: (*FAIL) force backtrack; synonym (*F)
1.1 misha 5334:
1.3 ! misha 5335: The following act only when a subsequent match failure causes a back-
1.1 misha 5336: track to reach them. They all force a match failure, but they differ in
5337: what happens afterwards. Those that advance the start-of-match point do
5338: so only if the pattern is not anchored.
5339:
1.3 ! misha 5340: (*COMMIT) overall failure, no advance of starting point
! 5341: (*PRUNE) advance to next starting character
! 5342: (*SKIP) advance start to current matching position
! 5343: (*THEN) local failure, backtrack to next alternation
1.1 misha 5344:
5345:
5346: NEWLINE CONVENTIONS
5347:
1.3 ! misha 5348: These are recognized only at the very start of the pattern or after a
! 5349: (*BSR_...) or (*UTF8) option.
1.1 misha 5350:
1.3 ! misha 5351: (*CR) carriage return only
! 5352: (*LF) linefeed only
! 5353: (*CRLF) carriage return followed by linefeed
! 5354: (*ANYCRLF) all three of the above
! 5355: (*ANY) any Unicode newline sequence
1.1 misha 5356:
5357:
5358: WHAT \R MATCHES
5359:
1.3 ! misha 5360: These are recognized only at the very start of the pattern or after a
! 5361: (*...) option that sets the newline convention or UTF-8 mode.
1.1 misha 5362:
1.3 ! misha 5363: (*BSR_ANYCRLF) CR, LF, or CRLF
! 5364: (*BSR_UNICODE) any Unicode newline sequence
1.1 misha 5365:
5366:
5367: CALLOUTS
5368:
5369: (?C) callout
5370: (?Cn) callout with data n
5371:
5372:
5373: SEE ALSO
5374:
5375: pcrepattern(3), pcreapi(3), pcrecallout(3), pcrematching(3), pcre(3).
5376:
5377:
5378: AUTHOR
5379:
5380: Philip Hazel
5381: University Computing Service
5382: Cambridge CB2 3QH, England.
5383:
5384:
5385: REVISION
5386:
1.3 ! misha 5387: Last updated: 11 April 2009
! 5388: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 5389: ------------------------------------------------------------------------------
5390:
5391:
5392: PCREPARTIAL(3) PCREPARTIAL(3)
5393:
5394:
5395: NAME
5396: PCRE - Perl-compatible regular expressions
5397:
5398:
5399: PARTIAL MATCHING IN PCRE
5400:
5401: In normal use of PCRE, if the subject string that is passed to
5402: pcre_exec() or pcre_dfa_exec() matches as far as it goes, but is too
5403: short to match the entire pattern, PCRE_ERROR_NOMATCH is returned.
5404: There are circumstances where it might be helpful to distinguish this
5405: case from other cases in which there is no match.
5406:
5407: Consider, for example, an application where a human is required to type
5408: in data for a field with specific formatting requirements. An example
5409: might be a date in the form ddmmmyy, defined by this pattern:
5410:
5411: ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$
5412:
5413: If the application sees the user's keystrokes one by one, and can check
5414: that what has been typed so far is potentially valid, it is able to
5415: raise an error as soon as a mistake is made, possibly beeping and not
5416: reflecting the character that has been typed. This immediate feedback
5417: is likely to be a better user interface than a check that is delayed
5418: until the entire string has been entered.
5419:
5420: PCRE supports the concept of partial matching by means of the PCRE_PAR-
5421: TIAL option, which can be set when calling pcre_exec() or
5422: pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code
5423: PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time
5424: during the matching process the last part of the subject string matched
5425: part of the pattern. Unfortunately, for non-anchored matching, it is
5426: not possible to obtain the position of the start of the partial match.
5427: No captured data is set when PCRE_ERROR_PARTIAL is returned.
5428:
5429: When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code
5430: PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of
5431: the subject is reached, there have been no complete matches, but there
5432: is still at least one matching possibility. The portion of the string
5433: that provided the partial match is set as the first matching string.
5434:
5435: Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers
5436: the last literal byte in a pattern, and abandons matching immediately
5437: if such a byte is not present in the subject string. This optimization
5438: cannot be used for a subject string that might match only partially.
5439:
5440:
5441: RESTRICTED PATTERNS FOR PCRE_PARTIAL
5442:
5443: Because of the way certain internal optimizations are implemented in
5444: the pcre_exec() function, the PCRE_PARTIAL option cannot be used with
5445: all patterns. These restrictions do not apply when pcre_dfa_exec() is
5446: used. For pcre_exec(), repeated single characters such as
5447:
5448: a{2,4}
5449:
5450: and repeated single metasequences such as
5451:
5452: \d+
5453:
5454: are not permitted if the maximum number of occurrences is greater than
5455: one. Optional items such as \d? (where the maximum is one) are permit-
5456: ted. Quantifiers with any values are permitted after parentheses, so
5457: the invalid examples above can be coded thus:
5458:
5459: (a){2,4}
5460: (\d)+
5461:
5462: These constructions run more slowly, but for the kinds of application
5463: that are envisaged for this facility, this is not felt to be a major
5464: restriction.
5465:
5466: If PCRE_PARTIAL is set for a pattern that does not conform to the
5467: restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL
5468: (-13). You can use the PCRE_INFO_OKPARTIAL call to pcre_fullinfo() to
5469: find out if a compiled pattern can be used for partial matching.
5470:
5471:
5472: EXAMPLE OF PARTIAL MATCHING USING PCRETEST
5473:
5474: If the escape sequence \P is present in a pcretest data line, the
5475: PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that
5476: uses the date example quoted above:
5477:
5478: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
5479: data> 25jun04\P
5480: 0: 25jun04
5481: 1: jun
5482: data> 25dec3\P
5483: Partial match
5484: data> 3ju\P
5485: Partial match
5486: data> 3juj\P
5487: No match
5488: data> j\P
5489: No match
5490:
5491: The first data string is matched completely, so pcretest shows the
5492: matched substrings. The remaining four strings do not match the com-
5493: plete pattern, but the first two are partial matches. The same test,
5494: using pcre_dfa_exec() matching (by means of the \D escape sequence),
5495: produces the following output:
5496:
5497: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
5498: data> 25jun04\P\D
5499: 0: 25jun04
5500: data> 23dec3\P\D
5501: Partial match: 23dec3
5502: data> 3ju\P\D
5503: Partial match: 3ju
5504: data> 3juj\P\D
5505: No match
5506: data> j\P\D
5507: No match
5508:
5509: Notice that in this case the portion of the string that was matched is
5510: made available.
5511:
5512:
5513: MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()
5514:
5515: When a partial match has been found using pcre_dfa_exec(), it is possi-
5516: ble to continue the match by providing additional subject data and
5517: calling pcre_dfa_exec() again with the same compiled regular expres-
5518: sion, this time setting the PCRE_DFA_RESTART option. You must also pass
5519: the same working space as before, because this is where details of the
5520: previous partial match are stored. Here is an example using pcretest,
5521: using the \R escape sequence to set the PCRE_DFA_RESTART option (\P and
5522: \D are as above):
5523:
5524: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
5525: data> 23ja\P\D
5526: Partial match: 23ja
5527: data> n05\R\D
5528: 0: n05
5529:
5530: The first call has "23ja" as the subject, and requests partial match-
5531: ing; the second call has "n05" as the subject for the continued
5532: (restarted) match. Notice that when the match is complete, only the
5533: last part is shown; PCRE does not retain the previously partially-
5534: matched string. It is up to the calling program to do that if it needs
5535: to.
5536:
5537: You can set PCRE_PARTIAL with PCRE_DFA_RESTART to continue partial
5538: matching over multiple segments. This facility can be used to pass very
5539: long subject strings to pcre_dfa_exec(). However, some care is needed
5540: for certain types of pattern.
5541:
5542: 1. If the pattern contains tests for the beginning or end of a line,
5543: you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri-
5544: ate, when the subject string for any call does not contain the begin-
5545: ning or end of a line.
5546:
5547: 2. If the pattern contains backward assertions (including \b or \B),
5548: you need to arrange for some overlap in the subject strings to allow
5549: for this. For example, you could pass the subject in chunks that are
5550: 500 bytes long, but in a buffer of 700 bytes, with the starting offset
5551: set to 200 and the previous 200 bytes at the start of the buffer.
5552:
5553: 3. Matching a subject string that is split into multiple segments does
5554: not always produce exactly the same result as matching over one single
5555: long string. The difference arises when there are multiple matching
5556: possibilities, because a partial match result is given only when there
5557: are no completed matches in a call to pcre_dfa_exec(). This means that
5558: as soon as the shortest match has been found, continuation to a new
5559: subject segment is no longer possible. Consider this pcretest example:
5560:
5561: re> /dog(sbody)?/
5562: data> do\P\D
5563: Partial match: do
5564: data> gsb\R\P\D
5565: 0: g
5566: data> dogsbody\D
5567: 0: dogsbody
5568: 1: dog
5569:
1.3 ! misha 5570: The pattern matches the words "dog" or "dogsbody". When the subject is
! 5571: presented in several parts ("do" and "gsb" being the first two) the
! 5572: match stops when "dog" has been found, and it is not possible to con-
! 5573: tinue. On the other hand, if "dogsbody" is presented as a single
1.1 misha 5574: string, both matches are found.
5575:
1.3 ! misha 5576: Because of this phenomenon, it does not usually make sense to end a
1.1 misha 5577: pattern that is going to be matched in this way with a variable repeat.
5578:
5579: 4. Patterns that contain alternatives at the top level which do not all
5580: start with the same pattern item may not work as expected. For example,
5581: consider this pattern:
5582:
5583: 1234|3789
5584:
5585: If the first part of the subject is "ABC123", a partial match of the
5586: first alternative is found at offset 3. There is no partial match for
5587: the second alternative, because such a match does not start at the same
5588: point in the subject string. Attempting to continue with the string
5589: "789" does not yield a match because only those alternatives that match
5590: at one point in the subject are remembered. The problem arises because
5591: the start of the second alternative matches within the first alterna-
5592: tive. There is no problem with anchored patterns or patterns such as:
5593:
5594: 1234|ABCD
5595:
5596: where no string can be a partial match for both alternatives.
5597:
5598:
5599: AUTHOR
5600:
5601: Philip Hazel
5602: University Computing Service
5603: Cambridge CB2 3QH, England.
5604:
5605:
5606: REVISION
5607:
5608: Last updated: 04 June 2007
5609: Copyright (c) 1997-2007 University of Cambridge.
5610: ------------------------------------------------------------------------------
5611:
5612:
5613: PCREPRECOMPILE(3) PCREPRECOMPILE(3)
5614:
5615:
5616: NAME
5617: PCRE - Perl-compatible regular expressions
5618:
5619:
5620: SAVING AND RE-USING PRECOMPILED PCRE PATTERNS
5621:
5622: If you are running an application that uses a large number of regular
5623: expression patterns, it may be useful to store them in a precompiled
5624: form instead of having to compile them every time the application is
5625: run. If you are not using any private character tables (see the
5626: pcre_maketables() documentation), this is relatively straightforward.
5627: If you are using private tables, it is a little bit more complicated.
5628:
5629: If you save compiled patterns to a file, you can copy them to a differ-
5630: ent host and run them there. This works even if the new host has the
5631: opposite endianness to the one on which the patterns were compiled.
5632: There may be a small performance penalty, but it should be insignifi-
5633: cant. However, compiling regular expressions with one version of PCRE
5634: for use with a different version is not guaranteed to work and may
5635: cause crashes.
5636:
5637:
5638: SAVING A COMPILED PATTERN
5639: The value returned by pcre_compile() points to a single block of memory
5640: that holds the compiled pattern and associated data. You can find the
5641: length of this block in bytes by calling pcre_fullinfo() with an argu-
5642: ment of PCRE_INFO_SIZE. You can then save the data in any appropriate
5643: manner. Here is sample code that compiles a pattern and writes it to a
5644: file. It assumes that the variable fd refers to a file that is open for
5645: output:
5646:
5647: int erroroffset, rc, size;
5648: char *error;
5649: pcre *re;
5650:
5651: re = pcre_compile("my pattern", 0, &error, &erroroffset, NULL);
5652: if (re == NULL) { ... handle errors ... }
5653: rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &size);
5654: if (rc < 0) { ... handle errors ... }
5655: rc = fwrite(re, 1, size, fd);
5656: if (rc != size) { ... handle errors ... }
5657:
5658: In this example, the bytes that comprise the compiled pattern are
5659: copied exactly. Note that this is binary data that may contain any of
5660: the 256 possible byte values. On systems that make a distinction
5661: between binary and non-binary data, be sure that the file is opened for
5662: binary output.
5663:
5664: If you want to write more than one pattern to a file, you will have to
5665: devise a way of separating them. For binary data, preceding each pat-
5666: tern with its length is probably the most straightforward approach.
5667: Another possibility is to write out the data in hexadecimal instead of
5668: binary, one pattern to a line.
5669:
5670: Saving compiled patterns in a file is only one possible way of storing
5671: them for later use. They could equally well be saved in a database, or
5672: in the memory of some daemon process that passes them via sockets to
5673: the processes that want them.
5674:
5675: If the pattern has been studied, it is also possible to save the study
5676: data in a similar way to the compiled pattern itself. When studying
5677: generates additional information, pcre_study() returns a pointer to a
5678: pcre_extra data block. Its format is defined in the section on matching
5679: a pattern in the pcreapi documentation. The study_data field points to
5680: the binary study data, and this is what you must save (not the
5681: pcre_extra block itself). The length of the study data can be obtained
5682: by calling pcre_fullinfo() with an argument of PCRE_INFO_STUDYSIZE.
5683: Remember to check that pcre_study() did return a non-NULL value before
5684: trying to save the study data.
5685:
5686:
5687: RE-USING A PRECOMPILED PATTERN
5688:
5689: Re-using a precompiled pattern is straightforward. Having reloaded it
5690: into main memory, you pass its pointer to pcre_exec() or
5691: pcre_dfa_exec() in the usual way. This should work even on another
5692: host, and even if that host has the opposite endianness to the one
5693: where the pattern was compiled.
5694:
5695: However, if you passed a pointer to custom character tables when the
5696: pattern was compiled (the tableptr argument of pcre_compile()), you
5697: must now pass a similar pointer to pcre_exec() or pcre_dfa_exec(),
5698: because the value saved with the compiled pattern will obviously be
5699: nonsense. A field in a pcre_extra() block is used to pass this data, as
5700: described in the section on matching a pattern in the pcreapi documen-
5701: tation.
5702:
5703: If you did not provide custom character tables when the pattern was
5704: compiled, the pointer in the compiled pattern is NULL, which causes
5705: pcre_exec() to use PCRE's internal tables. Thus, you do not need to
5706: take any special action at run time in this case.
5707:
5708: If you saved study data with the compiled pattern, you need to create
5709: your own pcre_extra data block and set the study_data field to point to
5710: the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA
5711: bit in the flags field to indicate that study data is present. Then
5712: pass the pcre_extra block to pcre_exec() or pcre_dfa_exec() in the
5713: usual way.
5714:
5715:
5716: COMPATIBILITY WITH DIFFERENT PCRE RELEASES
5717:
5718: In general, it is safest to recompile all saved patterns when you
5719: update to a new PCRE release, though not all updates actually require
5720: this. Recompiling is definitely needed for release 7.2.
5721:
5722:
5723: AUTHOR
5724:
5725: Philip Hazel
5726: University Computing Service
5727: Cambridge CB2 3QH, England.
5728:
5729:
5730: REVISION
5731:
5732: Last updated: 13 June 2007
5733: Copyright (c) 1997-2007 University of Cambridge.
5734: ------------------------------------------------------------------------------
5735:
5736:
5737: PCREPERFORM(3) PCREPERFORM(3)
5738:
5739:
5740: NAME
5741: PCRE - Perl-compatible regular expressions
5742:
5743:
5744: PCRE PERFORMANCE
5745:
5746: Two aspects of performance are discussed below: memory usage and pro-
5747: cessing time. The way you express your pattern as a regular expression
5748: can affect both of them.
5749:
5750:
5751: MEMORY USAGE
5752:
5753: Patterns are compiled by PCRE into a reasonably efficient byte code, so
5754: that most simple patterns do not use much memory. However, there is one
5755: case where memory usage can be unexpectedly large. When a parenthesized
5756: subpattern has a quantifier with a minimum greater than 1 and/or a lim-
5757: ited maximum, the whole subpattern is repeated in the compiled code.
5758: For example, the pattern
5759:
5760: (abc|def){2,4}
5761:
5762: is compiled as if it were
5763:
5764: (abc|def)(abc|def)((abc|def)(abc|def)?)?
5765:
5766: (Technical aside: It is done this way so that backtrack points within
5767: each of the repetitions can be independently maintained.)
5768:
5769: For regular expressions whose quantifiers use only small numbers, this
5770: is not usually a problem. However, if the numbers are large, and par-
5771: ticularly if such repetitions are nested, the memory usage can become
5772: an embarrassment. For example, the very simple pattern
5773:
5774: ((ab){1,1000}c){1,3}
5775:
5776: uses 51K bytes when compiled. When PCRE is compiled with its default
5777: internal pointer size of two bytes, the size limit on a compiled pat-
5778: tern is 64K, and this is reached with the above pattern if the outer
5779: repetition is increased from 3 to 4. PCRE can be compiled to use larger
5780: internal pointers and thus handle larger compiled patterns, but it is
5781: better to try to rewrite your pattern to use less memory if you can.
5782:
5783: One way of reducing the memory usage for such patterns is to make use
5784: of PCRE's "subroutine" facility. Re-writing the above pattern as
5785:
5786: ((ab)(?2){0,999}c)(?1){0,2}
5787:
5788: reduces the memory requirements to 18K, and indeed it remains under 20K
5789: even with the outer repetition increased to 100. However, this pattern
5790: is not exactly equivalent, because the "subroutine" calls are treated
5791: as atomic groups into which there can be no backtracking if there is a
5792: subsequent matching failure. Therefore, PCRE cannot do this kind of
5793: rewriting automatically. Furthermore, there is a noticeable loss of
5794: speed when executing the modified pattern. Nevertheless, if the atomic
5795: grouping is not a problem and the loss of speed is acceptable, this
5796: kind of rewriting will allow you to process patterns that PCRE cannot
5797: otherwise handle.
5798:
5799:
5800: PROCESSING TIME
5801:
5802: Certain items in regular expression patterns are processed more effi-
5803: ciently than others. It is more efficient to use a character class like
5804: [aeiou] than a set of single-character alternatives such as
5805: (a|e|i|o|u). In general, the simplest construction that provides the
5806: required behaviour is usually the most efficient. Jeffrey Friedl's book
5807: contains a lot of useful general discussion about optimizing regular
5808: expressions for efficient performance. This document contains a few
5809: observations about PCRE.
5810:
5811: Using Unicode character properties (the \p, \P, and \X escapes) is
5812: slow, because PCRE has to scan a structure that contains data for over
5813: fifteen thousand characters whenever it needs a character's property.
5814: If you can find an alternative pattern that does not use character
5815: properties, it will probably be faster.
5816:
5817: When a pattern begins with .* not in parentheses, or in parentheses
5818: that are not the subject of a backreference, and the PCRE_DOTALL option
5819: is set, the pattern is implicitly anchored by PCRE, since it can match
5820: only at the start of a subject string. However, if PCRE_DOTALL is not
5821: set, PCRE cannot make this optimization, because the . metacharacter
5822: does not then match a newline, and if the subject string contains new-
5823: lines, the pattern may match from the character immediately following
5824: one of them instead of from the very start. For example, the pattern
5825:
5826: .*second
5827:
5828: matches the subject "first\nand second" (where \n stands for a newline
5829: character), with the match starting at the seventh character. In order
5830: to do this, PCRE has to retry the match starting after every newline in
5831: the subject.
5832:
5833: If you are using such a pattern with subject strings that do not con-
5834: tain newlines, the best performance is obtained by setting PCRE_DOTALL,
5835: or starting the pattern with ^.* or ^.*? to indicate explicit anchor-
5836: ing. That saves PCRE from having to scan along the subject looking for
5837: a newline to restart at.
5838:
5839: Beware of patterns that contain nested indefinite repeats. These can
5840: take a long time to run when applied to a string that does not match.
5841: Consider the pattern fragment
5842:
5843: ^(a+)*
5844:
5845: This can match "aaaa" in 16 different ways, and this number increases
5846: very rapidly as the string gets longer. (The * repeat can match 0, 1,
5847: 2, 3, or 4 times, and for each of those cases other than 0 or 4, the +
5848: repeats can match different numbers of times.) When the remainder of
5849: the pattern is such that the entire match is going to fail, PCRE has in
5850: principle to try every possible variation, and this can take an
5851: extremely long time, even for relatively short strings.
5852:
5853: An optimization catches some of the more simple cases such as
5854:
5855: (a+)*b
5856:
5857: where a literal character follows. Before embarking on the standard
5858: matching procedure, PCRE checks that there is a "b" later in the sub-
5859: ject string, and if there is not, it fails the match immediately. How-
5860: ever, when there is no following literal this optimization cannot be
5861: used. You can see the difference by comparing the behaviour of
5862:
5863: (a+)*\d
5864:
5865: with the pattern above. The former gives a failure almost instantly
5866: when applied to a whole line of "a" characters, whereas the latter
5867: takes an appreciable time with strings longer than about 20 characters.
5868:
5869: In many cases, the solution to this kind of performance issue is to use
5870: an atomic group or a possessive quantifier.
5871:
5872:
5873: AUTHOR
5874:
5875: Philip Hazel
5876: University Computing Service
5877: Cambridge CB2 3QH, England.
5878:
5879:
5880: REVISION
5881:
5882: Last updated: 06 March 2007
5883: Copyright (c) 1997-2007 University of Cambridge.
5884: ------------------------------------------------------------------------------
5885:
5886:
5887: PCREPOSIX(3) PCREPOSIX(3)
5888:
5889:
5890: NAME
5891: PCRE - Perl-compatible regular expressions.
5892:
5893:
5894: SYNOPSIS OF POSIX API
5895:
5896: #include <pcreposix.h>
5897:
5898: int regcomp(regex_t *preg, const char *pattern,
5899: int cflags);
5900:
5901: int regexec(regex_t *preg, const char *string,
5902: size_t nmatch, regmatch_t pmatch[], int eflags);
5903:
5904: size_t regerror(int errcode, const regex_t *preg,
5905: char *errbuf, size_t errbuf_size);
5906:
5907: void regfree(regex_t *preg);
5908:
5909:
5910: DESCRIPTION
5911:
5912: This set of functions provides a POSIX-style API to the PCRE regular
5913: expression package. See the pcreapi documentation for a description of
5914: PCRE's native API, which contains much additional functionality.
5915:
5916: The functions described here are just wrapper functions that ultimately
5917: call the PCRE native API. Their prototypes are defined in the
5918: pcreposix.h header file, and on Unix systems the library itself is
5919: called pcreposix.a, so can be accessed by adding -lpcreposix to the
5920: command for linking an application that uses them. Because the POSIX
5921: functions call the native ones, it is also necessary to add -lpcre.
5922:
1.3 ! misha 5923: I have implemented only those POSIX option bits that can be reasonably
! 5924: mapped to PCRE native options. In addition, the option REG_EXTENDED is
! 5925: defined with the value zero. This has no effect, but since programs
! 5926: that are written to the POSIX interface often use it, this makes it
! 5927: easier to slot in PCRE as a replacement library. Other POSIX options
! 5928: are not even defined.
1.1 misha 5929:
5930: When PCRE is called via these functions, it is only the API that is
5931: POSIX-like in style. The syntax and semantics of the regular expres-
5932: sions themselves are still those of Perl, subject to the setting of
5933: various PCRE options, as described below. "POSIX-like in style" means
5934: that the API approximates to the POSIX definition; it is not fully
5935: POSIX-compatible, and in multi-byte encoding domains it is probably
5936: even less compatible.
5937:
5938: The header for these functions is supplied as pcreposix.h to avoid any
5939: potential clash with other POSIX libraries. It can, of course, be
5940: renamed or aliased as regex.h, which is the "correct" name. It provides
5941: two structure types, regex_t for compiled internal forms, and reg-
5942: match_t for returning captured substrings. It also defines some con-
5943: stants whose names start with "REG_"; these are used for setting
5944: options and identifying error codes.
5945:
5946:
5947: COMPILING A PATTERN
5948:
5949: The function regcomp() is called to compile a pattern into an internal
5950: form. The pattern is a C string terminated by a binary zero, and is
5951: passed in the argument pattern. The preg argument is a pointer to a
5952: regex_t structure that is used as a base for storing information about
5953: the compiled regular expression.
5954:
5955: The argument cflags is either zero, or contains one or more of the bits
5956: defined by the following macros:
5957:
5958: REG_DOTALL
5959:
5960: The PCRE_DOTALL option is set when the regular expression is passed for
5961: compilation to the native function. Note that REG_DOTALL is not part of
5962: the POSIX standard.
5963:
5964: REG_ICASE
5965:
5966: The PCRE_CASELESS option is set when the regular expression is passed
5967: for compilation to the native function.
5968:
5969: REG_NEWLINE
5970:
5971: The PCRE_MULTILINE option is set when the regular expression is passed
5972: for compilation to the native function. Note that this does not mimic
5973: the defined POSIX behaviour for REG_NEWLINE (see the following sec-
5974: tion).
5975:
5976: REG_NOSUB
5977:
5978: The PCRE_NO_AUTO_CAPTURE option is set when the regular expression is
5979: passed for compilation to the native function. In addition, when a pat-
5980: tern that is compiled with this flag is passed to regexec() for match-
5981: ing, the nmatch and pmatch arguments are ignored, and no captured
5982: strings are returned.
5983:
5984: REG_UTF8
5985:
5986: The PCRE_UTF8 option is set when the regular expression is passed for
5987: compilation to the native function. This causes the pattern itself and
5988: all data strings used for matching it to be treated as UTF-8 strings.
5989: Note that REG_UTF8 is not part of the POSIX standard.
5990:
5991: In the absence of these flags, no options are passed to the native
5992: function. This means the the regex is compiled with PCRE default
5993: semantics. In particular, the way it handles newline characters in the
5994: subject string is the Perl way, not the POSIX way. Note that setting
5995: PCRE_MULTILINE has only some of the effects specified for REG_NEWLINE.
5996: It does not affect the way newlines are matched by . (they aren't) or
5997: by a negative class such as [^a] (they are).
5998:
5999: The yield of regcomp() is zero on success, and non-zero otherwise. The
6000: preg structure is filled in on success, and one member of the structure
6001: is public: re_nsub contains the number of capturing subpatterns in the
6002: regular expression. Various error codes are defined in the header file.
6003:
6004:
6005: MATCHING NEWLINE CHARACTERS
6006:
6007: This area is not simple, because POSIX and Perl take different views of
1.3 ! misha 6008: things. It is not possible to get PCRE to obey POSIX semantics, but
! 6009: then PCRE was never intended to be a POSIX engine. The following table
! 6010: lists the different possibilities for matching newline characters in
1.1 misha 6011: PCRE:
6012:
6013: Default Change with
6014:
6015: . matches newline no PCRE_DOTALL
6016: newline matches [^a] yes not changeable
6017: $ matches \n at end yes PCRE_DOLLARENDONLY
6018: $ matches \n in middle no PCRE_MULTILINE
6019: ^ matches \n in middle no PCRE_MULTILINE
6020:
6021: This is the equivalent table for POSIX:
6022:
6023: Default Change with
6024:
6025: . matches newline yes REG_NEWLINE
6026: newline matches [^a] yes REG_NEWLINE
6027: $ matches \n at end no REG_NEWLINE
6028: $ matches \n in middle no REG_NEWLINE
6029: ^ matches \n in middle no REG_NEWLINE
6030:
6031: PCRE's behaviour is the same as Perl's, except that there is no equiva-
1.3 ! misha 6032: lent for PCRE_DOLLAR_ENDONLY in Perl. In both PCRE and Perl, there is
1.1 misha 6033: no way to stop newline from matching [^a].
6034:
1.3 ! misha 6035: The default POSIX newline handling can be obtained by setting
! 6036: PCRE_DOTALL and PCRE_DOLLAR_ENDONLY, but there is no way to make PCRE
1.1 misha 6037: behave exactly as for the REG_NEWLINE action.
6038:
6039:
6040: MATCHING A PATTERN
6041:
1.3 ! misha 6042: The function regexec() is called to match a compiled pattern preg
! 6043: against a given string, which is by default terminated by a zero byte
! 6044: (but see REG_STARTEND below), subject to the options in eflags. These
1.1 misha 6045: can be:
6046:
6047: REG_NOTBOL
6048:
6049: The PCRE_NOTBOL option is set when calling the underlying PCRE matching
6050: function.
6051:
1.3 ! misha 6052: REG_NOTEMPTY
! 6053:
! 6054: The PCRE_NOTEMPTY option is set when calling the underlying PCRE match-
! 6055: ing function. Note that REG_NOTEMPTY is not part of the POSIX standard.
! 6056: However, setting this option can give more POSIX-like behaviour in some
! 6057: situations.
! 6058:
1.1 misha 6059: REG_NOTEOL
6060:
6061: The PCRE_NOTEOL option is set when calling the underlying PCRE matching
6062: function.
6063:
6064: REG_STARTEND
6065:
6066: The string is considered to start at string + pmatch[0].rm_so and to
6067: have a terminating NUL located at string + pmatch[0].rm_eo (there need
6068: not actually be a NUL at that location), regardless of the value of
6069: nmatch. This is a BSD extension, compatible with but not specified by
6070: IEEE Standard 1003.2 (POSIX.2), and should be used with caution in
6071: software intended to be portable to other systems. Note that a non-zero
6072: rm_so does not imply REG_NOTBOL; REG_STARTEND affects only the location
6073: of the string, not how it is matched.
6074:
6075: If the pattern was compiled with the REG_NOSUB flag, no data about any
6076: matched strings is returned. The nmatch and pmatch arguments of
6077: regexec() are ignored.
6078:
6079: Otherwise,the portion of the string that was matched, and also any cap-
6080: tured substrings, are returned via the pmatch argument, which points to
6081: an array of nmatch structures of type regmatch_t, containing the mem-
6082: bers rm_so and rm_eo. These contain the offset to the first character
6083: of each substring and the offset to the first character after the end
6084: of each substring, respectively. The 0th element of the vector relates
6085: to the entire portion of string that was matched; subsequent elements
6086: relate to the capturing subpatterns of the regular expression. Unused
6087: entries in the array have both structure members set to -1.
6088:
6089: A successful match yields a zero return; various error codes are
6090: defined in the header file, of which REG_NOMATCH is the "expected"
6091: failure code.
6092:
6093:
6094: ERROR MESSAGES
6095:
6096: The regerror() function maps a non-zero errorcode from either regcomp()
6097: or regexec() to a printable message. If preg is not NULL, the error
6098: should have arisen from the use of that structure. A message terminated
6099: by a binary zero is placed in errbuf. The length of the message,
6100: including the zero, is limited to errbuf_size. The yield of the func-
6101: tion is the size of buffer needed to hold the whole message.
6102:
6103:
6104: MEMORY USAGE
6105:
6106: Compiling a regular expression causes memory to be allocated and asso-
6107: ciated with the preg structure. The function regfree() frees all such
6108: memory, after which preg may no longer be used as a compiled expres-
6109: sion.
6110:
6111:
6112: AUTHOR
6113:
6114: Philip Hazel
6115: University Computing Service
6116: Cambridge CB2 3QH, England.
6117:
6118:
6119: REVISION
6120:
1.3 ! misha 6121: Last updated: 11 March 2009
! 6122: Copyright (c) 1997-2009 University of Cambridge.
1.1 misha 6123: ------------------------------------------------------------------------------
6124:
6125:
6126: PCRECPP(3) PCRECPP(3)
6127:
6128:
6129: NAME
6130: PCRE - Perl-compatible regular expressions.
6131:
6132:
6133: SYNOPSIS OF C++ WRAPPER
6134:
6135: #include <pcrecpp.h>
6136:
6137:
6138: DESCRIPTION
6139:
6140: The C++ wrapper for PCRE was provided by Google Inc. Some additional
6141: functionality was added by Giuseppe Maxia. This brief man page was con-
6142: structed from the notes in the pcrecpp.h file, which should be con-
6143: sulted for further details.
6144:
6145:
6146: MATCHING INTERFACE
6147:
6148: The "FullMatch" operation checks that supplied text matches a supplied
6149: pattern exactly. If pointer arguments are supplied, it copies matched
6150: sub-strings that match sub-patterns into them.
6151:
6152: Example: successful match
6153: pcrecpp::RE re("h.*o");
6154: re.FullMatch("hello");
6155:
6156: Example: unsuccessful match (requires full match):
6157: pcrecpp::RE re("e");
6158: !re.FullMatch("hello");
6159:
6160: Example: creating a temporary RE object:
6161: pcrecpp::RE("h.*o").FullMatch("hello");
6162:
6163: You can pass in a "const char*" or a "string" for "text". The examples
6164: below tend to use a const char*. You can, as in the different examples
6165: above, store the RE object explicitly in a variable or use a temporary
6166: RE object. The examples below use one mode or the other arbitrarily.
6167: Either could correctly be used for any of these examples.
6168:
6169: You must supply extra pointer arguments to extract matched subpieces.
6170:
6171: Example: extracts "ruby" into "s" and 1234 into "i"
6172: int i;
6173: string s;
6174: pcrecpp::RE re("(\\w+):(\\d+)");
6175: re.FullMatch("ruby:1234", &s, &i);
6176:
6177: Example: does not try to extract any extra sub-patterns
6178: re.FullMatch("ruby:1234", &s);
6179:
6180: Example: does not try to extract into NULL
6181: re.FullMatch("ruby:1234", NULL, &i);
6182:
6183: Example: integer overflow causes failure
6184: !re.FullMatch("ruby:1234567891234", NULL, &i);
6185:
6186: Example: fails because there aren't enough sub-patterns:
6187: !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s);
6188:
6189: Example: fails because string cannot be stored in integer
6190: !pcrecpp::RE("(.*)").FullMatch("ruby", &i);
6191:
6192: The provided pointer arguments can be pointers to any scalar numeric
6193: type, or one of:
6194:
6195: string (matched piece is copied to string)
6196: StringPiece (StringPiece is mutated to point to matched piece)
6197: T (where "bool T::ParseFrom(const char*, int)" exists)
6198: NULL (the corresponding matched sub-pattern is not copied)
6199:
6200: The function returns true iff all of the following conditions are sat-
6201: isfied:
6202:
6203: a. "text" matches "pattern" exactly;
6204:
6205: b. The number of matched sub-patterns is >= number of supplied
6206: pointers;
6207:
6208: c. The "i"th argument has a suitable type for holding the
6209: string captured as the "i"th sub-pattern. If you pass in
6210: void * NULL for the "i"th argument, or a non-void * NULL
6211: of the correct type, or pass fewer arguments than the
6212: number of sub-patterns, "i"th captured sub-pattern is
6213: ignored.
6214:
6215: CAVEAT: An optional sub-pattern that does not exist in the matched
6216: string is assigned the empty string. Therefore, the following will
6217: return false (because the empty string is not a valid number):
6218:
6219: int number;
6220: pcrecpp::RE::FullMatch("abc", "[a-z]+(\\d+)?", &number);
6221:
6222: The matching interface supports at most 16 arguments per call. If you
6223: need more, consider using the more general interface
6224: pcrecpp::RE::DoMatch. See pcrecpp.h for the signature for DoMatch.
6225:
1.3 ! misha 6226: NOTE: Do not use no_arg, which is used internally to mark the end of a
! 6227: list of optional arguments, as a placeholder for missing arguments, as
! 6228: this can lead to segfaults.
! 6229:
1.1 misha 6230:
6231: QUOTING METACHARACTERS
6232:
6233: You can use the "QuoteMeta" operation to insert backslashes before all
6234: potentially meaningful characters in a string. The returned string,
6235: used as a regular expression, will exactly match the original string.
6236:
6237: Example:
6238: string quoted = RE::QuoteMeta(unquoted);
6239:
6240: Note that it's legal to escape a character even if it has no special
6241: meaning in a regular expression -- so this function does that. (This
6242: also makes it identical to the perl function of the same name; see
6243: "perldoc -f quotemeta".) For example, "1.5-2.0?" becomes
6244: "1\.5\-2\.0\?".
6245:
6246:
6247: PARTIAL MATCHES
6248:
6249: You can use the "PartialMatch" operation when you want the pattern to
6250: match any substring of the text.
6251:
6252: Example: simple search for a string:
6253: pcrecpp::RE("ell").PartialMatch("hello");
6254:
6255: Example: find first number in a string:
6256: int number;
6257: pcrecpp::RE re("(\\d+)");
6258: re.PartialMatch("x*100 + 20", &number);
6259: assert(number == 100);
6260:
6261:
6262: UTF-8 AND THE MATCHING INTERFACE
6263:
6264: By default, pattern and text are plain text, one byte per character.
6265: The UTF8 flag, passed to the constructor, causes both pattern and
6266: string to be treated as UTF-8 text, still a byte stream but potentially
6267: multiple bytes per character. In practice, the text is likelier to be
6268: UTF-8 than the pattern, but the match returned may depend on the UTF8
6269: flag, so always use it when matching UTF8 text. For example, "." will
6270: match one byte normally but with UTF8 set may match up to three bytes
6271: of a multi-byte character.
6272:
6273: Example:
6274: pcrecpp::RE_Options options;
6275: options.set_utf8();
6276: pcrecpp::RE re(utf8_pattern, options);
6277: re.FullMatch(utf8_string);
6278:
6279: Example: using the convenience function UTF8():
6280: pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8());
6281: re.FullMatch(utf8_string);
6282:
6283: NOTE: The UTF8 flag is ignored if pcre was not configured with the
6284: --enable-utf8 flag.
6285:
6286:
6287: PASSING MODIFIERS TO THE REGULAR EXPRESSION ENGINE
6288:
6289: PCRE defines some modifiers to change the behavior of the regular
6290: expression engine. The C++ wrapper defines an auxiliary class,
6291: RE_Options, as a vehicle to pass such modifiers to a RE class. Cur-
6292: rently, the following modifiers are supported:
6293:
6294: modifier description Perl corresponding
6295:
6296: PCRE_CASELESS case insensitive match /i
6297: PCRE_MULTILINE multiple lines match /m
6298: PCRE_DOTALL dot matches newlines /s
6299: PCRE_DOLLAR_ENDONLY $ matches only at end N/A
6300: PCRE_EXTRA strict escape parsing N/A
6301: PCRE_EXTENDED ignore whitespaces /x
6302: PCRE_UTF8 handles UTF8 chars built-in
6303: PCRE_UNGREEDY reverses * and *? N/A
6304: PCRE_NO_AUTO_CAPTURE disables capturing parens N/A (*)
6305:
6306: (*) Both Perl and PCRE allow non capturing parentheses by means of the
6307: "?:" modifier within the pattern itself. e.g. (?:ab|cd) does not cap-
6308: ture, while (ab|cd) does.
6309:
6310: For a full account on how each modifier works, please check the PCRE
6311: API reference page.
6312:
6313: For each modifier, there are two member functions whose name is made
6314: out of the modifier in lowercase, without the "PCRE_" prefix. For
6315: instance, PCRE_CASELESS is handled by
6316:
6317: bool caseless()
6318:
6319: which returns true if the modifier is set, and
6320:
6321: RE_Options & set_caseless(bool)
6322:
6323: which sets or unsets the modifier. Moreover, PCRE_EXTRA_MATCH_LIMIT can
6324: be accessed through the set_match_limit() and match_limit() member
6325: functions. Setting match_limit to a non-zero value will limit the exe-
6326: cution of pcre to keep it from doing bad things like blowing the stack
6327: or taking an eternity to return a result. A value of 5000 is good
6328: enough to stop stack blowup in a 2MB thread stack. Setting match_limit
6329: to zero disables match limiting. Alternatively, you can call
6330: match_limit_recursion() which uses PCRE_EXTRA_MATCH_LIMIT_RECURSION to
6331: limit how much PCRE recurses. match_limit() limits the number of
6332: matches PCRE does; match_limit_recursion() limits the depth of internal
6333: recursion, and therefore the amount of stack that is used.
6334:
6335: Normally, to pass one or more modifiers to a RE class, you declare a
6336: RE_Options object, set the appropriate options, and pass this object to
6337: a RE constructor. Example:
6338:
6339: RE_options opt;
6340: opt.set_caseless(true);
6341: if (RE("HELLO", opt).PartialMatch("hello world")) ...
6342:
6343: RE_options has two constructors. The default constructor takes no argu-
6344: ments and creates a set of flags that are off by default. The optional
6345: parameter option_flags is to facilitate transfer of legacy code from C
6346: programs. This lets you do
6347:
6348: RE(pattern,
6349: RE_Options(PCRE_CASELESS|PCRE_MULTILINE)).PartialMatch(str);
6350:
6351: However, new code is better off doing
6352:
6353: RE(pattern,
6354: RE_Options().set_caseless(true).set_multiline(true))
6355: .PartialMatch(str);
6356:
6357: If you are going to pass one of the most used modifiers, there are some
6358: convenience functions that return a RE_Options class with the appropri-
6359: ate modifier already set: CASELESS(), UTF8(), MULTILINE(), DOTALL(),
6360: and EXTENDED().
6361:
6362: If you need to set several options at once, and you don't want to go
6363: through the pains of declaring a RE_Options object and setting several
6364: options, there is a parallel method that give you such ability on the
6365: fly. You can concatenate several set_xxxxx() member functions, since
6366: each of them returns a reference to its class object. For example, to
6367: pass PCRE_CASELESS, PCRE_EXTENDED, and PCRE_MULTILINE to a RE with one
6368: statement, you may write:
6369:
6370: RE(" ^ xyz \\s+ .* blah$",
6371: RE_Options()
6372: .set_caseless(true)
6373: .set_extended(true)
6374: .set_multiline(true)).PartialMatch(sometext);
6375:
6376:
6377: SCANNING TEXT INCREMENTALLY
6378:
6379: The "Consume" operation may be useful if you want to repeatedly match
6380: regular expressions at the front of a string and skip over them as they
6381: match. This requires use of the "StringPiece" type, which represents a
6382: sub-range of a real string. Like RE, StringPiece is defined in the
6383: pcrecpp namespace.
6384:
6385: Example: read lines of the form "var = value" from a string.
6386: string contents = ...; // Fill string somehow
6387: pcrecpp::StringPiece input(contents); // Wrap in a StringPiece
6388:
6389: string var;
6390: int value;
6391: pcrecpp::RE re("(\\w+) = (\\d+)\n");
6392: while (re.Consume(&input, &var, &value)) {
6393: ...;
6394: }
6395:
6396: Each successful call to "Consume" will set "var/value", and also
6397: advance "input" so it points past the matched text.
6398:
6399: The "FindAndConsume" operation is similar to "Consume" but does not
6400: anchor your match at the beginning of the string. For example, you
6401: could extract all words from a string by repeatedly calling
6402:
6403: pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word)
6404:
6405:
6406: PARSING HEX/OCTAL/C-RADIX NUMBERS
6407:
6408: By default, if you pass a pointer to a numeric value, the corresponding
6409: text is interpreted as a base-10 number. You can instead wrap the
6410: pointer with a call to one of the operators Hex(), Octal(), or CRadix()
6411: to interpret the text in another base. The CRadix operator interprets
6412: C-style "0" (base-8) and "0x" (base-16) prefixes, but defaults to
6413: base-10.
6414:
6415: Example:
6416: int a, b, c, d;
6417: pcrecpp::RE re("(.*) (.*) (.*) (.*)");
6418: re.FullMatch("100 40 0100 0x40",
6419: pcrecpp::Octal(&a), pcrecpp::Hex(&b),
6420: pcrecpp::CRadix(&c), pcrecpp::CRadix(&d));
6421:
6422: will leave 64 in a, b, c, and d.
6423:
6424:
6425: REPLACING PARTS OF STRINGS
6426:
6427: You can replace the first match of "pattern" in "str" with "rewrite".
6428: Within "rewrite", backslash-escaped digits (\1 to \9) can be used to
6429: insert text matching corresponding parenthesized group from the pat-
6430: tern. \0 in "rewrite" refers to the entire matching text. For example:
6431:
6432: string s = "yabba dabba doo";
6433: pcrecpp::RE("b+").Replace("d", &s);
6434:
6435: will leave "s" containing "yada dabba doo". The result is true if the
6436: pattern matches and a replacement occurs, false otherwise.
6437:
6438: GlobalReplace is like Replace except that it replaces all occurrences
6439: of the pattern in the string with the rewrite. Replacements are not
6440: subject to re-matching. For example:
6441:
6442: string s = "yabba dabba doo";
6443: pcrecpp::RE("b+").GlobalReplace("d", &s);
6444:
6445: will leave "s" containing "yada dada doo". It returns the number of
6446: replacements made.
6447:
6448: Extract is like Replace, except that if the pattern matches, "rewrite"
6449: is copied into "out" (an additional argument) with substitutions. The
6450: non-matching portions of "text" are ignored. Returns true iff a match
6451: occurred and the extraction happened successfully; if no match occurs,
6452: the string is left unaffected.
6453:
6454:
6455: AUTHOR
6456:
6457: The C++ wrapper was contributed by Google Inc.
6458: Copyright (c) 2007 Google Inc.
6459:
6460:
6461: REVISION
6462:
1.3 ! misha 6463: Last updated: 17 March 2009
1.1 misha 6464: ------------------------------------------------------------------------------
6465:
6466:
6467: PCRESAMPLE(3) PCRESAMPLE(3)
6468:
6469:
6470: NAME
6471: PCRE - Perl-compatible regular expressions
6472:
6473:
6474: PCRE SAMPLE PROGRAM
6475:
6476: A simple, complete demonstration program, to get you started with using
6477: PCRE, is supplied in the file pcredemo.c in the PCRE distribution.
6478:
6479: The program compiles the regular expression that is its first argument,
6480: and matches it against the subject string in its second argument. No
6481: PCRE options are set, and default character tables are used. If match-
6482: ing succeeds, the program outputs the portion of the subject that
6483: matched, together with the contents of any captured substrings.
6484:
6485: If the -g option is given on the command line, the program then goes on
6486: to check for further matches of the same regular expression in the same
6487: subject string. The logic is a little bit tricky because of the possi-
6488: bility of matching an empty string. Comments in the code explain what
6489: is going on.
6490:
6491: If PCRE is installed in the standard include and library directories
6492: for your system, you should be able to compile the demonstration pro-
6493: gram using this command:
6494:
6495: gcc -o pcredemo pcredemo.c -lpcre
6496:
6497: If PCRE is installed elsewhere, you may need to add additional options
6498: to the command line. For example, on a Unix-like system that has PCRE
6499: installed in /usr/local, you can compile the demonstration program
6500: using a command like this:
6501:
6502: gcc -o pcredemo -I/usr/local/include pcredemo.c \
6503: -L/usr/local/lib -lpcre
6504:
6505: Once you have compiled the demonstration program, you can run simple
6506: tests like this:
6507:
6508: ./pcredemo 'cat|dog' 'the cat sat on the mat'
6509: ./pcredemo -g 'cat|dog' 'the dog sat on the cat'
6510:
6511: Note that there is a much more comprehensive test program, called
6512: pcretest, which supports many more facilities for testing regular
6513: expressions and the PCRE library. The pcredemo program is provided as a
6514: simple coding example.
6515:
6516: On some operating systems (e.g. Solaris), when PCRE is not installed in
6517: the standard library directory, you may get an error like this when you
6518: try to run pcredemo:
6519:
6520: ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or
6521: directory
6522:
6523: This is caused by the way shared library support works on those sys-
6524: tems. You need to add
6525:
6526: -R/usr/local/lib
6527:
6528: (for example) to the compile command to get round this problem.
6529:
6530:
6531: AUTHOR
6532:
6533: Philip Hazel
6534: University Computing Service
6535: Cambridge CB2 3QH, England.
6536:
6537:
6538: REVISION
6539:
6540: Last updated: 23 January 2008
6541: Copyright (c) 1997-2008 University of Cambridge.
6542: ------------------------------------------------------------------------------
6543: PCRESTACK(3) PCRESTACK(3)
6544:
6545:
6546: NAME
6547: PCRE - Perl-compatible regular expressions
6548:
6549:
6550: PCRE DISCUSSION OF STACK USAGE
6551:
6552: When you call pcre_exec(), it makes use of an internal function called
6553: match(). This calls itself recursively at branch points in the pattern,
6554: in order to remember the state of the match so that it can back up and
6555: try a different alternative if the first one fails. As matching pro-
6556: ceeds deeper and deeper into the tree of possibilities, the recursion
6557: depth increases.
6558:
6559: Not all calls of match() increase the recursion depth; for an item such
6560: as a* it may be called several times at the same level, after matching
6561: different numbers of a's. Furthermore, in a number of cases where the
6562: result of the recursive call would immediately be passed back as the
6563: result of the current call (a "tail recursion"), the function is just
6564: restarted instead.
6565:
6566: The pcre_dfa_exec() function operates in an entirely different way, and
6567: hardly uses recursion at all. The limit on its complexity is the amount
6568: of workspace it is given. The comments that follow do NOT apply to
6569: pcre_dfa_exec(); they are relevant only for pcre_exec().
6570:
6571: You can set limits on the number of times that match() is called, both
6572: in total and recursively. If the limit is exceeded, an error occurs.
6573: For details, see the section on extra data for pcre_exec() in the
6574: pcreapi documentation.
6575:
6576: Each time that match() is actually called recursively, it uses memory
6577: from the process stack. For certain kinds of pattern and data, very
6578: large amounts of stack may be needed, despite the recognition of "tail
6579: recursion". You can often reduce the amount of recursion, and there-
6580: fore the amount of stack used, by modifying the pattern that is being
6581: matched. Consider, for example, this pattern:
6582:
6583: ([^<]|<(?!inet))+
6584:
6585: It matches from wherever it starts until it encounters "<inet" or the
6586: end of the data, and is the kind of pattern that might be used when
6587: processing an XML file. Each iteration of the outer parentheses matches
6588: either one character that is not "<" or a "<" that is not followed by
6589: "inet". However, each time a parenthesis is processed, a recursion
6590: occurs, so this formulation uses a stack frame for each matched charac-
6591: ter. For a long string, a lot of stack is required. Consider now this
6592: rewritten pattern, which matches exactly the same strings:
6593:
6594: ([^<]++|<(?!inet))+
6595:
6596: This uses very much less stack, because runs of characters that do not
6597: contain "<" are "swallowed" in one item inside the parentheses. Recur-
6598: sion happens only when a "<" character that is not followed by "inet"
6599: is encountered (and we assume this is relatively rare). A possessive
6600: quantifier is used to stop any backtracking into the runs of non-"<"
6601: characters, but that is not related to stack usage.
6602:
6603: This example shows that one way of avoiding stack problems when match-
6604: ing long subject strings is to write repeated parenthesized subpatterns
6605: to match more than one character whenever possible.
6606:
1.2 misha 6607: Compiling PCRE to use heap instead of stack
6608:
1.1 misha 6609: In environments where stack memory is constrained, you might want to
6610: compile PCRE to use heap memory instead of stack for remembering back-
6611: up points. This makes it run a lot more slowly, however. Details of how
6612: to do this are given in the pcrebuild documentation. When built in this
6613: way, instead of using the stack, PCRE obtains and frees memory by call-
6614: ing the functions that are pointed to by the pcre_stack_malloc and
6615: pcre_stack_free variables. By default, these point to malloc() and
6616: free(), but you can replace the pointers to cause PCRE to use your own
6617: functions. Since the block sizes are always the same, and are always
6618: freed in reverse order, it may be possible to implement customized mem-
6619: ory handlers that are more efficient than the standard functions.
6620:
1.2 misha 6621: Limiting PCRE's stack usage
6622:
6623: PCRE has an internal counter that can be used to limit the depth of
6624: recursion, and thus cause pcre_exec() to give an error code before it
6625: runs out of stack. By default, the limit is very large, and unlikely
6626: ever to operate. It can be changed when PCRE is built, and it can also
6627: be set when pcre_exec() is called. For details of these interfaces, see
6628: the pcrebuild and pcreapi documentation.
6629:
6630: As a very rough rule of thumb, you should reckon on about 500 bytes per
6631: recursion. Thus, if you want to limit your stack usage to 8Mb, you
6632: should set the limit at 16000 recursions. A 64Mb stack, on the other
6633: hand, can support around 128000 recursions. The pcretest test program
6634: has a command line option (-S) that can be used to increase the size of
6635: its stack.
6636:
6637: Changing stack size in Unix-like systems
6638:
1.1 misha 6639: In Unix-like environments, there is not often a problem with the stack
6640: unless very long strings are involved, though the default limit on
6641: stack size varies from system to system. Values from 8Mb to 64Mb are
6642: common. You can find your default limit by running the command:
6643:
6644: ulimit -s
6645:
6646: Unfortunately, the effect of running out of stack is often SIGSEGV,
6647: though sometimes a more explicit error message is given. You can nor-
6648: mally increase the limit on stack size by code such as this:
6649:
6650: struct rlimit rlim;
6651: getrlimit(RLIMIT_STACK, &rlim);
6652: rlim.rlim_cur = 100*1024*1024;
6653: setrlimit(RLIMIT_STACK, &rlim);
6654:
6655: This reads the current limits (soft and hard) using getrlimit(), then
6656: attempts to increase the soft limit to 100Mb using setrlimit(). You
6657: must do this before calling pcre_exec().
6658:
1.2 misha 6659: Changing stack size in Mac OS X
1.1 misha 6660:
1.2 misha 6661: Using setrlimit(), as described above, should also work on Mac OS X. It
6662: is also possible to set a stack size when linking a program. There is a
6663: discussion about stack sizes in Mac OS X at this web site:
6664: http://developer.apple.com/qa/qa2005/qa1419.html.
1.1 misha 6665:
6666:
6667: AUTHOR
6668:
6669: Philip Hazel
6670: University Computing Service
6671: Cambridge CB2 3QH, England.
6672:
6673:
6674: REVISION
6675:
1.2 misha 6676: Last updated: 09 July 2008
6677: Copyright (c) 1997-2008 University of Cambridge.
1.1 misha 6678: ------------------------------------------------------------------------------
6679:
6680:
E-mail:
![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to pcre.txt CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [parser3project] / win32 / pcre