Annotation of win32/pcre/pcre.txt, revision 1.1
1.1 ! misha 1: -----------------------------------------------------------------------------
! 2: This file contains a concatenation of the PCRE man pages, converted to plain
! 3: text format for ease of searching with a text editor, or for use on systems
! 4: that do not have a man page processor. The small individual files that give
! 5: synopses of each function in the library have not been included. There are
! 6: separate text files for the pcregrep and pcretest commands.
! 7: -----------------------------------------------------------------------------
! 8:
! 9:
! 10: PCRE(3) PCRE(3)
! 11:
! 12:
! 13: NAME
! 14: PCRE - Perl-compatible regular expressions
! 15:
! 16:
! 17: INTRODUCTION
! 18:
! 19: The PCRE library is a set of functions that implement regular expres-
! 20: sion pattern matching using the same syntax and semantics as Perl, with
! 21: just a few differences. Certain features that appeared in Python and
! 22: PCRE before they appeared in Perl are also available using the Python
! 23: syntax. There is also some support for certain .NET and Oniguruma syn-
! 24: tax items, and there is an option for requesting some minor changes
! 25: that give better JavaScript compatibility.
! 26:
! 27: The current implementation of PCRE (release 7.x) corresponds approxi-
! 28: mately with Perl 5.10, including support for UTF-8 encoded strings and
! 29: Unicode general category properties. However, UTF-8 and Unicode support
! 30: has to be explicitly enabled; it is not the default. The Unicode tables
! 31: correspond to Unicode release 5.0.0.
! 32:
! 33: In addition to the Perl-compatible matching function, PCRE contains an
! 34: alternative matching function that matches the same compiled patterns
! 35: in a different way. In certain circumstances, the alternative function
! 36: has some advantages. For a discussion of the two matching algorithms,
! 37: see the pcrematching page.
! 38:
! 39: PCRE is written in C and released as a C library. A number of people
! 40: have written wrappers and interfaces of various kinds. In particular,
! 41: Google Inc. have provided a comprehensive C++ wrapper. This is now
! 42: included as part of the PCRE distribution. The pcrecpp page has details
! 43: of this interface. Other people's contributions can be found in the
! 44: Contrib directory at the primary FTP site, which is:
! 45:
! 46: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
! 47:
! 48: Details of exactly which Perl regular expression features are and are
! 49: not supported by PCRE are given in separate documents. See the pcrepat-
! 50: tern and pcrecompat pages. There is a syntax summary in the pcresyntax
! 51: page.
! 52:
! 53: Some features of PCRE can be included, excluded, or changed when the
! 54: library is built. The pcre_config() function makes it possible for a
! 55: client to discover which features are available. The features them-
! 56: selves are described in the pcrebuild page. Documentation about build-
! 57: ing PCRE for various operating systems can be found in the README file
! 58: in the source distribution.
! 59:
! 60: The library contains a number of undocumented internal functions and
! 61: data tables that are used by more than one of the exported external
! 62: functions, but which are not intended for use by external callers.
! 63: Their names all begin with "_pcre_", which hopefully will not provoke
! 64: any name clashes. In some environments, it is possible to control which
! 65: external symbols are exported when a shared library is built, and in
! 66: these cases the undocumented symbols are not exported.
! 67:
! 68:
! 69: USER DOCUMENTATION
! 70:
! 71: The user documentation for PCRE comprises a number of different sec-
! 72: tions. In the "man" format, each of these is a separate "man page". In
! 73: the HTML format, each is a separate page, linked from the index page.
! 74: In the plain text format, all the sections are concatenated, for ease
! 75: of searching. The sections are as follows:
! 76:
! 77: pcre this document
! 78: pcre-config show PCRE installation configuration information
! 79: pcreapi details of PCRE's native C API
! 80: pcrebuild options for building PCRE
! 81: pcrecallout details of the callout feature
! 82: pcrecompat discussion of Perl compatibility
! 83: pcrecpp details of the C++ wrapper
! 84: pcregrep description of the pcregrep command
! 85: pcrematching discussion of the two matching algorithms
! 86: pcrepartial details of the partial matching facility
! 87: pcrepattern syntax and semantics of supported
! 88: regular expressions
! 89: pcresyntax quick syntax reference
! 90: pcreperform discussion of performance issues
! 91: pcreposix the POSIX-compatible C API
! 92: pcreprecompile details of saving and re-using precompiled patterns
! 93: pcresample discussion of the sample program
! 94: pcrestack discussion of stack usage
! 95: pcretest description of the pcretest testing command
! 96:
! 97: In addition, in the "man" and HTML formats, there is a short page for
! 98: each C library function, listing its arguments and results.
! 99:
! 100:
! 101: LIMITATIONS
! 102:
! 103: There are some size limitations in PCRE but it is hoped that they will
! 104: never in practice be relevant.
! 105:
! 106: The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
! 107: is compiled with the default internal linkage size of 2. If you want to
! 108: process regular expressions that are truly enormous, you can compile
! 109: PCRE with an internal linkage size of 3 or 4 (see the README file in
! 110: the source distribution and the pcrebuild documentation for details).
! 111: In these cases the limit is substantially larger. However, the speed
! 112: of execution is slower.
! 113:
! 114: All values in repeating quantifiers must be less than 65536.
! 115:
! 116: There is no limit to the number of parenthesized subpatterns, but there
! 117: can be no more than 65535 capturing subpatterns.
! 118:
! 119: The maximum length of name for a named subpattern is 32 characters, and
! 120: the maximum number of named subpatterns is 10000.
! 121:
! 122: The maximum length of a subject string is the largest positive number
! 123: that an integer variable can hold. However, when using the traditional
! 124: matching function, PCRE uses recursion to handle subpatterns and indef-
! 125: inite repetition. This means that the available stack space may limit
! 126: the size of a subject string that can be processed by certain patterns.
! 127: For a discussion of stack issues, see the pcrestack documentation.
! 128:
! 129:
! 130: UTF-8 AND UNICODE PROPERTY SUPPORT
! 131:
! 132: From release 3.3, PCRE has had some support for character strings
! 133: encoded in the UTF-8 format. For release 4.0 this was greatly extended
! 134: to cover most common requirements, and in release 5.0 additional sup-
! 135: port for Unicode general category properties was added.
! 136:
! 137: In order process UTF-8 strings, you must build PCRE to include UTF-8
! 138: support in the code, and, in addition, you must call pcre_compile()
! 139: with the PCRE_UTF8 option flag. When you do this, both the pattern and
! 140: any subject strings that are matched against it are treated as UTF-8
! 141: strings instead of just strings of bytes.
! 142:
! 143: If you compile PCRE with UTF-8 support, but do not use it at run time,
! 144: the library will be a bit bigger, but the additional run time overhead
! 145: is limited to testing the PCRE_UTF8 flag occasionally, so should not be
! 146: very big.
! 147:
! 148: If PCRE is built with Unicode character property support (which implies
! 149: UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
! 150: ported. The available properties that can be tested are limited to the
! 151: general category properties such as Lu for an upper case letter or Nd
! 152: for a decimal number, the Unicode script names such as Arabic or Han,
! 153: and the derived properties Any and L&. A full list is given in the
! 154: pcrepattern documentation. Only the short names for properties are sup-
! 155: ported. For example, \p{L} matches a letter. Its Perl synonym, \p{Let-
! 156: ter}, is not supported. Furthermore, in Perl, many properties may
! 157: optionally be prefixed by "Is", for compatibility with Perl 5.6. PCRE
! 158: does not support this.
! 159:
! 160: Validity of UTF-8 strings
! 161:
! 162: When you set the PCRE_UTF8 flag, the strings passed as patterns and
! 163: subjects are (by default) checked for validity on entry to the relevant
! 164: functions. From release 7.3 of PCRE, the check is according the rules
! 165: of RFC 3629, which are themselves derived from the Unicode specifica-
! 166: tion. Earlier releases of PCRE followed the rules of RFC 2279, which
! 167: allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current
! 168: check allows only values in the range U+0 to U+10FFFF, excluding U+D800
! 169: to U+DFFF.
! 170:
! 171: The excluded code points are the "Low Surrogate Area" of Unicode, of
! 172: which the Unicode Standard says this: "The Low Surrogate Area does not
! 173: contain any character assignments, consequently no character code
! 174: charts or namelists are provided for this area. Surrogates are reserved
! 175: for use with UTF-16 and then must be used in pairs." The code points
! 176: that are encoded by UTF-16 pairs are available as independent code
! 177: points in the UTF-8 encoding. (In other words, the whole surrogate
! 178: thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
! 179:
! 180: If an invalid UTF-8 string is passed to PCRE, an error return
! 181: (PCRE_ERROR_BADUTF8) is given. In some situations, you may already know
! 182: that your strings are valid, and therefore want to skip these checks in
! 183: order to improve performance. If you set the PCRE_NO_UTF8_CHECK flag at
! 184: compile time or at run time, PCRE assumes that the pattern or subject
! 185: it is given (respectively) contains only valid UTF-8 codes. In this
! 186: case, it does not diagnose an invalid UTF-8 string.
! 187:
! 188: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set,
! 189: what happens depends on why the string is invalid. If the string con-
! 190: forms to the "old" definition of UTF-8 (RFC 2279), it is processed as a
! 191: string of characters in the range 0 to 0x7FFFFFFF. In other words,
! 192: apart from the initial validity test, PCRE (when in UTF-8 mode) handles
! 193: strings according to the more liberal rules of RFC 2279. However, if
! 194: the string does not even conform to RFC 2279, the result is undefined.
! 195: Your program may crash.
! 196:
! 197: If you want to process strings of values in the full range 0 to
! 198: 0x7FFFFFFF, encoded in a UTF-8-like manner as per the old RFC, you can
! 199: set PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in
! 200: this situation, you will have to apply your own validity check.
! 201:
! 202: General comments about UTF-8 mode
! 203:
! 204: 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a
! 205: two-byte UTF-8 character if the value is greater than 127.
! 206:
! 207: 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
! 208: characters for values greater than \177.
! 209:
! 210: 3. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
! 211: vidual bytes, for example: \x{100}{3}.
! 212:
! 213: 4. The dot metacharacter matches one UTF-8 character instead of a sin-
! 214: gle byte.
! 215:
! 216: 5. The escape sequence \C can be used to match a single byte in UTF-8
! 217: mode, but its use can lead to some strange effects. This facility is
! 218: not available in the alternative matching function, pcre_dfa_exec().
! 219:
! 220: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
! 221: test characters of any code value, but the characters that PCRE recog-
! 222: nizes as digits, spaces, or word characters remain the same set as
! 223: before, all with values less than 256. This remains true even when PCRE
! 224: includes Unicode property support, because to do otherwise would slow
! 225: down PCRE in many common cases. If you really want to test for a wider
! 226: sense of, say, "digit", you must use Unicode property tests such as
! 227: \p{Nd}.
! 228:
! 229: 7. Similarly, characters that match the POSIX named character classes
! 230: are all low-valued characters.
! 231:
! 232: 8. However, the Perl 5.10 horizontal and vertical whitespace matching
! 233: escapes (\h, \H, \v, and \V) do match all the appropriate Unicode char-
! 234: acters.
! 235:
! 236: 9. Case-insensitive matching applies only to characters whose values
! 237: are less than 128, unless PCRE is built with Unicode property support.
! 238: Even when Unicode property support is available, PCRE still uses its
! 239: own character tables when checking the case of low-valued characters,
! 240: so as not to degrade performance. The Unicode property information is
! 241: used only for characters with higher values. Even when Unicode property
! 242: support is available, PCRE supports case-insensitive matching only when
! 243: there is a one-to-one mapping between a letter's cases. There are a
! 244: small number of many-to-one mappings in Unicode; these are not sup-
! 245: ported by PCRE.
! 246:
! 247:
! 248: AUTHOR
! 249:
! 250: Philip Hazel
! 251: University Computing Service
! 252: Cambridge CB2 3QH, England.
! 253:
! 254: Putting an actual email address here seems to have been a spam magnet,
! 255: so I've taken it away. If you want to email me, use my two initials,
! 256: followed by the two digits 10, at the domain cam.ac.uk.
! 257:
! 258:
! 259: REVISION
! 260:
! 261: Last updated: 12 April 2008
! 262: Copyright (c) 1997-2008 University of Cambridge.
! 263: ------------------------------------------------------------------------------
! 264:
! 265:
! 266: PCREBUILD(3) PCREBUILD(3)
! 267:
! 268:
! 269: NAME
! 270: PCRE - Perl-compatible regular expressions
! 271:
! 272:
! 273: PCRE BUILD-TIME OPTIONS
! 274:
! 275: This document describes the optional features of PCRE that can be
! 276: selected when the library is compiled. It assumes use of the configure
! 277: script, where the optional features are selected or deselected by pro-
! 278: viding options to configure before running the make command. However,
! 279: the same options can be selected in both Unix-like and non-Unix-like
! 280: environments using the GUI facility of CMakeSetup if you are using
! 281: CMake instead of configure to build PCRE.
! 282:
! 283: The complete list of options for configure (which includes the standard
! 284: ones such as the selection of the installation directory) can be
! 285: obtained by running
! 286:
! 287: ./configure --help
! 288:
! 289: The following sections include descriptions of options whose names
! 290: begin with --enable or --disable. These settings specify changes to the
! 291: defaults for the configure command. Because of the way that configure
! 292: works, --enable and --disable always come in pairs, so the complemen-
! 293: tary option always exists as well, but as it specifies the default, it
! 294: is not described.
! 295:
! 296:
! 297: C++ SUPPORT
! 298:
! 299: By default, the configure script will search for a C++ compiler and C++
! 300: header files. If it finds them, it automatically builds the C++ wrapper
! 301: library for PCRE. You can disable this by adding
! 302:
! 303: --disable-cpp
! 304:
! 305: to the configure command.
! 306:
! 307:
! 308: UTF-8 SUPPORT
! 309:
! 310: To build PCRE with support for UTF-8 character strings, add
! 311:
! 312: --enable-utf8
! 313:
! 314: to the configure command. Of itself, this does not make PCRE treat
! 315: strings as UTF-8. As well as compiling PCRE with this option, you also
! 316: have have to set the PCRE_UTF8 option when you call the pcre_compile()
! 317: function.
! 318:
! 319:
! 320: UNICODE CHARACTER PROPERTY SUPPORT
! 321:
! 322: UTF-8 support allows PCRE to process character values greater than 255
! 323: in the strings that it handles. On its own, however, it does not pro-
! 324: vide any facilities for accessing the properties of such characters. If
! 325: you want to be able to use the pattern escapes \P, \p, and \X, which
! 326: refer to Unicode character properties, you must add
! 327:
! 328: --enable-unicode-properties
! 329:
! 330: to the configure command. This implies UTF-8 support, even if you have
! 331: not explicitly requested it.
! 332:
! 333: Including Unicode property support adds around 30K of tables to the
! 334: PCRE library. Only the general category properties such as Lu and Nd
! 335: are supported. Details are given in the pcrepattern documentation.
! 336:
! 337:
! 338: CODE VALUE OF NEWLINE
! 339:
! 340: By default, PCRE interprets character 10 (linefeed, LF) as indicating
! 341: the end of a line. This is the normal newline character on Unix-like
! 342: systems. You can compile PCRE to use character 13 (carriage return, CR)
! 343: instead, by adding
! 344:
! 345: --enable-newline-is-cr
! 346:
! 347: to the configure command. There is also a --enable-newline-is-lf
! 348: option, which explicitly specifies linefeed as the newline character.
! 349:
! 350: Alternatively, you can specify that line endings are to be indicated by
! 351: the two character sequence CRLF. If you want this, add
! 352:
! 353: --enable-newline-is-crlf
! 354:
! 355: to the configure command. There is a fourth option, specified by
! 356:
! 357: --enable-newline-is-anycrlf
! 358:
! 359: which causes PCRE to recognize any of the three sequences CR, LF, or
! 360: CRLF as indicating a line ending. Finally, a fifth option, specified by
! 361:
! 362: --enable-newline-is-any
! 363:
! 364: causes PCRE to recognize any Unicode newline sequence.
! 365:
! 366: Whatever line ending convention is selected when PCRE is built can be
! 367: overridden when the library functions are called. At build time it is
! 368: conventional to use the standard for your operating system.
! 369:
! 370:
! 371: WHAT \R MATCHES
! 372:
! 373: By default, the sequence \R in a pattern matches any Unicode newline
! 374: sequence, whatever has been selected as the line ending sequence. If
! 375: you specify
! 376:
! 377: --enable-bsr-anycrlf
! 378:
! 379: the default is changed so that \R matches only CR, LF, or CRLF. What-
! 380: ever is selected when PCRE is built can be overridden when the library
! 381: functions are called.
! 382:
! 383:
! 384: BUILDING SHARED AND STATIC LIBRARIES
! 385:
! 386: The PCRE building process uses libtool to build both shared and static
! 387: Unix libraries by default. You can suppress one of these by adding one
! 388: of
! 389:
! 390: --disable-shared
! 391: --disable-static
! 392:
! 393: to the configure command, as required.
! 394:
! 395:
! 396: POSIX MALLOC USAGE
! 397:
! 398: When PCRE is called through the POSIX interface (see the pcreposix doc-
! 399: umentation), additional working storage is required for holding the
! 400: pointers to capturing substrings, because PCRE requires three integers
! 401: per substring, whereas the POSIX interface provides only two. If the
! 402: number of expected substrings is small, the wrapper function uses space
! 403: on the stack, because this is faster than using malloc() for each call.
! 404: The default threshold above which the stack is no longer used is 10; it
! 405: can be changed by adding a setting such as
! 406:
! 407: --with-posix-malloc-threshold=20
! 408:
! 409: to the configure command.
! 410:
! 411:
! 412: HANDLING VERY LARGE PATTERNS
! 413:
! 414: Within a compiled pattern, offset values are used to point from one
! 415: part to another (for example, from an opening parenthesis to an alter-
! 416: nation metacharacter). By default, two-byte values are used for these
! 417: offsets, leading to a maximum size for a compiled pattern of around
! 418: 64K. This is sufficient to handle all but the most gigantic patterns.
! 419: Nevertheless, some people do want to process enormous patterns, so it
! 420: is possible to compile PCRE to use three-byte or four-byte offsets by
! 421: adding a setting such as
! 422:
! 423: --with-link-size=3
! 424:
! 425: to the configure command. The value given must be 2, 3, or 4. Using
! 426: longer offsets slows down the operation of PCRE because it has to load
! 427: additional bytes when handling them.
! 428:
! 429:
! 430: AVOIDING EXCESSIVE STACK USAGE
! 431:
! 432: When matching with the pcre_exec() function, PCRE implements backtrack-
! 433: ing by making recursive calls to an internal function called match().
! 434: In environments where the size of the stack is limited, this can se-
! 435: verely limit PCRE's operation. (The Unix environment does not usually
! 436: suffer from this problem, but it may sometimes be necessary to increase
! 437: the maximum stack size. There is a discussion in the pcrestack docu-
! 438: mentation.) An alternative approach to recursion that uses memory from
! 439: the heap to remember data, instead of using recursive function calls,
! 440: has been implemented to work round the problem of limited stack size.
! 441: If you want to build a version of PCRE that works this way, add
! 442:
! 443: --disable-stack-for-recursion
! 444:
! 445: to the configure command. With this configuration, PCRE will use the
! 446: pcre_stack_malloc and pcre_stack_free variables to call memory manage-
! 447: ment functions. By default these point to malloc() and free(), but you
! 448: can replace the pointers so that your own functions are used.
! 449:
! 450: Separate functions are provided rather than using pcre_malloc and
! 451: pcre_free because the usage is very predictable: the block sizes
! 452: requested are always the same, and the blocks are always freed in
! 453: reverse order. A calling program might be able to implement optimized
! 454: functions that perform better than malloc() and free(). PCRE runs
! 455: noticeably more slowly when built in this way. This option affects only
! 456: the pcre_exec() function; it is not relevant for the the
! 457: pcre_dfa_exec() function.
! 458:
! 459:
! 460: LIMITING PCRE RESOURCE USAGE
! 461:
! 462: Internally, PCRE has a function called match(), which it calls repeat-
! 463: edly (sometimes recursively) when matching a pattern with the
! 464: pcre_exec() function. By controlling the maximum number of times this
! 465: function may be called during a single matching operation, a limit can
! 466: be placed on the resources used by a single call to pcre_exec(). The
! 467: limit can be changed at run time, as described in the pcreapi documen-
! 468: tation. The default is 10 million, but this can be changed by adding a
! 469: setting such as
! 470:
! 471: --with-match-limit=500000
! 472:
! 473: to the configure command. This setting has no effect on the
! 474: pcre_dfa_exec() matching function.
! 475:
! 476: In some environments it is desirable to limit the depth of recursive
! 477: calls of match() more strictly than the total number of calls, in order
! 478: to restrict the maximum amount of stack (or heap, if --disable-stack-
! 479: for-recursion is specified) that is used. A second limit controls this;
! 480: it defaults to the value that is set for --with-match-limit, which
! 481: imposes no additional constraints. However, you can set a lower limit
! 482: by adding, for example,
! 483:
! 484: --with-match-limit-recursion=10000
! 485:
! 486: to the configure command. This value can also be overridden at run
! 487: time.
! 488:
! 489:
! 490: CREATING CHARACTER TABLES AT BUILD TIME
! 491:
! 492: PCRE uses fixed tables for processing characters whose code values are
! 493: less than 256. By default, PCRE is built with a set of tables that are
! 494: distributed in the file pcre_chartables.c.dist. These tables are for
! 495: ASCII codes only. If you add
! 496:
! 497: --enable-rebuild-chartables
! 498:
! 499: to the configure command, the distributed tables are no longer used.
! 500: Instead, a program called dftables is compiled and run. This outputs
! 501: the source for new set of tables, created in the default locale of your
! 502: C runtime system. (This method of replacing the tables does not work if
! 503: you are cross compiling, because dftables is run on the local host. If
! 504: you need to create alternative tables when cross compiling, you will
! 505: have to do so "by hand".)
! 506:
! 507:
! 508: USING EBCDIC CODE
! 509:
! 510: PCRE assumes by default that it will run in an environment where the
! 511: character code is ASCII (or Unicode, which is a superset of ASCII).
! 512: This is the case for most computer operating systems. PCRE can, how-
! 513: ever, be compiled to run in an EBCDIC environment by adding
! 514:
! 515: --enable-ebcdic
! 516:
! 517: to the configure command. This setting implies --enable-rebuild-charta-
! 518: bles. You should only use it if you know that you are in an EBCDIC
! 519: environment (for example, an IBM mainframe operating system).
! 520:
! 521:
! 522: PCREGREP OPTIONS FOR COMPRESSED FILE SUPPORT
! 523:
! 524: By default, pcregrep reads all files as plain text. You can build it so
! 525: that it recognizes files whose names end in .gz or .bz2, and reads them
! 526: with libz or libbz2, respectively, by adding one or both of
! 527:
! 528: --enable-pcregrep-libz
! 529: --enable-pcregrep-libbz2
! 530:
! 531: to the configure command. These options naturally require that the rel-
! 532: evant libraries are installed on your system. Configuration will fail
! 533: if they are not.
! 534:
! 535:
! 536: PCRETEST OPTION FOR LIBREADLINE SUPPORT
! 537:
! 538: If you add
! 539:
! 540: --enable-pcretest-libreadline
! 541:
! 542: to the configure command, pcretest is linked with the libreadline
! 543: library, and when its input is from a terminal, it reads it using the
! 544: readline() function. This provides line-editing and history facilities.
! 545: Note that libreadline is GPL-licenced, so if you distribute a binary of
! 546: pcretest linked in this way, there may be licensing issues.
! 547:
! 548: Setting this option causes the -lreadline option to be added to the
! 549: pcretest build. In many operating environments with a sytem-installed
! 550: libreadline this is sufficient. However, in some environments (e.g. if
! 551: an unmodified distribution version of readline is in use), some extra
! 552: configuration may be necessary. The INSTALL file for libreadline says
! 553: this:
! 554:
! 555: "Readline uses the termcap functions, but does not link with the
! 556: termcap or curses library itself, allowing applications which link
! 557: with readline the to choose an appropriate library."
! 558:
! 559: If your environment has not been set up so that an appropriate library
! 560: is automatically included, you may need to add something like
! 561:
! 562: LIBS="-ncurses"
! 563:
! 564: immediately before the configure command.
! 565:
! 566:
! 567: SEE ALSO
! 568:
! 569: pcreapi(3), pcre_config(3).
! 570:
! 571:
! 572: AUTHOR
! 573:
! 574: Philip Hazel
! 575: University Computing Service
! 576: Cambridge CB2 3QH, England.
! 577:
! 578:
! 579: REVISION
! 580:
! 581: Last updated: 13 April 2008
! 582: Copyright (c) 1997-2008 University of Cambridge.
! 583: ------------------------------------------------------------------------------
! 584:
! 585:
! 586: PCREMATCHING(3) PCREMATCHING(3)
! 587:
! 588:
! 589: NAME
! 590: PCRE - Perl-compatible regular expressions
! 591:
! 592:
! 593: PCRE MATCHING ALGORITHMS
! 594:
! 595: This document describes the two different algorithms that are available
! 596: in PCRE for matching a compiled regular expression against a given sub-
! 597: ject string. The "standard" algorithm is the one provided by the
! 598: pcre_exec() function. This works in the same was as Perl's matching
! 599: function, and provides a Perl-compatible matching operation.
! 600:
! 601: An alternative algorithm is provided by the pcre_dfa_exec() function;
! 602: this operates in a different way, and is not Perl-compatible. It has
! 603: advantages and disadvantages compared with the standard algorithm, and
! 604: these are described below.
! 605:
! 606: When there is only one possible way in which a given subject string can
! 607: match a pattern, the two algorithms give the same answer. A difference
! 608: arises, however, when there are multiple possibilities. For example, if
! 609: the pattern
! 610:
! 611: ^<.*>
! 612:
! 613: is matched against the string
! 614:
! 615: <something> <something else> <something further>
! 616:
! 617: there are three possible answers. The standard algorithm finds only one
! 618: of them, whereas the alternative algorithm finds all three.
! 619:
! 620:
! 621: REGULAR EXPRESSIONS AS TREES
! 622:
! 623: The set of strings that are matched by a regular expression can be rep-
! 624: resented as a tree structure. An unlimited repetition in the pattern
! 625: makes the tree of infinite size, but it is still a tree. Matching the
! 626: pattern to a given subject string (from a given starting point) can be
! 627: thought of as a search of the tree. There are two ways to search a
! 628: tree: depth-first and breadth-first, and these correspond to the two
! 629: matching algorithms provided by PCRE.
! 630:
! 631:
! 632: THE STANDARD MATCHING ALGORITHM
! 633:
! 634: In the terminology of Jeffrey Friedl's book "Mastering Regular Expres-
! 635: sions", the standard algorithm is an "NFA algorithm". It conducts a
! 636: depth-first search of the pattern tree. That is, it proceeds along a
! 637: single path through the tree, checking that the subject matches what is
! 638: required. When there is a mismatch, the algorithm tries any alterna-
! 639: tives at the current point, and if they all fail, it backs up to the
! 640: previous branch point in the tree, and tries the next alternative
! 641: branch at that level. This often involves backing up (moving to the
! 642: left) in the subject string as well. The order in which repetition
! 643: branches are tried is controlled by the greedy or ungreedy nature of
! 644: the quantifier.
! 645:
! 646: If a leaf node is reached, a matching string has been found, and at
! 647: that point the algorithm stops. Thus, if there is more than one possi-
! 648: ble match, this algorithm returns the first one that it finds. Whether
! 649: this is the shortest, the longest, or some intermediate length depends
! 650: on the way the greedy and ungreedy repetition quantifiers are specified
! 651: in the pattern.
! 652:
! 653: Because it ends up with a single path through the tree, it is rela-
! 654: tively straightforward for this algorithm to keep track of the sub-
! 655: strings that are matched by portions of the pattern in parentheses.
! 656: This provides support for capturing parentheses and back references.
! 657:
! 658:
! 659: THE ALTERNATIVE MATCHING ALGORITHM
! 660:
! 661: This algorithm conducts a breadth-first search of the tree. Starting
! 662: from the first matching point in the subject, it scans the subject
! 663: string from left to right, once, character by character, and as it does
! 664: this, it remembers all the paths through the tree that represent valid
! 665: matches. In Friedl's terminology, this is a kind of "DFA algorithm",
! 666: though it is not implemented as a traditional finite state machine (it
! 667: keeps multiple states active simultaneously).
! 668:
! 669: The scan continues until either the end of the subject is reached, or
! 670: there are no more unterminated paths. At this point, terminated paths
! 671: represent the different matching possibilities (if there are none, the
! 672: match has failed). Thus, if there is more than one possible match,
! 673: this algorithm finds all of them, and in particular, it finds the long-
! 674: est. In PCRE, there is an option to stop the algorithm after the first
! 675: match (which is necessarily the shortest) has been found.
! 676:
! 677: Note that all the matches that are found start at the same point in the
! 678: subject. If the pattern
! 679:
! 680: cat(er(pillar)?)
! 681:
! 682: is matched against the string "the caterpillar catchment", the result
! 683: will be the three strings "cat", "cater", and "caterpillar" that start
! 684: at the fourth character of the subject. The algorithm does not automat-
! 685: ically move on to find matches that start at later positions.
! 686:
! 687: There are a number of features of PCRE regular expressions that are not
! 688: supported by the alternative matching algorithm. They are as follows:
! 689:
! 690: 1. Because the algorithm finds all possible matches, the greedy or
! 691: ungreedy nature of repetition quantifiers is not relevant. Greedy and
! 692: ungreedy quantifiers are treated in exactly the same way. However, pos-
! 693: sessive quantifiers can make a difference when what follows could also
! 694: match what is quantified, for example in a pattern like this:
! 695:
! 696: ^a++\w!
! 697:
! 698: This pattern matches "aaab!" but not "aaa!", which would be matched by
! 699: a non-possessive quantifier. Similarly, if an atomic group is present,
! 700: it is matched as if it were a standalone pattern at the current point,
! 701: and the longest match is then "locked in" for the rest of the overall
! 702: pattern.
! 703:
! 704: 2. When dealing with multiple paths through the tree simultaneously, it
! 705: is not straightforward to keep track of captured substrings for the
! 706: different matching possibilities, and PCRE's implementation of this
! 707: algorithm does not attempt to do this. This means that no captured sub-
! 708: strings are available.
! 709:
! 710: 3. Because no substrings are captured, back references within the pat-
! 711: tern are not supported, and cause errors if encountered.
! 712:
! 713: 4. For the same reason, conditional expressions that use a backrefer-
! 714: ence as the condition or test for a specific group recursion are not
! 715: supported.
! 716:
! 717: 5. Because many paths through the tree may be active, the \K escape
! 718: sequence, which resets the start of the match when encountered (but may
! 719: be on some paths and not on others), is not supported. It causes an
! 720: error if encountered.
! 721:
! 722: 6. Callouts are supported, but the value of the capture_top field is
! 723: always 1, and the value of the capture_last field is always -1.
! 724:
! 725: 7. The \C escape sequence, which (in the standard algorithm) matches a
! 726: single byte, even in UTF-8 mode, is not supported because the alterna-
! 727: tive algorithm moves through the subject string one character at a
! 728: time, for all active paths through the tree.
! 729:
! 730: 8. Except for (*FAIL), the backtracking control verbs such as (*PRUNE)
! 731: are not supported. (*FAIL) is supported, and behaves like a failing
! 732: negative assertion.
! 733:
! 734:
! 735: ADVANTAGES OF THE ALTERNATIVE ALGORITHM
! 736:
! 737: Using the alternative matching algorithm provides the following advan-
! 738: tages:
! 739:
! 740: 1. All possible matches (at a single point in the subject) are automat-
! 741: ically found, and in particular, the longest match is found. To find
! 742: more than one match using the standard algorithm, you have to do kludgy
! 743: things with callouts.
! 744:
! 745: 2. There is much better support for partial matching. The restrictions
! 746: on the content of the pattern that apply when using the standard algo-
! 747: rithm for partial matching do not apply to the alternative algorithm.
! 748: For non-anchored patterns, the starting position of a partial match is
! 749: available.
! 750:
! 751: 3. Because the alternative algorithm scans the subject string just
! 752: once, and never needs to backtrack, it is possible to pass very long
! 753: subject strings to the matching function in several pieces, checking
! 754: for partial matching each time.
! 755:
! 756:
! 757: DISADVANTAGES OF THE ALTERNATIVE ALGORITHM
! 758:
! 759: The alternative algorithm suffers from a number of disadvantages:
! 760:
! 761: 1. It is substantially slower than the standard algorithm. This is
! 762: partly because it has to search for all possible matches, but is also
! 763: because it is less susceptible to optimization.
! 764:
! 765: 2. Capturing parentheses and back references are not supported.
! 766:
! 767: 3. Although atomic groups are supported, their use does not provide the
! 768: performance advantage that it does for the standard algorithm.
! 769:
! 770:
! 771: AUTHOR
! 772:
! 773: Philip Hazel
! 774: University Computing Service
! 775: Cambridge CB2 3QH, England.
! 776:
! 777:
! 778: REVISION
! 779:
! 780: Last updated: 19 April 2008
! 781: Copyright (c) 1997-2008 University of Cambridge.
! 782: ------------------------------------------------------------------------------
! 783:
! 784:
! 785: PCREAPI(3) PCREAPI(3)
! 786:
! 787:
! 788: NAME
! 789: PCRE - Perl-compatible regular expressions
! 790:
! 791:
! 792: PCRE NATIVE API
! 793:
! 794: #include <pcre.h>
! 795:
! 796: pcre *pcre_compile(const char *pattern, int options,
! 797: const char **errptr, int *erroffset,
! 798: const unsigned char *tableptr);
! 799:
! 800: pcre *pcre_compile2(const char *pattern, int options,
! 801: int *errorcodeptr,
! 802: const char **errptr, int *erroffset,
! 803: const unsigned char *tableptr);
! 804:
! 805: pcre_extra *pcre_study(const pcre *code, int options,
! 806: const char **errptr);
! 807:
! 808: int pcre_exec(const pcre *code, const pcre_extra *extra,
! 809: const char *subject, int length, int startoffset,
! 810: int options, int *ovector, int ovecsize);
! 811:
! 812: int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
! 813: const char *subject, int length, int startoffset,
! 814: int options, int *ovector, int ovecsize,
! 815: int *workspace, int wscount);
! 816:
! 817: int pcre_copy_named_substring(const pcre *code,
! 818: const char *subject, int *ovector,
! 819: int stringcount, const char *stringname,
! 820: char *buffer, int buffersize);
! 821:
! 822: int pcre_copy_substring(const char *subject, int *ovector,
! 823: int stringcount, int stringnumber, char *buffer,
! 824: int buffersize);
! 825:
! 826: int pcre_get_named_substring(const pcre *code,
! 827: const char *subject, int *ovector,
! 828: int stringcount, const char *stringname,
! 829: const char **stringptr);
! 830:
! 831: int pcre_get_stringnumber(const pcre *code,
! 832: const char *name);
! 833:
! 834: int pcre_get_stringtable_entries(const pcre *code,
! 835: const char *name, char **first, char **last);
! 836:
! 837: int pcre_get_substring(const char *subject, int *ovector,
! 838: int stringcount, int stringnumber,
! 839: const char **stringptr);
! 840:
! 841: int pcre_get_substring_list(const char *subject,
! 842: int *ovector, int stringcount, const char ***listptr);
! 843:
! 844: void pcre_free_substring(const char *stringptr);
! 845:
! 846: void pcre_free_substring_list(const char **stringptr);
! 847:
! 848: const unsigned char *pcre_maketables(void);
! 849:
! 850: int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
! 851: int what, void *where);
! 852:
! 853: int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
! 854:
! 855: int pcre_refcount(pcre *code, int adjust);
! 856:
! 857: int pcre_config(int what, void *where);
! 858:
! 859: char *pcre_version(void);
! 860:
! 861: void *(*pcre_malloc)(size_t);
! 862:
! 863: void (*pcre_free)(void *);
! 864:
! 865: void *(*pcre_stack_malloc)(size_t);
! 866:
! 867: void (*pcre_stack_free)(void *);
! 868:
! 869: int (*pcre_callout)(pcre_callout_block *);
! 870:
! 871:
! 872: PCRE API OVERVIEW
! 873:
! 874: PCRE has its own native API, which is described in this document. There
! 875: are also some wrapper functions that correspond to the POSIX regular
! 876: expression API. These are described in the pcreposix documentation.
! 877: Both of these APIs define a set of C function calls. A C++ wrapper is
! 878: distributed with PCRE. It is documented in the pcrecpp page.
! 879:
! 880: The native API C function prototypes are defined in the header file
! 881: pcre.h, and on Unix systems the library itself is called libpcre. It
! 882: can normally be accessed by adding -lpcre to the command for linking an
! 883: application that uses PCRE. The header file defines the macros
! 884: PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
! 885: bers for the library. Applications can use these to include support
! 886: for different releases of PCRE.
! 887:
! 888: The functions pcre_compile(), pcre_compile2(), pcre_study(), and
! 889: pcre_exec() are used for compiling and matching regular expressions in
! 890: a Perl-compatible manner. A sample program that demonstrates the sim-
! 891: plest way of using them is provided in the file called pcredemo.c in
! 892: the source distribution. The pcresample documentation describes how to
! 893: compile and run it.
! 894:
! 895: A second matching function, pcre_dfa_exec(), which is not Perl-compati-
! 896: ble, is also provided. This uses a different algorithm for the match-
! 897: ing. The alternative algorithm finds all possible matches (at a given
! 898: point in the subject), and scans the subject just once. However, this
! 899: algorithm does not return captured substrings. A description of the two
! 900: matching algorithms and their advantages and disadvantages is given in
! 901: the pcrematching documentation.
! 902:
! 903: In addition to the main compiling and matching functions, there are
! 904: convenience functions for extracting captured substrings from a subject
! 905: string that is matched by pcre_exec(). They are:
! 906:
! 907: pcre_copy_substring()
! 908: pcre_copy_named_substring()
! 909: pcre_get_substring()
! 910: pcre_get_named_substring()
! 911: pcre_get_substring_list()
! 912: pcre_get_stringnumber()
! 913: pcre_get_stringtable_entries()
! 914:
! 915: pcre_free_substring() and pcre_free_substring_list() are also provided,
! 916: to free the memory used for extracted strings.
! 917:
! 918: The function pcre_maketables() is used to build a set of character
! 919: tables in the current locale for passing to pcre_compile(),
! 920: pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
! 921: provided for specialist use. Most commonly, no special tables are
! 922: passed, in which case internal tables that are generated when PCRE is
! 923: built are used.
! 924:
! 925: The function pcre_fullinfo() is used to find out information about a
! 926: compiled pattern; pcre_info() is an obsolete version that returns only
! 927: some of the available information, but is retained for backwards com-
! 928: patibility. The function pcre_version() returns a pointer to a string
! 929: containing the version of PCRE and its date of release.
! 930:
! 931: The function pcre_refcount() maintains a reference count in a data
! 932: block containing a compiled pattern. This is provided for the benefit
! 933: of object-oriented applications.
! 934:
! 935: The global variables pcre_malloc and pcre_free initially contain the
! 936: entry points of the standard malloc() and free() functions, respec-
! 937: tively. PCRE calls the memory management functions via these variables,
! 938: so a calling program can replace them if it wishes to intercept the
! 939: calls. This should be done before calling any PCRE functions.
! 940:
! 941: The global variables pcre_stack_malloc and pcre_stack_free are also
! 942: indirections to memory management functions. These special functions
! 943: are used only when PCRE is compiled to use the heap for remembering
! 944: data, instead of recursive function calls, when running the pcre_exec()
! 945: function. See the pcrebuild documentation for details of how to do
! 946: this. It is a non-standard way of building PCRE, for use in environ-
! 947: ments that have limited stacks. Because of the greater use of memory
! 948: management, it runs more slowly. Separate functions are provided so
! 949: that special-purpose external code can be used for this case. When
! 950: used, these functions are always called in a stack-like manner (last
! 951: obtained, first freed), and always for memory blocks of the same size.
! 952: There is a discussion about PCRE's stack usage in the pcrestack docu-
! 953: mentation.
! 954:
! 955: The global variable pcre_callout initially contains NULL. It can be set
! 956: by the caller to a "callout" function, which PCRE will then call at
! 957: specified points during a matching operation. Details are given in the
! 958: pcrecallout documentation.
! 959:
! 960:
! 961: NEWLINES
! 962:
! 963: PCRE supports five different conventions for indicating line breaks in
! 964: strings: a single CR (carriage return) character, a single LF (line-
! 965: feed) character, the two-character sequence CRLF, any of the three pre-
! 966: ceding, or any Unicode newline sequence. The Unicode newline sequences
! 967: are the three just mentioned, plus the single characters VT (vertical
! 968: tab, U+000B), FF (formfeed, U+000C), NEL (next line, U+0085), LS (line
! 969: separator, U+2028), and PS (paragraph separator, U+2029).
! 970:
! 971: Each of the first three conventions is used by at least one operating
! 972: system as its standard newline sequence. When PCRE is built, a default
! 973: can be specified. The default default is LF, which is the Unix stan-
! 974: dard. When PCRE is run, the default can be overridden, either when a
! 975: pattern is compiled, or when it is matched.
! 976:
! 977: At compile time, the newline convention can be specified by the options
! 978: argument of pcre_compile(), or it can be specified by special text at
! 979: the start of the pattern itself; this overrides any other settings. See
! 980: the pcrepattern page for details of the special character sequences.
! 981:
! 982: In the PCRE documentation the word "newline" is used to mean "the char-
! 983: acter or pair of characters that indicate a line break". The choice of
! 984: newline convention affects the handling of the dot, circumflex, and
! 985: dollar metacharacters, the handling of #-comments in /x mode, and, when
! 986: CRLF is a recognized line ending sequence, the match position advance-
! 987: ment for a non-anchored pattern. There is more detail about this in the
! 988: section on pcre_exec() options below.
! 989:
! 990: The choice of newline convention does not affect the interpretation of
! 991: the \n or \r escape sequences, nor does it affect what \R matches,
! 992: which is controlled in a similar way, but by separate options.
! 993:
! 994:
! 995: MULTITHREADING
! 996:
! 997: The PCRE functions can be used in multi-threading applications, with
! 998: the proviso that the memory management functions pointed to by
! 999: pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
! 1000: callout function pointed to by pcre_callout, are shared by all threads.
! 1001:
! 1002: The compiled form of a regular expression is not altered during match-
! 1003: ing, so the same compiled pattern can safely be used by several threads
! 1004: at once.
! 1005:
! 1006:
! 1007: SAVING PRECOMPILED PATTERNS FOR LATER USE
! 1008:
! 1009: The compiled form of a regular expression can be saved and re-used at a
! 1010: later time, possibly by a different program, and even on a host other
! 1011: than the one on which it was compiled. Details are given in the
! 1012: pcreprecompile documentation. However, compiling a regular expression
! 1013: with one version of PCRE for use with a different version is not guar-
! 1014: anteed to work and may cause crashes.
! 1015:
! 1016:
! 1017: CHECKING BUILD-TIME OPTIONS
! 1018:
! 1019: int pcre_config(int what, void *where);
! 1020:
! 1021: The function pcre_config() makes it possible for a PCRE client to dis-
! 1022: cover which optional features have been compiled into the PCRE library.
! 1023: The pcrebuild documentation has more details about these optional fea-
! 1024: tures.
! 1025:
! 1026: The first argument for pcre_config() is an integer, specifying which
! 1027: information is required; the second argument is a pointer to a variable
! 1028: into which the information is placed. The following information is
! 1029: available:
! 1030:
! 1031: PCRE_CONFIG_UTF8
! 1032:
! 1033: The output is an integer that is set to one if UTF-8 support is avail-
! 1034: able; otherwise it is set to zero.
! 1035:
! 1036: PCRE_CONFIG_UNICODE_PROPERTIES
! 1037:
! 1038: The output is an integer that is set to one if support for Unicode
! 1039: character properties is available; otherwise it is set to zero.
! 1040:
! 1041: PCRE_CONFIG_NEWLINE
! 1042:
! 1043: The output is an integer whose value specifies the default character
! 1044: sequence that is recognized as meaning "newline". The four values that
! 1045: are supported are: 10 for LF, 13 for CR, 3338 for CRLF, -2 for ANYCRLF,
! 1046: and -1 for ANY. The default should normally be the standard sequence
! 1047: for your operating system.
! 1048:
! 1049: PCRE_CONFIG_BSR
! 1050:
! 1051: The output is an integer whose value indicates what character sequences
! 1052: the \R escape sequence matches by default. A value of 0 means that \R
! 1053: matches any Unicode line ending sequence; a value of 1 means that \R
! 1054: matches only CR, LF, or CRLF. The default can be overridden when a pat-
! 1055: tern is compiled or matched.
! 1056:
! 1057: PCRE_CONFIG_LINK_SIZE
! 1058:
! 1059: The output is an integer that contains the number of bytes used for
! 1060: internal linkage in compiled regular expressions. The value is 2, 3, or
! 1061: 4. Larger values allow larger regular expressions to be compiled, at
! 1062: the expense of slower matching. The default value of 2 is sufficient
! 1063: for all but the most massive patterns, since it allows the compiled
! 1064: pattern to be up to 64K in size.
! 1065:
! 1066: PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
! 1067:
! 1068: The output is an integer that contains the threshold above which the
! 1069: POSIX interface uses malloc() for output vectors. Further details are
! 1070: given in the pcreposix documentation.
! 1071:
! 1072: PCRE_CONFIG_MATCH_LIMIT
! 1073:
! 1074: The output is an integer that gives the default limit for the number of
! 1075: internal matching function calls in a pcre_exec() execution. Further
! 1076: details are given with pcre_exec() below.
! 1077:
! 1078: PCRE_CONFIG_MATCH_LIMIT_RECURSION
! 1079:
! 1080: The output is an integer that gives the default limit for the depth of
! 1081: recursion when calling the internal matching function in a pcre_exec()
! 1082: execution. Further details are given with pcre_exec() below.
! 1083:
! 1084: PCRE_CONFIG_STACKRECURSE
! 1085:
! 1086: The output is an integer that is set to one if internal recursion when
! 1087: running pcre_exec() is implemented by recursive function calls that use
! 1088: the stack to remember their state. This is the usual way that PCRE is
! 1089: compiled. The output is zero if PCRE was compiled to use blocks of data
! 1090: on the heap instead of recursive function calls. In this case,
! 1091: pcre_stack_malloc and pcre_stack_free are called to manage memory
! 1092: blocks on the heap, thus avoiding the use of the stack.
! 1093:
! 1094:
! 1095: COMPILING A PATTERN
! 1096:
! 1097: pcre *pcre_compile(const char *pattern, int options,
! 1098: const char **errptr, int *erroffset,
! 1099: const unsigned char *tableptr);
! 1100:
! 1101: pcre *pcre_compile2(const char *pattern, int options,
! 1102: int *errorcodeptr,
! 1103: const char **errptr, int *erroffset,
! 1104: const unsigned char *tableptr);
! 1105:
! 1106: Either of the functions pcre_compile() or pcre_compile2() can be called
! 1107: to compile a pattern into an internal form. The only difference between
! 1108: the two interfaces is that pcre_compile2() has an additional argument,
! 1109: errorcodeptr, via which a numerical error code can be returned.
! 1110:
! 1111: The pattern is a C string terminated by a binary zero, and is passed in
! 1112: the pattern argument. A pointer to a single block of memory that is
! 1113: obtained via pcre_malloc is returned. This contains the compiled code
! 1114: and related data. The pcre type is defined for the returned block; this
! 1115: is a typedef for a structure whose contents are not externally defined.
! 1116: It is up to the caller to free the memory (via pcre_free) when it is no
! 1117: longer required.
! 1118:
! 1119: Although the compiled code of a PCRE regex is relocatable, that is, it
! 1120: does not depend on memory location, the complete pcre data block is not
! 1121: fully relocatable, because it may contain a copy of the tableptr argu-
! 1122: ment, which is an address (see below).
! 1123:
! 1124: The options argument contains various bit settings that affect the com-
! 1125: pilation. It should be zero if no options are required. The available
! 1126: options are described below. Some of them, in particular, those that
! 1127: are compatible with Perl, can also be set and unset from within the
! 1128: pattern (see the detailed description in the pcrepattern documenta-
! 1129: tion). For these options, the contents of the options argument speci-
! 1130: fies their initial settings at the start of compilation and execution.
! 1131: The PCRE_ANCHORED and PCRE_NEWLINE_xxx options can be set at the time
! 1132: of matching as well as at compile time.
! 1133:
! 1134: If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
! 1135: if compilation of a pattern fails, pcre_compile() returns NULL, and
! 1136: sets the variable pointed to by errptr to point to a textual error mes-
! 1137: sage. This is a static string that is part of the library. You must not
! 1138: try to free it. The offset from the start of the pattern to the charac-
! 1139: ter where the error was discovered is placed in the variable pointed to
! 1140: by erroffset, which must not be NULL. If it is, an immediate error is
! 1141: given.
! 1142:
! 1143: If pcre_compile2() is used instead of pcre_compile(), and the error-
! 1144: codeptr argument is not NULL, a non-zero error code number is returned
! 1145: via this argument in the event of an error. This is in addition to the
! 1146: textual error message. Error codes and messages are listed below.
! 1147:
! 1148: If the final argument, tableptr, is NULL, PCRE uses a default set of
! 1149: character tables that are built when PCRE is compiled, using the
! 1150: default C locale. Otherwise, tableptr must be an address that is the
! 1151: result of a call to pcre_maketables(). This value is stored with the
! 1152: compiled pattern, and used again by pcre_exec(), unless another table
! 1153: pointer is passed to it. For more discussion, see the section on locale
! 1154: support below.
! 1155:
! 1156: This code fragment shows a typical straightforward call to pcre_com-
! 1157: pile():
! 1158:
! 1159: pcre *re;
! 1160: const char *error;
! 1161: int erroffset;
! 1162: re = pcre_compile(
! 1163: "^A.*Z", /* the pattern */
! 1164: 0, /* default options */
! 1165: &error, /* for error message */
! 1166: &erroffset, /* for error offset */
! 1167: NULL); /* use default character tables */
! 1168:
! 1169: The following names for option bits are defined in the pcre.h header
! 1170: file:
! 1171:
! 1172: PCRE_ANCHORED
! 1173:
! 1174: If this bit is set, the pattern is forced to be "anchored", that is, it
! 1175: is constrained to match only at the first matching point in the string
! 1176: that is being searched (the "subject string"). This effect can also be
! 1177: achieved by appropriate constructs in the pattern itself, which is the
! 1178: only way to do it in Perl.
! 1179:
! 1180: PCRE_AUTO_CALLOUT
! 1181:
! 1182: If this bit is set, pcre_compile() automatically inserts callout items,
! 1183: all with number 255, before each pattern item. For discussion of the
! 1184: callout facility, see the pcrecallout documentation.
! 1185:
! 1186: PCRE_BSR_ANYCRLF
! 1187: PCRE_BSR_UNICODE
! 1188:
! 1189: These options (which are mutually exclusive) control what the \R escape
! 1190: sequence matches. The choice is either to match only CR, LF, or CRLF,
! 1191: or to match any Unicode newline sequence. The default is specified when
! 1192: PCRE is built. It can be overridden from within the pattern, or by set-
! 1193: ting an option when a compiled pattern is matched.
! 1194:
! 1195: PCRE_CASELESS
! 1196:
! 1197: If this bit is set, letters in the pattern match both upper and lower
! 1198: case letters. It is equivalent to Perl's /i option, and it can be
! 1199: changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
! 1200: always understands the concept of case for characters whose values are
! 1201: less than 128, so caseless matching is always possible. For characters
! 1202: with higher values, the concept of case is supported if PCRE is com-
! 1203: piled with Unicode property support, but not otherwise. If you want to
! 1204: use caseless matching for characters 128 and above, you must ensure
! 1205: that PCRE is compiled with Unicode property support as well as with
! 1206: UTF-8 support.
! 1207:
! 1208: PCRE_DOLLAR_ENDONLY
! 1209:
! 1210: If this bit is set, a dollar metacharacter in the pattern matches only
! 1211: at the end of the subject string. Without this option, a dollar also
! 1212: matches immediately before a newline at the end of the string (but not
! 1213: before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored
! 1214: if PCRE_MULTILINE is set. There is no equivalent to this option in
! 1215: Perl, and no way to set it within a pattern.
! 1216:
! 1217: PCRE_DOTALL
! 1218:
! 1219: If this bit is set, a dot metacharater in the pattern matches all char-
! 1220: acters, including those that indicate newline. Without it, a dot does
! 1221: not match when the current position is at a newline. This option is
! 1222: equivalent to Perl's /s option, and it can be changed within a pattern
! 1223: by a (?s) option setting. A negative class such as [^a] always matches
! 1224: newline characters, independent of the setting of this option.
! 1225:
! 1226: PCRE_DUPNAMES
! 1227:
! 1228: If this bit is set, names used to identify capturing subpatterns need
! 1229: not be unique. This can be helpful for certain types of pattern when it
! 1230: is known that only one instance of the named subpattern can ever be
! 1231: matched. There are more details of named subpatterns below; see also
! 1232: the pcrepattern documentation.
! 1233:
! 1234: PCRE_EXTENDED
! 1235:
! 1236: If this bit is set, whitespace data characters in the pattern are
! 1237: totally ignored except when escaped or inside a character class. White-
! 1238: space does not include the VT character (code 11). In addition, charac-
! 1239: ters between an unescaped # outside a character class and the next new-
! 1240: line, inclusive, are also ignored. This is equivalent to Perl's /x
! 1241: option, and it can be changed within a pattern by a (?x) option set-
! 1242: ting.
! 1243:
! 1244: This option makes it possible to include comments inside complicated
! 1245: patterns. Note, however, that this applies only to data characters.
! 1246: Whitespace characters may never appear within special character
! 1247: sequences in a pattern, for example within the sequence (?( which
! 1248: introduces a conditional subpattern.
! 1249:
! 1250: PCRE_EXTRA
! 1251:
! 1252: This option was invented in order to turn on additional functionality
! 1253: of PCRE that is incompatible with Perl, but it is currently of very
! 1254: little use. When set, any backslash in a pattern that is followed by a
! 1255: letter that has no special meaning causes an error, thus reserving
! 1256: these combinations for future expansion. By default, as in Perl, a
! 1257: backslash followed by a letter with no special meaning is treated as a
! 1258: literal. (Perl can, however, be persuaded to give a warning for this.)
! 1259: There are at present no other features controlled by this option. It
! 1260: can also be set by a (?X) option setting within a pattern.
! 1261:
! 1262: PCRE_FIRSTLINE
! 1263:
! 1264: If this option is set, an unanchored pattern is required to match
! 1265: before or at the first newline in the subject string, though the
! 1266: matched text may continue over the newline.
! 1267:
! 1268: PCRE_JAVASCRIPT_COMPAT
! 1269:
! 1270: If this option is set, PCRE's behaviour is changed in some ways so that
! 1271: it is compatible with JavaScript rather than Perl. The changes are as
! 1272: follows:
! 1273:
! 1274: (1) A lone closing square bracket in a pattern causes a compile-time
! 1275: error, because this is illegal in JavaScript (by default it is treated
! 1276: as a data character). Thus, the pattern AB]CD becomes illegal when this
! 1277: option is set.
! 1278:
! 1279: (2) At run time, a back reference to an unset subpattern group matches
! 1280: an empty string (by default this causes the current matching alterna-
! 1281: tive to fail). A pattern such as (\1)(a) succeeds when this option is
! 1282: set (assuming it can find an "a" in the subject), whereas it fails by
! 1283: default, for Perl compatibility.
! 1284:
! 1285: PCRE_MULTILINE
! 1286:
! 1287: By default, PCRE treats the subject string as consisting of a single
! 1288: line of characters (even if it actually contains newlines). The "start
! 1289: of line" metacharacter (^) matches only at the start of the string,
! 1290: while the "end of line" metacharacter ($) matches only at the end of
! 1291: the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
! 1292: is set). This is the same as Perl.
! 1293:
! 1294: When PCRE_MULTILINE it is set, the "start of line" and "end of line"
! 1295: constructs match immediately following or immediately before internal
! 1296: newlines in the subject string, respectively, as well as at the very
! 1297: start and end. This is equivalent to Perl's /m option, and it can be
! 1298: changed within a pattern by a (?m) option setting. If there are no new-
! 1299: lines in a subject string, or no occurrences of ^ or $ in a pattern,
! 1300: setting PCRE_MULTILINE has no effect.
! 1301:
! 1302: PCRE_NEWLINE_CR
! 1303: PCRE_NEWLINE_LF
! 1304: PCRE_NEWLINE_CRLF
! 1305: PCRE_NEWLINE_ANYCRLF
! 1306: PCRE_NEWLINE_ANY
! 1307:
! 1308: These options override the default newline definition that was chosen
! 1309: when PCRE was built. Setting the first or the second specifies that a
! 1310: newline is indicated by a single character (CR or LF, respectively).
! 1311: Setting PCRE_NEWLINE_CRLF specifies that a newline is indicated by the
! 1312: two-character CRLF sequence. Setting PCRE_NEWLINE_ANYCRLF specifies
! 1313: that any of the three preceding sequences should be recognized. Setting
! 1314: PCRE_NEWLINE_ANY specifies that any Unicode newline sequence should be
! 1315: recognized. The Unicode newline sequences are the three just mentioned,
! 1316: plus the single characters VT (vertical tab, U+000B), FF (formfeed,
! 1317: U+000C), NEL (next line, U+0085), LS (line separator, U+2028), and PS
! 1318: (paragraph separator, U+2029). The last two are recognized only in
! 1319: UTF-8 mode.
! 1320:
! 1321: The newline setting in the options word uses three bits that are
! 1322: treated as a number, giving eight possibilities. Currently only six are
! 1323: used (default plus the five values above). This means that if you set
! 1324: more than one newline option, the combination may or may not be sensi-
! 1325: ble. For example, PCRE_NEWLINE_CR with PCRE_NEWLINE_LF is equivalent to
! 1326: PCRE_NEWLINE_CRLF, but other combinations may yield unused numbers and
! 1327: cause an error.
! 1328:
! 1329: The only time that a line break is specially recognized when compiling
! 1330: a pattern is if PCRE_EXTENDED is set, and an unescaped # outside a
! 1331: character class is encountered. This indicates a comment that lasts
! 1332: until after the next line break sequence. In other circumstances, line
! 1333: break sequences are treated as literal data, except that in
! 1334: PCRE_EXTENDED mode, both CR and LF are treated as whitespace characters
! 1335: and are therefore ignored.
! 1336:
! 1337: The newline option that is set at compile time becomes the default that
! 1338: is used for pcre_exec() and pcre_dfa_exec(), but it can be overridden.
! 1339:
! 1340: PCRE_NO_AUTO_CAPTURE
! 1341:
! 1342: If this option is set, it disables the use of numbered capturing paren-
! 1343: theses in the pattern. Any opening parenthesis that is not followed by
! 1344: ? behaves as if it were followed by ?: but named parentheses can still
! 1345: be used for capturing (and they acquire numbers in the usual way).
! 1346: There is no equivalent of this option in Perl.
! 1347:
! 1348: PCRE_UNGREEDY
! 1349:
! 1350: This option inverts the "greediness" of the quantifiers so that they
! 1351: are not greedy by default, but become greedy if followed by "?". It is
! 1352: not compatible with Perl. It can also be set by a (?U) option setting
! 1353: within the pattern.
! 1354:
! 1355: PCRE_UTF8
! 1356:
! 1357: This option causes PCRE to regard both the pattern and the subject as
! 1358: strings of UTF-8 characters instead of single-byte character strings.
! 1359: However, it is available only when PCRE is built to include UTF-8 sup-
! 1360: port. If not, the use of this option provokes an error. Details of how
! 1361: this option changes the behaviour of PCRE are given in the section on
! 1362: UTF-8 support in the main pcre page.
! 1363:
! 1364: PCRE_NO_UTF8_CHECK
! 1365:
! 1366: When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
! 1367: automatically checked. There is a discussion about the validity of
! 1368: UTF-8 strings in the main pcre page. If an invalid UTF-8 sequence of
! 1369: bytes is found, pcre_compile() returns an error. If you already know
! 1370: that your pattern is valid, and you want to skip this check for perfor-
! 1371: mance reasons, you can set the PCRE_NO_UTF8_CHECK option. When it is
! 1372: set, the effect of passing an invalid UTF-8 string as a pattern is
! 1373: undefined. It may cause your program to crash. Note that this option
! 1374: can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the
! 1375: UTF-8 validity checking of subject strings.
! 1376:
! 1377:
! 1378: COMPILATION ERROR CODES
! 1379:
! 1380: The following table lists the error codes than may be returned by
! 1381: pcre_compile2(), along with the error messages that may be returned by
! 1382: both compiling functions. As PCRE has developed, some error codes have
! 1383: fallen out of use. To avoid confusion, they have not been re-used.
! 1384:
! 1385: 0 no error
! 1386: 1 \ at end of pattern
! 1387: 2 \c at end of pattern
! 1388: 3 unrecognized character follows \
! 1389: 4 numbers out of order in {} quantifier
! 1390: 5 number too big in {} quantifier
! 1391: 6 missing terminating ] for character class
! 1392: 7 invalid escape sequence in character class
! 1393: 8 range out of order in character class
! 1394: 9 nothing to repeat
! 1395: 10 [this code is not in use]
! 1396: 11 internal error: unexpected repeat
! 1397: 12 unrecognized character after (? or (?-
! 1398: 13 POSIX named classes are supported only within a class
! 1399: 14 missing )
! 1400: 15 reference to non-existent subpattern
! 1401: 16 erroffset passed as NULL
! 1402: 17 unknown option bit(s) set
! 1403: 18 missing ) after comment
! 1404: 19 [this code is not in use]
! 1405: 20 regular expression is too large
! 1406: 21 failed to get memory
! 1407: 22 unmatched parentheses
! 1408: 23 internal error: code overflow
! 1409: 24 unrecognized character after (?<
! 1410: 25 lookbehind assertion is not fixed length
! 1411: 26 malformed number or name after (?(
! 1412: 27 conditional group contains more than two branches
! 1413: 28 assertion expected after (?(
! 1414: 29 (?R or (?[+-]digits must be followed by )
! 1415: 30 unknown POSIX class name
! 1416: 31 POSIX collating elements are not supported
! 1417: 32 this version of PCRE is not compiled with PCRE_UTF8 support
! 1418: 33 [this code is not in use]
! 1419: 34 character value in \x{...} sequence is too large
! 1420: 35 invalid condition (?(0)
! 1421: 36 \C not allowed in lookbehind assertion
! 1422: 37 PCRE does not support \L, \l, \N, \U, or \u
! 1423: 38 number after (?C is > 255
! 1424: 39 closing ) for (?C expected
! 1425: 40 recursive call could loop indefinitely
! 1426: 41 unrecognized character after (?P
! 1427: 42 syntax error in subpattern name (missing terminator)
! 1428: 43 two named subpatterns have the same name
! 1429: 44 invalid UTF-8 string
! 1430: 45 support for \P, \p, and \X has not been compiled
! 1431: 46 malformed \P or \p sequence
! 1432: 47 unknown property name after \P or \p
! 1433: 48 subpattern name is too long (maximum 32 characters)
! 1434: 49 too many named subpatterns (maximum 10000)
! 1435: 50 [this code is not in use]
! 1436: 51 octal value is greater than \377 (not in UTF-8 mode)
! 1437: 52 internal error: overran compiling workspace
! 1438: 53 internal error: previously-checked referenced subpattern not
! 1439: found
! 1440: 54 DEFINE group contains more than one branch
! 1441: 55 repeating a DEFINE group is not allowed
! 1442: 56 inconsistent NEWLINE options
! 1443: 57 \g is not followed by a braced, angle-bracketed, or quoted
! 1444: name/number or by a plain number
! 1445: 58 a numbered reference must not be zero
! 1446: 59 (*VERB) with an argument is not supported
! 1447: 60 (*VERB) not recognized
! 1448: 61 number is too big
! 1449: 62 subpattern name expected
! 1450: 63 digit expected after (?+
! 1451: 64 ] is an invalid data character in JavaScript compatibility mode
! 1452:
! 1453: The numbers 32 and 10000 in errors 48 and 49 are defaults; different
! 1454: values may be used if the limits were changed when PCRE was built.
! 1455:
! 1456:
! 1457: STUDYING A PATTERN
! 1458:
! 1459: pcre_extra *pcre_study(const pcre *code, int options
! 1460: const char **errptr);
! 1461:
! 1462: If a compiled pattern is going to be used several times, it is worth
! 1463: spending more time analyzing it in order to speed up the time taken for
! 1464: matching. The function pcre_study() takes a pointer to a compiled pat-
! 1465: tern as its first argument. If studying the pattern produces additional
! 1466: information that will help speed up matching, pcre_study() returns a
! 1467: pointer to a pcre_extra block, in which the study_data field points to
! 1468: the results of the study.
! 1469:
! 1470: The returned value from pcre_study() can be passed directly to
! 1471: pcre_exec(). However, a pcre_extra block also contains other fields
! 1472: that can be set by the caller before the block is passed; these are
! 1473: described below in the section on matching a pattern.
! 1474:
! 1475: If studying the pattern does not produce any additional information
! 1476: pcre_study() returns NULL. In that circumstance, if the calling program
! 1477: wants to pass any of the other fields to pcre_exec(), it must set up
! 1478: its own pcre_extra block.
! 1479:
! 1480: The second argument of pcre_study() contains option bits. At present,
! 1481: no options are defined, and this argument should always be zero.
! 1482:
! 1483: The third argument for pcre_study() is a pointer for an error message.
! 1484: If studying succeeds (even if no data is returned), the variable it
! 1485: points to is set to NULL. Otherwise it is set to point to a textual
! 1486: error message. This is a static string that is part of the library. You
! 1487: must not try to free it. You should test the error pointer for NULL
! 1488: after calling pcre_study(), to be sure that it has run successfully.
! 1489:
! 1490: This is a typical call to pcre_study():
! 1491:
! 1492: pcre_extra *pe;
! 1493: pe = pcre_study(
! 1494: re, /* result of pcre_compile() */
! 1495: 0, /* no options exist */
! 1496: &error); /* set to NULL or points to a message */
! 1497:
! 1498: At present, studying a pattern is useful only for non-anchored patterns
! 1499: that do not have a single fixed starting character. A bitmap of possi-
! 1500: ble starting bytes is created.
! 1501:
! 1502:
! 1503: LOCALE SUPPORT
! 1504:
! 1505: PCRE handles caseless matching, and determines whether characters are
! 1506: letters, digits, or whatever, by reference to a set of tables, indexed
! 1507: by character value. When running in UTF-8 mode, this applies only to
! 1508: characters with codes less than 128. Higher-valued codes never match
! 1509: escapes such as \w or \d, but can be tested with \p if PCRE is built
! 1510: with Unicode character property support. The use of locales with Uni-
! 1511: code is discouraged. If you are handling characters with codes greater
! 1512: than 128, you should either use UTF-8 and Unicode, or use locales, but
! 1513: not try to mix the two.
! 1514:
! 1515: PCRE contains an internal set of tables that are used when the final
! 1516: argument of pcre_compile() is NULL. These are sufficient for many
! 1517: applications. Normally, the internal tables recognize only ASCII char-
! 1518: acters. However, when PCRE is built, it is possible to cause the inter-
! 1519: nal tables to be rebuilt in the default "C" locale of the local system,
! 1520: which may cause them to be different.
! 1521:
! 1522: The internal tables can always be overridden by tables supplied by the
! 1523: application that calls PCRE. These may be created in a different locale
! 1524: from the default. As more and more applications change to using Uni-
! 1525: code, the need for this locale support is expected to die away.
! 1526:
! 1527: External tables are built by calling the pcre_maketables() function,
! 1528: which has no arguments, in the relevant locale. The result can then be
! 1529: passed to pcre_compile() or pcre_exec() as often as necessary. For
! 1530: example, to build and use tables that are appropriate for the French
! 1531: locale (where accented characters with values greater than 128 are
! 1532: treated as letters), the following code could be used:
! 1533:
! 1534: setlocale(LC_CTYPE, "fr_FR");
! 1535: tables = pcre_maketables();
! 1536: re = pcre_compile(..., tables);
! 1537:
! 1538: The locale name "fr_FR" is used on Linux and other Unix-like systems;
! 1539: if you are using Windows, the name for the French locale is "french".
! 1540:
! 1541: When pcre_maketables() runs, the tables are built in memory that is
! 1542: obtained via pcre_malloc. It is the caller's responsibility to ensure
! 1543: that the memory containing the tables remains available for as long as
! 1544: it is needed.
! 1545:
! 1546: The pointer that is passed to pcre_compile() is saved with the compiled
! 1547: pattern, and the same tables are used via this pointer by pcre_study()
! 1548: and normally also by pcre_exec(). Thus, by default, for any single pat-
! 1549: tern, compilation, studying and matching all happen in the same locale,
! 1550: but different patterns can be compiled in different locales.
! 1551:
! 1552: It is possible to pass a table pointer or NULL (indicating the use of
! 1553: the internal tables) to pcre_exec(). Although not intended for this
! 1554: purpose, this facility could be used to match a pattern in a different
! 1555: locale from the one in which it was compiled. Passing table pointers at
! 1556: run time is discussed below in the section on matching a pattern.
! 1557:
! 1558:
! 1559: INFORMATION ABOUT A PATTERN
! 1560:
! 1561: int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
! 1562: int what, void *where);
! 1563:
! 1564: The pcre_fullinfo() function returns information about a compiled pat-
! 1565: tern. It replaces the obsolete pcre_info() function, which is neverthe-
! 1566: less retained for backwards compability (and is documented below).
! 1567:
! 1568: The first argument for pcre_fullinfo() is a pointer to the compiled
! 1569: pattern. The second argument is the result of pcre_study(), or NULL if
! 1570: the pattern was not studied. The third argument specifies which piece
! 1571: of information is required, and the fourth argument is a pointer to a
! 1572: variable to receive the data. The yield of the function is zero for
! 1573: success, or one of the following negative numbers:
! 1574:
! 1575: PCRE_ERROR_NULL the argument code was NULL
! 1576: the argument where was NULL
! 1577: PCRE_ERROR_BADMAGIC the "magic number" was not found
! 1578: PCRE_ERROR_BADOPTION the value of what was invalid
! 1579:
! 1580: The "magic number" is placed at the start of each compiled pattern as
! 1581: an simple check against passing an arbitrary memory pointer. Here is a
! 1582: typical call of pcre_fullinfo(), to obtain the length of the compiled
! 1583: pattern:
! 1584:
! 1585: int rc;
! 1586: size_t length;
! 1587: rc = pcre_fullinfo(
! 1588: re, /* result of pcre_compile() */
! 1589: pe, /* result of pcre_study(), or NULL */
! 1590: PCRE_INFO_SIZE, /* what is required */
! 1591: &length); /* where to put the data */
! 1592:
! 1593: The possible values for the third argument are defined in pcre.h, and
! 1594: are as follows:
! 1595:
! 1596: PCRE_INFO_BACKREFMAX
! 1597:
! 1598: Return the number of the highest back reference in the pattern. The
! 1599: fourth argument should point to an int variable. Zero is returned if
! 1600: there are no back references.
! 1601:
! 1602: PCRE_INFO_CAPTURECOUNT
! 1603:
! 1604: Return the number of capturing subpatterns in the pattern. The fourth
! 1605: argument should point to an int variable.
! 1606:
! 1607: PCRE_INFO_DEFAULT_TABLES
! 1608:
! 1609: Return a pointer to the internal default character tables within PCRE.
! 1610: The fourth argument should point to an unsigned char * variable. This
! 1611: information call is provided for internal use by the pcre_study() func-
! 1612: tion. External callers can cause PCRE to use its internal tables by
! 1613: passing a NULL table pointer.
! 1614:
! 1615: PCRE_INFO_FIRSTBYTE
! 1616:
! 1617: Return information about the first byte of any matched string, for a
! 1618: non-anchored pattern. The fourth argument should point to an int vari-
! 1619: able. (This option used to be called PCRE_INFO_FIRSTCHAR; the old name
! 1620: is still recognized for backwards compatibility.)
! 1621:
! 1622: If there is a fixed first byte, for example, from a pattern such as
! 1623: (cat|cow|coyote), its value is returned. Otherwise, if either
! 1624:
! 1625: (a) the pattern was compiled with the PCRE_MULTILINE option, and every
! 1626: branch starts with "^", or
! 1627:
! 1628: (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
! 1629: set (if it were set, the pattern would be anchored),
! 1630:
! 1631: -1 is returned, indicating that the pattern matches only at the start
! 1632: of a subject string or after any newline within the string. Otherwise
! 1633: -2 is returned. For anchored patterns, -2 is returned.
! 1634:
! 1635: PCRE_INFO_FIRSTTABLE
! 1636:
! 1637: If the pattern was studied, and this resulted in the construction of a
! 1638: 256-bit table indicating a fixed set of bytes for the first byte in any
! 1639: matching string, a pointer to the table is returned. Otherwise NULL is
! 1640: returned. The fourth argument should point to an unsigned char * vari-
! 1641: able.
! 1642:
! 1643: PCRE_INFO_HASCRORLF
! 1644:
! 1645: Return 1 if the pattern contains any explicit matches for CR or LF
! 1646: characters, otherwise 0. The fourth argument should point to an int
! 1647: variable. An explicit match is either a literal CR or LF character, or
! 1648: \r or \n.
! 1649:
! 1650: PCRE_INFO_JCHANGED
! 1651:
! 1652: Return 1 if the (?J) or (?-J) option setting is used in the pattern,
! 1653: otherwise 0. The fourth argument should point to an int variable. (?J)
! 1654: and (?-J) set and unset the local PCRE_DUPNAMES option, respectively.
! 1655:
! 1656: PCRE_INFO_LASTLITERAL
! 1657:
! 1658: Return the value of the rightmost literal byte that must exist in any
! 1659: matched string, other than at its start, if such a byte has been
! 1660: recorded. The fourth argument should point to an int variable. If there
! 1661: is no such byte, -1 is returned. For anchored patterns, a last literal
! 1662: byte is recorded only if it follows something of variable length. For
! 1663: example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
! 1664: /^a\dz\d/ the returned value is -1.
! 1665:
! 1666: PCRE_INFO_NAMECOUNT
! 1667: PCRE_INFO_NAMEENTRYSIZE
! 1668: PCRE_INFO_NAMETABLE
! 1669:
! 1670: PCRE supports the use of named as well as numbered capturing parenthe-
! 1671: ses. The names are just an additional way of identifying the parenthe-
! 1672: ses, which still acquire numbers. Several convenience functions such as
! 1673: pcre_get_named_substring() are provided for extracting captured sub-
! 1674: strings by name. It is also possible to extract the data directly, by
! 1675: first converting the name to a number in order to access the correct
! 1676: pointers in the output vector (described with pcre_exec() below). To do
! 1677: the conversion, you need to use the name-to-number map, which is
! 1678: described by these three values.
! 1679:
! 1680: The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
! 1681: gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
! 1682: of each entry; both of these return an int value. The entry size
! 1683: depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
! 1684: a pointer to the first entry of the table (a pointer to char). The
! 1685: first two bytes of each entry are the number of the capturing parenthe-
! 1686: sis, most significant byte first. The rest of the entry is the corre-
! 1687: sponding name, zero terminated. The names are in alphabetical order.
! 1688: When PCRE_DUPNAMES is set, duplicate names are in order of their paren-
! 1689: theses numbers. For example, consider the following pattern (assume
! 1690: PCRE_EXTENDED is set, so white space - including newlines - is
! 1691: ignored):
! 1692:
! 1693: (?<date> (?<year>(\d\d)?\d\d) -
! 1694: (?<month>\d\d) - (?<day>\d\d) )
! 1695:
! 1696: There are four named subpatterns, so the table has four entries, and
! 1697: each entry in the table is eight bytes long. The table is as follows,
! 1698: with non-printing bytes shows in hexadecimal, and undefined bytes shown
! 1699: as ??:
! 1700:
! 1701: 00 01 d a t e 00 ??
! 1702: 00 05 d a y 00 ?? ??
! 1703: 00 04 m o n t h 00
! 1704: 00 02 y e a r 00 ??
! 1705:
! 1706: When writing code to extract data from named subpatterns using the
! 1707: name-to-number map, remember that the length of the entries is likely
! 1708: to be different for each compiled pattern.
! 1709:
! 1710: PCRE_INFO_OKPARTIAL
! 1711:
! 1712: Return 1 if the pattern can be used for partial matching, otherwise 0.
! 1713: The fourth argument should point to an int variable. The pcrepartial
! 1714: documentation lists the restrictions that apply to patterns when par-
! 1715: tial matching is used.
! 1716:
! 1717: PCRE_INFO_OPTIONS
! 1718:
! 1719: Return a copy of the options with which the pattern was compiled. The
! 1720: fourth argument should point to an unsigned long int variable. These
! 1721: option bits are those specified in the call to pcre_compile(), modified
! 1722: by any top-level option settings at the start of the pattern itself. In
! 1723: other words, they are the options that will be in force when matching
! 1724: starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with
! 1725: the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE,
! 1726: and PCRE_EXTENDED.
! 1727:
! 1728: A pattern is automatically anchored by PCRE if all of its top-level
! 1729: alternatives begin with one of the following:
! 1730:
! 1731: ^ unless PCRE_MULTILINE is set
! 1732: \A always
! 1733: \G always
! 1734: .* if PCRE_DOTALL is set and there are no back
! 1735: references to the subpattern in which .* appears
! 1736:
! 1737: For such patterns, the PCRE_ANCHORED bit is set in the options returned
! 1738: by pcre_fullinfo().
! 1739:
! 1740: PCRE_INFO_SIZE
! 1741:
! 1742: Return the size of the compiled pattern, that is, the value that was
! 1743: passed as the argument to pcre_malloc() when PCRE was getting memory in
! 1744: which to place the compiled data. The fourth argument should point to a
! 1745: size_t variable.
! 1746:
! 1747: PCRE_INFO_STUDYSIZE
! 1748:
! 1749: Return the size of the data block pointed to by the study_data field in
! 1750: a pcre_extra block. That is, it is the value that was passed to
! 1751: pcre_malloc() when PCRE was getting memory into which to place the data
! 1752: created by pcre_study(). The fourth argument should point to a size_t
! 1753: variable.
! 1754:
! 1755:
! 1756: OBSOLETE INFO FUNCTION
! 1757:
! 1758: int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
! 1759:
! 1760: The pcre_info() function is now obsolete because its interface is too
! 1761: restrictive to return all the available data about a compiled pattern.
! 1762: New programs should use pcre_fullinfo() instead. The yield of
! 1763: pcre_info() is the number of capturing subpatterns, or one of the fol-
! 1764: lowing negative numbers:
! 1765:
! 1766: PCRE_ERROR_NULL the argument code was NULL
! 1767: PCRE_ERROR_BADMAGIC the "magic number" was not found
! 1768:
! 1769: If the optptr argument is not NULL, a copy of the options with which
! 1770: the pattern was compiled is placed in the integer it points to (see
! 1771: PCRE_INFO_OPTIONS above).
! 1772:
! 1773: If the pattern is not anchored and the firstcharptr argument is not
! 1774: NULL, it is used to pass back information about the first character of
! 1775: any matched string (see PCRE_INFO_FIRSTBYTE above).
! 1776:
! 1777:
! 1778: REFERENCE COUNTS
! 1779:
! 1780: int pcre_refcount(pcre *code, int adjust);
! 1781:
! 1782: The pcre_refcount() function is used to maintain a reference count in
! 1783: the data block that contains a compiled pattern. It is provided for the
! 1784: benefit of applications that operate in an object-oriented manner,
! 1785: where different parts of the application may be using the same compiled
! 1786: pattern, but you want to free the block when they are all done.
! 1787:
! 1788: When a pattern is compiled, the reference count field is initialized to
! 1789: zero. It is changed only by calling this function, whose action is to
! 1790: add the adjust value (which may be positive or negative) to it. The
! 1791: yield of the function is the new value. However, the value of the count
! 1792: is constrained to lie between 0 and 65535, inclusive. If the new value
! 1793: is outside these limits, it is forced to the appropriate limit value.
! 1794:
! 1795: Except when it is zero, the reference count is not correctly preserved
! 1796: if a pattern is compiled on one host and then transferred to a host
! 1797: whose byte-order is different. (This seems a highly unlikely scenario.)
! 1798:
! 1799:
! 1800: MATCHING A PATTERN: THE TRADITIONAL FUNCTION
! 1801:
! 1802: int pcre_exec(const pcre *code, const pcre_extra *extra,
! 1803: const char *subject, int length, int startoffset,
! 1804: int options, int *ovector, int ovecsize);
! 1805:
! 1806: The function pcre_exec() is called to match a subject string against a
! 1807: compiled pattern, which is passed in the code argument. If the pattern
! 1808: has been studied, the result of the study should be passed in the extra
! 1809: argument. This function is the main matching facility of the library,
! 1810: and it operates in a Perl-like manner. For specialist use there is also
! 1811: an alternative matching function, which is described below in the sec-
! 1812: tion about the pcre_dfa_exec() function.
! 1813:
! 1814: In most applications, the pattern will have been compiled (and option-
! 1815: ally studied) in the same process that calls pcre_exec(). However, it
! 1816: is possible to save compiled patterns and study data, and then use them
! 1817: later in different processes, possibly even on different hosts. For a
! 1818: discussion about this, see the pcreprecompile documentation.
! 1819:
! 1820: Here is an example of a simple call to pcre_exec():
! 1821:
! 1822: int rc;
! 1823: int ovector[30];
! 1824: rc = pcre_exec(
! 1825: re, /* result of pcre_compile() */
! 1826: NULL, /* we didn't study the pattern */
! 1827: "some string", /* the subject string */
! 1828: 11, /* the length of the subject string */
! 1829: 0, /* start at offset 0 in the subject */
! 1830: 0, /* default options */
! 1831: ovector, /* vector of integers for substring information */
! 1832: 30); /* number of elements (NOT size in bytes) */
! 1833:
! 1834: Extra data for pcre_exec()
! 1835:
! 1836: If the extra argument is not NULL, it must point to a pcre_extra data
! 1837: block. The pcre_study() function returns such a block (when it doesn't
! 1838: return NULL), but you can also create one for yourself, and pass addi-
! 1839: tional information in it. The pcre_extra block contains the following
! 1840: fields (not necessarily in this order):
! 1841:
! 1842: unsigned long int flags;
! 1843: void *study_data;
! 1844: unsigned long int match_limit;
! 1845: unsigned long int match_limit_recursion;
! 1846: void *callout_data;
! 1847: const unsigned char *tables;
! 1848:
! 1849: The flags field is a bitmap that specifies which of the other fields
! 1850: are set. The flag bits are:
! 1851:
! 1852: PCRE_EXTRA_STUDY_DATA
! 1853: PCRE_EXTRA_MATCH_LIMIT
! 1854: PCRE_EXTRA_MATCH_LIMIT_RECURSION
! 1855: PCRE_EXTRA_CALLOUT_DATA
! 1856: PCRE_EXTRA_TABLES
! 1857:
! 1858: Other flag bits should be set to zero. The study_data field is set in
! 1859: the pcre_extra block that is returned by pcre_study(), together with
! 1860: the appropriate flag bit. You should not set this yourself, but you may
! 1861: add to the block by setting the other fields and their corresponding
! 1862: flag bits.
! 1863:
! 1864: The match_limit field provides a means of preventing PCRE from using up
! 1865: a vast amount of resources when running patterns that are not going to
! 1866: match, but which have a very large number of possibilities in their
! 1867: search trees. The classic example is the use of nested unlimited
! 1868: repeats.
! 1869:
! 1870: Internally, PCRE uses a function called match() which it calls repeat-
! 1871: edly (sometimes recursively). The limit set by match_limit is imposed
! 1872: on the number of times this function is called during a match, which
! 1873: has the effect of limiting the amount of backtracking that can take
! 1874: place. For patterns that are not anchored, the count restarts from zero
! 1875: for each position in the subject string.
! 1876:
! 1877: The default value for the limit can be set when PCRE is built; the
! 1878: default default is 10 million, which handles all but the most extreme
! 1879: cases. You can override the default by suppling pcre_exec() with a
! 1880: pcre_extra block in which match_limit is set, and
! 1881: PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
! 1882: exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
! 1883:
! 1884: The match_limit_recursion field is similar to match_limit, but instead
! 1885: of limiting the total number of times that match() is called, it limits
! 1886: the depth of recursion. The recursion depth is a smaller number than
! 1887: the total number of calls, because not all calls to match() are recur-
! 1888: sive. This limit is of use only if it is set smaller than match_limit.
! 1889:
! 1890: Limiting the recursion depth limits the amount of stack that can be
! 1891: used, or, when PCRE has been compiled to use memory on the heap instead
! 1892: of the stack, the amount of heap memory that can be used.
! 1893:
! 1894: The default value for match_limit_recursion can be set when PCRE is
! 1895: built; the default default is the same value as the default for
! 1896: match_limit. You can override the default by suppling pcre_exec() with
! 1897: a pcre_extra block in which match_limit_recursion is set, and
! 1898: PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the
! 1899: limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT.
! 1900:
! 1901: The pcre_callout field is used in conjunction with the "callout" fea-
! 1902: ture, which is described in the pcrecallout documentation.
! 1903:
! 1904: The tables field is used to pass a character tables pointer to
! 1905: pcre_exec(); this overrides the value that is stored with the compiled
! 1906: pattern. A non-NULL value is stored with the compiled pattern only if
! 1907: custom tables were supplied to pcre_compile() via its tableptr argu-
! 1908: ment. If NULL is passed to pcre_exec() using this mechanism, it forces
! 1909: PCRE's internal tables to be used. This facility is helpful when re-
! 1910: using patterns that have been saved after compiling with an external
! 1911: set of tables, because the external tables might be at a different
! 1912: address when pcre_exec() is called. See the pcreprecompile documenta-
! 1913: tion for a discussion of saving compiled patterns for later use.
! 1914:
! 1915: Option bits for pcre_exec()
! 1916:
! 1917: The unused bits of the options argument for pcre_exec() must be zero.
! 1918: The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx,
! 1919: PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and
! 1920: PCRE_PARTIAL.
! 1921:
! 1922: PCRE_ANCHORED
! 1923:
! 1924: The PCRE_ANCHORED option limits pcre_exec() to matching at the first
! 1925: matching position. If a pattern was compiled with PCRE_ANCHORED, or
! 1926: turned out to be anchored by virtue of its contents, it cannot be made
! 1927: unachored at matching time.
! 1928:
! 1929: PCRE_BSR_ANYCRLF
! 1930: PCRE_BSR_UNICODE
! 1931:
! 1932: These options (which are mutually exclusive) control what the \R escape
! 1933: sequence matches. The choice is either to match only CR, LF, or CRLF,
! 1934: or to match any Unicode newline sequence. These options override the
! 1935: choice that was made or defaulted when the pattern was compiled.
! 1936:
! 1937: PCRE_NEWLINE_CR
! 1938: PCRE_NEWLINE_LF
! 1939: PCRE_NEWLINE_CRLF
! 1940: PCRE_NEWLINE_ANYCRLF
! 1941: PCRE_NEWLINE_ANY
! 1942:
! 1943: These options override the newline definition that was chosen or
! 1944: defaulted when the pattern was compiled. For details, see the descrip-
! 1945: tion of pcre_compile() above. During matching, the newline choice
! 1946: affects the behaviour of the dot, circumflex, and dollar metacharac-
! 1947: ters. It may also alter the way the match position is advanced after a
! 1948: match failure for an unanchored pattern.
! 1949:
! 1950: When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is
! 1951: set, and a match attempt for an unanchored pattern fails when the cur-
! 1952: rent position is at a CRLF sequence, and the pattern contains no
! 1953: explicit matches for CR or LF characters, the match position is
! 1954: advanced by two characters instead of one, in other words, to after the
! 1955: CRLF.
! 1956:
! 1957: The above rule is a compromise that makes the most common cases work as
! 1958: expected. For example, if the pattern is .+A (and the PCRE_DOTALL
! 1959: option is not set), it does not match the string "\r\nA" because, after
! 1960: failing at the start, it skips both the CR and the LF before retrying.
! 1961: However, the pattern [\r\n]A does match that string, because it con-
! 1962: tains an explicit CR or LF reference, and so advances only by one char-
! 1963: acter after the first failure.
! 1964:
! 1965: An explicit match for CR of LF is either a literal appearance of one of
! 1966: those characters, or one of the \r or \n escape sequences. Implicit
! 1967: matches such as [^X] do not count, nor does \s (which includes CR and
! 1968: LF in the characters that it matches).
! 1969:
! 1970: Notwithstanding the above, anomalous effects may still occur when CRLF
! 1971: is a valid newline sequence and explicit \r or \n escapes appear in the
! 1972: pattern.
! 1973:
! 1974: PCRE_NOTBOL
! 1975:
! 1976: This option specifies that first character of the subject string is not
! 1977: the beginning of a line, so the circumflex metacharacter should not
! 1978: match before it. Setting this without PCRE_MULTILINE (at compile time)
! 1979: causes circumflex never to match. This option affects only the behav-
! 1980: iour of the circumflex metacharacter. It does not affect \A.
! 1981:
! 1982: PCRE_NOTEOL
! 1983:
! 1984: This option specifies that the end of the subject string is not the end
! 1985: of a line, so the dollar metacharacter should not match it nor (except
! 1986: in multiline mode) a newline immediately before it. Setting this with-
! 1987: out PCRE_MULTILINE (at compile time) causes dollar never to match. This
! 1988: option affects only the behaviour of the dollar metacharacter. It does
! 1989: not affect \Z or \z.
! 1990:
! 1991: PCRE_NOTEMPTY
! 1992:
! 1993: An empty string is not considered to be a valid match if this option is
! 1994: set. If there are alternatives in the pattern, they are tried. If all
! 1995: the alternatives match the empty string, the entire match fails. For
! 1996: example, if the pattern
! 1997:
! 1998: a?b?
! 1999:
! 2000: is applied to a string not beginning with "a" or "b", it matches the
! 2001: empty string at the start of the subject. With PCRE_NOTEMPTY set, this
! 2002: match is not valid, so PCRE searches further into the string for occur-
! 2003: rences of "a" or "b".
! 2004:
! 2005: Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
! 2006: cial case of a pattern match of the empty string within its split()
! 2007: function, and when using the /g modifier. It is possible to emulate
! 2008: Perl's behaviour after matching a null string by first trying the match
! 2009: again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
! 2010: if that fails by advancing the starting offset (see below) and trying
! 2011: an ordinary match again. There is some code that demonstrates how to do
! 2012: this in the pcredemo.c sample program.
! 2013:
! 2014: PCRE_NO_UTF8_CHECK
! 2015:
! 2016: When PCRE_UTF8 is set at compile time, the validity of the subject as a
! 2017: UTF-8 string is automatically checked when pcre_exec() is subsequently
! 2018: called. The value of startoffset is also checked to ensure that it
! 2019: points to the start of a UTF-8 character. There is a discussion about
! 2020: the validity of UTF-8 strings in the section on UTF-8 support in the
! 2021: main pcre page. If an invalid UTF-8 sequence of bytes is found,
! 2022: pcre_exec() returns the error PCRE_ERROR_BADUTF8. If startoffset con-
! 2023: tains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned.
! 2024:
! 2025: If you already know that your subject is valid, and you want to skip
! 2026: these checks for performance reasons, you can set the
! 2027: PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
! 2028: do this for the second and subsequent calls to pcre_exec() if you are
! 2029: making repeated calls to find all the matches in a single subject
! 2030: string. However, you should be sure that the value of startoffset
! 2031: points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
! 2032: set, the effect of passing an invalid UTF-8 string as a subject, or a
! 2033: value of startoffset that does not point to the start of a UTF-8 char-
! 2034: acter, is undefined. Your program may crash.
! 2035:
! 2036: PCRE_PARTIAL
! 2037:
! 2038: This option turns on the partial matching feature. If the subject
! 2039: string fails to match the pattern, but at some point during the match-
! 2040: ing process the end of the subject was reached (that is, the subject
! 2041: partially matches the pattern and the failure to match occurred only
! 2042: because there were not enough subject characters), pcre_exec() returns
! 2043: PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
! 2044: used, there are restrictions on what may appear in the pattern. These
! 2045: are discussed in the pcrepartial documentation.
! 2046:
! 2047: The string to be matched by pcre_exec()
! 2048:
! 2049: The subject string is passed to pcre_exec() as a pointer in subject, a
! 2050: length in length, and a starting byte offset in startoffset. In UTF-8
! 2051: mode, the byte offset must point to the start of a UTF-8 character.
! 2052: Unlike the pattern string, the subject may contain binary zero bytes.
! 2053: When the starting offset is zero, the search for a match starts at the
! 2054: beginning of the subject, and this is by far the most common case.
! 2055:
! 2056: A non-zero starting offset is useful when searching for another match
! 2057: in the same subject by calling pcre_exec() again after a previous suc-
! 2058: cess. Setting startoffset differs from just passing over a shortened
! 2059: string and setting PCRE_NOTBOL in the case of a pattern that begins
! 2060: with any kind of lookbehind. For example, consider the pattern
! 2061:
! 2062: \Biss\B
! 2063:
! 2064: which finds occurrences of "iss" in the middle of words. (\B matches
! 2065: only if the current position in the subject is not a word boundary.)
! 2066: When applied to the string "Mississipi" the first call to pcre_exec()
! 2067: finds the first occurrence. If pcre_exec() is called again with just
! 2068: the remainder of the subject, namely "issipi", it does not match,
! 2069: because \B is always false at the start of the subject, which is deemed
! 2070: to be a word boundary. However, if pcre_exec() is passed the entire
! 2071: string again, but with startoffset set to 4, it finds the second occur-
! 2072: rence of "iss" because it is able to look behind the starting point to
! 2073: discover that it is preceded by a letter.
! 2074:
! 2075: If a non-zero starting offset is passed when the pattern is anchored,
! 2076: one attempt to match at the given offset is made. This can only succeed
! 2077: if the pattern does not require the match to be at the start of the
! 2078: subject.
! 2079:
! 2080: How pcre_exec() returns captured substrings
! 2081:
! 2082: In general, a pattern matches a certain portion of the subject, and in
! 2083: addition, further substrings from the subject may be picked out by
! 2084: parts of the pattern. Following the usage in Jeffrey Friedl's book,
! 2085: this is called "capturing" in what follows, and the phrase "capturing
! 2086: subpattern" is used for a fragment of a pattern that picks out a sub-
! 2087: string. PCRE supports several other kinds of parenthesized subpattern
! 2088: that do not cause substrings to be captured.
! 2089:
! 2090: Captured substrings are returned to the caller via a vector of integer
! 2091: offsets whose address is passed in ovector. The number of elements in
! 2092: the vector is passed in ovecsize, which must be a non-negative number.
! 2093: Note: this argument is NOT the size of ovector in bytes.
! 2094:
! 2095: The first two-thirds of the vector is used to pass back captured sub-
! 2096: strings, each substring using a pair of integers. The remaining third
! 2097: of the vector is used as workspace by pcre_exec() while matching cap-
! 2098: turing subpatterns, and is not available for passing back information.
! 2099: The length passed in ovecsize should always be a multiple of three. If
! 2100: it is not, it is rounded down.
! 2101:
! 2102: When a match is successful, information about captured substrings is
! 2103: returned in pairs of integers, starting at the beginning of ovector,
! 2104: and continuing up to two-thirds of its length at the most. The first
! 2105: element of a pair is set to the offset of the first character in a sub-
! 2106: string, and the second is set to the offset of the first character
! 2107: after the end of a substring. The first pair, ovector[0] and ovec-
! 2108: tor[1], identify the portion of the subject string matched by the
! 2109: entire pattern. The next pair is used for the first capturing subpat-
! 2110: tern, and so on. The value returned by pcre_exec() is one more than the
! 2111: highest numbered pair that has been set. For example, if two substrings
! 2112: have been captured, the returned value is 3. If there are no capturing
! 2113: subpatterns, the return value from a successful match is 1, indicating
! 2114: that just the first pair of offsets has been set.
! 2115:
! 2116: If a capturing subpattern is matched repeatedly, it is the last portion
! 2117: of the string that it matched that is returned.
! 2118:
! 2119: If the vector is too small to hold all the captured substring offsets,
! 2120: it is used as far as possible (up to two-thirds of its length), and the
! 2121: function returns a value of zero. In particular, if the substring off-
! 2122: sets are not of interest, pcre_exec() may be called with ovector passed
! 2123: as NULL and ovecsize as zero. However, if the pattern contains back
! 2124: references and the ovector is not big enough to remember the related
! 2125: substrings, PCRE has to get additional memory for use during matching.
! 2126: Thus it is usually advisable to supply an ovector.
! 2127:
! 2128: The pcre_info() function can be used to find out how many capturing
! 2129: subpatterns there are in a compiled pattern. The smallest size for
! 2130: ovector that will allow for n captured substrings, in addition to the
! 2131: offsets of the substring matched by the whole pattern, is (n+1)*3.
! 2132:
! 2133: It is possible for capturing subpattern number n+1 to match some part
! 2134: of the subject when subpattern n has not been used at all. For example,
! 2135: if the string "abc" is matched against the pattern (a|(z))(bc) the
! 2136: return from the function is 4, and subpatterns 1 and 3 are matched, but
! 2137: 2 is not. When this happens, both values in the offset pairs corre-
! 2138: sponding to unused subpatterns are set to -1.
! 2139:
! 2140: Offset values that correspond to unused subpatterns at the end of the
! 2141: expression are also set to -1. For example, if the string "abc" is
! 2142: matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not
! 2143: matched. The return from the function is 2, because the highest used
! 2144: capturing subpattern number is 1. However, you can refer to the offsets
! 2145: for the second and third capturing subpatterns if you wish (assuming
! 2146: the vector is large enough, of course).
! 2147:
! 2148: Some convenience functions are provided for extracting the captured
! 2149: substrings as separate strings. These are described below.
! 2150:
! 2151: Error return values from pcre_exec()
! 2152:
! 2153: If pcre_exec() fails, it returns a negative number. The following are
! 2154: defined in the header file:
! 2155:
! 2156: PCRE_ERROR_NOMATCH (-1)
! 2157:
! 2158: The subject string did not match the pattern.
! 2159:
! 2160: PCRE_ERROR_NULL (-2)
! 2161:
! 2162: Either code or subject was passed as NULL, or ovector was NULL and
! 2163: ovecsize was not zero.
! 2164:
! 2165: PCRE_ERROR_BADOPTION (-3)
! 2166:
! 2167: An unrecognized bit was set in the options argument.
! 2168:
! 2169: PCRE_ERROR_BADMAGIC (-4)
! 2170:
! 2171: PCRE stores a 4-byte "magic number" at the start of the compiled code,
! 2172: to catch the case when it is passed a junk pointer and to detect when a
! 2173: pattern that was compiled in an environment of one endianness is run in
! 2174: an environment with the other endianness. This is the error that PCRE
! 2175: gives when the magic number is not present.
! 2176:
! 2177: PCRE_ERROR_UNKNOWN_OPCODE (-5)
! 2178:
! 2179: While running the pattern match, an unknown item was encountered in the
! 2180: compiled pattern. This error could be caused by a bug in PCRE or by
! 2181: overwriting of the compiled pattern.
! 2182:
! 2183: PCRE_ERROR_NOMEMORY (-6)
! 2184:
! 2185: If a pattern contains back references, but the ovector that is passed
! 2186: to pcre_exec() is not big enough to remember the referenced substrings,
! 2187: PCRE gets a block of memory at the start of matching to use for this
! 2188: purpose. If the call via pcre_malloc() fails, this error is given. The
! 2189: memory is automatically freed at the end of matching.
! 2190:
! 2191: PCRE_ERROR_NOSUBSTRING (-7)
! 2192:
! 2193: This error is used by the pcre_copy_substring(), pcre_get_substring(),
! 2194: and pcre_get_substring_list() functions (see below). It is never
! 2195: returned by pcre_exec().
! 2196:
! 2197: PCRE_ERROR_MATCHLIMIT (-8)
! 2198:
! 2199: The backtracking limit, as specified by the match_limit field in a
! 2200: pcre_extra structure (or defaulted) was reached. See the description
! 2201: above.
! 2202:
! 2203: PCRE_ERROR_CALLOUT (-9)
! 2204:
! 2205: This error is never generated by pcre_exec() itself. It is provided for
! 2206: use by callout functions that want to yield a distinctive error code.
! 2207: See the pcrecallout documentation for details.
! 2208:
! 2209: PCRE_ERROR_BADUTF8 (-10)
! 2210:
! 2211: A string that contains an invalid UTF-8 byte sequence was passed as a
! 2212: subject.
! 2213:
! 2214: PCRE_ERROR_BADUTF8_OFFSET (-11)
! 2215:
! 2216: The UTF-8 byte sequence that was passed as a subject was valid, but the
! 2217: value of startoffset did not point to the beginning of a UTF-8 charac-
! 2218: ter.
! 2219:
! 2220: PCRE_ERROR_PARTIAL (-12)
! 2221:
! 2222: The subject string did not match, but it did match partially. See the
! 2223: pcrepartial documentation for details of partial matching.
! 2224:
! 2225: PCRE_ERROR_BADPARTIAL (-13)
! 2226:
! 2227: The PCRE_PARTIAL option was used with a compiled pattern containing
! 2228: items that are not supported for partial matching. See the pcrepartial
! 2229: documentation for details of partial matching.
! 2230:
! 2231: PCRE_ERROR_INTERNAL (-14)
! 2232:
! 2233: An unexpected internal error has occurred. This error could be caused
! 2234: by a bug in PCRE or by overwriting of the compiled pattern.
! 2235:
! 2236: PCRE_ERROR_BADCOUNT (-15)
! 2237:
! 2238: This error is given if the value of the ovecsize argument is negative.
! 2239:
! 2240: PCRE_ERROR_RECURSIONLIMIT (-21)
! 2241:
! 2242: The internal recursion limit, as specified by the match_limit_recursion
! 2243: field in a pcre_extra structure (or defaulted) was reached. See the
! 2244: description above.
! 2245:
! 2246: PCRE_ERROR_BADNEWLINE (-23)
! 2247:
! 2248: An invalid combination of PCRE_NEWLINE_xxx options was given.
! 2249:
! 2250: Error numbers -16 to -20 and -22 are not used by pcre_exec().
! 2251:
! 2252:
! 2253: EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
! 2254:
! 2255: int pcre_copy_substring(const char *subject, int *ovector,
! 2256: int stringcount, int stringnumber, char *buffer,
! 2257: int buffersize);
! 2258:
! 2259: int pcre_get_substring(const char *subject, int *ovector,
! 2260: int stringcount, int stringnumber,
! 2261: const char **stringptr);
! 2262:
! 2263: int pcre_get_substring_list(const char *subject,
! 2264: int *ovector, int stringcount, const char ***listptr);
! 2265:
! 2266: Captured substrings can be accessed directly by using the offsets
! 2267: returned by pcre_exec() in ovector. For convenience, the functions
! 2268: pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
! 2269: string_list() are provided for extracting captured substrings as new,
! 2270: separate, zero-terminated strings. These functions identify substrings
! 2271: by number. The next section describes functions for extracting named
! 2272: substrings.
! 2273:
! 2274: A substring that contains a binary zero is correctly extracted and has
! 2275: a further zero added on the end, but the result is not, of course, a C
! 2276: string. However, you can process such a string by referring to the
! 2277: length that is returned by pcre_copy_substring() and pcre_get_sub-
! 2278: string(). Unfortunately, the interface to pcre_get_substring_list() is
! 2279: not adequate for handling strings containing binary zeros, because the
! 2280: end of the final string is not independently indicated.
! 2281:
! 2282: The first three arguments are the same for all three of these func-
! 2283: tions: subject is the subject string that has just been successfully
! 2284: matched, ovector is a pointer to the vector of integer offsets that was
! 2285: passed to pcre_exec(), and stringcount is the number of substrings that
! 2286: were captured by the match, including the substring that matched the
! 2287: entire regular expression. This is the value returned by pcre_exec() if
! 2288: it is greater than zero. If pcre_exec() returned zero, indicating that
! 2289: it ran out of space in ovector, the value passed as stringcount should
! 2290: be the number of elements in the vector divided by three.
! 2291:
! 2292: The functions pcre_copy_substring() and pcre_get_substring() extract a
! 2293: single substring, whose number is given as stringnumber. A value of
! 2294: zero extracts the substring that matched the entire pattern, whereas
! 2295: higher values extract the captured substrings. For pcre_copy_sub-
! 2296: string(), the string is placed in buffer, whose length is given by
! 2297: buffersize, while for pcre_get_substring() a new block of memory is
! 2298: obtained via pcre_malloc, and its address is returned via stringptr.
! 2299: The yield of the function is the length of the string, not including
! 2300: the terminating zero, or one of these error codes:
! 2301:
! 2302: PCRE_ERROR_NOMEMORY (-6)
! 2303:
! 2304: The buffer was too small for pcre_copy_substring(), or the attempt to
! 2305: get memory failed for pcre_get_substring().
! 2306:
! 2307: PCRE_ERROR_NOSUBSTRING (-7)
! 2308:
! 2309: There is no substring whose number is stringnumber.
! 2310:
! 2311: The pcre_get_substring_list() function extracts all available sub-
! 2312: strings and builds a list of pointers to them. All this is done in a
! 2313: single block of memory that is obtained via pcre_malloc. The address of
! 2314: the memory block is returned via listptr, which is also the start of
! 2315: the list of string pointers. The end of the list is marked by a NULL
! 2316: pointer. The yield of the function is zero if all went well, or the
! 2317: error code
! 2318:
! 2319: PCRE_ERROR_NOMEMORY (-6)
! 2320:
! 2321: if the attempt to get the memory block failed.
! 2322:
! 2323: When any of these functions encounter a substring that is unset, which
! 2324: can happen when capturing subpattern number n+1 matches some part of
! 2325: the subject, but subpattern n has not been used at all, they return an
! 2326: empty string. This can be distinguished from a genuine zero-length sub-
! 2327: string by inspecting the appropriate offset in ovector, which is nega-
! 2328: tive for unset substrings.
! 2329:
! 2330: The two convenience functions pcre_free_substring() and pcre_free_sub-
! 2331: string_list() can be used to free the memory returned by a previous
! 2332: call of pcre_get_substring() or pcre_get_substring_list(), respec-
! 2333: tively. They do nothing more than call the function pointed to by
! 2334: pcre_free, which of course could be called directly from a C program.
! 2335: However, PCRE is used in some situations where it is linked via a spe-
! 2336: cial interface to another programming language that cannot use
! 2337: pcre_free directly; it is for these cases that the functions are pro-
! 2338: vided.
! 2339:
! 2340:
! 2341: EXTRACTING CAPTURED SUBSTRINGS BY NAME
! 2342:
! 2343: int pcre_get_stringnumber(const pcre *code,
! 2344: const char *name);
! 2345:
! 2346: int pcre_copy_named_substring(const pcre *code,
! 2347: const char *subject, int *ovector,
! 2348: int stringcount, const char *stringname,
! 2349: char *buffer, int buffersize);
! 2350:
! 2351: int pcre_get_named_substring(const pcre *code,
! 2352: const char *subject, int *ovector,
! 2353: int stringcount, const char *stringname,
! 2354: const char **stringptr);
! 2355:
! 2356: To extract a substring by name, you first have to find associated num-
! 2357: ber. For example, for this pattern
! 2358:
! 2359: (a+)b(?<xxx>\d+)...
! 2360:
! 2361: the number of the subpattern called "xxx" is 2. If the name is known to
! 2362: be unique (PCRE_DUPNAMES was not set), you can find the number from the
! 2363: name by calling pcre_get_stringnumber(). The first argument is the com-
! 2364: piled pattern, and the second is the name. The yield of the function is
! 2365: the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no
! 2366: subpattern of that name.
! 2367:
! 2368: Given the number, you can extract the substring directly, or use one of
! 2369: the functions described in the previous section. For convenience, there
! 2370: are also two functions that do the whole job.
! 2371:
! 2372: Most of the arguments of pcre_copy_named_substring() and
! 2373: pcre_get_named_substring() are the same as those for the similarly
! 2374: named functions that extract by number. As these are described in the
! 2375: previous section, they are not re-described here. There are just two
! 2376: differences:
! 2377:
! 2378: First, instead of a substring number, a substring name is given. Sec-
! 2379: ond, there is an extra argument, given at the start, which is a pointer
! 2380: to the compiled pattern. This is needed in order to gain access to the
! 2381: name-to-number translation table.
! 2382:
! 2383: These functions call pcre_get_stringnumber(), and if it succeeds, they
! 2384: then call pcre_copy_substring() or pcre_get_substring(), as appropri-
! 2385: ate. NOTE: If PCRE_DUPNAMES is set and there are duplicate names, the
! 2386: behaviour may not be what you want (see the next section).
! 2387:
! 2388:
! 2389: DUPLICATE SUBPATTERN NAMES
! 2390:
! 2391: int pcre_get_stringtable_entries(const pcre *code,
! 2392: const char *name, char **first, char **last);
! 2393:
! 2394: When a pattern is compiled with the PCRE_DUPNAMES option, names for
! 2395: subpatterns are not required to be unique. Normally, patterns with
! 2396: duplicate names are such that in any one match, only one of the named
! 2397: subpatterns participates. An example is shown in the pcrepattern docu-
! 2398: mentation.
! 2399:
! 2400: When duplicates are present, pcre_copy_named_substring() and
! 2401: pcre_get_named_substring() return the first substring corresponding to
! 2402: the given name that is set. If none are set, PCRE_ERROR_NOSUBSTRING
! 2403: (-7) is returned; no data is returned. The pcre_get_stringnumber()
! 2404: function returns one of the numbers that are associated with the name,
! 2405: but it is not defined which it is.
! 2406:
! 2407: If you want to get full details of all captured substrings for a given
! 2408: name, you must use the pcre_get_stringtable_entries() function. The
! 2409: first argument is the compiled pattern, and the second is the name. The
! 2410: third and fourth are pointers to variables which are updated by the
! 2411: function. After it has run, they point to the first and last entries in
! 2412: the name-to-number table for the given name. The function itself
! 2413: returns the length of each entry, or PCRE_ERROR_NOSUBSTRING (-7) if
! 2414: there are none. The format of the table is described above in the sec-
! 2415: tion entitled Information about a pattern. Given all the relevant
! 2416: entries for the name, you can extract each of their numbers, and hence
! 2417: the captured data, if any.
! 2418:
! 2419:
! 2420: FINDING ALL POSSIBLE MATCHES
! 2421:
! 2422: The traditional matching function uses a similar algorithm to Perl,
! 2423: which stops when it finds the first match, starting at a given point in
! 2424: the subject. If you want to find all possible matches, or the longest
! 2425: possible match, consider using the alternative matching function (see
! 2426: below) instead. If you cannot use the alternative function, but still
! 2427: need to find all possible matches, you can kludge it up by making use
! 2428: of the callout facility, which is described in the pcrecallout documen-
! 2429: tation.
! 2430:
! 2431: What you have to do is to insert a callout right at the end of the pat-
! 2432: tern. When your callout function is called, extract and save the cur-
! 2433: rent matched substring. Then return 1, which forces pcre_exec() to
! 2434: backtrack and try other alternatives. Ultimately, when it runs out of
! 2435: matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
! 2436:
! 2437:
! 2438: MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
! 2439:
! 2440: int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
! 2441: const char *subject, int length, int startoffset,
! 2442: int options, int *ovector, int ovecsize,
! 2443: int *workspace, int wscount);
! 2444:
! 2445: The function pcre_dfa_exec() is called to match a subject string
! 2446: against a compiled pattern, using a matching algorithm that scans the
! 2447: subject string just once, and does not backtrack. This has different
! 2448: characteristics to the normal algorithm, and is not compatible with
! 2449: Perl. Some of the features of PCRE patterns are not supported. Never-
! 2450: theless, there are times when this kind of matching can be useful. For
! 2451: a discussion of the two matching algorithms, see the pcrematching docu-
! 2452: mentation.
! 2453:
! 2454: The arguments for the pcre_dfa_exec() function are the same as for
! 2455: pcre_exec(), plus two extras. The ovector argument is used in a differ-
! 2456: ent way, and this is described below. The other common arguments are
! 2457: used in the same way as for pcre_exec(), so their description is not
! 2458: repeated here.
! 2459:
! 2460: The two additional arguments provide workspace for the function. The
! 2461: workspace vector should contain at least 20 elements. It is used for
! 2462: keeping track of multiple paths through the pattern tree. More
! 2463: workspace will be needed for patterns and subjects where there are a
! 2464: lot of potential matches.
! 2465:
! 2466: Here is an example of a simple call to pcre_dfa_exec():
! 2467:
! 2468: int rc;
! 2469: int ovector[10];
! 2470: int wspace[20];
! 2471: rc = pcre_dfa_exec(
! 2472: re, /* result of pcre_compile() */
! 2473: NULL, /* we didn't study the pattern */
! 2474: "some string", /* the subject string */
! 2475: 11, /* the length of the subject string */
! 2476: 0, /* start at offset 0 in the subject */
! 2477: 0, /* default options */
! 2478: ovector, /* vector of integers for substring information */
! 2479: 10, /* number of elements (NOT size in bytes) */
! 2480: wspace, /* working space vector */
! 2481: 20); /* number of elements (NOT size in bytes) */
! 2482:
! 2483: Option bits for pcre_dfa_exec()
! 2484:
! 2485: The unused bits of the options argument for pcre_dfa_exec() must be
! 2486: zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NEW-
! 2487: LINE_xxx, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK,
! 2488: PCRE_PARTIAL, PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last
! 2489: three of these are the same as for pcre_exec(), so their description is
! 2490: not repeated here.
! 2491:
! 2492: PCRE_PARTIAL
! 2493:
! 2494: This has the same general effect as it does for pcre_exec(), but the
! 2495: details are slightly different. When PCRE_PARTIAL is set for
! 2496: pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
! 2497: PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
! 2498: been no complete matches, but there is still at least one matching pos-
! 2499: sibility. The portion of the string that provided the partial match is
! 2500: set as the first matching string.
! 2501:
! 2502: PCRE_DFA_SHORTEST
! 2503:
! 2504: Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
! 2505: stop as soon as it has found one match. Because of the way the alterna-
! 2506: tive algorithm works, this is necessarily the shortest possible match
! 2507: at the first possible matching point in the subject string.
! 2508:
! 2509: PCRE_DFA_RESTART
! 2510:
! 2511: When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
! 2512: returns a partial match, it is possible to call it again, with addi-
! 2513: tional subject characters, and have it continue with the same match.
! 2514: The PCRE_DFA_RESTART option requests this action; when it is set, the
! 2515: workspace and wscount options must reference the same vector as before
! 2516: because data about the match so far is left in them after a partial
! 2517: match. There is more discussion of this facility in the pcrepartial
! 2518: documentation.
! 2519:
! 2520: Successful returns from pcre_dfa_exec()
! 2521:
! 2522: When pcre_dfa_exec() succeeds, it may have matched more than one sub-
! 2523: string in the subject. Note, however, that all the matches from one run
! 2524: of the function start at the same point in the subject. The shorter
! 2525: matches are all initial substrings of the longer matches. For example,
! 2526: if the pattern
! 2527:
! 2528: <.*>
! 2529:
! 2530: is matched against the string
! 2531:
! 2532: This is <something> <something else> <something further> no more
! 2533:
! 2534: the three matched strings are
! 2535:
! 2536: <something>
! 2537: <something> <something else>
! 2538: <something> <something else> <something further>
! 2539:
! 2540: On success, the yield of the function is a number greater than zero,
! 2541: which is the number of matched substrings. The substrings themselves
! 2542: are returned in ovector. Each string uses two elements; the first is
! 2543: the offset to the start, and the second is the offset to the end. In
! 2544: fact, all the strings have the same start offset. (Space could have
! 2545: been saved by giving this only once, but it was decided to retain some
! 2546: compatibility with the way pcre_exec() returns data, even though the
! 2547: meaning of the strings is different.)
! 2548:
! 2549: The strings are returned in reverse order of length; that is, the long-
! 2550: est matching string is given first. If there were too many matches to
! 2551: fit into ovector, the yield of the function is zero, and the vector is
! 2552: filled with the longest matches.
! 2553:
! 2554: Error returns from pcre_dfa_exec()
! 2555:
! 2556: The pcre_dfa_exec() function returns a negative number when it fails.
! 2557: Many of the errors are the same as for pcre_exec(), and these are
! 2558: described above. There are in addition the following errors that are
! 2559: specific to pcre_dfa_exec():
! 2560:
! 2561: PCRE_ERROR_DFA_UITEM (-16)
! 2562:
! 2563: This return is given if pcre_dfa_exec() encounters an item in the pat-
! 2564: tern that it does not support, for instance, the use of \C or a back
! 2565: reference.
! 2566:
! 2567: PCRE_ERROR_DFA_UCOND (-17)
! 2568:
! 2569: This return is given if pcre_dfa_exec() encounters a condition item
! 2570: that uses a back reference for the condition, or a test for recursion
! 2571: in a specific group. These are not supported.
! 2572:
! 2573: PCRE_ERROR_DFA_UMLIMIT (-18)
! 2574:
! 2575: This return is given if pcre_dfa_exec() is called with an extra block
! 2576: that contains a setting of the match_limit field. This is not supported
! 2577: (it is meaningless).
! 2578:
! 2579: PCRE_ERROR_DFA_WSSIZE (-19)
! 2580:
! 2581: This return is given if pcre_dfa_exec() runs out of space in the
! 2582: workspace vector.
! 2583:
! 2584: PCRE_ERROR_DFA_RECURSE (-20)
! 2585:
! 2586: When a recursive subpattern is processed, the matching function calls
! 2587: itself recursively, using private vectors for ovector and workspace.
! 2588: This error is given if the output vector is not large enough. This
! 2589: should be extremely rare, as a vector of size 1000 is used.
! 2590:
! 2591:
! 2592: SEE ALSO
! 2593:
! 2594: pcrebuild(3), pcrecallout(3), pcrecpp(3)(3), pcrematching(3), pcrepar-
! 2595: tial(3), pcreposix(3), pcreprecompile(3), pcresample(3), pcrestack(3).
! 2596:
! 2597:
! 2598: AUTHOR
! 2599:
! 2600: Philip Hazel
! 2601: University Computing Service
! 2602: Cambridge CB2 3QH, England.
! 2603:
! 2604:
! 2605: REVISION
! 2606:
! 2607: Last updated: 12 April 2008
! 2608: Copyright (c) 1997-2008 University of Cambridge.
! 2609: ------------------------------------------------------------------------------
! 2610:
! 2611:
! 2612: PCRECALLOUT(3) PCRECALLOUT(3)
! 2613:
! 2614:
! 2615: NAME
! 2616: PCRE - Perl-compatible regular expressions
! 2617:
! 2618:
! 2619: PCRE CALLOUTS
! 2620:
! 2621: int (*pcre_callout)(pcre_callout_block *);
! 2622:
! 2623: PCRE provides a feature called "callout", which is a means of temporar-
! 2624: ily passing control to the caller of PCRE in the middle of pattern
! 2625: matching. The caller of PCRE provides an external function by putting
! 2626: its entry point in the global variable pcre_callout. By default, this
! 2627: variable contains NULL, which disables all calling out.
! 2628:
! 2629: Within a regular expression, (?C) indicates the points at which the
! 2630: external function is to be called. Different callout points can be
! 2631: identified by putting a number less than 256 after the letter C. The
! 2632: default value is zero. For example, this pattern has two callout
! 2633: points:
! 2634:
! 2635: (?C1)abc(?C2)def
! 2636:
! 2637: If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
! 2638: called, PCRE automatically inserts callouts, all with number 255,
! 2639: before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
! 2640: used with the pattern
! 2641:
! 2642: A(\d{2}|--)
! 2643:
! 2644: it is processed as if it were
! 2645:
! 2646: (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
! 2647:
! 2648: Notice that there is a callout before and after each parenthesis and
! 2649: alternation bar. Automatic callouts can be used for tracking the
! 2650: progress of pattern matching. The pcretest command has an option that
! 2651: sets automatic callouts; when it is used, the output indicates how the
! 2652: pattern is matched. This is useful information when you are trying to
! 2653: optimize the performance of a particular pattern.
! 2654:
! 2655:
! 2656: MISSING CALLOUTS
! 2657:
! 2658: You should be aware that, because of optimizations in the way PCRE
! 2659: matches patterns, callouts sometimes do not happen. For example, if the
! 2660: pattern is
! 2661:
! 2662: ab(?C4)cd
! 2663:
! 2664: PCRE knows that any matching string must contain the letter "d". If the
! 2665: subject string is "abyz", the lack of "d" means that matching doesn't
! 2666: ever start, and the callout is never reached. However, with "abyd",
! 2667: though the result is still no match, the callout is obeyed.
! 2668:
! 2669:
! 2670: THE CALLOUT INTERFACE
! 2671:
! 2672: During matching, when PCRE reaches a callout point, the external func-
! 2673: tion defined by pcre_callout is called (if it is set). This applies to
! 2674: both the pcre_exec() and the pcre_dfa_exec() matching functions. The
! 2675: only argument to the callout function is a pointer to a pcre_callout
! 2676: block. This structure contains the following fields:
! 2677:
! 2678: int version;
! 2679: int callout_number;
! 2680: int *offset_vector;
! 2681: const char *subject;
! 2682: int subject_length;
! 2683: int start_match;
! 2684: int current_position;
! 2685: int capture_top;
! 2686: int capture_last;
! 2687: void *callout_data;
! 2688: int pattern_position;
! 2689: int next_item_length;
! 2690:
! 2691: The version field is an integer containing the version number of the
! 2692: block format. The initial version was 0; the current version is 1. The
! 2693: version number will change again in future if additional fields are
! 2694: added, but the intention is never to remove any of the existing fields.
! 2695:
! 2696: The callout_number field contains the number of the callout, as com-
! 2697: piled into the pattern (that is, the number after ?C for manual call-
! 2698: outs, and 255 for automatically generated callouts).
! 2699:
! 2700: The offset_vector field is a pointer to the vector of offsets that was
! 2701: passed by the caller to pcre_exec() or pcre_dfa_exec(). When
! 2702: pcre_exec() is used, the contents can be inspected in order to extract
! 2703: substrings that have been matched so far, in the same way as for
! 2704: extracting substrings after a match has completed. For pcre_dfa_exec()
! 2705: this field is not useful.
! 2706:
! 2707: The subject and subject_length fields contain copies of the values that
! 2708: were passed to pcre_exec().
! 2709:
! 2710: The start_match field normally contains the offset within the subject
! 2711: at which the current match attempt started. However, if the escape
! 2712: sequence \K has been encountered, this value is changed to reflect the
! 2713: modified starting point. If the pattern is not anchored, the callout
! 2714: function may be called several times from the same point in the pattern
! 2715: for different starting points in the subject.
! 2716:
! 2717: The current_position field contains the offset within the subject of
! 2718: the current match pointer.
! 2719:
! 2720: When the pcre_exec() function is used, the capture_top field contains
! 2721: one more than the number of the highest numbered captured substring so
! 2722: far. If no substrings have been captured, the value of capture_top is
! 2723: one. This is always the case when pcre_dfa_exec() is used, because it
! 2724: does not support captured substrings.
! 2725:
! 2726: The capture_last field contains the number of the most recently cap-
! 2727: tured substring. If no substrings have been captured, its value is -1.
! 2728: This is always the case when pcre_dfa_exec() is used.
! 2729:
! 2730: The callout_data field contains a value that is passed to pcre_exec()
! 2731: or pcre_dfa_exec() specifically so that it can be passed back in call-
! 2732: outs. It is passed in the pcre_callout field of the pcre_extra data
! 2733: structure. If no such data was passed, the value of callout_data in a
! 2734: pcre_callout block is NULL. There is a description of the pcre_extra
! 2735: structure in the pcreapi documentation.
! 2736:
! 2737: The pattern_position field is present from version 1 of the pcre_call-
! 2738: out structure. It contains the offset to the next item to be matched in
! 2739: the pattern string.
! 2740:
! 2741: The next_item_length field is present from version 1 of the pcre_call-
! 2742: out structure. It contains the length of the next item to be matched in
! 2743: the pattern string. When the callout immediately precedes an alterna-
! 2744: tion bar, a closing parenthesis, or the end of the pattern, the length
! 2745: is zero. When the callout precedes an opening parenthesis, the length
! 2746: is that of the entire subpattern.
! 2747:
! 2748: The pattern_position and next_item_length fields are intended to help
! 2749: in distinguishing between different automatic callouts, which all have
! 2750: the same callout number. However, they are set for all callouts.
! 2751:
! 2752:
! 2753: RETURN VALUES
! 2754:
! 2755: The external callout function returns an integer to PCRE. If the value
! 2756: is zero, matching proceeds as normal. If the value is greater than
! 2757: zero, matching fails at the current point, but the testing of other
! 2758: matching possibilities goes ahead, just as if a lookahead assertion had
! 2759: failed. If the value is less than zero, the match is abandoned, and
! 2760: pcre_exec() (or pcre_dfa_exec()) returns the negative value.
! 2761:
! 2762: Negative values should normally be chosen from the set of
! 2763: PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
! 2764: dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
! 2765: reserved for use by callout functions; it will never be used by PCRE
! 2766: itself.
! 2767:
! 2768:
! 2769: AUTHOR
! 2770:
! 2771: Philip Hazel
! 2772: University Computing Service
! 2773: Cambridge CB2 3QH, England.
! 2774:
! 2775:
! 2776: REVISION
! 2777:
! 2778: Last updated: 29 May 2007
! 2779: Copyright (c) 1997-2007 University of Cambridge.
! 2780: ------------------------------------------------------------------------------
! 2781:
! 2782:
! 2783: PCRECOMPAT(3) PCRECOMPAT(3)
! 2784:
! 2785:
! 2786: NAME
! 2787: PCRE - Perl-compatible regular expressions
! 2788:
! 2789:
! 2790: DIFFERENCES BETWEEN PCRE AND PERL
! 2791:
! 2792: This document describes the differences in the ways that PCRE and Perl
! 2793: handle regular expressions. The differences described here are mainly
! 2794: with respect to Perl 5.8, though PCRE versions 7.0 and later contain
! 2795: some features that are expected to be in the forthcoming Perl 5.10.
! 2796:
! 2797: 1. PCRE has only a subset of Perl's UTF-8 and Unicode support. Details
! 2798: of what it does have are given in the section on UTF-8 support in the
! 2799: main pcre page.
! 2800:
! 2801: 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
! 2802: permits them, but they do not mean what you might think. For example,
! 2803: (?!a){3} does not assert that the next three characters are not "a". It
! 2804: just asserts that the next character is not "a" three times.
! 2805:
! 2806: 3. Capturing subpatterns that occur inside negative lookahead asser-
! 2807: tions are counted, but their entries in the offsets vector are never
! 2808: set. Perl sets its numerical variables from any such patterns that are
! 2809: matched before the assertion fails to match something (thereby succeed-
! 2810: ing), but only if the negative lookahead assertion contains just one
! 2811: branch.
! 2812:
! 2813: 4. Though binary zero characters are supported in the subject string,
! 2814: they are not allowed in a pattern string because it is passed as a nor-
! 2815: mal C string, terminated by zero. The escape sequence \0 can be used in
! 2816: the pattern to represent a binary zero.
! 2817:
! 2818: 5. The following Perl escape sequences are not supported: \l, \u, \L,
! 2819: \U, and \N. In fact these are implemented by Perl's general string-han-
! 2820: dling and are not part of its pattern matching engine. If any of these
! 2821: are encountered by PCRE, an error is generated.
! 2822:
! 2823: 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
! 2824: is built with Unicode character property support. The properties that
! 2825: can be tested with \p and \P are limited to the general category prop-
! 2826: erties such as Lu and Nd, script names such as Greek or Han, and the
! 2827: derived properties Any and L&.
! 2828:
! 2829: 7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
! 2830: ters in between are treated as literals. This is slightly different
! 2831: from Perl in that $ and @ are also handled as literals inside the
! 2832: quotes. In Perl, they cause variable interpolation (but of course PCRE
! 2833: does not have variables). Note the following examples:
! 2834:
! 2835: Pattern PCRE matches Perl matches
! 2836:
! 2837: \Qabc$xyz\E abc$xyz abc followed by the
! 2838: contents of $xyz
! 2839: \Qabc\$xyz\E abc\$xyz abc\$xyz
! 2840: \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
! 2841:
! 2842: The \Q...\E sequence is recognized both inside and outside character
! 2843: classes.
! 2844:
! 2845: 8. Fairly obviously, PCRE does not support the (?{code}) and (??{code})
! 2846: constructions. However, there is support for recursive patterns. This
! 2847: is not available in Perl 5.8, but will be in Perl 5.10. Also, the PCRE
! 2848: "callout" feature allows an external function to be called during pat-
! 2849: tern matching. See the pcrecallout documentation for details.
! 2850:
! 2851: 9. Subpatterns that are called recursively or as "subroutines" are
! 2852: always treated as atomic groups in PCRE. This is like Python, but
! 2853: unlike Perl.
! 2854:
! 2855: 10. There are some differences that are concerned with the settings of
! 2856: captured strings when part of a pattern is repeated. For example,
! 2857: matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
! 2858: unset, but in PCRE it is set to "b".
! 2859:
! 2860: 11. PCRE does support Perl 5.10's backtracking verbs (*ACCEPT),
! 2861: (*FAIL), (*F), (*COMMIT), (*PRUNE), (*SKIP), and (*THEN), but only in
! 2862: the forms without an argument. PCRE does not support (*MARK). If
! 2863: (*ACCEPT) is within capturing parentheses, PCRE does not set that cap-
! 2864: ture group; this is different to Perl.
! 2865:
! 2866: 12. PCRE provides some extensions to the Perl regular expression facil-
! 2867: ities. Perl 5.10 will include new features that are not in earlier
! 2868: versions, some of which (such as named parentheses) have been in PCRE
! 2869: for some time. This list is with respect to Perl 5.10:
! 2870:
! 2871: (a) Although lookbehind assertions must match fixed length strings,
! 2872: each alternative branch of a lookbehind assertion can match a different
! 2873: length of string. Perl requires them all to have the same length.
! 2874:
! 2875: (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
! 2876: meta-character matches only at the very end of the string.
! 2877:
! 2878: (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
! 2879: cial meaning is faulted. Otherwise, like Perl, the backslash is quietly
! 2880: ignored. (Perl can be made to issue a warning.)
! 2881:
! 2882: (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
! 2883: fiers is inverted, that is, by default they are not greedy, but if fol-
! 2884: lowed by a question mark they are.
! 2885:
! 2886: (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
! 2887: tried only at the first matching position in the subject string.
! 2888:
! 2889: (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
! 2890: TURE options for pcre_exec() have no Perl equivalents.
! 2891:
! 2892: (g) The \R escape sequence can be restricted to match only CR, LF, or
! 2893: CRLF by the PCRE_BSR_ANYCRLF option.
! 2894:
! 2895: (h) The callout facility is PCRE-specific.
! 2896:
! 2897: (i) The partial matching facility is PCRE-specific.
! 2898:
! 2899: (j) Patterns compiled by PCRE can be saved and re-used at a later time,
! 2900: even on different hosts that have the other endianness.
! 2901:
! 2902: (k) The alternative matching function (pcre_dfa_exec()) matches in a
! 2903: different way and is not Perl-compatible.
! 2904:
! 2905: (l) PCRE recognizes some special sequences such as (*CR) at the start
! 2906: of a pattern that set overall options that cannot be changed within the
! 2907: pattern.
! 2908:
! 2909:
! 2910: AUTHOR
! 2911:
! 2912: Philip Hazel
! 2913: University Computing Service
! 2914: Cambridge CB2 3QH, England.
! 2915:
! 2916:
! 2917: REVISION
! 2918:
! 2919: Last updated: 11 September 2007
! 2920: Copyright (c) 1997-2007 University of Cambridge.
! 2921: ------------------------------------------------------------------------------
! 2922:
! 2923:
! 2924: PCREPATTERN(3) PCREPATTERN(3)
! 2925:
! 2926:
! 2927: NAME
! 2928: PCRE - Perl-compatible regular expressions
! 2929:
! 2930:
! 2931: PCRE REGULAR EXPRESSION DETAILS
! 2932:
! 2933: The syntax and semantics of the regular expressions that are supported
! 2934: by PCRE are described in detail below. There is a quick-reference syn-
! 2935: tax summary in the pcresyntax page. PCRE tries to match Perl syntax and
! 2936: semantics as closely as it can. PCRE also supports some alternative
! 2937: regular expression syntax (which does not conflict with the Perl syn-
! 2938: tax) in order to provide some compatibility with regular expressions in
! 2939: Python, .NET, and Oniguruma.
! 2940:
! 2941: Perl's regular expressions are described in its own documentation, and
! 2942: regular expressions in general are covered in a number of books, some
! 2943: of which have copious examples. Jeffrey Friedl's "Mastering Regular
! 2944: Expressions", published by O'Reilly, covers regular expressions in
! 2945: great detail. This description of PCRE's regular expressions is
! 2946: intended as reference material.
! 2947:
! 2948: The original operation of PCRE was on strings of one-byte characters.
! 2949: However, there is now also support for UTF-8 character strings. To use
! 2950: this, you must build PCRE to include UTF-8 support, and then call
! 2951: pcre_compile() with the PCRE_UTF8 option. How this affects pattern
! 2952: matching is mentioned in several places below. There is also a summary
! 2953: of UTF-8 features in the section on UTF-8 support in the main pcre
! 2954: page.
! 2955:
! 2956: The remainder of this document discusses the patterns that are sup-
! 2957: ported by PCRE when its main matching function, pcre_exec(), is used.
! 2958: From release 6.0, PCRE offers a second matching function,
! 2959: pcre_dfa_exec(), which matches using a different algorithm that is not
! 2960: Perl-compatible. Some of the features discussed below are not available
! 2961: when pcre_dfa_exec() is used. The advantages and disadvantages of the
! 2962: alternative function, and how it differs from the normal function, are
! 2963: discussed in the pcrematching page.
! 2964:
! 2965:
! 2966: NEWLINE CONVENTIONS
! 2967:
! 2968: PCRE supports five different conventions for indicating line breaks in
! 2969: strings: a single CR (carriage return) character, a single LF (line-
! 2970: feed) character, the two-character sequence CRLF, any of the three pre-
! 2971: ceding, or any Unicode newline sequence. The pcreapi page has further
! 2972: discussion about newlines, and shows how to set the newline convention
! 2973: in the options arguments for the compiling and matching functions.
! 2974:
! 2975: It is also possible to specify a newline convention by starting a pat-
! 2976: tern string with one of the following five sequences:
! 2977:
! 2978: (*CR) carriage return
! 2979: (*LF) linefeed
! 2980: (*CRLF) carriage return, followed by linefeed
! 2981: (*ANYCRLF) any of the three above
! 2982: (*ANY) all Unicode newline sequences
! 2983:
! 2984: These override the default and the options given to pcre_compile(). For
! 2985: example, on a Unix system where LF is the default newline sequence, the
! 2986: pattern
! 2987:
! 2988: (*CR)a.b
! 2989:
! 2990: changes the convention to CR. That pattern matches "a\nb" because LF is
! 2991: no longer a newline. Note that these special settings, which are not
! 2992: Perl-compatible, are recognized only at the very start of a pattern,
! 2993: and that they must be in upper case. If more than one of them is
! 2994: present, the last one is used.
! 2995:
! 2996: The newline convention does not affect what the \R escape sequence
! 2997: matches. By default, this is any Unicode newline sequence, for Perl
! 2998: compatibility. However, this can be changed; see the description of \R
! 2999: in the section entitled "Newline sequences" below. A change of \R set-
! 3000: ting can be combined with a change of newline convention.
! 3001:
! 3002:
! 3003: CHARACTERS AND METACHARACTERS
! 3004:
! 3005: A regular expression is a pattern that is matched against a subject
! 3006: string from left to right. Most characters stand for themselves in a
! 3007: pattern, and match the corresponding characters in the subject. As a
! 3008: trivial example, the pattern
! 3009:
! 3010: The quick brown fox
! 3011:
! 3012: matches a portion of a subject string that is identical to itself. When
! 3013: caseless matching is specified (the PCRE_CASELESS option), letters are
! 3014: matched independently of case. In UTF-8 mode, PCRE always understands
! 3015: the concept of case for characters whose values are less than 128, so
! 3016: caseless matching is always possible. For characters with higher val-
! 3017: ues, the concept of case is supported if PCRE is compiled with Unicode
! 3018: property support, but not otherwise. If you want to use caseless
! 3019: matching for characters 128 and above, you must ensure that PCRE is
! 3020: compiled with Unicode property support as well as with UTF-8 support.
! 3021:
! 3022: The power of regular expressions comes from the ability to include
! 3023: alternatives and repetitions in the pattern. These are encoded in the
! 3024: pattern by the use of metacharacters, which do not stand for themselves
! 3025: but instead are interpreted in some special way.
! 3026:
! 3027: There are two different sets of metacharacters: those that are recog-
! 3028: nized anywhere in the pattern except within square brackets, and those
! 3029: that are recognized within square brackets. Outside square brackets,
! 3030: the metacharacters are as follows:
! 3031:
! 3032: \ general escape character with several uses
! 3033: ^ assert start of string (or line, in multiline mode)
! 3034: $ assert end of string (or line, in multiline mode)
! 3035: . match any character except newline (by default)
! 3036: [ start character class definition
! 3037: | start of alternative branch
! 3038: ( start subpattern
! 3039: ) end subpattern
! 3040: ? extends the meaning of (
! 3041: also 0 or 1 quantifier
! 3042: also quantifier minimizer
! 3043: * 0 or more quantifier
! 3044: + 1 or more quantifier
! 3045: also "possessive quantifier"
! 3046: { start min/max quantifier
! 3047:
! 3048: Part of a pattern that is in square brackets is called a "character
! 3049: class". In a character class the only metacharacters are:
! 3050:
! 3051: \ general escape character
! 3052: ^ negate the class, but only if the first character
! 3053: - indicates character range
! 3054: [ POSIX character class (only if followed by POSIX
! 3055: syntax)
! 3056: ] terminates the character class
! 3057:
! 3058: The following sections describe the use of each of the metacharacters.
! 3059:
! 3060:
! 3061: BACKSLASH
! 3062:
! 3063: The backslash character has several uses. Firstly, if it is followed by
! 3064: a non-alphanumeric character, it takes away any special meaning that
! 3065: character may have. This use of backslash as an escape character
! 3066: applies both inside and outside character classes.
! 3067:
! 3068: For example, if you want to match a * character, you write \* in the
! 3069: pattern. This escaping action applies whether or not the following
! 3070: character would otherwise be interpreted as a metacharacter, so it is
! 3071: always safe to precede a non-alphanumeric with backslash to specify
! 3072: that it stands for itself. In particular, if you want to match a back-
! 3073: slash, you write \\.
! 3074:
! 3075: If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
! 3076: the pattern (other than in a character class) and characters between a
! 3077: # outside a character class and the next newline are ignored. An escap-
! 3078: ing backslash can be used to include a whitespace or # character as
! 3079: part of the pattern.
! 3080:
! 3081: If you want to remove the special meaning from a sequence of charac-
! 3082: ters, you can do so by putting them between \Q and \E. This is differ-
! 3083: ent from Perl in that $ and @ are handled as literals in \Q...\E
! 3084: sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
! 3085: tion. Note the following examples:
! 3086:
! 3087: Pattern PCRE matches Perl matches
! 3088:
! 3089: \Qabc$xyz\E abc$xyz abc followed by the
! 3090: contents of $xyz
! 3091: \Qabc\$xyz\E abc\$xyz abc\$xyz
! 3092: \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
! 3093:
! 3094: The \Q...\E sequence is recognized both inside and outside character
! 3095: classes.
! 3096:
! 3097: Non-printing characters
! 3098:
! 3099: A second use of backslash provides a way of encoding non-printing char-
! 3100: acters in patterns in a visible manner. There is no restriction on the
! 3101: appearance of non-printing characters, apart from the binary zero that
! 3102: terminates a pattern, but when a pattern is being prepared by text
! 3103: editing, it is usually easier to use one of the following escape
! 3104: sequences than the binary character it represents:
! 3105:
! 3106: \a alarm, that is, the BEL character (hex 07)
! 3107: \cx "control-x", where x is any character
! 3108: \e escape (hex 1B)
! 3109: \f formfeed (hex 0C)
! 3110: \n linefeed (hex 0A)
! 3111: \r carriage return (hex 0D)
! 3112: \t tab (hex 09)
! 3113: \ddd character with octal code ddd, or backreference
! 3114: \xhh character with hex code hh
! 3115: \x{hhh..} character with hex code hhh..
! 3116:
! 3117: The precise effect of \cx is as follows: if x is a lower case letter,
! 3118: it is converted to upper case. Then bit 6 of the character (hex 40) is
! 3119: inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
! 3120: becomes hex 7B.
! 3121:
! 3122: After \x, from zero to two hexadecimal digits are read (letters can be
! 3123: in upper or lower case). Any number of hexadecimal digits may appear
! 3124: between \x{ and }, but the value of the character code must be less
! 3125: than 256 in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is,
! 3126: the maximum value in hexadecimal is 7FFFFFFF. Note that this is bigger
! 3127: than the largest Unicode code point, which is 10FFFF.
! 3128:
! 3129: If characters other than hexadecimal digits appear between \x{ and },
! 3130: or if there is no terminating }, this form of escape is not recognized.
! 3131: Instead, the initial \x will be interpreted as a basic hexadecimal
! 3132: escape, with no following digits, giving a character whose value is
! 3133: zero.
! 3134:
! 3135: Characters whose value is less than 256 can be defined by either of the
! 3136: two syntaxes for \x. There is no difference in the way they are han-
! 3137: dled. For example, \xdc is exactly the same as \x{dc}.
! 3138:
! 3139: After \0 up to two further octal digits are read. If there are fewer
! 3140: than two digits, just those that are present are used. Thus the
! 3141: sequence \0\x\07 specifies two binary zeros followed by a BEL character
! 3142: (code value 7). Make sure you supply two digits after the initial zero
! 3143: if the pattern character that follows is itself an octal digit.
! 3144:
! 3145: The handling of a backslash followed by a digit other than 0 is compli-
! 3146: cated. Outside a character class, PCRE reads it and any following dig-
! 3147: its as a decimal number. If the number is less than 10, or if there
! 3148: have been at least that many previous capturing left parentheses in the
! 3149: expression, the entire sequence is taken as a back reference. A
! 3150: description of how this works is given later, following the discussion
! 3151: of parenthesized subpatterns.
! 3152:
! 3153: Inside a character class, or if the decimal number is greater than 9
! 3154: and there have not been that many capturing subpatterns, PCRE re-reads
! 3155: up to three octal digits following the backslash, and uses them to gen-
! 3156: erate a data character. Any subsequent digits stand for themselves. In
! 3157: non-UTF-8 mode, the value of a character specified in octal must be
! 3158: less than \400. In UTF-8 mode, values up to \777 are permitted. For
! 3159: example:
! 3160:
! 3161: \040 is another way of writing a space
! 3162: \40 is the same, provided there are fewer than 40
! 3163: previous capturing subpatterns
! 3164: \7 is always a back reference
! 3165: \11 might be a back reference, or another way of
! 3166: writing a tab
! 3167: \011 is always a tab
! 3168: \0113 is a tab followed by the character "3"
! 3169: \113 might be a back reference, otherwise the
! 3170: character with octal code 113
! 3171: \377 might be a back reference, otherwise
! 3172: the byte consisting entirely of 1 bits
! 3173: \81 is either a back reference, or a binary zero
! 3174: followed by the two characters "8" and "1"
! 3175:
! 3176: Note that octal values of 100 or greater must not be introduced by a
! 3177: leading zero, because no more than three octal digits are ever read.
! 3178:
! 3179: All the sequences that define a single character value can be used both
! 3180: inside and outside character classes. In addition, inside a character
! 3181: class, the sequence \b is interpreted as the backspace character (hex
! 3182: 08), and the sequences \R and \X are interpreted as the characters "R"
! 3183: and "X", respectively. Outside a character class, these sequences have
! 3184: different meanings (see below).
! 3185:
! 3186: Absolute and relative back references
! 3187:
! 3188: The sequence \g followed by an unsigned or a negative number, option-
! 3189: ally enclosed in braces, is an absolute or relative back reference. A
! 3190: named back reference can be coded as \g{name}. Back references are dis-
! 3191: cussed later, following the discussion of parenthesized subpatterns.
! 3192:
! 3193: Absolute and relative subroutine calls
! 3194:
! 3195: For compatibility with Oniguruma, the non-Perl syntax \g followed by a
! 3196: name or a number enclosed either in angle brackets or single quotes, is
! 3197: an alternative syntax for referencing a subpattern as a "subroutine".
! 3198: Details are discussed later. Note that \g{...} (Perl syntax) and
! 3199: \g<...> (Oniguruma syntax) are not synonymous. The former is a back
! 3200: reference; the latter is a subroutine call.
! 3201:
! 3202: Generic character types
! 3203:
! 3204: Another use of backslash is for specifying generic character types. The
! 3205: following are always recognized:
! 3206:
! 3207: \d any decimal digit
! 3208: \D any character that is not a decimal digit
! 3209: \h any horizontal whitespace character
! 3210: \H any character that is not a horizontal whitespace character
! 3211: \s any whitespace character
! 3212: \S any character that is not a whitespace character
! 3213: \v any vertical whitespace character
! 3214: \V any character that is not a vertical whitespace character
! 3215: \w any "word" character
! 3216: \W any "non-word" character
! 3217:
! 3218: Each pair of escape sequences partitions the complete set of characters
! 3219: into two disjoint sets. Any given character matches one, and only one,
! 3220: of each pair.
! 3221:
! 3222: These character type sequences can appear both inside and outside char-
! 3223: acter classes. They each match one character of the appropriate type.
! 3224: If the current matching point is at the end of the subject string, all
! 3225: of them fail, since there is no character to match.
! 3226:
! 3227: For compatibility with Perl, \s does not match the VT character (code
! 3228: 11). This makes it different from the the POSIX "space" class. The \s
! 3229: characters are HT (9), LF (10), FF (12), CR (13), and space (32). If
! 3230: "use locale;" is included in a Perl script, \s may match the VT charac-
! 3231: ter. In PCRE, it never does.
! 3232:
! 3233: In UTF-8 mode, characters with values greater than 128 never match \d,
! 3234: \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
! 3235: code character property support is available. These sequences retain
! 3236: their original meanings from before UTF-8 support was available, mainly
! 3237: for efficiency reasons.
! 3238:
! 3239: The sequences \h, \H, \v, and \V are Perl 5.10 features. In contrast to
! 3240: the other sequences, these do match certain high-valued codepoints in
! 3241: UTF-8 mode. The horizontal space characters are:
! 3242:
! 3243: U+0009 Horizontal tab
! 3244: U+0020 Space
! 3245: U+00A0 Non-break space
! 3246: U+1680 Ogham space mark
! 3247: U+180E Mongolian vowel separator
! 3248: U+2000 En quad
! 3249: U+2001 Em quad
! 3250: U+2002 En space
! 3251: U+2003 Em space
! 3252: U+2004 Three-per-em space
! 3253: U+2005 Four-per-em space
! 3254: U+2006 Six-per-em space
! 3255: U+2007 Figure space
! 3256: U+2008 Punctuation space
! 3257: U+2009 Thin space
! 3258: U+200A Hair space
! 3259: U+202F Narrow no-break space
! 3260: U+205F Medium mathematical space
! 3261: U+3000 Ideographic space
! 3262:
! 3263: The vertical space characters are:
! 3264:
! 3265: U+000A Linefeed
! 3266: U+000B Vertical tab
! 3267: U+000C Formfeed
! 3268: U+000D Carriage return
! 3269: U+0085 Next line
! 3270: U+2028 Line separator
! 3271: U+2029 Paragraph separator
! 3272:
! 3273: A "word" character is an underscore or any character less than 256 that
! 3274: is a letter or digit. The definition of letters and digits is con-
! 3275: trolled by PCRE's low-valued character tables, and may vary if locale-
! 3276: specific matching is taking place (see "Locale support" in the pcreapi
! 3277: page). For example, in a French locale such as "fr_FR" in Unix-like
! 3278: systems, or "french" in Windows, some character codes greater than 128
! 3279: are used for accented letters, and these are matched by \w. The use of
! 3280: locales with Unicode is discouraged.
! 3281:
! 3282: Newline sequences
! 3283:
! 3284: Outside a character class, by default, the escape sequence \R matches
! 3285: any Unicode newline sequence. This is a Perl 5.10 feature. In non-UTF-8
! 3286: mode \R is equivalent to the following:
! 3287:
! 3288: (?>\r\n|\n|\x0b|\f|\r|\x85)
! 3289:
! 3290: This is an example of an "atomic group", details of which are given
! 3291: below. This particular group matches either the two-character sequence
! 3292: CR followed by LF, or one of the single characters LF (linefeed,
! 3293: U+000A), VT (vertical tab, U+000B), FF (formfeed, U+000C), CR (carriage
! 3294: return, U+000D), or NEL (next line, U+0085). The two-character sequence
! 3295: is treated as a single unit that cannot be split.
! 3296:
! 3297: In UTF-8 mode, two additional characters whose codepoints are greater
! 3298: than 255 are added: LS (line separator, U+2028) and PS (paragraph sepa-
! 3299: rator, U+2029). Unicode character property support is not needed for
! 3300: these characters to be recognized.
! 3301:
! 3302: It is possible to restrict \R to match only CR, LF, or CRLF (instead of
! 3303: the complete set of Unicode line endings) by setting the option
! 3304: PCRE_BSR_ANYCRLF either at compile time or when the pattern is matched.
! 3305: (BSR is an abbrevation for "backslash R".) This can be made the default
! 3306: when PCRE is built; if this is the case, the other behaviour can be
! 3307: requested via the PCRE_BSR_UNICODE option. It is also possible to
! 3308: specify these settings by starting a pattern string with one of the
! 3309: following sequences:
! 3310:
! 3311: (*BSR_ANYCRLF) CR, LF, or CRLF only
! 3312: (*BSR_UNICODE) any Unicode newline sequence
! 3313:
! 3314: These override the default and the options given to pcre_compile(), but
! 3315: they can be overridden by options given to pcre_exec(). Note that these
! 3316: special settings, which are not Perl-compatible, are recognized only at
! 3317: the very start of a pattern, and that they must be in upper case. If
! 3318: more than one of them is present, the last one is used. They can be
! 3319: combined with a change of newline convention, for example, a pattern
! 3320: can start with:
! 3321:
! 3322: (*ANY)(*BSR_ANYCRLF)
! 3323:
! 3324: Inside a character class, \R matches the letter "R".
! 3325:
! 3326: Unicode character properties
! 3327:
! 3328: When PCRE is built with Unicode character property support, three addi-
! 3329: tional escape sequences that match characters with specific properties
! 3330: are available. When not in UTF-8 mode, these sequences are of course
! 3331: limited to testing characters whose codepoints are less than 256, but
! 3332: they do work in this mode. The extra escape sequences are:
! 3333:
! 3334: \p{xx} a character with the xx property
! 3335: \P{xx} a character without the xx property
! 3336: \X an extended Unicode sequence
! 3337:
! 3338: The property names represented by xx above are limited to the Unicode
! 3339: script names, the general category properties, and "Any", which matches
! 3340: any character (including newline). Other properties such as "InMusical-
! 3341: Symbols" are not currently supported by PCRE. Note that \P{Any} does
! 3342: not match any characters, so always causes a match failure.
! 3343:
! 3344: Sets of Unicode characters are defined as belonging to certain scripts.
! 3345: A character from one of these sets can be matched using a script name.
! 3346: For example:
! 3347:
! 3348: \p{Greek}
! 3349: \P{Han}
! 3350:
! 3351: Those that are not part of an identified script are lumped together as
! 3352: "Common". The current list of scripts is:
! 3353:
! 3354: Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
! 3355: Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
! 3356: Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
! 3357: Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hira-
! 3358: gana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
! 3359: Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko,
! 3360: Ogham, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician,
! 3361: Runic, Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa,
! 3362: Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.
! 3363:
! 3364: Each character has exactly one general category property, specified by
! 3365: a two-letter abbreviation. For compatibility with Perl, negation can be
! 3366: specified by including a circumflex between the opening brace and the
! 3367: property name. For example, \p{^Lu} is the same as \P{Lu}.
! 3368:
! 3369: If only one letter is specified with \p or \P, it includes all the gen-
! 3370: eral category properties that start with that letter. In this case, in
! 3371: the absence of negation, the curly brackets in the escape sequence are
! 3372: optional; these two examples have the same effect:
! 3373:
! 3374: \p{L}
! 3375: \pL
! 3376:
! 3377: The following general category property codes are supported:
! 3378:
! 3379: C Other
! 3380: Cc Control
! 3381: Cf Format
! 3382: Cn Unassigned
! 3383: Co Private use
! 3384: Cs Surrogate
! 3385:
! 3386: L Letter
! 3387: Ll Lower case letter
! 3388: Lm Modifier letter
! 3389: Lo Other letter
! 3390: Lt Title case letter
! 3391: Lu Upper case letter
! 3392:
! 3393: M Mark
! 3394: Mc Spacing mark
! 3395: Me Enclosing mark
! 3396: Mn Non-spacing mark
! 3397:
! 3398: N Number
! 3399: Nd Decimal number
! 3400: Nl Letter number
! 3401: No Other number
! 3402:
! 3403: P Punctuation
! 3404: Pc Connector punctuation
! 3405: Pd Dash punctuation
! 3406: Pe Close punctuation
! 3407: Pf Final punctuation
! 3408: Pi Initial punctuation
! 3409: Po Other punctuation
! 3410: Ps Open punctuation
! 3411:
! 3412: S Symbol
! 3413: Sc Currency symbol
! 3414: Sk Modifier symbol
! 3415: Sm Mathematical symbol
! 3416: So Other symbol
! 3417:
! 3418: Z Separator
! 3419: Zl Line separator
! 3420: Zp Paragraph separator
! 3421: Zs Space separator
! 3422:
! 3423: The special property L& is also supported: it matches a character that
! 3424: has the Lu, Ll, or Lt property, in other words, a letter that is not
! 3425: classified as a modifier or "other".
! 3426:
! 3427: The Cs (Surrogate) property applies only to characters in the range
! 3428: U+D800 to U+DFFF. Such characters are not valid in UTF-8 strings (see
! 3429: RFC 3629) and so cannot be tested by PCRE, unless UTF-8 validity check-
! 3430: ing has been turned off (see the discussion of PCRE_NO_UTF8_CHECK in
! 3431: the pcreapi page).
! 3432:
! 3433: The long synonyms for these properties that Perl supports (such as
! 3434: \p{Letter}) are not supported by PCRE, nor is it permitted to prefix
! 3435: any of these properties with "Is".
! 3436:
! 3437: No character that is in the Unicode table has the Cn (unassigned) prop-
! 3438: erty. Instead, this property is assumed for any code point that is not
! 3439: in the Unicode table.
! 3440:
! 3441: Specifying caseless matching does not affect these escape sequences.
! 3442: For example, \p{Lu} always matches only upper case letters.
! 3443:
! 3444: The \X escape matches any number of Unicode characters that form an
! 3445: extended Unicode sequence. \X is equivalent to
! 3446:
! 3447: (?>\PM\pM*)
! 3448:
! 3449: That is, it matches a character without the "mark" property, followed
! 3450: by zero or more characters with the "mark" property, and treats the
! 3451: sequence as an atomic group (see below). Characters with the "mark"
! 3452: property are typically accents that affect the preceding character.
! 3453: None of them have codepoints less than 256, so in non-UTF-8 mode \X
! 3454: matches any one character.
! 3455:
! 3456: Matching characters by Unicode property is not fast, because PCRE has
! 3457: to search a structure that contains data for over fifteen thousand
! 3458: characters. That is why the traditional escape sequences such as \d and
! 3459: \w do not use Unicode properties in PCRE.
! 3460:
! 3461: Resetting the match start
! 3462:
! 3463: The escape sequence \K, which is a Perl 5.10 feature, causes any previ-
! 3464: ously matched characters not to be included in the final matched
! 3465: sequence. For example, the pattern:
! 3466:
! 3467: foo\Kbar
! 3468:
! 3469: matches "foobar", but reports that it has matched "bar". This feature
! 3470: is similar to a lookbehind assertion (described below). However, in
! 3471: this case, the part of the subject before the real match does not have
! 3472: to be of fixed length, as lookbehind assertions do. The use of \K does
! 3473: not interfere with the setting of captured substrings. For example,
! 3474: when the pattern
! 3475:
! 3476: (foo)\Kbar
! 3477:
! 3478: matches "foobar", the first substring is still set to "foo".
! 3479:
! 3480: Simple assertions
! 3481:
! 3482: The final use of backslash is for certain simple assertions. An asser-
! 3483: tion specifies a condition that has to be met at a particular point in
! 3484: a match, without consuming any characters from the subject string. The
! 3485: use of subpatterns for more complicated assertions is described below.
! 3486: The backslashed assertions are:
! 3487:
! 3488: \b matches at a word boundary
! 3489: \B matches when not at a word boundary
! 3490: \A matches at the start of the subject
! 3491: \Z matches at the end of the subject
! 3492: also matches before a newline at the end of the subject
! 3493: \z matches only at the end of the subject
! 3494: \G matches at the first matching position in the subject
! 3495:
! 3496: These assertions may not appear in character classes (but note that \b
! 3497: has a different meaning, namely the backspace character, inside a char-
! 3498: acter class).
! 3499:
! 3500: A word boundary is a position in the subject string where the current
! 3501: character and the previous character do not both match \w or \W (i.e.
! 3502: one matches \w and the other matches \W), or the start or end of the
! 3503: string if the first or last character matches \w, respectively.
! 3504:
! 3505: The \A, \Z, and \z assertions differ from the traditional circumflex
! 3506: and dollar (described in the next section) in that they only ever match
! 3507: at the very start and end of the subject string, whatever options are
! 3508: set. Thus, they are independent of multiline mode. These three asser-
! 3509: tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which
! 3510: affect only the behaviour of the circumflex and dollar metacharacters.
! 3511: However, if the startoffset argument of pcre_exec() is non-zero, indi-
! 3512: cating that matching is to start at a point other than the beginning of
! 3513: the subject, \A can never match. The difference between \Z and \z is
! 3514: that \Z matches before a newline at the end of the string as well as at
! 3515: the very end, whereas \z matches only at the end.
! 3516:
! 3517: The \G assertion is true only when the current matching position is at
! 3518: the start point of the match, as specified by the startoffset argument
! 3519: of pcre_exec(). It differs from \A when the value of startoffset is
! 3520: non-zero. By calling pcre_exec() multiple times with appropriate argu-
! 3521: ments, you can mimic Perl's /g option, and it is in this kind of imple-
! 3522: mentation where \G can be useful.
! 3523:
! 3524: Note, however, that PCRE's interpretation of \G, as the start of the
! 3525: current match, is subtly different from Perl's, which defines it as the
! 3526: end of the previous match. In Perl, these can be different when the
! 3527: previously matched string was empty. Because PCRE does just one match
! 3528: at a time, it cannot reproduce this behaviour.
! 3529:
! 3530: If all the alternatives of a pattern begin with \G, the expression is
! 3531: anchored to the starting match position, and the "anchored" flag is set
! 3532: in the compiled regular expression.
! 3533:
! 3534:
! 3535: CIRCUMFLEX AND DOLLAR
! 3536:
! 3537: Outside a character class, in the default matching mode, the circumflex
! 3538: character is an assertion that is true only if the current matching
! 3539: point is at the start of the subject string. If the startoffset argu-
! 3540: ment of pcre_exec() is non-zero, circumflex can never match if the
! 3541: PCRE_MULTILINE option is unset. Inside a character class, circumflex
! 3542: has an entirely different meaning (see below).
! 3543:
! 3544: Circumflex need not be the first character of the pattern if a number
! 3545: of alternatives are involved, but it should be the first thing in each
! 3546: alternative in which it appears if the pattern is ever to match that
! 3547: branch. If all possible alternatives start with a circumflex, that is,
! 3548: if the pattern is constrained to match only at the start of the sub-
! 3549: ject, it is said to be an "anchored" pattern. (There are also other
! 3550: constructs that can cause a pattern to be anchored.)
! 3551:
! 3552: A dollar character is an assertion that is true only if the current
! 3553: matching point is at the end of the subject string, or immediately
! 3554: before a newline at the end of the string (by default). Dollar need not
! 3555: be the last character of the pattern if a number of alternatives are
! 3556: involved, but it should be the last item in any branch in which it
! 3557: appears. Dollar has no special meaning in a character class.
! 3558:
! 3559: The meaning of dollar can be changed so that it matches only at the
! 3560: very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at
! 3561: compile time. This does not affect the \Z assertion.
! 3562:
! 3563: The meanings of the circumflex and dollar characters are changed if the
! 3564: PCRE_MULTILINE option is set. When this is the case, a circumflex
! 3565: matches immediately after internal newlines as well as at the start of
! 3566: the subject string. It does not match after a newline that ends the
! 3567: string. A dollar matches before any newlines in the string, as well as
! 3568: at the very end, when PCRE_MULTILINE is set. When newline is specified
! 3569: as the two-character sequence CRLF, isolated CR and LF characters do
! 3570: not indicate newlines.
! 3571:
! 3572: For example, the pattern /^abc$/ matches the subject string "def\nabc"
! 3573: (where \n represents a newline) in multiline mode, but not otherwise.
! 3574: Consequently, patterns that are anchored in single line mode because
! 3575: all branches start with ^ are not anchored in multiline mode, and a
! 3576: match for circumflex is possible when the startoffset argument of
! 3577: pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if
! 3578: PCRE_MULTILINE is set.
! 3579:
! 3580: Note that the sequences \A, \Z, and \z can be used to match the start
! 3581: and end of the subject in both modes, and if all branches of a pattern
! 3582: start with \A it is always anchored, whether or not PCRE_MULTILINE is
! 3583: set.
! 3584:
! 3585:
! 3586: FULL STOP (PERIOD, DOT)
! 3587:
! 3588: Outside a character class, a dot in the pattern matches any one charac-
! 3589: ter in the subject string except (by default) a character that signi-
! 3590: fies the end of a line. In UTF-8 mode, the matched character may be
! 3591: more than one byte long.
! 3592:
! 3593: When a line ending is defined as a single character, dot never matches
! 3594: that character; when the two-character sequence CRLF is used, dot does
! 3595: not match CR if it is immediately followed by LF, but otherwise it
! 3596: matches all characters (including isolated CRs and LFs). When any Uni-
! 3597: code line endings are being recognized, dot does not match CR or LF or
! 3598: any of the other line ending characters.
! 3599:
! 3600: The behaviour of dot with regard to newlines can be changed. If the
! 3601: PCRE_DOTALL option is set, a dot matches any one character, without
! 3602: exception. If the two-character sequence CRLF is present in the subject
! 3603: string, it takes two dots to match it.
! 3604:
! 3605: The handling of dot is entirely independent of the handling of circum-
! 3606: flex and dollar, the only relationship being that they both involve
! 3607: newlines. Dot has no special meaning in a character class.
! 3608:
! 3609:
! 3610: MATCHING A SINGLE BYTE
! 3611:
! 3612: Outside a character class, the escape sequence \C matches any one byte,
! 3613: both in and out of UTF-8 mode. Unlike a dot, it always matches any
! 3614: line-ending characters. The feature is provided in Perl in order to
! 3615: match individual bytes in UTF-8 mode. Because it breaks up UTF-8 char-
! 3616: acters into individual bytes, what remains in the string may be a mal-
! 3617: formed UTF-8 string. For this reason, the \C escape sequence is best
! 3618: avoided.
! 3619:
! 3620: PCRE does not allow \C to appear in lookbehind assertions (described
! 3621: below), because in UTF-8 mode this would make it impossible to calcu-
! 3622: late the length of the lookbehind.
! 3623:
! 3624:
! 3625: SQUARE BRACKETS AND CHARACTER CLASSES
! 3626:
! 3627: An opening square bracket introduces a character class, terminated by a
! 3628: closing square bracket. A closing square bracket on its own is not spe-
! 3629: cial. If a closing square bracket is required as a member of the class,
! 3630: it should be the first data character in the class (after an initial
! 3631: circumflex, if present) or escaped with a backslash.
! 3632:
! 3633: A character class matches a single character in the subject. In UTF-8
! 3634: mode, the character may occupy more than one byte. A matched character
! 3635: must be in the set of characters defined by the class, unless the first
! 3636: character in the class definition is a circumflex, in which case the
! 3637: subject character must not be in the set defined by the class. If a
! 3638: circumflex is actually required as a member of the class, ensure it is
! 3639: not the first character, or escape it with a backslash.
! 3640:
! 3641: For example, the character class [aeiou] matches any lower case vowel,
! 3642: while [^aeiou] matches any character that is not a lower case vowel.
! 3643: Note that a circumflex is just a convenient notation for specifying the
! 3644: characters that are in the class by enumerating those that are not. A
! 3645: class that starts with a circumflex is not an assertion: it still con-
! 3646: sumes a character from the subject string, and therefore it fails if
! 3647: the current pointer is at the end of the string.
! 3648:
! 3649: In UTF-8 mode, characters with values greater than 255 can be included
! 3650: in a class as a literal string of bytes, or by using the \x{ escaping
! 3651: mechanism.
! 3652:
! 3653: When caseless matching is set, any letters in a class represent both
! 3654: their upper case and lower case versions, so for example, a caseless
! 3655: [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not
! 3656: match "A", whereas a caseful version would. In UTF-8 mode, PCRE always
! 3657: understands the concept of case for characters whose values are less
! 3658: than 128, so caseless matching is always possible. For characters with
! 3659: higher values, the concept of case is supported if PCRE is compiled
! 3660: with Unicode property support, but not otherwise. If you want to use
! 3661: caseless matching for characters 128 and above, you must ensure that
! 3662: PCRE is compiled with Unicode property support as well as with UTF-8
! 3663: support.
! 3664:
! 3665: Characters that might indicate line breaks are never treated in any
! 3666: special way when matching character classes, whatever line-ending
! 3667: sequence is in use, and whatever setting of the PCRE_DOTALL and
! 3668: PCRE_MULTILINE options is used. A class such as [^a] always matches one
! 3669: of these characters.
! 3670:
! 3671: The minus (hyphen) character can be used to specify a range of charac-
! 3672: ters in a character class. For example, [d-m] matches any letter
! 3673: between d and m, inclusive. If a minus character is required in a
! 3674: class, it must be escaped with a backslash or appear in a position
! 3675: where it cannot be interpreted as indicating a range, typically as the
! 3676: first or last character in the class.
! 3677:
! 3678: It is not possible to have the literal character "]" as the end charac-
! 3679: ter of a range. A pattern such as [W-]46] is interpreted as a class of
! 3680: two characters ("W" and "-") followed by a literal string "46]", so it
! 3681: would match "W46]" or "-46]". However, if the "]" is escaped with a
! 3682: backslash it is interpreted as the end of range, so [W-\]46] is inter-
! 3683: preted as a class containing a range followed by two other characters.
! 3684: The octal or hexadecimal representation of "]" can also be used to end
! 3685: a range.
! 3686:
! 3687: Ranges operate in the collating sequence of character values. They can
! 3688: also be used for characters specified numerically, for example
! 3689: [\000-\037]. In UTF-8 mode, ranges can include characters whose values
! 3690: are greater than 255, for example [\x{100}-\x{2ff}].
! 3691:
! 3692: If a range that includes letters is used when caseless matching is set,
! 3693: it matches the letters in either case. For example, [W-c] is equivalent
! 3694: to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if
! 3695: character tables for a French locale are in use, [\xc8-\xcb] matches
! 3696: accented E characters in both cases. In UTF-8 mode, PCRE supports the
! 3697: concept of case for characters with values greater than 128 only when
! 3698: it is compiled with Unicode property support.
! 3699:
! 3700: The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear
! 3701: in a character class, and add the characters that they match to the
! 3702: class. For example, [\dABCDEF] matches any hexadecimal digit. A circum-
! 3703: flex can conveniently be used with the upper case character types to
! 3704: specify a more restricted set of characters than the matching lower
! 3705: case type. For example, the class [^\W_] matches any letter or digit,
! 3706: but not underscore.
! 3707:
! 3708: The only metacharacters that are recognized in character classes are
! 3709: backslash, hyphen (only where it can be interpreted as specifying a
! 3710: range), circumflex (only at the start), opening square bracket (only
! 3711: when it can be interpreted as introducing a POSIX class name - see the
! 3712: next section), and the terminating closing square bracket. However,
! 3713: escaping other non-alphanumeric characters does no harm.
! 3714:
! 3715:
! 3716: POSIX CHARACTER CLASSES
! 3717:
! 3718: Perl supports the POSIX notation for character classes. This uses names
! 3719: enclosed by [: and :] within the enclosing square brackets. PCRE also
! 3720: supports this notation. For example,
! 3721:
! 3722: [01[:alpha:]%]
! 3723:
! 3724: matches "0", "1", any alphabetic character, or "%". The supported class
! 3725: names are
! 3726:
! 3727: alnum letters and digits
! 3728: alpha letters
! 3729: ascii character codes 0 - 127
! 3730: blank space or tab only
! 3731: cntrl control characters
! 3732: digit decimal digits (same as \d)
! 3733: graph printing characters, excluding space
! 3734: lower lower case letters
! 3735: print printing characters, including space
! 3736: punct printing characters, excluding letters and digits
! 3737: space white space (not quite the same as \s)
! 3738: upper upper case letters
! 3739: word "word" characters (same as \w)
! 3740: xdigit hexadecimal digits
! 3741:
! 3742: The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13),
! 3743: and space (32). Notice that this list includes the VT character (code
! 3744: 11). This makes "space" different to \s, which does not include VT (for
! 3745: Perl compatibility).
! 3746:
! 3747: The name "word" is a Perl extension, and "blank" is a GNU extension
! 3748: from Perl 5.8. Another Perl extension is negation, which is indicated
! 3749: by a ^ character after the colon. For example,
! 3750:
! 3751: [12[:^digit:]]
! 3752:
! 3753: matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the
! 3754: POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but
! 3755: these are not supported, and an error is given if they are encountered.
! 3756:
! 3757: In UTF-8 mode, characters with values greater than 128 do not match any
! 3758: of the POSIX character classes.
! 3759:
! 3760:
! 3761: VERTICAL BAR
! 3762:
! 3763: Vertical bar characters are used to separate alternative patterns. For
! 3764: example, the pattern
! 3765:
! 3766: gilbert|sullivan
! 3767:
! 3768: matches either "gilbert" or "sullivan". Any number of alternatives may
! 3769: appear, and an empty alternative is permitted (matching the empty
! 3770: string). The matching process tries each alternative in turn, from left
! 3771: to right, and the first one that succeeds is used. If the alternatives
! 3772: are within a subpattern (defined below), "succeeds" means matching the
! 3773: rest of the main pattern as well as the alternative in the subpattern.
! 3774:
! 3775:
! 3776: INTERNAL OPTION SETTING
! 3777:
! 3778: The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and
! 3779: PCRE_EXTENDED options (which are Perl-compatible) can be changed from
! 3780: within the pattern by a sequence of Perl option letters enclosed
! 3781: between "(?" and ")". The option letters are
! 3782:
! 3783: i for PCRE_CASELESS
! 3784: m for PCRE_MULTILINE
! 3785: s for PCRE_DOTALL
! 3786: x for PCRE_EXTENDED
! 3787:
! 3788: For example, (?im) sets caseless, multiline matching. It is also possi-
! 3789: ble to unset these options by preceding the letter with a hyphen, and a
! 3790: combined setting and unsetting such as (?im-sx), which sets PCRE_CASE-
! 3791: LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED,
! 3792: is also permitted. If a letter appears both before and after the
! 3793: hyphen, the option is unset.
! 3794:
! 3795: The PCRE-specific options PCRE_DUPNAMES, PCRE_UNGREEDY, and PCRE_EXTRA
! 3796: can be changed in the same way as the Perl-compatible options by using
! 3797: the characters J, U and X respectively.
! 3798:
! 3799: When an option change occurs at top level (that is, not inside subpat-
! 3800: tern parentheses), the change applies to the remainder of the pattern
! 3801: that follows. If the change is placed right at the start of a pattern,
! 3802: PCRE extracts it into the global options (and it will therefore show up
! 3803: in data extracted by the pcre_fullinfo() function).
! 3804:
! 3805: An option change within a subpattern (see below for a description of
! 3806: subpatterns) affects only that part of the current pattern that follows
! 3807: it, so
! 3808:
! 3809: (a(?i)b)c
! 3810:
! 3811: matches abc and aBc and no other strings (assuming PCRE_CASELESS is not
! 3812: used). By this means, options can be made to have different settings
! 3813: in different parts of the pattern. Any changes made in one alternative
! 3814: do carry on into subsequent branches within the same subpattern. For
! 3815: example,
! 3816:
! 3817: (a(?i)b|c)
! 3818:
! 3819: matches "ab", "aB", "c", and "C", even though when matching "C" the
! 3820: first branch is abandoned before the option setting. This is because
! 3821: the effects of option settings happen at compile time. There would be
! 3822: some very weird behaviour otherwise.
! 3823:
! 3824: Note: There are other PCRE-specific options that can be set by the
! 3825: application when the compile or match functions are called. In some
! 3826: cases the pattern can contain special leading sequences to override
! 3827: what the application has set or what has been defaulted. Details are
! 3828: given in the section entitled "Newline sequences" above.
! 3829:
! 3830:
! 3831: SUBPATTERNS
! 3832:
! 3833: Subpatterns are delimited by parentheses (round brackets), which can be
! 3834: nested. Turning part of a pattern into a subpattern does two things:
! 3835:
! 3836: 1. It localizes a set of alternatives. For example, the pattern
! 3837:
! 3838: cat(aract|erpillar|)
! 3839:
! 3840: matches one of the words "cat", "cataract", or "caterpillar". Without
! 3841: the parentheses, it would match "cataract", "erpillar" or an empty
! 3842: string.
! 3843:
! 3844: 2. It sets up the subpattern as a capturing subpattern. This means
! 3845: that, when the whole pattern matches, that portion of the subject
! 3846: string that matched the subpattern is passed back to the caller via the
! 3847: ovector argument of pcre_exec(). Opening parentheses are counted from
! 3848: left to right (starting from 1) to obtain numbers for the capturing
! 3849: subpatterns.
! 3850:
! 3851: For example, if the string "the red king" is matched against the pat-
! 3852: tern
! 3853:
! 3854: the ((red|white) (king|queen))
! 3855:
! 3856: the captured substrings are "red king", "red", and "king", and are num-
! 3857: bered 1, 2, and 3, respectively.
! 3858:
! 3859: The fact that plain parentheses fulfil two functions is not always
! 3860: helpful. There are often times when a grouping subpattern is required
! 3861: without a capturing requirement. If an opening parenthesis is followed
! 3862: by a question mark and a colon, the subpattern does not do any captur-
! 3863: ing, and is not counted when computing the number of any subsequent
! 3864: capturing subpatterns. For example, if the string "the white queen" is
! 3865: matched against the pattern
! 3866:
! 3867: the ((?:red|white) (king|queen))
! 3868:
! 3869: the captured substrings are "white queen" and "queen", and are numbered
! 3870: 1 and 2. The maximum number of capturing subpatterns is 65535.
! 3871:
! 3872: As a convenient shorthand, if any option settings are required at the
! 3873: start of a non-capturing subpattern, the option letters may appear
! 3874: between the "?" and the ":". Thus the two patterns
! 3875:
! 3876: (?i:saturday|sunday)
! 3877: (?:(?i)saturday|sunday)
! 3878:
! 3879: match exactly the same set of strings. Because alternative branches are
! 3880: tried from left to right, and options are not reset until the end of
! 3881: the subpattern is reached, an option setting in one branch does affect
! 3882: subsequent branches, so the above patterns match "SUNDAY" as well as
! 3883: "Saturday".
! 3884:
! 3885:
! 3886: DUPLICATE SUBPATTERN NUMBERS
! 3887:
! 3888: Perl 5.10 introduced a feature whereby each alternative in a subpattern
! 3889: uses the same numbers for its capturing parentheses. Such a subpattern
! 3890: starts with (?| and is itself a non-capturing subpattern. For example,
! 3891: consider this pattern:
! 3892:
! 3893: (?|(Sat)ur|(Sun))day
! 3894:
! 3895: Because the two alternatives are inside a (?| group, both sets of cap-
! 3896: turing parentheses are numbered one. Thus, when the pattern matches,
! 3897: you can look at captured substring number one, whichever alternative
! 3898: matched. This construct is useful when you want to capture part, but
! 3899: not all, of one of a number of alternatives. Inside a (?| group, paren-
! 3900: theses are numbered as usual, but the number is reset at the start of
! 3901: each branch. The numbers of any capturing buffers that follow the sub-
! 3902: pattern start after the highest number used in any branch. The follow-
! 3903: ing example is taken from the Perl documentation. The numbers under-
! 3904: neath show in which buffer the captured content will be stored.
! 3905:
! 3906: # before ---------------branch-reset----------- after
! 3907: / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
! 3908: # 1 2 2 3 2 3 4
! 3909:
! 3910: A backreference or a recursive call to a numbered subpattern always
! 3911: refers to the first one in the pattern with the given number.
! 3912:
! 3913: An alternative approach to using this "branch reset" feature is to use
! 3914: duplicate named subpatterns, as described in the next section.
! 3915:
! 3916:
! 3917: NAMED SUBPATTERNS
! 3918:
! 3919: Identifying capturing parentheses by number is simple, but it can be
! 3920: very hard to keep track of the numbers in complicated regular expres-
! 3921: sions. Furthermore, if an expression is modified, the numbers may
! 3922: change. To help with this difficulty, PCRE supports the naming of sub-
! 3923: patterns. This feature was not added to Perl until release 5.10. Python
! 3924: had the feature earlier, and PCRE introduced it at release 4.0, using
! 3925: the Python syntax. PCRE now supports both the Perl and the Python syn-
! 3926: tax.
! 3927:
! 3928: In PCRE, a subpattern can be named in one of three ways: (?<name>...)
! 3929: or (?'name'...) as in Perl, or (?P<name>...) as in Python. References
! 3930: to capturing parentheses from other parts of the pattern, such as back-
! 3931: references, recursion, and conditions, can be made by name as well as
! 3932: by number.
! 3933:
! 3934: Names consist of up to 32 alphanumeric characters and underscores.
! 3935: Named capturing parentheses are still allocated numbers as well as
! 3936: names, exactly as if the names were not present. The PCRE API provides
! 3937: function calls for extracting the name-to-number translation table from
! 3938: a compiled pattern. There is also a convenience function for extracting
! 3939: a captured substring by name.
! 3940:
! 3941: By default, a name must be unique within a pattern, but it is possible
! 3942: to relax this constraint by setting the PCRE_DUPNAMES option at compile
! 3943: time. This can be useful for patterns where only one instance of the
! 3944: named parentheses can match. Suppose you want to match the name of a
! 3945: weekday, either as a 3-letter abbreviation or as the full name, and in
! 3946: both cases you want to extract the abbreviation. This pattern (ignoring
! 3947: the line breaks) does the job:
! 3948:
! 3949: (?<DN>Mon|Fri|Sun)(?:day)?|
! 3950: (?<DN>Tue)(?:sday)?|
! 3951: (?<DN>Wed)(?:nesday)?|
! 3952: (?<DN>Thu)(?:rsday)?|
! 3953: (?<DN>Sat)(?:urday)?
! 3954:
! 3955: There are five capturing substrings, but only one is ever set after a
! 3956: match. (An alternative way of solving this problem is to use a "branch
! 3957: reset" subpattern, as described in the previous section.)
! 3958:
! 3959: The convenience function for extracting the data by name returns the
! 3960: substring for the first (and in this example, the only) subpattern of
! 3961: that name that matched. This saves searching to find which numbered
! 3962: subpattern it was. If you make a reference to a non-unique named sub-
! 3963: pattern from elsewhere in the pattern, the one that corresponds to the
! 3964: lowest number is used. For further details of the interfaces for han-
! 3965: dling named subpatterns, see the pcreapi documentation.
! 3966:
! 3967:
! 3968: REPETITION
! 3969:
! 3970: Repetition is specified by quantifiers, which can follow any of the
! 3971: following items:
! 3972:
! 3973: a literal data character
! 3974: the dot metacharacter
! 3975: the \C escape sequence
! 3976: the \X escape sequence (in UTF-8 mode with Unicode properties)
! 3977: the \R escape sequence
! 3978: an escape such as \d that matches a single character
! 3979: a character class
! 3980: a back reference (see next section)
! 3981: a parenthesized subpattern (unless it is an assertion)
! 3982:
! 3983: The general repetition quantifier specifies a minimum and maximum num-
! 3984: ber of permitted matches, by giving the two numbers in curly brackets
! 3985: (braces), separated by a comma. The numbers must be less than 65536,
! 3986: and the first must be less than or equal to the second. For example:
! 3987:
! 3988: z{2,4}
! 3989:
! 3990: matches "zz", "zzz", or "zzzz". A closing brace on its own is not a
! 3991: special character. If the second number is omitted, but the comma is
! 3992: present, there is no upper limit; if the second number and the comma
! 3993: are both omitted, the quantifier specifies an exact number of required
! 3994: matches. Thus
! 3995:
! 3996: [aeiou]{3,}
! 3997:
! 3998: matches at least 3 successive vowels, but may match many more, while
! 3999:
! 4000: \d{8}
! 4001:
! 4002: matches exactly 8 digits. An opening curly bracket that appears in a
! 4003: position where a quantifier is not allowed, or one that does not match
! 4004: the syntax of a quantifier, is taken as a literal character. For exam-
! 4005: ple, {,6} is not a quantifier, but a literal string of four characters.
! 4006:
! 4007: In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to
! 4008: individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char-
! 4009: acters, each of which is represented by a two-byte sequence. Similarly,
! 4010: when Unicode property support is available, \X{3} matches three Unicode
! 4011: extended sequences, each of which may be several bytes long (and they
! 4012: may be of different lengths).
! 4013:
! 4014: The quantifier {0} is permitted, causing the expression to behave as if
! 4015: the previous item and the quantifier were not present. This may be use-
! 4016: ful for subpatterns that are referenced as subroutines from elsewhere
! 4017: in the pattern. Items other than subpatterns that have a {0} quantifier
! 4018: are omitted from the compiled pattern.
! 4019:
! 4020: For convenience, the three most common quantifiers have single-charac-
! 4021: ter abbreviations:
! 4022:
! 4023: * is equivalent to {0,}
! 4024: + is equivalent to {1,}
! 4025: ? is equivalent to {0,1}
! 4026:
! 4027: It is possible to construct infinite loops by following a subpattern
! 4028: that can match no characters with a quantifier that has no upper limit,
! 4029: for example:
! 4030:
! 4031: (a?)*
! 4032:
! 4033: Earlier versions of Perl and PCRE used to give an error at compile time
! 4034: for such patterns. However, because there are cases where this can be
! 4035: useful, such patterns are now accepted, but if any repetition of the
! 4036: subpattern does in fact match no characters, the loop is forcibly bro-
! 4037: ken.
! 4038:
! 4039: By default, the quantifiers are "greedy", that is, they match as much
! 4040: as possible (up to the maximum number of permitted times), without
! 4041: causing the rest of the pattern to fail. The classic example of where
! 4042: this gives problems is in trying to match comments in C programs. These
! 4043: appear between /* and */ and within the comment, individual * and /
! 4044: characters may appear. An attempt to match C comments by applying the
! 4045: pattern
! 4046:
! 4047: /\*.*\*/
! 4048:
! 4049: to the string
! 4050:
! 4051: /* first comment */ not comment /* second comment */
! 4052:
! 4053: fails, because it matches the entire string owing to the greediness of
! 4054: the .* item.
! 4055:
! 4056: However, if a quantifier is followed by a question mark, it ceases to
! 4057: be greedy, and instead matches the minimum number of times possible, so
! 4058: the pattern
! 4059:
! 4060: /\*.*?\*/
! 4061:
! 4062: does the right thing with the C comments. The meaning of the various
! 4063: quantifiers is not otherwise changed, just the preferred number of
! 4064: matches. Do not confuse this use of question mark with its use as a
! 4065: quantifier in its own right. Because it has two uses, it can sometimes
! 4066: appear doubled, as in
! 4067:
! 4068: \d??\d
! 4069:
! 4070: which matches one digit by preference, but can match two if that is the
! 4071: only way the rest of the pattern matches.
! 4072:
! 4073: If the PCRE_UNGREEDY option is set (an option that is not available in
! 4074: Perl), the quantifiers are not greedy by default, but individual ones
! 4075: can be made greedy by following them with a question mark. In other
! 4076: words, it inverts the default behaviour.
! 4077:
! 4078: When a parenthesized subpattern is quantified with a minimum repeat
! 4079: count that is greater than 1 or with a limited maximum, more memory is
! 4080: required for the compiled pattern, in proportion to the size of the
! 4081: minimum or maximum.
! 4082:
! 4083: If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv-
! 4084: alent to Perl's /s) is set, thus allowing the dot to match newlines,
! 4085: the pattern is implicitly anchored, because whatever follows will be
! 4086: tried against every character position in the subject string, so there
! 4087: is no point in retrying the overall match at any position after the
! 4088: first. PCRE normally treats such a pattern as though it were preceded
! 4089: by \A.
! 4090:
! 4091: In cases where it is known that the subject string contains no new-
! 4092: lines, it is worth setting PCRE_DOTALL in order to obtain this opti-
! 4093: mization, or alternatively using ^ to indicate anchoring explicitly.
! 4094:
! 4095: However, there is one situation where the optimization cannot be used.
! 4096: When .* is inside capturing parentheses that are the subject of a
! 4097: backreference elsewhere in the pattern, a match at the start may fail
! 4098: where a later one succeeds. Consider, for example:
! 4099:
! 4100: (.*)abc\1
! 4101:
! 4102: If the subject is "xyz123abc123" the match point is the fourth charac-
! 4103: ter. For this reason, such a pattern is not implicitly anchored.
! 4104:
! 4105: When a capturing subpattern is repeated, the value captured is the sub-
! 4106: string that matched the final iteration. For example, after
! 4107:
! 4108: (tweedle[dume]{3}\s*)+
! 4109:
! 4110: has matched "tweedledum tweedledee" the value of the captured substring
! 4111: is "tweedledee". However, if there are nested capturing subpatterns,
! 4112: the corresponding captured values may have been set in previous itera-
! 4113: tions. For example, after
! 4114:
! 4115: /(a|(b))+/
! 4116:
! 4117: matches "aba" the value of the second captured substring is "b".
! 4118:
! 4119:
! 4120: ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
! 4121:
! 4122: With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy")
! 4123: repetition, failure of what follows normally causes the repeated item
! 4124: to be re-evaluated to see if a different number of repeats allows the
! 4125: rest of the pattern to match. Sometimes it is useful to prevent this,
! 4126: either to change the nature of the match, or to cause it fail earlier
! 4127: than it otherwise might, when the author of the pattern knows there is
! 4128: no point in carrying on.
! 4129:
! 4130: Consider, for example, the pattern \d+foo when applied to the subject
! 4131: line
! 4132:
! 4133: 123456bar
! 4134:
! 4135: After matching all 6 digits and then failing to match "foo", the normal
! 4136: action of the matcher is to try again with only 5 digits matching the
! 4137: \d+ item, and then with 4, and so on, before ultimately failing.
! 4138: "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides
! 4139: the means for specifying that once a subpattern has matched, it is not
! 4140: to be re-evaluated in this way.
! 4141:
! 4142: If we use atomic grouping for the previous example, the matcher gives
! 4143: up immediately on failing to match "foo" the first time. The notation
! 4144: is a kind of special parenthesis, starting with (?> as in this example:
! 4145:
! 4146: (?>\d+)foo
! 4147:
! 4148: This kind of parenthesis "locks up" the part of the pattern it con-
! 4149: tains once it has matched, and a failure further into the pattern is
! 4150: prevented from backtracking into it. Backtracking past it to previous
! 4151: items, however, works as normal.
! 4152:
! 4153: An alternative description is that a subpattern of this type matches
! 4154: the string of characters that an identical standalone pattern would
! 4155: match, if anchored at the current point in the subject string.
! 4156:
! 4157: Atomic grouping subpatterns are not capturing subpatterns. Simple cases
! 4158: such as the above example can be thought of as a maximizing repeat that
! 4159: must swallow everything it can. So, while both \d+ and \d+? are pre-
! 4160: pared to adjust the number of digits they match in order to make the
! 4161: rest of the pattern match, (?>\d+) can only match an entire sequence of
! 4162: digits.
! 4163:
! 4164: Atomic groups in general can of course contain arbitrarily complicated
! 4165: subpatterns, and can be nested. However, when the subpattern for an
! 4166: atomic group is just a single repeated item, as in the example above, a
! 4167: simpler notation, called a "possessive quantifier" can be used. This
! 4168: consists of an additional + character following a quantifier. Using
! 4169: this notation, the previous example can be rewritten as
! 4170:
! 4171: \d++foo
! 4172:
! 4173: Note that a possessive quantifier can be used with an entire group, for
! 4174: example:
! 4175:
! 4176: (abc|xyz){2,3}+
! 4177:
! 4178: Possessive quantifiers are always greedy; the setting of the
! 4179: PCRE_UNGREEDY option is ignored. They are a convenient notation for the
! 4180: simpler forms of atomic group. However, there is no difference in the
! 4181: meaning of a possessive quantifier and the equivalent atomic group,
! 4182: though there may be a performance difference; possessive quantifiers
! 4183: should be slightly faster.
! 4184:
! 4185: The possessive quantifier syntax is an extension to the Perl 5.8 syn-
! 4186: tax. Jeffrey Friedl originated the idea (and the name) in the first
! 4187: edition of his book. Mike McCloskey liked it, so implemented it when he
! 4188: built Sun's Java package, and PCRE copied it from there. It ultimately
! 4189: found its way into Perl at release 5.10.
! 4190:
! 4191: PCRE has an optimization that automatically "possessifies" certain sim-
! 4192: ple pattern constructs. For example, the sequence A+B is treated as
! 4193: A++B because there is no point in backtracking into a sequence of A's
! 4194: when B must follow.
! 4195:
! 4196: When a pattern contains an unlimited repeat inside a subpattern that
! 4197: can itself be repeated an unlimited number of times, the use of an
! 4198: atomic group is the only way to avoid some failing matches taking a
! 4199: very long time indeed. The pattern
! 4200:
! 4201: (\D+|<\d+>)*[!?]
! 4202:
! 4203: matches an unlimited number of substrings that either consist of non-
! 4204: digits, or digits enclosed in <>, followed by either ! or ?. When it
! 4205: matches, it runs quickly. However, if it is applied to
! 4206:
! 4207: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
! 4208:
! 4209: it takes a long time before reporting failure. This is because the
! 4210: string can be divided between the internal \D+ repeat and the external
! 4211: * repeat in a large number of ways, and all have to be tried. (The
! 4212: example uses [!?] rather than a single character at the end, because
! 4213: both PCRE and Perl have an optimization that allows for fast failure
! 4214: when a single character is used. They remember the last single charac-
! 4215: ter that is required for a match, and fail early if it is not present
! 4216: in the string.) If the pattern is changed so that it uses an atomic
! 4217: group, like this:
! 4218:
! 4219: ((?>\D+)|<\d+>)*[!?]
! 4220:
! 4221: sequences of non-digits cannot be broken, and failure happens quickly.
! 4222:
! 4223:
! 4224: BACK REFERENCES
! 4225:
! 4226: Outside a character class, a backslash followed by a digit greater than
! 4227: 0 (and possibly further digits) is a back reference to a capturing sub-
! 4228: pattern earlier (that is, to its left) in the pattern, provided there
! 4229: have been that many previous capturing left parentheses.
! 4230:
! 4231: However, if the decimal number following the backslash is less than 10,
! 4232: it is always taken as a back reference, and causes an error only if
! 4233: there are not that many capturing left parentheses in the entire pat-
! 4234: tern. In other words, the parentheses that are referenced need not be
! 4235: to the left of the reference for numbers less than 10. A "forward back
! 4236: reference" of this type can make sense when a repetition is involved
! 4237: and the subpattern to the right has participated in an earlier itera-
! 4238: tion.
! 4239:
! 4240: It is not possible to have a numerical "forward back reference" to a
! 4241: subpattern whose number is 10 or more using this syntax because a
! 4242: sequence such as \50 is interpreted as a character defined in octal.
! 4243: See the subsection entitled "Non-printing characters" above for further
! 4244: details of the handling of digits following a backslash. There is no
! 4245: such problem when named parentheses are used. A back reference to any
! 4246: subpattern is possible using named parentheses (see below).
! 4247:
! 4248: Another way of avoiding the ambiguity inherent in the use of digits
! 4249: following a backslash is to use the \g escape sequence, which is a fea-
! 4250: ture introduced in Perl 5.10. This escape must be followed by an
! 4251: unsigned number or a negative number, optionally enclosed in braces.
! 4252: These examples are all identical:
! 4253:
! 4254: (ring), \1
! 4255: (ring), \g1
! 4256: (ring), \g{1}
! 4257:
! 4258: An unsigned number specifies an absolute reference without the ambigu-
! 4259: ity that is present in the older syntax. It is also useful when literal
! 4260: digits follow the reference. A negative number is a relative reference.
! 4261: Consider this example:
! 4262:
! 4263: (abc(def)ghi)\g{-1}
! 4264:
! 4265: The sequence \g{-1} is a reference to the most recently started captur-
! 4266: ing subpattern before \g, that is, is it equivalent to \2. Similarly,
! 4267: \g{-2} would be equivalent to \1. The use of relative references can be
! 4268: helpful in long patterns, and also in patterns that are created by
! 4269: joining together fragments that contain references within themselves.
! 4270:
! 4271: A back reference matches whatever actually matched the capturing sub-
! 4272: pattern in the current subject string, rather than anything matching
! 4273: the subpattern itself (see "Subpatterns as subroutines" below for a way
! 4274: of doing that). So the pattern
! 4275:
! 4276: (sens|respons)e and \1ibility
! 4277:
! 4278: matches "sense and sensibility" and "response and responsibility", but
! 4279: not "sense and responsibility". If caseful matching is in force at the
! 4280: time of the back reference, the case of letters is relevant. For exam-
! 4281: ple,
! 4282:
! 4283: ((?i)rah)\s+\1
! 4284:
! 4285: matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
! 4286: original capturing subpattern is matched caselessly.
! 4287:
! 4288: There are several different ways of writing back references to named
! 4289: subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or
! 4290: \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's
! 4291: unified back reference syntax, in which \g can be used for both numeric
! 4292: and named references, is also supported. We could rewrite the above
! 4293: example in any of the following ways:
! 4294:
! 4295: (?<p1>(?i)rah)\s+\k<p1>
! 4296: (?'p1'(?i)rah)\s+\k{p1}
! 4297: (?P<p1>(?i)rah)\s+(?P=p1)
! 4298: (?<p1>(?i)rah)\s+\g{p1}
! 4299:
! 4300: A subpattern that is referenced by name may appear in the pattern
! 4301: before or after the reference.
! 4302:
! 4303: There may be more than one back reference to the same subpattern. If a
! 4304: subpattern has not actually been used in a particular match, any back
! 4305: references to it always fail. For example, the pattern
! 4306:
! 4307: (a|(bc))\2
! 4308:
! 4309: always fails if it starts to match "a" rather than "bc". Because there
! 4310: may be many capturing parentheses in a pattern, all digits following
! 4311: the backslash are taken as part of a potential back reference number.
! 4312: If the pattern continues with a digit character, some delimiter must be
! 4313: used to terminate the back reference. If the PCRE_EXTENDED option is
! 4314: set, this can be whitespace. Otherwise an empty comment (see "Com-
! 4315: ments" below) can be used.
! 4316:
! 4317: A back reference that occurs inside the parentheses to which it refers
! 4318: fails when the subpattern is first used, so, for example, (a\1) never
! 4319: matches. However, such references can be useful inside repeated sub-
! 4320: patterns. For example, the pattern
! 4321:
! 4322: (a|b\1)+
! 4323:
! 4324: matches any number of "a"s and also "aba", "ababbaa" etc. At each iter-
! 4325: ation of the subpattern, the back reference matches the character
! 4326: string corresponding to the previous iteration. In order for this to
! 4327: work, the pattern must be such that the first iteration does not need
! 4328: to match the back reference. This can be done using alternation, as in
! 4329: the example above, or by a quantifier with a minimum of zero.
! 4330:
! 4331:
! 4332: ASSERTIONS
! 4333:
! 4334: An assertion is a test on the characters following or preceding the
! 4335: current matching point that does not actually consume any characters.
! 4336: The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are
! 4337: described above.
! 4338:
! 4339: More complicated assertions are coded as subpatterns. There are two
! 4340: kinds: those that look ahead of the current position in the subject
! 4341: string, and those that look behind it. An assertion subpattern is
! 4342: matched in the normal way, except that it does not cause the current
! 4343: matching position to be changed.
! 4344:
! 4345: Assertion subpatterns are not capturing subpatterns, and may not be
! 4346: repeated, because it makes no sense to assert the same thing several
! 4347: times. If any kind of assertion contains capturing subpatterns within
! 4348: it, these are counted for the purposes of numbering the capturing sub-
! 4349: patterns in the whole pattern. However, substring capturing is carried
! 4350: out only for positive assertions, because it does not make sense for
! 4351: negative assertions.
! 4352:
! 4353: Lookahead assertions
! 4354:
! 4355: Lookahead assertions start with (?= for positive assertions and (?! for
! 4356: negative assertions. For example,
! 4357:
! 4358: \w+(?=;)
! 4359:
! 4360: matches a word followed by a semicolon, but does not include the semi-
! 4361: colon in the match, and
! 4362:
! 4363: foo(?!bar)
! 4364:
! 4365: matches any occurrence of "foo" that is not followed by "bar". Note
! 4366: that the apparently similar pattern
! 4367:
! 4368: (?!foo)bar
! 4369:
! 4370: does not find an occurrence of "bar" that is preceded by something
! 4371: other than "foo"; it finds any occurrence of "bar" whatsoever, because
! 4372: the assertion (?!foo) is always true when the next three characters are
! 4373: "bar". A lookbehind assertion is needed to achieve the other effect.
! 4374:
! 4375: If you want to force a matching failure at some point in a pattern, the
! 4376: most convenient way to do it is with (?!) because an empty string
! 4377: always matches, so an assertion that requires there not to be an empty
! 4378: string must always fail.
! 4379:
! 4380: Lookbehind assertions
! 4381:
! 4382: Lookbehind assertions start with (?<= for positive assertions and (?<!
! 4383: for negative assertions. For example,
! 4384:
! 4385: (?<!foo)bar
! 4386:
! 4387: does find an occurrence of "bar" that is not preceded by "foo". The
! 4388: contents of a lookbehind assertion are restricted such that all the
! 4389: strings it matches must have a fixed length. However, if there are sev-
! 4390: eral top-level alternatives, they do not all have to have the same
! 4391: fixed length. Thus
! 4392:
! 4393: (?<=bullock|donkey)
! 4394:
! 4395: is permitted, but
! 4396:
! 4397: (?<!dogs?|cats?)
! 4398:
! 4399: causes an error at compile time. Branches that match different length
! 4400: strings are permitted only at the top level of a lookbehind assertion.
! 4401: This is an extension compared with Perl (at least for 5.8), which
! 4402: requires all branches to match the same length of string. An assertion
! 4403: such as
! 4404:
! 4405: (?<=ab(c|de))
! 4406:
! 4407: is not permitted, because its single top-level branch can match two
! 4408: different lengths, but it is acceptable if rewritten to use two top-
! 4409: level branches:
! 4410:
! 4411: (?<=abc|abde)
! 4412:
! 4413: In some cases, the Perl 5.10 escape sequence \K (see above) can be used
! 4414: instead of a lookbehind assertion; this is not restricted to a fixed-
! 4415: length.
! 4416:
! 4417: The implementation of lookbehind assertions is, for each alternative,
! 4418: to temporarily move the current position back by the fixed length and
! 4419: then try to match. If there are insufficient characters before the cur-
! 4420: rent position, the assertion fails.
! 4421:
! 4422: PCRE does not allow the \C escape (which matches a single byte in UTF-8
! 4423: mode) to appear in lookbehind assertions, because it makes it impossi-
! 4424: ble to calculate the length of the lookbehind. The \X and \R escapes,
! 4425: which can match different numbers of bytes, are also not permitted.
! 4426:
! 4427: Possessive quantifiers can be used in conjunction with lookbehind
! 4428: assertions to specify efficient matching at the end of the subject
! 4429: string. Consider a simple pattern such as
! 4430:
! 4431: abcd$
! 4432:
! 4433: when applied to a long string that does not match. Because matching
! 4434: proceeds from left to right, PCRE will look for each "a" in the subject
! 4435: and then see if what follows matches the rest of the pattern. If the
! 4436: pattern is specified as
! 4437:
! 4438: ^.*abcd$
! 4439:
! 4440: the initial .* matches the entire string at first, but when this fails
! 4441: (because there is no following "a"), it backtracks to match all but the
! 4442: last character, then all but the last two characters, and so on. Once
! 4443: again the search for "a" covers the entire string, from right to left,
! 4444: so we are no better off. However, if the pattern is written as
! 4445:
! 4446: ^.*+(?<=abcd)
! 4447:
! 4448: there can be no backtracking for the .*+ item; it can match only the
! 4449: entire string. The subsequent lookbehind assertion does a single test
! 4450: on the last four characters. If it fails, the match fails immediately.
! 4451: For long strings, this approach makes a significant difference to the
! 4452: processing time.
! 4453:
! 4454: Using multiple assertions
! 4455:
! 4456: Several assertions (of any sort) may occur in succession. For example,
! 4457:
! 4458: (?<=\d{3})(?<!999)foo
! 4459:
! 4460: matches "foo" preceded by three digits that are not "999". Notice that
! 4461: each of the assertions is applied independently at the same point in
! 4462: the subject string. First there is a check that the previous three
! 4463: characters are all digits, and then there is a check that the same
! 4464: three characters are not "999". This pattern does not match "foo" pre-
! 4465: ceded by six characters, the first of which are digits and the last
! 4466: three of which are not "999". For example, it doesn't match "123abc-
! 4467: foo". A pattern to do that is
! 4468:
! 4469: (?<=\d{3}...)(?<!999)foo
! 4470:
! 4471: This time the first assertion looks at the preceding six characters,
! 4472: checking that the first three are digits, and then the second assertion
! 4473: checks that the preceding three characters are not "999".
! 4474:
! 4475: Assertions can be nested in any combination. For example,
! 4476:
! 4477: (?<=(?<!foo)bar)baz
! 4478:
! 4479: matches an occurrence of "baz" that is preceded by "bar" which in turn
! 4480: is not preceded by "foo", while
! 4481:
! 4482: (?<=\d{3}(?!999)...)foo
! 4483:
! 4484: is another pattern that matches "foo" preceded by three digits and any
! 4485: three characters that are not "999".
! 4486:
! 4487:
! 4488: CONDITIONAL SUBPATTERNS
! 4489:
! 4490: It is possible to cause the matching process to obey a subpattern con-
! 4491: ditionally or to choose between two alternative subpatterns, depending
! 4492: on the result of an assertion, or whether a previous capturing subpat-
! 4493: tern matched or not. The two possible forms of conditional subpattern
! 4494: are
! 4495:
! 4496: (?(condition)yes-pattern)
! 4497: (?(condition)yes-pattern|no-pattern)
! 4498:
! 4499: If the condition is satisfied, the yes-pattern is used; otherwise the
! 4500: no-pattern (if present) is used. If there are more than two alterna-
! 4501: tives in the subpattern, a compile-time error occurs.
! 4502:
! 4503: There are four kinds of condition: references to subpatterns, refer-
! 4504: ences to recursion, a pseudo-condition called DEFINE, and assertions.
! 4505:
! 4506: Checking for a used subpattern by number
! 4507:
! 4508: If the text between the parentheses consists of a sequence of digits,
! 4509: the condition is true if the capturing subpattern of that number has
! 4510: previously matched. An alternative notation is to precede the digits
! 4511: with a plus or minus sign. In this case, the subpattern number is rela-
! 4512: tive rather than absolute. The most recently opened parentheses can be
! 4513: referenced by (?(-1), the next most recent by (?(-2), and so on. In
! 4514: looping constructs it can also make sense to refer to subsequent groups
! 4515: with constructs such as (?(+2).
! 4516:
! 4517: Consider the following pattern, which contains non-significant white
! 4518: space to make it more readable (assume the PCRE_EXTENDED option) and to
! 4519: divide it into three parts for ease of discussion:
! 4520:
! 4521: ( \( )? [^()]+ (?(1) \) )
! 4522:
! 4523: The first part matches an optional opening parenthesis, and if that
! 4524: character is present, sets it as the first captured substring. The sec-
! 4525: ond part matches one or more characters that are not parentheses. The
! 4526: third part is a conditional subpattern that tests whether the first set
! 4527: of parentheses matched or not. If they did, that is, if subject started
! 4528: with an opening parenthesis, the condition is true, and so the yes-pat-
! 4529: tern is executed and a closing parenthesis is required. Otherwise,
! 4530: since no-pattern is not present, the subpattern matches nothing. In
! 4531: other words, this pattern matches a sequence of non-parentheses,
! 4532: optionally enclosed in parentheses.
! 4533:
! 4534: If you were embedding this pattern in a larger one, you could use a
! 4535: relative reference:
! 4536:
! 4537: ...other stuff... ( \( )? [^()]+ (?(-1) \) ) ...
! 4538:
! 4539: This makes the fragment independent of the parentheses in the larger
! 4540: pattern.
! 4541:
! 4542: Checking for a used subpattern by name
! 4543:
! 4544: Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a
! 4545: used subpattern by name. For compatibility with earlier versions of
! 4546: PCRE, which had this facility before Perl, the syntax (?(name)...) is
! 4547: also recognized. However, there is a possible ambiguity with this syn-
! 4548: tax, because subpattern names may consist entirely of digits. PCRE
! 4549: looks first for a named subpattern; if it cannot find one and the name
! 4550: consists entirely of digits, PCRE looks for a subpattern of that num-
! 4551: ber, which must be greater than zero. Using subpattern names that con-
! 4552: sist entirely of digits is not recommended.
! 4553:
! 4554: Rewriting the above example to use a named subpattern gives this:
! 4555:
! 4556: (?<OPEN> \( )? [^()]+ (?(<OPEN>) \) )
! 4557:
! 4558:
! 4559: Checking for pattern recursion
! 4560:
! 4561: If the condition is the string (R), and there is no subpattern with the
! 4562: name R, the condition is true if a recursive call to the whole pattern
! 4563: or any subpattern has been made. If digits or a name preceded by amper-
! 4564: sand follow the letter R, for example:
! 4565:
! 4566: (?(R3)...) or (?(R&name)...)
! 4567:
! 4568: the condition is true if the most recent recursion is into the subpat-
! 4569: tern whose number or name is given. This condition does not check the
! 4570: entire recursion stack.
! 4571:
! 4572: At "top level", all these recursion test conditions are false. Recur-
! 4573: sive patterns are described below.
! 4574:
! 4575: Defining subpatterns for use by reference only
! 4576:
! 4577: If the condition is the string (DEFINE), and there is no subpattern
! 4578: with the name DEFINE, the condition is always false. In this case,
! 4579: there may be only one alternative in the subpattern. It is always
! 4580: skipped if control reaches this point in the pattern; the idea of
! 4581: DEFINE is that it can be used to define "subroutines" that can be ref-
! 4582: erenced from elsewhere. (The use of "subroutines" is described below.)
! 4583: For example, a pattern to match an IPv4 address could be written like
! 4584: this (ignore whitespace and line breaks):
! 4585:
! 4586: (?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) )
! 4587: \b (?&byte) (\.(?&byte)){3} \b
! 4588:
! 4589: The first part of the pattern is a DEFINE group inside which a another
! 4590: group named "byte" is defined. This matches an individual component of
! 4591: an IPv4 address (a number less than 256). When matching takes place,
! 4592: this part of the pattern is skipped because DEFINE acts like a false
! 4593: condition.
! 4594:
! 4595: The rest of the pattern uses references to the named group to match the
! 4596: four dot-separated components of an IPv4 address, insisting on a word
! 4597: boundary at each end.
! 4598:
! 4599: Assertion conditions
! 4600:
! 4601: If the condition is not in any of the above formats, it must be an
! 4602: assertion. This may be a positive or negative lookahead or lookbehind
! 4603: assertion. Consider this pattern, again containing non-significant
! 4604: white space, and with the two alternatives on the second line:
! 4605:
! 4606: (?(?=[^a-z]*[a-z])
! 4607: \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} )
! 4608:
! 4609: The condition is a positive lookahead assertion that matches an
! 4610: optional sequence of non-letters followed by a letter. In other words,
! 4611: it tests for the presence of at least one letter in the subject. If a
! 4612: letter is found, the subject is matched against the first alternative;
! 4613: otherwise it is matched against the second. This pattern matches
! 4614: strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are
! 4615: letters and dd are digits.
! 4616:
! 4617:
! 4618: COMMENTS
! 4619:
! 4620: The sequence (?# marks the start of a comment that continues up to the
! 4621: next closing parenthesis. Nested parentheses are not permitted. The
! 4622: characters that make up a comment play no part in the pattern matching
! 4623: at all.
! 4624:
! 4625: If the PCRE_EXTENDED option is set, an unescaped # character outside a
! 4626: character class introduces a comment that continues to immediately
! 4627: after the next newline in the pattern.
! 4628:
! 4629:
! 4630: RECURSIVE PATTERNS
! 4631:
! 4632: Consider the problem of matching a string in parentheses, allowing for
! 4633: unlimited nested parentheses. Without the use of recursion, the best
! 4634: that can be done is to use a pattern that matches up to some fixed
! 4635: depth of nesting. It is not possible to handle an arbitrary nesting
! 4636: depth.
! 4637:
! 4638: For some time, Perl has provided a facility that allows regular expres-
! 4639: sions to recurse (amongst other things). It does this by interpolating
! 4640: Perl code in the expression at run time, and the code can refer to the
! 4641: expression itself. A Perl pattern using code interpolation to solve the
! 4642: parentheses problem can be created like this:
! 4643:
! 4644: $re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x;
! 4645:
! 4646: The (?p{...}) item interpolates Perl code at run time, and in this case
! 4647: refers recursively to the pattern in which it appears.
! 4648:
! 4649: Obviously, PCRE cannot support the interpolation of Perl code. Instead,
! 4650: it supports special syntax for recursion of the entire pattern, and
! 4651: also for individual subpattern recursion. After its introduction in
! 4652: PCRE and Python, this kind of recursion was introduced into Perl at
! 4653: release 5.10.
! 4654:
! 4655: A special item that consists of (? followed by a number greater than
! 4656: zero and a closing parenthesis is a recursive call of the subpattern of
! 4657: the given number, provided that it occurs inside that subpattern. (If
! 4658: not, it is a "subroutine" call, which is described in the next sec-
! 4659: tion.) The special item (?R) or (?0) is a recursive call of the entire
! 4660: regular expression.
! 4661:
! 4662: In PCRE (like Python, but unlike Perl), a recursive subpattern call is
! 4663: always treated as an atomic group. That is, once it has matched some of
! 4664: the subject string, it is never re-entered, even if it contains untried
! 4665: alternatives and there is a subsequent matching failure.
! 4666:
! 4667: This PCRE pattern solves the nested parentheses problem (assume the
! 4668: PCRE_EXTENDED option is set so that white space is ignored):
! 4669:
! 4670: \( ( (?>[^()]+) | (?R) )* \)
! 4671:
! 4672: First it matches an opening parenthesis. Then it matches any number of
! 4673: substrings which can either be a sequence of non-parentheses, or a
! 4674: recursive match of the pattern itself (that is, a correctly parenthe-
! 4675: sized substring). Finally there is a closing parenthesis.
! 4676:
! 4677: If this were part of a larger pattern, you would not want to recurse
! 4678: the entire pattern, so instead you could use this:
! 4679:
! 4680: ( \( ( (?>[^()]+) | (?1) )* \) )
! 4681:
! 4682: We have put the pattern into parentheses, and caused the recursion to
! 4683: refer to them instead of the whole pattern.
! 4684:
! 4685: In a larger pattern, keeping track of parenthesis numbers can be
! 4686: tricky. This is made easier by the use of relative references. (A Perl
! 4687: 5.10 feature.) Instead of (?1) in the pattern above you can write
! 4688: (?-2) to refer to the second most recently opened parentheses preceding
! 4689: the recursion. In other words, a negative number counts capturing
! 4690: parentheses leftwards from the point at which it is encountered.
! 4691:
! 4692: It is also possible to refer to subsequently opened parentheses, by
! 4693: writing references such as (?+2). However, these cannot be recursive
! 4694: because the reference is not inside the parentheses that are refer-
! 4695: enced. They are always "subroutine" calls, as described in the next
! 4696: section.
! 4697:
! 4698: An alternative approach is to use named parentheses instead. The Perl
! 4699: syntax for this is (?&name); PCRE's earlier syntax (?P>name) is also
! 4700: supported. We could rewrite the above example as follows:
! 4701:
! 4702: (?<pn> \( ( (?>[^()]+) | (?&pn) )* \) )
! 4703:
! 4704: If there is more than one subpattern with the same name, the earliest
! 4705: one is used.
! 4706:
! 4707: This particular example pattern that we have been looking at contains
! 4708: nested unlimited repeats, and so the use of atomic grouping for match-
! 4709: ing strings of non-parentheses is important when applying the pattern
! 4710: to strings that do not match. For example, when this pattern is applied
! 4711: to
! 4712:
! 4713: (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
! 4714:
! 4715: it yields "no match" quickly. However, if atomic grouping is not used,
! 4716: the match runs for a very long time indeed because there are so many
! 4717: different ways the + and * repeats can carve up the subject, and all
! 4718: have to be tested before failure can be reported.
! 4719:
! 4720: At the end of a match, the values set for any capturing subpatterns are
! 4721: those from the outermost level of the recursion at which the subpattern
! 4722: value is set. If you want to obtain intermediate values, a callout
! 4723: function can be used (see below and the pcrecallout documentation). If
! 4724: the pattern above is matched against
! 4725:
! 4726: (ab(cd)ef)
! 4727:
! 4728: the value for the capturing parentheses is "ef", which is the last
! 4729: value taken on at the top level. If additional parentheses are added,
! 4730: giving
! 4731:
! 4732: \( ( ( (?>[^()]+) | (?R) )* ) \)
! 4733: ^ ^
! 4734: ^ ^
! 4735:
! 4736: the string they capture is "ab(cd)ef", the contents of the top level
! 4737: parentheses. If there are more than 15 capturing parentheses in a pat-
! 4738: tern, PCRE has to obtain extra memory to store data during a recursion,
! 4739: which it does by using pcre_malloc, freeing it via pcre_free after-
! 4740: wards. If no memory can be obtained, the match fails with the
! 4741: PCRE_ERROR_NOMEMORY error.
! 4742:
! 4743: Do not confuse the (?R) item with the condition (R), which tests for
! 4744: recursion. Consider this pattern, which matches text in angle brack-
! 4745: ets, allowing for arbitrary nesting. Only digits are allowed in nested
! 4746: brackets (that is, when recursing), whereas any characters are permit-
! 4747: ted at the outer level.
! 4748:
! 4749: < (?: (?(R) \d++ | [^<>]*+) | (?R)) * >
! 4750:
! 4751: In this pattern, (?(R) is the start of a conditional subpattern, with
! 4752: two different alternatives for the recursive and non-recursive cases.
! 4753: The (?R) item is the actual recursive call.
! 4754:
! 4755:
! 4756: SUBPATTERNS AS SUBROUTINES
! 4757:
! 4758: If the syntax for a recursive subpattern reference (either by number or
! 4759: by name) is used outside the parentheses to which it refers, it oper-
! 4760: ates like a subroutine in a programming language. The "called" subpat-
! 4761: tern may be defined before or after the reference. A numbered reference
! 4762: can be absolute or relative, as in these examples:
! 4763:
! 4764: (...(absolute)...)...(?2)...
! 4765: (...(relative)...)...(?-1)...
! 4766: (...(?+1)...(relative)...
! 4767:
! 4768: An earlier example pointed out that the pattern
! 4769:
! 4770: (sens|respons)e and \1ibility
! 4771:
! 4772: matches "sense and sensibility" and "response and responsibility", but
! 4773: not "sense and responsibility". If instead the pattern
! 4774:
! 4775: (sens|respons)e and (?1)ibility
! 4776:
! 4777: is used, it does match "sense and responsibility" as well as the other
! 4778: two strings. Another example is given in the discussion of DEFINE
! 4779: above.
! 4780:
! 4781: Like recursive subpatterns, a "subroutine" call is always treated as an
! 4782: atomic group. That is, once it has matched some of the subject string,
! 4783: it is never re-entered, even if it contains untried alternatives and
! 4784: there is a subsequent matching failure.
! 4785:
! 4786: When a subpattern is used as a subroutine, processing options such as
! 4787: case-independence are fixed when the subpattern is defined. They cannot
! 4788: be changed for different calls. For example, consider this pattern:
! 4789:
! 4790: (abc)(?i:(?-1))
! 4791:
! 4792: It matches "abcabc". It does not match "abcABC" because the change of
! 4793: processing option does not affect the called subpattern.
! 4794:
! 4795:
! 4796: ONIGURUMA SUBROUTINE SYNTAX
! 4797:
! 4798: For compatibility with Oniguruma, the non-Perl syntax \g followed by a
! 4799: name or a number enclosed either in angle brackets or single quotes, is
! 4800: an alternative syntax for referencing a subpattern as a subroutine,
! 4801: possibly recursively. Here are two of the examples used above, rewrit-
! 4802: ten using this syntax:
! 4803:
! 4804: (?<pn> \( ( (?>[^()]+) | \g<pn> )* \) )
! 4805: (sens|respons)e and \g'1'ibility
! 4806:
! 4807: PCRE supports an extension to Oniguruma: if a number is preceded by a
! 4808: plus or a minus sign it is taken as a relative reference. For example:
! 4809:
! 4810: (abc)(?i:\g<-1>)
! 4811:
! 4812: Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not
! 4813: synonymous. The former is a back reference; the latter is a subroutine
! 4814: call.
! 4815:
! 4816:
! 4817: CALLOUTS
! 4818:
! 4819: Perl has a feature whereby using the sequence (?{...}) causes arbitrary
! 4820: Perl code to be obeyed in the middle of matching a regular expression.
! 4821: This makes it possible, amongst other things, to extract different sub-
! 4822: strings that match the same pair of parentheses when there is a repeti-
! 4823: tion.
! 4824:
! 4825: PCRE provides a similar feature, but of course it cannot obey arbitrary
! 4826: Perl code. The feature is called "callout". The caller of PCRE provides
! 4827: an external function by putting its entry point in the global variable
! 4828: pcre_callout. By default, this variable contains NULL, which disables
! 4829: all calling out.
! 4830:
! 4831: Within a regular expression, (?C) indicates the points at which the
! 4832: external function is to be called. If you want to identify different
! 4833: callout points, you can put a number less than 256 after the letter C.
! 4834: The default value is zero. For example, this pattern has two callout
! 4835: points:
! 4836:
! 4837: (?C1)abc(?C2)def
! 4838:
! 4839: If the PCRE_AUTO_CALLOUT flag is passed to pcre_compile(), callouts are
! 4840: automatically installed before each item in the pattern. They are all
! 4841: numbered 255.
! 4842:
! 4843: During matching, when PCRE reaches a callout point (and pcre_callout is
! 4844: set), the external function is called. It is provided with the number
! 4845: of the callout, the position in the pattern, and, optionally, one item
! 4846: of data originally supplied by the caller of pcre_exec(). The callout
! 4847: function may cause matching to proceed, to backtrack, or to fail alto-
! 4848: gether. A complete description of the interface to the callout function
! 4849: is given in the pcrecallout documentation.
! 4850:
! 4851:
! 4852: BACKTRACKING CONTROL
! 4853:
! 4854: Perl 5.10 introduced a number of "Special Backtracking Control Verbs",
! 4855: which are described in the Perl documentation as "experimental and sub-
! 4856: ject to change or removal in a future version of Perl". It goes on to
! 4857: say: "Their usage in production code should be noted to avoid problems
! 4858: during upgrades." The same remarks apply to the PCRE features described
! 4859: in this section.
! 4860:
! 4861: Since these verbs are specifically related to backtracking, most of
! 4862: them can be used only when the pattern is to be matched using
! 4863: pcre_exec(), which uses a backtracking algorithm. With the exception of
! 4864: (*FAIL), which behaves like a failing negative assertion, they cause an
! 4865: error if encountered by pcre_dfa_exec().
! 4866:
! 4867: The new verbs make use of what was previously invalid syntax: an open-
! 4868: ing parenthesis followed by an asterisk. In Perl, they are generally of
! 4869: the form (*VERB:ARG) but PCRE does not support the use of arguments, so
! 4870: its general form is just (*VERB). Any number of these verbs may occur
! 4871: in a pattern. There are two kinds:
! 4872:
! 4873: Verbs that act immediately
! 4874:
! 4875: The following verbs act as soon as they are encountered:
! 4876:
! 4877: (*ACCEPT)
! 4878:
! 4879: This verb causes the match to end successfully, skipping the remainder
! 4880: of the pattern. When inside a recursion, only the innermost pattern is
! 4881: ended immediately. PCRE differs from Perl in what happens if the
! 4882: (*ACCEPT) is inside capturing parentheses. In Perl, the data so far is
! 4883: captured: in PCRE no data is captured. For example:
! 4884:
! 4885: A(A|B(*ACCEPT)|C)D
! 4886:
! 4887: This matches "AB", "AAD", or "ACD", but when it matches "AB", no data
! 4888: is captured.
! 4889:
! 4890: (*FAIL) or (*F)
! 4891:
! 4892: This verb causes the match to fail, forcing backtracking to occur. It
! 4893: is equivalent to (?!) but easier to read. The Perl documentation notes
! 4894: that it is probably useful only when combined with (?{}) or (??{}).
! 4895: Those are, of course, Perl features that are not present in PCRE. The
! 4896: nearest equivalent is the callout feature, as for example in this pat-
! 4897: tern:
! 4898:
! 4899: a+(?C)(*FAIL)
! 4900:
! 4901: A match with the string "aaaa" always fails, but the callout is taken
! 4902: before each backtrack happens (in this example, 10 times).
! 4903:
! 4904: Verbs that act after backtracking
! 4905:
! 4906: The following verbs do nothing when they are encountered. Matching con-
! 4907: tinues with what follows, but if there is no subsequent match, a fail-
! 4908: ure is forced. The verbs differ in exactly what kind of failure
! 4909: occurs.
! 4910:
! 4911: (*COMMIT)
! 4912:
! 4913: This verb causes the whole match to fail outright if the rest of the
! 4914: pattern does not match. Even if the pattern is unanchored, no further
! 4915: attempts to find a match by advancing the start point take place. Once
! 4916: (*COMMIT) has been passed, pcre_exec() is committed to finding a match
! 4917: at the current starting point, or not at all. For example:
! 4918:
! 4919: a+(*COMMIT)b
! 4920:
! 4921: This matches "xxaab" but not "aacaab". It can be thought of as a kind
! 4922: of dynamic anchor, or "I've started, so I must finish."
! 4923:
! 4924: (*PRUNE)
! 4925:
! 4926: This verb causes the match to fail at the current position if the rest
! 4927: of the pattern does not match. If the pattern is unanchored, the normal
! 4928: "bumpalong" advance to the next starting character then happens. Back-
! 4929: tracking can occur as usual to the left of (*PRUNE), or when matching
! 4930: to the right of (*PRUNE), but if there is no match to the right, back-
! 4931: tracking cannot cross (*PRUNE). In simple cases, the use of (*PRUNE)
! 4932: is just an alternative to an atomic group or possessive quantifier, but
! 4933: there are some uses of (*PRUNE) that cannot be expressed in any other
! 4934: way.
! 4935:
! 4936: (*SKIP)
! 4937:
! 4938: This verb is like (*PRUNE), except that if the pattern is unanchored,
! 4939: the "bumpalong" advance is not to the next character, but to the posi-
! 4940: tion in the subject where (*SKIP) was encountered. (*SKIP) signifies
! 4941: that whatever text was matched leading up to it cannot be part of a
! 4942: successful match. Consider:
! 4943:
! 4944: a+(*SKIP)b
! 4945:
! 4946: If the subject is "aaaac...", after the first match attempt fails
! 4947: (starting at the first character in the string), the starting point
! 4948: skips on to start the next attempt at "c". Note that a possessive quan-
! 4949: tifer does not have the same effect in this example; although it would
! 4950: suppress backtracking during the first match attempt, the second
! 4951: attempt would start at the second character instead of skipping on to
! 4952: "c".
! 4953:
! 4954: (*THEN)
! 4955:
! 4956: This verb causes a skip to the next alternation if the rest of the pat-
! 4957: tern does not match. That is, it cancels pending backtracking, but only
! 4958: within the current alternation. Its name comes from the observation
! 4959: that it can be used for a pattern-based if-then-else block:
! 4960:
! 4961: ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ...
! 4962:
! 4963: If the COND1 pattern matches, FOO is tried (and possibly further items
! 4964: after the end of the group if FOO succeeds); on failure the matcher
! 4965: skips to the second alternative and tries COND2, without backtracking
! 4966: into COND1. If (*THEN) is used outside of any alternation, it acts
! 4967: exactly like (*PRUNE).
! 4968:
! 4969:
! 4970: SEE ALSO
! 4971:
! 4972: pcreapi(3), pcrecallout(3), pcrematching(3), pcre(3).
! 4973:
! 4974:
! 4975: AUTHOR
! 4976:
! 4977: Philip Hazel
! 4978: University Computing Service
! 4979: Cambridge CB2 3QH, England.
! 4980:
! 4981:
! 4982: REVISION
! 4983:
! 4984: Last updated: 19 April 2008
! 4985: Copyright (c) 1997-2008 University of Cambridge.
! 4986: ------------------------------------------------------------------------------
! 4987:
! 4988:
! 4989: PCRESYNTAX(3) PCRESYNTAX(3)
! 4990:
! 4991:
! 4992: NAME
! 4993: PCRE - Perl-compatible regular expressions
! 4994:
! 4995:
! 4996: PCRE REGULAR EXPRESSION SYNTAX SUMMARY
! 4997:
! 4998: The full syntax and semantics of the regular expressions that are sup-
! 4999: ported by PCRE are described in the pcrepattern documentation. This
! 5000: document contains just a quick-reference summary of the syntax.
! 5001:
! 5002:
! 5003: QUOTING
! 5004:
! 5005: \x where x is non-alphanumeric is a literal x
! 5006: \Q...\E treat enclosed characters as literal
! 5007:
! 5008:
! 5009: CHARACTERS
! 5010:
! 5011: \a alarm, that is, the BEL character (hex 07)
! 5012: \cx "control-x", where x is any character
! 5013: \e escape (hex 1B)
! 5014: \f formfeed (hex 0C)
! 5015: \n newline (hex 0A)
! 5016: \r carriage return (hex 0D)
! 5017: \t tab (hex 09)
! 5018: \ddd character with octal code ddd, or backreference
! 5019: \xhh character with hex code hh
! 5020: \x{hhh..} character with hex code hhh..
! 5021:
! 5022:
! 5023: CHARACTER TYPES
! 5024:
! 5025: . any character except newline;
! 5026: in dotall mode, any character whatsoever
! 5027: \C one byte, even in UTF-8 mode (best avoided)
! 5028: \d a decimal digit
! 5029: \D a character that is not a decimal digit
! 5030: \h a horizontal whitespace character
! 5031: \H a character that is not a horizontal whitespace character
! 5032: \p{xx} a character with the xx property
! 5033: \P{xx} a character without the xx property
! 5034: \R a newline sequence
! 5035: \s a whitespace character
! 5036: \S a character that is not a whitespace character
! 5037: \v a vertical whitespace character
! 5038: \V a character that is not a vertical whitespace character
! 5039: \w a "word" character
! 5040: \W a "non-word" character
! 5041: \X an extended Unicode sequence
! 5042:
! 5043: In PCRE, \d, \D, \s, \S, \w, and \W recognize only ASCII characters.
! 5044:
! 5045:
! 5046: GENERAL CATEGORY PROPERTY CODES FOR \p and \P
! 5047:
! 5048: C Other
! 5049: Cc Control
! 5050: Cf Format
! 5051: Cn Unassigned
! 5052: Co Private use
! 5053: Cs Surrogate
! 5054:
! 5055: L Letter
! 5056: Ll Lower case letter
! 5057: Lm Modifier letter
! 5058: Lo Other letter
! 5059: Lt Title case letter
! 5060: Lu Upper case letter
! 5061: L& Ll, Lu, or Lt
! 5062:
! 5063: M Mark
! 5064: Mc Spacing mark
! 5065: Me Enclosing mark
! 5066: Mn Non-spacing mark
! 5067:
! 5068: N Number
! 5069: Nd Decimal number
! 5070: Nl Letter number
! 5071: No Other number
! 5072:
! 5073: P Punctuation
! 5074: Pc Connector punctuation
! 5075: Pd Dash punctuation
! 5076: Pe Close punctuation
! 5077: Pf Final punctuation
! 5078: Pi Initial punctuation
! 5079: Po Other punctuation
! 5080: Ps Open punctuation
! 5081:
! 5082: S Symbol
! 5083: Sc Currency symbol
! 5084: Sk Modifier symbol
! 5085: Sm Mathematical symbol
! 5086: So Other symbol
! 5087:
! 5088: Z Separator
! 5089: Zl Line separator
! 5090: Zp Paragraph separator
! 5091: Zs Space separator
! 5092:
! 5093:
! 5094: SCRIPT NAMES FOR \p AND \P
! 5095:
! 5096: Arabic, Armenian, Balinese, Bengali, Bopomofo, Braille, Buginese,
! 5097: Buhid, Canadian_Aboriginal, Cherokee, Common, Coptic, Cuneiform,
! 5098: Cypriot, Cyrillic, Deseret, Devanagari, Ethiopic, Georgian, Glagolitic,
! 5099: Gothic, Greek, Gujarati, Gurmukhi, Han, Hangul, Hanunoo, Hebrew, Hira-
! 5100: gana, Inherited, Kannada, Katakana, Kharoshthi, Khmer, Lao, Latin,
! 5101: Limbu, Linear_B, Malayalam, Mongolian, Myanmar, New_Tai_Lue, Nko,
! 5102: Ogham, Old_Italic, Old_Persian, Oriya, Osmanya, Phags_Pa, Phoenician,
! 5103: Runic, Shavian, Sinhala, Syloti_Nagri, Syriac, Tagalog, Tagbanwa,
! 5104: Tai_Le, Tamil, Telugu, Thaana, Thai, Tibetan, Tifinagh, Ugaritic, Yi.
! 5105:
! 5106:
! 5107: CHARACTER CLASSES
! 5108:
! 5109: [...] positive character class
! 5110: [^...] negative character class
! 5111: [x-y] range (can be used for hex characters)
! 5112: [[:xxx:]] positive POSIX named set
! 5113: [[:^xxx:]] negative POSIX named set
! 5114:
! 5115: alnum alphanumeric
! 5116: alpha alphabetic
! 5117: ascii 0-127
! 5118: blank space or tab
! 5119: cntrl control character
! 5120: digit decimal digit
! 5121: graph printing, excluding space
! 5122: lower lower case letter
! 5123: print printing, including space
! 5124: punct printing, excluding alphanumeric
! 5125: space whitespace
! 5126: upper upper case letter
! 5127: word same as \w
! 5128: xdigit hexadecimal digit
! 5129:
! 5130: In PCRE, POSIX character set names recognize only ASCII characters. You
! 5131: can use \Q...\E inside a character class.
! 5132:
! 5133:
! 5134: QUANTIFIERS
! 5135:
! 5136: ? 0 or 1, greedy
! 5137: ?+ 0 or 1, possessive
! 5138: ?? 0 or 1, lazy
! 5139: * 0 or more, greedy
! 5140: *+ 0 or more, possessive
! 5141: *? 0 or more, lazy
! 5142: + 1 or more, greedy
! 5143: ++ 1 or more, possessive
! 5144: +? 1 or more, lazy
! 5145: {n} exactly n
! 5146: {n,m} at least n, no more than m, greedy
! 5147: {n,m}+ at least n, no more than m, possessive
! 5148: {n,m}? at least n, no more than m, lazy
! 5149: {n,} n or more, greedy
! 5150: {n,}+ n or more, possessive
! 5151: {n,}? n or more, lazy
! 5152:
! 5153:
! 5154: ANCHORS AND SIMPLE ASSERTIONS
! 5155:
! 5156: \b word boundary
! 5157: \B not a word boundary
! 5158: ^ start of subject
! 5159: also after internal newline in multiline mode
! 5160: \A start of subject
! 5161: $ end of subject
! 5162: also before newline at end of subject
! 5163: also before internal newline in multiline mode
! 5164: \Z end of subject
! 5165: also before newline at end of subject
! 5166: \z end of subject
! 5167: \G first matching position in subject
! 5168:
! 5169:
! 5170: MATCH POINT RESET
! 5171:
! 5172: \K reset start of match
! 5173:
! 5174:
! 5175: ALTERNATION
! 5176:
! 5177: expr|expr|expr...
! 5178:
! 5179:
! 5180: CAPTURING
! 5181:
! 5182: (...) capturing group
! 5183: (?<name>...) named capturing group (Perl)
! 5184: (?'name'...) named capturing group (Perl)
! 5185: (?P<name>...) named capturing group (Python)
! 5186: (?:...) non-capturing group
! 5187: (?|...) non-capturing group; reset group numbers for
! 5188: capturing groups in each alternative
! 5189:
! 5190:
! 5191: ATOMIC GROUPS
! 5192:
! 5193: (?>...) atomic, non-capturing group
! 5194:
! 5195:
! 5196: COMMENT
! 5197:
! 5198: (?#....) comment (not nestable)
! 5199:
! 5200:
! 5201: OPTION SETTING
! 5202:
! 5203: (?i) caseless
! 5204: (?J) allow duplicate names
! 5205: (?m) multiline
! 5206: (?s) single line (dotall)
! 5207: (?U) default ungreedy (lazy)
! 5208: (?x) extended (ignore white space)
! 5209: (?-...) unset option(s)
! 5210:
! 5211:
! 5212: LOOKAHEAD AND LOOKBEHIND ASSERTIONS
! 5213:
! 5214: (?=...) positive look ahead
! 5215: (?!...) negative look ahead
! 5216: (?<=...) positive look behind
! 5217: (?<!...) negative look behind
! 5218:
! 5219: Each top-level branch of a look behind must be of a fixed length.
! 5220:
! 5221:
! 5222: BACKREFERENCES
! 5223:
! 5224: \n reference by number (can be ambiguous)
! 5225: \gn reference by number
! 5226: \g{n} reference by number
! 5227: \g{-n} relative reference by number
! 5228: \k<name> reference by name (Perl)
! 5229: \k'name' reference by name (Perl)
! 5230: \g{name} reference by name (Perl)
! 5231: \k{name} reference by name (.NET)
! 5232: (?P=name) reference by name (Python)
! 5233:
! 5234:
! 5235: SUBROUTINE REFERENCES (POSSIBLY RECURSIVE)
! 5236:
! 5237: (?R) recurse whole pattern
! 5238: (?n) call subpattern by absolute number
! 5239: (?+n) call subpattern by relative number
! 5240: (?-n) call subpattern by relative number
! 5241: (?&name) call subpattern by name (Perl)
! 5242: (?P>name) call subpattern by name (Python)
! 5243: \g<name> call subpattern by name (Oniguruma)
! 5244: \g'name' call subpattern by name (Oniguruma)
! 5245: \g<n> call subpattern by absolute number (Oniguruma)
! 5246: \g'n' call subpattern by absolute number (Oniguruma)
! 5247: \g<+n> call subpattern by relative number (PCRE extension)
! 5248: \g'+n' call subpattern by relative number (PCRE extension)
! 5249: \g<-n> call subpattern by relative number (PCRE extension)
! 5250: \g'-n' call subpattern by relative number (PCRE extension)
! 5251:
! 5252:
! 5253: CONDITIONAL PATTERNS
! 5254:
! 5255: (?(condition)yes-pattern)
! 5256: (?(condition)yes-pattern|no-pattern)
! 5257:
! 5258: (?(n)... absolute reference condition
! 5259: (?(+n)... relative reference condition
! 5260: (?(-n)... relative reference condition
! 5261: (?(<name>)... named reference condition (Perl)
! 5262: (?('name')... named reference condition (Perl)
! 5263: (?(name)... named reference condition (PCRE)
! 5264: (?(R)... overall recursion condition
! 5265: (?(Rn)... specific group recursion condition
! 5266: (?(R&name)... specific recursion condition
! 5267: (?(DEFINE)... define subpattern for reference
! 5268: (?(assert)... assertion condition
! 5269:
! 5270:
! 5271: BACKTRACKING CONTROL
! 5272:
! 5273: The following act immediately they are reached:
! 5274:
! 5275: (*ACCEPT) force successful match
! 5276: (*FAIL) force backtrack; synonym (*F)
! 5277:
! 5278: The following act only when a subsequent match failure causes a back-
! 5279: track to reach them. They all force a match failure, but they differ in
! 5280: what happens afterwards. Those that advance the start-of-match point do
! 5281: so only if the pattern is not anchored.
! 5282:
! 5283: (*COMMIT) overall failure, no advance of starting point
! 5284: (*PRUNE) advance to next starting character
! 5285: (*SKIP) advance start to current matching position
! 5286: (*THEN) local failure, backtrack to next alternation
! 5287:
! 5288:
! 5289: NEWLINE CONVENTIONS
! 5290:
! 5291: These are recognized only at the very start of the pattern or after a
! 5292: (*BSR_...) option.
! 5293:
! 5294: (*CR)
! 5295: (*LF)
! 5296: (*CRLF)
! 5297: (*ANYCRLF)
! 5298: (*ANY)
! 5299:
! 5300:
! 5301: WHAT \R MATCHES
! 5302:
! 5303: These are recognized only at the very start of the pattern or after a
! 5304: (*...) option that sets the newline convention.
! 5305:
! 5306: (*BSR_ANYCRLF)
! 5307: (*BSR_UNICODE)
! 5308:
! 5309:
! 5310: CALLOUTS
! 5311:
! 5312: (?C) callout
! 5313: (?Cn) callout with data n
! 5314:
! 5315:
! 5316: SEE ALSO
! 5317:
! 5318: pcrepattern(3), pcreapi(3), pcrecallout(3), pcrematching(3), pcre(3).
! 5319:
! 5320:
! 5321: AUTHOR
! 5322:
! 5323: Philip Hazel
! 5324: University Computing Service
! 5325: Cambridge CB2 3QH, England.
! 5326:
! 5327:
! 5328: REVISION
! 5329:
! 5330: Last updated: 09 April 2008
! 5331: Copyright (c) 1997-2008 University of Cambridge.
! 5332: ------------------------------------------------------------------------------
! 5333:
! 5334:
! 5335: PCREPARTIAL(3) PCREPARTIAL(3)
! 5336:
! 5337:
! 5338: NAME
! 5339: PCRE - Perl-compatible regular expressions
! 5340:
! 5341:
! 5342: PARTIAL MATCHING IN PCRE
! 5343:
! 5344: In normal use of PCRE, if the subject string that is passed to
! 5345: pcre_exec() or pcre_dfa_exec() matches as far as it goes, but is too
! 5346: short to match the entire pattern, PCRE_ERROR_NOMATCH is returned.
! 5347: There are circumstances where it might be helpful to distinguish this
! 5348: case from other cases in which there is no match.
! 5349:
! 5350: Consider, for example, an application where a human is required to type
! 5351: in data for a field with specific formatting requirements. An example
! 5352: might be a date in the form ddmmmyy, defined by this pattern:
! 5353:
! 5354: ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$
! 5355:
! 5356: If the application sees the user's keystrokes one by one, and can check
! 5357: that what has been typed so far is potentially valid, it is able to
! 5358: raise an error as soon as a mistake is made, possibly beeping and not
! 5359: reflecting the character that has been typed. This immediate feedback
! 5360: is likely to be a better user interface than a check that is delayed
! 5361: until the entire string has been entered.
! 5362:
! 5363: PCRE supports the concept of partial matching by means of the PCRE_PAR-
! 5364: TIAL option, which can be set when calling pcre_exec() or
! 5365: pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code
! 5366: PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time
! 5367: during the matching process the last part of the subject string matched
! 5368: part of the pattern. Unfortunately, for non-anchored matching, it is
! 5369: not possible to obtain the position of the start of the partial match.
! 5370: No captured data is set when PCRE_ERROR_PARTIAL is returned.
! 5371:
! 5372: When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code
! 5373: PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of
! 5374: the subject is reached, there have been no complete matches, but there
! 5375: is still at least one matching possibility. The portion of the string
! 5376: that provided the partial match is set as the first matching string.
! 5377:
! 5378: Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers
! 5379: the last literal byte in a pattern, and abandons matching immediately
! 5380: if such a byte is not present in the subject string. This optimization
! 5381: cannot be used for a subject string that might match only partially.
! 5382:
! 5383:
! 5384: RESTRICTED PATTERNS FOR PCRE_PARTIAL
! 5385:
! 5386: Because of the way certain internal optimizations are implemented in
! 5387: the pcre_exec() function, the PCRE_PARTIAL option cannot be used with
! 5388: all patterns. These restrictions do not apply when pcre_dfa_exec() is
! 5389: used. For pcre_exec(), repeated single characters such as
! 5390:
! 5391: a{2,4}
! 5392:
! 5393: and repeated single metasequences such as
! 5394:
! 5395: \d+
! 5396:
! 5397: are not permitted if the maximum number of occurrences is greater than
! 5398: one. Optional items such as \d? (where the maximum is one) are permit-
! 5399: ted. Quantifiers with any values are permitted after parentheses, so
! 5400: the invalid examples above can be coded thus:
! 5401:
! 5402: (a){2,4}
! 5403: (\d)+
! 5404:
! 5405: These constructions run more slowly, but for the kinds of application
! 5406: that are envisaged for this facility, this is not felt to be a major
! 5407: restriction.
! 5408:
! 5409: If PCRE_PARTIAL is set for a pattern that does not conform to the
! 5410: restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL
! 5411: (-13). You can use the PCRE_INFO_OKPARTIAL call to pcre_fullinfo() to
! 5412: find out if a compiled pattern can be used for partial matching.
! 5413:
! 5414:
! 5415: EXAMPLE OF PARTIAL MATCHING USING PCRETEST
! 5416:
! 5417: If the escape sequence \P is present in a pcretest data line, the
! 5418: PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that
! 5419: uses the date example quoted above:
! 5420:
! 5421: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
! 5422: data> 25jun04\P
! 5423: 0: 25jun04
! 5424: 1: jun
! 5425: data> 25dec3\P
! 5426: Partial match
! 5427: data> 3ju\P
! 5428: Partial match
! 5429: data> 3juj\P
! 5430: No match
! 5431: data> j\P
! 5432: No match
! 5433:
! 5434: The first data string is matched completely, so pcretest shows the
! 5435: matched substrings. The remaining four strings do not match the com-
! 5436: plete pattern, but the first two are partial matches. The same test,
! 5437: using pcre_dfa_exec() matching (by means of the \D escape sequence),
! 5438: produces the following output:
! 5439:
! 5440: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
! 5441: data> 25jun04\P\D
! 5442: 0: 25jun04
! 5443: data> 23dec3\P\D
! 5444: Partial match: 23dec3
! 5445: data> 3ju\P\D
! 5446: Partial match: 3ju
! 5447: data> 3juj\P\D
! 5448: No match
! 5449: data> j\P\D
! 5450: No match
! 5451:
! 5452: Notice that in this case the portion of the string that was matched is
! 5453: made available.
! 5454:
! 5455:
! 5456: MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()
! 5457:
! 5458: When a partial match has been found using pcre_dfa_exec(), it is possi-
! 5459: ble to continue the match by providing additional subject data and
! 5460: calling pcre_dfa_exec() again with the same compiled regular expres-
! 5461: sion, this time setting the PCRE_DFA_RESTART option. You must also pass
! 5462: the same working space as before, because this is where details of the
! 5463: previous partial match are stored. Here is an example using pcretest,
! 5464: using the \R escape sequence to set the PCRE_DFA_RESTART option (\P and
! 5465: \D are as above):
! 5466:
! 5467: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
! 5468: data> 23ja\P\D
! 5469: Partial match: 23ja
! 5470: data> n05\R\D
! 5471: 0: n05
! 5472:
! 5473: The first call has "23ja" as the subject, and requests partial match-
! 5474: ing; the second call has "n05" as the subject for the continued
! 5475: (restarted) match. Notice that when the match is complete, only the
! 5476: last part is shown; PCRE does not retain the previously partially-
! 5477: matched string. It is up to the calling program to do that if it needs
! 5478: to.
! 5479:
! 5480: You can set PCRE_PARTIAL with PCRE_DFA_RESTART to continue partial
! 5481: matching over multiple segments. This facility can be used to pass very
! 5482: long subject strings to pcre_dfa_exec(). However, some care is needed
! 5483: for certain types of pattern.
! 5484:
! 5485: 1. If the pattern contains tests for the beginning or end of a line,
! 5486: you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri-
! 5487: ate, when the subject string for any call does not contain the begin-
! 5488: ning or end of a line.
! 5489:
! 5490: 2. If the pattern contains backward assertions (including \b or \B),
! 5491: you need to arrange for some overlap in the subject strings to allow
! 5492: for this. For example, you could pass the subject in chunks that are
! 5493: 500 bytes long, but in a buffer of 700 bytes, with the starting offset
! 5494: set to 200 and the previous 200 bytes at the start of the buffer.
! 5495:
! 5496: 3. Matching a subject string that is split into multiple segments does
! 5497: not always produce exactly the same result as matching over one single
! 5498: long string. The difference arises when there are multiple matching
! 5499: possibilities, because a partial match result is given only when there
! 5500: are no completed matches in a call to pcre_dfa_exec(). This means that
! 5501: as soon as the shortest match has been found, continuation to a new
! 5502: subject segment is no longer possible. Consider this pcretest example:
! 5503:
! 5504: re> /dog(sbody)?/
! 5505: data> do\P\D
! 5506: Partial match: do
! 5507: data> gsb\R\P\D
! 5508: 0: g
! 5509: data> dogsbody\D
! 5510: 0: dogsbody
! 5511: 1: dog
! 5512:
! 5513: The pattern matches the words "dog" or "dogsbody". When the subject is
! 5514: presented in several parts ("do" and "gsb" being the first two) the
! 5515: match stops when "dog" has been found, and it is not possible to con-
! 5516: tinue. On the other hand, if "dogsbody" is presented as a single
! 5517: string, both matches are found.
! 5518:
! 5519: Because of this phenomenon, it does not usually make sense to end a
! 5520: pattern that is going to be matched in this way with a variable repeat.
! 5521:
! 5522: 4. Patterns that contain alternatives at the top level which do not all
! 5523: start with the same pattern item may not work as expected. For example,
! 5524: consider this pattern:
! 5525:
! 5526: 1234|3789
! 5527:
! 5528: If the first part of the subject is "ABC123", a partial match of the
! 5529: first alternative is found at offset 3. There is no partial match for
! 5530: the second alternative, because such a match does not start at the same
! 5531: point in the subject string. Attempting to continue with the string
! 5532: "789" does not yield a match because only those alternatives that match
! 5533: at one point in the subject are remembered. The problem arises because
! 5534: the start of the second alternative matches within the first alterna-
! 5535: tive. There is no problem with anchored patterns or patterns such as:
! 5536:
! 5537: 1234|ABCD
! 5538:
! 5539: where no string can be a partial match for both alternatives.
! 5540:
! 5541:
! 5542: AUTHOR
! 5543:
! 5544: Philip Hazel
! 5545: University Computing Service
! 5546: Cambridge CB2 3QH, England.
! 5547:
! 5548:
! 5549: REVISION
! 5550:
! 5551: Last updated: 04 June 2007
! 5552: Copyright (c) 1997-2007 University of Cambridge.
! 5553: ------------------------------------------------------------------------------
! 5554:
! 5555:
! 5556: PCREPRECOMPILE(3) PCREPRECOMPILE(3)
! 5557:
! 5558:
! 5559: NAME
! 5560: PCRE - Perl-compatible regular expressions
! 5561:
! 5562:
! 5563: SAVING AND RE-USING PRECOMPILED PCRE PATTERNS
! 5564:
! 5565: If you are running an application that uses a large number of regular
! 5566: expression patterns, it may be useful to store them in a precompiled
! 5567: form instead of having to compile them every time the application is
! 5568: run. If you are not using any private character tables (see the
! 5569: pcre_maketables() documentation), this is relatively straightforward.
! 5570: If you are using private tables, it is a little bit more complicated.
! 5571:
! 5572: If you save compiled patterns to a file, you can copy them to a differ-
! 5573: ent host and run them there. This works even if the new host has the
! 5574: opposite endianness to the one on which the patterns were compiled.
! 5575: There may be a small performance penalty, but it should be insignifi-
! 5576: cant. However, compiling regular expressions with one version of PCRE
! 5577: for use with a different version is not guaranteed to work and may
! 5578: cause crashes.
! 5579:
! 5580:
! 5581: SAVING A COMPILED PATTERN
! 5582: The value returned by pcre_compile() points to a single block of memory
! 5583: that holds the compiled pattern and associated data. You can find the
! 5584: length of this block in bytes by calling pcre_fullinfo() with an argu-
! 5585: ment of PCRE_INFO_SIZE. You can then save the data in any appropriate
! 5586: manner. Here is sample code that compiles a pattern and writes it to a
! 5587: file. It assumes that the variable fd refers to a file that is open for
! 5588: output:
! 5589:
! 5590: int erroroffset, rc, size;
! 5591: char *error;
! 5592: pcre *re;
! 5593:
! 5594: re = pcre_compile("my pattern", 0, &error, &erroroffset, NULL);
! 5595: if (re == NULL) { ... handle errors ... }
! 5596: rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &size);
! 5597: if (rc < 0) { ... handle errors ... }
! 5598: rc = fwrite(re, 1, size, fd);
! 5599: if (rc != size) { ... handle errors ... }
! 5600:
! 5601: In this example, the bytes that comprise the compiled pattern are
! 5602: copied exactly. Note that this is binary data that may contain any of
! 5603: the 256 possible byte values. On systems that make a distinction
! 5604: between binary and non-binary data, be sure that the file is opened for
! 5605: binary output.
! 5606:
! 5607: If you want to write more than one pattern to a file, you will have to
! 5608: devise a way of separating them. For binary data, preceding each pat-
! 5609: tern with its length is probably the most straightforward approach.
! 5610: Another possibility is to write out the data in hexadecimal instead of
! 5611: binary, one pattern to a line.
! 5612:
! 5613: Saving compiled patterns in a file is only one possible way of storing
! 5614: them for later use. They could equally well be saved in a database, or
! 5615: in the memory of some daemon process that passes them via sockets to
! 5616: the processes that want them.
! 5617:
! 5618: If the pattern has been studied, it is also possible to save the study
! 5619: data in a similar way to the compiled pattern itself. When studying
! 5620: generates additional information, pcre_study() returns a pointer to a
! 5621: pcre_extra data block. Its format is defined in the section on matching
! 5622: a pattern in the pcreapi documentation. The study_data field points to
! 5623: the binary study data, and this is what you must save (not the
! 5624: pcre_extra block itself). The length of the study data can be obtained
! 5625: by calling pcre_fullinfo() with an argument of PCRE_INFO_STUDYSIZE.
! 5626: Remember to check that pcre_study() did return a non-NULL value before
! 5627: trying to save the study data.
! 5628:
! 5629:
! 5630: RE-USING A PRECOMPILED PATTERN
! 5631:
! 5632: Re-using a precompiled pattern is straightforward. Having reloaded it
! 5633: into main memory, you pass its pointer to pcre_exec() or
! 5634: pcre_dfa_exec() in the usual way. This should work even on another
! 5635: host, and even if that host has the opposite endianness to the one
! 5636: where the pattern was compiled.
! 5637:
! 5638: However, if you passed a pointer to custom character tables when the
! 5639: pattern was compiled (the tableptr argument of pcre_compile()), you
! 5640: must now pass a similar pointer to pcre_exec() or pcre_dfa_exec(),
! 5641: because the value saved with the compiled pattern will obviously be
! 5642: nonsense. A field in a pcre_extra() block is used to pass this data, as
! 5643: described in the section on matching a pattern in the pcreapi documen-
! 5644: tation.
! 5645:
! 5646: If you did not provide custom character tables when the pattern was
! 5647: compiled, the pointer in the compiled pattern is NULL, which causes
! 5648: pcre_exec() to use PCRE's internal tables. Thus, you do not need to
! 5649: take any special action at run time in this case.
! 5650:
! 5651: If you saved study data with the compiled pattern, you need to create
! 5652: your own pcre_extra data block and set the study_data field to point to
! 5653: the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA
! 5654: bit in the flags field to indicate that study data is present. Then
! 5655: pass the pcre_extra block to pcre_exec() or pcre_dfa_exec() in the
! 5656: usual way.
! 5657:
! 5658:
! 5659: COMPATIBILITY WITH DIFFERENT PCRE RELEASES
! 5660:
! 5661: In general, it is safest to recompile all saved patterns when you
! 5662: update to a new PCRE release, though not all updates actually require
! 5663: this. Recompiling is definitely needed for release 7.2.
! 5664:
! 5665:
! 5666: AUTHOR
! 5667:
! 5668: Philip Hazel
! 5669: University Computing Service
! 5670: Cambridge CB2 3QH, England.
! 5671:
! 5672:
! 5673: REVISION
! 5674:
! 5675: Last updated: 13 June 2007
! 5676: Copyright (c) 1997-2007 University of Cambridge.
! 5677: ------------------------------------------------------------------------------
! 5678:
! 5679:
! 5680: PCREPERFORM(3) PCREPERFORM(3)
! 5681:
! 5682:
! 5683: NAME
! 5684: PCRE - Perl-compatible regular expressions
! 5685:
! 5686:
! 5687: PCRE PERFORMANCE
! 5688:
! 5689: Two aspects of performance are discussed below: memory usage and pro-
! 5690: cessing time. The way you express your pattern as a regular expression
! 5691: can affect both of them.
! 5692:
! 5693:
! 5694: MEMORY USAGE
! 5695:
! 5696: Patterns are compiled by PCRE into a reasonably efficient byte code, so
! 5697: that most simple patterns do not use much memory. However, there is one
! 5698: case where memory usage can be unexpectedly large. When a parenthesized
! 5699: subpattern has a quantifier with a minimum greater than 1 and/or a lim-
! 5700: ited maximum, the whole subpattern is repeated in the compiled code.
! 5701: For example, the pattern
! 5702:
! 5703: (abc|def){2,4}
! 5704:
! 5705: is compiled as if it were
! 5706:
! 5707: (abc|def)(abc|def)((abc|def)(abc|def)?)?
! 5708:
! 5709: (Technical aside: It is done this way so that backtrack points within
! 5710: each of the repetitions can be independently maintained.)
! 5711:
! 5712: For regular expressions whose quantifiers use only small numbers, this
! 5713: is not usually a problem. However, if the numbers are large, and par-
! 5714: ticularly if such repetitions are nested, the memory usage can become
! 5715: an embarrassment. For example, the very simple pattern
! 5716:
! 5717: ((ab){1,1000}c){1,3}
! 5718:
! 5719: uses 51K bytes when compiled. When PCRE is compiled with its default
! 5720: internal pointer size of two bytes, the size limit on a compiled pat-
! 5721: tern is 64K, and this is reached with the above pattern if the outer
! 5722: repetition is increased from 3 to 4. PCRE can be compiled to use larger
! 5723: internal pointers and thus handle larger compiled patterns, but it is
! 5724: better to try to rewrite your pattern to use less memory if you can.
! 5725:
! 5726: One way of reducing the memory usage for such patterns is to make use
! 5727: of PCRE's "subroutine" facility. Re-writing the above pattern as
! 5728:
! 5729: ((ab)(?2){0,999}c)(?1){0,2}
! 5730:
! 5731: reduces the memory requirements to 18K, and indeed it remains under 20K
! 5732: even with the outer repetition increased to 100. However, this pattern
! 5733: is not exactly equivalent, because the "subroutine" calls are treated
! 5734: as atomic groups into which there can be no backtracking if there is a
! 5735: subsequent matching failure. Therefore, PCRE cannot do this kind of
! 5736: rewriting automatically. Furthermore, there is a noticeable loss of
! 5737: speed when executing the modified pattern. Nevertheless, if the atomic
! 5738: grouping is not a problem and the loss of speed is acceptable, this
! 5739: kind of rewriting will allow you to process patterns that PCRE cannot
! 5740: otherwise handle.
! 5741:
! 5742:
! 5743: PROCESSING TIME
! 5744:
! 5745: Certain items in regular expression patterns are processed more effi-
! 5746: ciently than others. It is more efficient to use a character class like
! 5747: [aeiou] than a set of single-character alternatives such as
! 5748: (a|e|i|o|u). In general, the simplest construction that provides the
! 5749: required behaviour is usually the most efficient. Jeffrey Friedl's book
! 5750: contains a lot of useful general discussion about optimizing regular
! 5751: expressions for efficient performance. This document contains a few
! 5752: observations about PCRE.
! 5753:
! 5754: Using Unicode character properties (the \p, \P, and \X escapes) is
! 5755: slow, because PCRE has to scan a structure that contains data for over
! 5756: fifteen thousand characters whenever it needs a character's property.
! 5757: If you can find an alternative pattern that does not use character
! 5758: properties, it will probably be faster.
! 5759:
! 5760: When a pattern begins with .* not in parentheses, or in parentheses
! 5761: that are not the subject of a backreference, and the PCRE_DOTALL option
! 5762: is set, the pattern is implicitly anchored by PCRE, since it can match
! 5763: only at the start of a subject string. However, if PCRE_DOTALL is not
! 5764: set, PCRE cannot make this optimization, because the . metacharacter
! 5765: does not then match a newline, and if the subject string contains new-
! 5766: lines, the pattern may match from the character immediately following
! 5767: one of them instead of from the very start. For example, the pattern
! 5768:
! 5769: .*second
! 5770:
! 5771: matches the subject "first\nand second" (where \n stands for a newline
! 5772: character), with the match starting at the seventh character. In order
! 5773: to do this, PCRE has to retry the match starting after every newline in
! 5774: the subject.
! 5775:
! 5776: If you are using such a pattern with subject strings that do not con-
! 5777: tain newlines, the best performance is obtained by setting PCRE_DOTALL,
! 5778: or starting the pattern with ^.* or ^.*? to indicate explicit anchor-
! 5779: ing. That saves PCRE from having to scan along the subject looking for
! 5780: a newline to restart at.
! 5781:
! 5782: Beware of patterns that contain nested indefinite repeats. These can
! 5783: take a long time to run when applied to a string that does not match.
! 5784: Consider the pattern fragment
! 5785:
! 5786: ^(a+)*
! 5787:
! 5788: This can match "aaaa" in 16 different ways, and this number increases
! 5789: very rapidly as the string gets longer. (The * repeat can match 0, 1,
! 5790: 2, 3, or 4 times, and for each of those cases other than 0 or 4, the +
! 5791: repeats can match different numbers of times.) When the remainder of
! 5792: the pattern is such that the entire match is going to fail, PCRE has in
! 5793: principle to try every possible variation, and this can take an
! 5794: extremely long time, even for relatively short strings.
! 5795:
! 5796: An optimization catches some of the more simple cases such as
! 5797:
! 5798: (a+)*b
! 5799:
! 5800: where a literal character follows. Before embarking on the standard
! 5801: matching procedure, PCRE checks that there is a "b" later in the sub-
! 5802: ject string, and if there is not, it fails the match immediately. How-
! 5803: ever, when there is no following literal this optimization cannot be
! 5804: used. You can see the difference by comparing the behaviour of
! 5805:
! 5806: (a+)*\d
! 5807:
! 5808: with the pattern above. The former gives a failure almost instantly
! 5809: when applied to a whole line of "a" characters, whereas the latter
! 5810: takes an appreciable time with strings longer than about 20 characters.
! 5811:
! 5812: In many cases, the solution to this kind of performance issue is to use
! 5813: an atomic group or a possessive quantifier.
! 5814:
! 5815:
! 5816: AUTHOR
! 5817:
! 5818: Philip Hazel
! 5819: University Computing Service
! 5820: Cambridge CB2 3QH, England.
! 5821:
! 5822:
! 5823: REVISION
! 5824:
! 5825: Last updated: 06 March 2007
! 5826: Copyright (c) 1997-2007 University of Cambridge.
! 5827: ------------------------------------------------------------------------------
! 5828:
! 5829:
! 5830: PCREPOSIX(3) PCREPOSIX(3)
! 5831:
! 5832:
! 5833: NAME
! 5834: PCRE - Perl-compatible regular expressions.
! 5835:
! 5836:
! 5837: SYNOPSIS OF POSIX API
! 5838:
! 5839: #include <pcreposix.h>
! 5840:
! 5841: int regcomp(regex_t *preg, const char *pattern,
! 5842: int cflags);
! 5843:
! 5844: int regexec(regex_t *preg, const char *string,
! 5845: size_t nmatch, regmatch_t pmatch[], int eflags);
! 5846:
! 5847: size_t regerror(int errcode, const regex_t *preg,
! 5848: char *errbuf, size_t errbuf_size);
! 5849:
! 5850: void regfree(regex_t *preg);
! 5851:
! 5852:
! 5853: DESCRIPTION
! 5854:
! 5855: This set of functions provides a POSIX-style API to the PCRE regular
! 5856: expression package. See the pcreapi documentation for a description of
! 5857: PCRE's native API, which contains much additional functionality.
! 5858:
! 5859: The functions described here are just wrapper functions that ultimately
! 5860: call the PCRE native API. Their prototypes are defined in the
! 5861: pcreposix.h header file, and on Unix systems the library itself is
! 5862: called pcreposix.a, so can be accessed by adding -lpcreposix to the
! 5863: command for linking an application that uses them. Because the POSIX
! 5864: functions call the native ones, it is also necessary to add -lpcre.
! 5865:
! 5866: I have implemented only those option bits that can be reasonably mapped
! 5867: to PCRE native options. In addition, the option REG_EXTENDED is defined
! 5868: with the value zero. This has no effect, but since programs that are
! 5869: written to the POSIX interface often use it, this makes it easier to
! 5870: slot in PCRE as a replacement library. Other POSIX options are not even
! 5871: defined.
! 5872:
! 5873: When PCRE is called via these functions, it is only the API that is
! 5874: POSIX-like in style. The syntax and semantics of the regular expres-
! 5875: sions themselves are still those of Perl, subject to the setting of
! 5876: various PCRE options, as described below. "POSIX-like in style" means
! 5877: that the API approximates to the POSIX definition; it is not fully
! 5878: POSIX-compatible, and in multi-byte encoding domains it is probably
! 5879: even less compatible.
! 5880:
! 5881: The header for these functions is supplied as pcreposix.h to avoid any
! 5882: potential clash with other POSIX libraries. It can, of course, be
! 5883: renamed or aliased as regex.h, which is the "correct" name. It provides
! 5884: two structure types, regex_t for compiled internal forms, and reg-
! 5885: match_t for returning captured substrings. It also defines some con-
! 5886: stants whose names start with "REG_"; these are used for setting
! 5887: options and identifying error codes.
! 5888:
! 5889:
! 5890: COMPILING A PATTERN
! 5891:
! 5892: The function regcomp() is called to compile a pattern into an internal
! 5893: form. The pattern is a C string terminated by a binary zero, and is
! 5894: passed in the argument pattern. The preg argument is a pointer to a
! 5895: regex_t structure that is used as a base for storing information about
! 5896: the compiled regular expression.
! 5897:
! 5898: The argument cflags is either zero, or contains one or more of the bits
! 5899: defined by the following macros:
! 5900:
! 5901: REG_DOTALL
! 5902:
! 5903: The PCRE_DOTALL option is set when the regular expression is passed for
! 5904: compilation to the native function. Note that REG_DOTALL is not part of
! 5905: the POSIX standard.
! 5906:
! 5907: REG_ICASE
! 5908:
! 5909: The PCRE_CASELESS option is set when the regular expression is passed
! 5910: for compilation to the native function.
! 5911:
! 5912: REG_NEWLINE
! 5913:
! 5914: The PCRE_MULTILINE option is set when the regular expression is passed
! 5915: for compilation to the native function. Note that this does not mimic
! 5916: the defined POSIX behaviour for REG_NEWLINE (see the following sec-
! 5917: tion).
! 5918:
! 5919: REG_NOSUB
! 5920:
! 5921: The PCRE_NO_AUTO_CAPTURE option is set when the regular expression is
! 5922: passed for compilation to the native function. In addition, when a pat-
! 5923: tern that is compiled with this flag is passed to regexec() for match-
! 5924: ing, the nmatch and pmatch arguments are ignored, and no captured
! 5925: strings are returned.
! 5926:
! 5927: REG_UTF8
! 5928:
! 5929: The PCRE_UTF8 option is set when the regular expression is passed for
! 5930: compilation to the native function. This causes the pattern itself and
! 5931: all data strings used for matching it to be treated as UTF-8 strings.
! 5932: Note that REG_UTF8 is not part of the POSIX standard.
! 5933:
! 5934: In the absence of these flags, no options are passed to the native
! 5935: function. This means the the regex is compiled with PCRE default
! 5936: semantics. In particular, the way it handles newline characters in the
! 5937: subject string is the Perl way, not the POSIX way. Note that setting
! 5938: PCRE_MULTILINE has only some of the effects specified for REG_NEWLINE.
! 5939: It does not affect the way newlines are matched by . (they aren't) or
! 5940: by a negative class such as [^a] (they are).
! 5941:
! 5942: The yield of regcomp() is zero on success, and non-zero otherwise. The
! 5943: preg structure is filled in on success, and one member of the structure
! 5944: is public: re_nsub contains the number of capturing subpatterns in the
! 5945: regular expression. Various error codes are defined in the header file.
! 5946:
! 5947:
! 5948: MATCHING NEWLINE CHARACTERS
! 5949:
! 5950: This area is not simple, because POSIX and Perl take different views of
! 5951: things. It is not possible to get PCRE to obey POSIX semantics, but
! 5952: then PCRE was never intended to be a POSIX engine. The following table
! 5953: lists the different possibilities for matching newline characters in
! 5954: PCRE:
! 5955:
! 5956: Default Change with
! 5957:
! 5958: . matches newline no PCRE_DOTALL
! 5959: newline matches [^a] yes not changeable
! 5960: $ matches \n at end yes PCRE_DOLLARENDONLY
! 5961: $ matches \n in middle no PCRE_MULTILINE
! 5962: ^ matches \n in middle no PCRE_MULTILINE
! 5963:
! 5964: This is the equivalent table for POSIX:
! 5965:
! 5966: Default Change with
! 5967:
! 5968: . matches newline yes REG_NEWLINE
! 5969: newline matches [^a] yes REG_NEWLINE
! 5970: $ matches \n at end no REG_NEWLINE
! 5971: $ matches \n in middle no REG_NEWLINE
! 5972: ^ matches \n in middle no REG_NEWLINE
! 5973:
! 5974: PCRE's behaviour is the same as Perl's, except that there is no equiva-
! 5975: lent for PCRE_DOLLAR_ENDONLY in Perl. In both PCRE and Perl, there is
! 5976: no way to stop newline from matching [^a].
! 5977:
! 5978: The default POSIX newline handling can be obtained by setting
! 5979: PCRE_DOTALL and PCRE_DOLLAR_ENDONLY, but there is no way to make PCRE
! 5980: behave exactly as for the REG_NEWLINE action.
! 5981:
! 5982:
! 5983: MATCHING A PATTERN
! 5984:
! 5985: The function regexec() is called to match a compiled pattern preg
! 5986: against a given string, which is by default terminated by a zero byte
! 5987: (but see REG_STARTEND below), subject to the options in eflags. These
! 5988: can be:
! 5989:
! 5990: REG_NOTBOL
! 5991:
! 5992: The PCRE_NOTBOL option is set when calling the underlying PCRE matching
! 5993: function.
! 5994:
! 5995: REG_NOTEOL
! 5996:
! 5997: The PCRE_NOTEOL option is set when calling the underlying PCRE matching
! 5998: function.
! 5999:
! 6000: REG_STARTEND
! 6001:
! 6002: The string is considered to start at string + pmatch[0].rm_so and to
! 6003: have a terminating NUL located at string + pmatch[0].rm_eo (there need
! 6004: not actually be a NUL at that location), regardless of the value of
! 6005: nmatch. This is a BSD extension, compatible with but not specified by
! 6006: IEEE Standard 1003.2 (POSIX.2), and should be used with caution in
! 6007: software intended to be portable to other systems. Note that a non-zero
! 6008: rm_so does not imply REG_NOTBOL; REG_STARTEND affects only the location
! 6009: of the string, not how it is matched.
! 6010:
! 6011: If the pattern was compiled with the REG_NOSUB flag, no data about any
! 6012: matched strings is returned. The nmatch and pmatch arguments of
! 6013: regexec() are ignored.
! 6014:
! 6015: Otherwise,the portion of the string that was matched, and also any cap-
! 6016: tured substrings, are returned via the pmatch argument, which points to
! 6017: an array of nmatch structures of type regmatch_t, containing the mem-
! 6018: bers rm_so and rm_eo. These contain the offset to the first character
! 6019: of each substring and the offset to the first character after the end
! 6020: of each substring, respectively. The 0th element of the vector relates
! 6021: to the entire portion of string that was matched; subsequent elements
! 6022: relate to the capturing subpatterns of the regular expression. Unused
! 6023: entries in the array have both structure members set to -1.
! 6024:
! 6025: A successful match yields a zero return; various error codes are
! 6026: defined in the header file, of which REG_NOMATCH is the "expected"
! 6027: failure code.
! 6028:
! 6029:
! 6030: ERROR MESSAGES
! 6031:
! 6032: The regerror() function maps a non-zero errorcode from either regcomp()
! 6033: or regexec() to a printable message. If preg is not NULL, the error
! 6034: should have arisen from the use of that structure. A message terminated
! 6035: by a binary zero is placed in errbuf. The length of the message,
! 6036: including the zero, is limited to errbuf_size. The yield of the func-
! 6037: tion is the size of buffer needed to hold the whole message.
! 6038:
! 6039:
! 6040: MEMORY USAGE
! 6041:
! 6042: Compiling a regular expression causes memory to be allocated and asso-
! 6043: ciated with the preg structure. The function regfree() frees all such
! 6044: memory, after which preg may no longer be used as a compiled expres-
! 6045: sion.
! 6046:
! 6047:
! 6048: AUTHOR
! 6049:
! 6050: Philip Hazel
! 6051: University Computing Service
! 6052: Cambridge CB2 3QH, England.
! 6053:
! 6054:
! 6055: REVISION
! 6056:
! 6057: Last updated: 05 April 2008
! 6058: Copyright (c) 1997-2008 University of Cambridge.
! 6059: ------------------------------------------------------------------------------
! 6060:
! 6061:
! 6062: PCRECPP(3) PCRECPP(3)
! 6063:
! 6064:
! 6065: NAME
! 6066: PCRE - Perl-compatible regular expressions.
! 6067:
! 6068:
! 6069: SYNOPSIS OF C++ WRAPPER
! 6070:
! 6071: #include <pcrecpp.h>
! 6072:
! 6073:
! 6074: DESCRIPTION
! 6075:
! 6076: The C++ wrapper for PCRE was provided by Google Inc. Some additional
! 6077: functionality was added by Giuseppe Maxia. This brief man page was con-
! 6078: structed from the notes in the pcrecpp.h file, which should be con-
! 6079: sulted for further details.
! 6080:
! 6081:
! 6082: MATCHING INTERFACE
! 6083:
! 6084: The "FullMatch" operation checks that supplied text matches a supplied
! 6085: pattern exactly. If pointer arguments are supplied, it copies matched
! 6086: sub-strings that match sub-patterns into them.
! 6087:
! 6088: Example: successful match
! 6089: pcrecpp::RE re("h.*o");
! 6090: re.FullMatch("hello");
! 6091:
! 6092: Example: unsuccessful match (requires full match):
! 6093: pcrecpp::RE re("e");
! 6094: !re.FullMatch("hello");
! 6095:
! 6096: Example: creating a temporary RE object:
! 6097: pcrecpp::RE("h.*o").FullMatch("hello");
! 6098:
! 6099: You can pass in a "const char*" or a "string" for "text". The examples
! 6100: below tend to use a const char*. You can, as in the different examples
! 6101: above, store the RE object explicitly in a variable or use a temporary
! 6102: RE object. The examples below use one mode or the other arbitrarily.
! 6103: Either could correctly be used for any of these examples.
! 6104:
! 6105: You must supply extra pointer arguments to extract matched subpieces.
! 6106:
! 6107: Example: extracts "ruby" into "s" and 1234 into "i"
! 6108: int i;
! 6109: string s;
! 6110: pcrecpp::RE re("(\\w+):(\\d+)");
! 6111: re.FullMatch("ruby:1234", &s, &i);
! 6112:
! 6113: Example: does not try to extract any extra sub-patterns
! 6114: re.FullMatch("ruby:1234", &s);
! 6115:
! 6116: Example: does not try to extract into NULL
! 6117: re.FullMatch("ruby:1234", NULL, &i);
! 6118:
! 6119: Example: integer overflow causes failure
! 6120: !re.FullMatch("ruby:1234567891234", NULL, &i);
! 6121:
! 6122: Example: fails because there aren't enough sub-patterns:
! 6123: !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s);
! 6124:
! 6125: Example: fails because string cannot be stored in integer
! 6126: !pcrecpp::RE("(.*)").FullMatch("ruby", &i);
! 6127:
! 6128: The provided pointer arguments can be pointers to any scalar numeric
! 6129: type, or one of:
! 6130:
! 6131: string (matched piece is copied to string)
! 6132: StringPiece (StringPiece is mutated to point to matched piece)
! 6133: T (where "bool T::ParseFrom(const char*, int)" exists)
! 6134: NULL (the corresponding matched sub-pattern is not copied)
! 6135:
! 6136: The function returns true iff all of the following conditions are sat-
! 6137: isfied:
! 6138:
! 6139: a. "text" matches "pattern" exactly;
! 6140:
! 6141: b. The number of matched sub-patterns is >= number of supplied
! 6142: pointers;
! 6143:
! 6144: c. The "i"th argument has a suitable type for holding the
! 6145: string captured as the "i"th sub-pattern. If you pass in
! 6146: void * NULL for the "i"th argument, or a non-void * NULL
! 6147: of the correct type, or pass fewer arguments than the
! 6148: number of sub-patterns, "i"th captured sub-pattern is
! 6149: ignored.
! 6150:
! 6151: CAVEAT: An optional sub-pattern that does not exist in the matched
! 6152: string is assigned the empty string. Therefore, the following will
! 6153: return false (because the empty string is not a valid number):
! 6154:
! 6155: int number;
! 6156: pcrecpp::RE::FullMatch("abc", "[a-z]+(\\d+)?", &number);
! 6157:
! 6158: The matching interface supports at most 16 arguments per call. If you
! 6159: need more, consider using the more general interface
! 6160: pcrecpp::RE::DoMatch. See pcrecpp.h for the signature for DoMatch.
! 6161:
! 6162:
! 6163: QUOTING METACHARACTERS
! 6164:
! 6165: You can use the "QuoteMeta" operation to insert backslashes before all
! 6166: potentially meaningful characters in a string. The returned string,
! 6167: used as a regular expression, will exactly match the original string.
! 6168:
! 6169: Example:
! 6170: string quoted = RE::QuoteMeta(unquoted);
! 6171:
! 6172: Note that it's legal to escape a character even if it has no special
! 6173: meaning in a regular expression -- so this function does that. (This
! 6174: also makes it identical to the perl function of the same name; see
! 6175: "perldoc -f quotemeta".) For example, "1.5-2.0?" becomes
! 6176: "1\.5\-2\.0\?".
! 6177:
! 6178:
! 6179: PARTIAL MATCHES
! 6180:
! 6181: You can use the "PartialMatch" operation when you want the pattern to
! 6182: match any substring of the text.
! 6183:
! 6184: Example: simple search for a string:
! 6185: pcrecpp::RE("ell").PartialMatch("hello");
! 6186:
! 6187: Example: find first number in a string:
! 6188: int number;
! 6189: pcrecpp::RE re("(\\d+)");
! 6190: re.PartialMatch("x*100 + 20", &number);
! 6191: assert(number == 100);
! 6192:
! 6193:
! 6194: UTF-8 AND THE MATCHING INTERFACE
! 6195:
! 6196: By default, pattern and text are plain text, one byte per character.
! 6197: The UTF8 flag, passed to the constructor, causes both pattern and
! 6198: string to be treated as UTF-8 text, still a byte stream but potentially
! 6199: multiple bytes per character. In practice, the text is likelier to be
! 6200: UTF-8 than the pattern, but the match returned may depend on the UTF8
! 6201: flag, so always use it when matching UTF8 text. For example, "." will
! 6202: match one byte normally but with UTF8 set may match up to three bytes
! 6203: of a multi-byte character.
! 6204:
! 6205: Example:
! 6206: pcrecpp::RE_Options options;
! 6207: options.set_utf8();
! 6208: pcrecpp::RE re(utf8_pattern, options);
! 6209: re.FullMatch(utf8_string);
! 6210:
! 6211: Example: using the convenience function UTF8():
! 6212: pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8());
! 6213: re.FullMatch(utf8_string);
! 6214:
! 6215: NOTE: The UTF8 flag is ignored if pcre was not configured with the
! 6216: --enable-utf8 flag.
! 6217:
! 6218:
! 6219: PASSING MODIFIERS TO THE REGULAR EXPRESSION ENGINE
! 6220:
! 6221: PCRE defines some modifiers to change the behavior of the regular
! 6222: expression engine. The C++ wrapper defines an auxiliary class,
! 6223: RE_Options, as a vehicle to pass such modifiers to a RE class. Cur-
! 6224: rently, the following modifiers are supported:
! 6225:
! 6226: modifier description Perl corresponding
! 6227:
! 6228: PCRE_CASELESS case insensitive match /i
! 6229: PCRE_MULTILINE multiple lines match /m
! 6230: PCRE_DOTALL dot matches newlines /s
! 6231: PCRE_DOLLAR_ENDONLY $ matches only at end N/A
! 6232: PCRE_EXTRA strict escape parsing N/A
! 6233: PCRE_EXTENDED ignore whitespaces /x
! 6234: PCRE_UTF8 handles UTF8 chars built-in
! 6235: PCRE_UNGREEDY reverses * and *? N/A
! 6236: PCRE_NO_AUTO_CAPTURE disables capturing parens N/A (*)
! 6237:
! 6238: (*) Both Perl and PCRE allow non capturing parentheses by means of the
! 6239: "?:" modifier within the pattern itself. e.g. (?:ab|cd) does not cap-
! 6240: ture, while (ab|cd) does.
! 6241:
! 6242: For a full account on how each modifier works, please check the PCRE
! 6243: API reference page.
! 6244:
! 6245: For each modifier, there are two member functions whose name is made
! 6246: out of the modifier in lowercase, without the "PCRE_" prefix. For
! 6247: instance, PCRE_CASELESS is handled by
! 6248:
! 6249: bool caseless()
! 6250:
! 6251: which returns true if the modifier is set, and
! 6252:
! 6253: RE_Options & set_caseless(bool)
! 6254:
! 6255: which sets or unsets the modifier. Moreover, PCRE_EXTRA_MATCH_LIMIT can
! 6256: be accessed through the set_match_limit() and match_limit() member
! 6257: functions. Setting match_limit to a non-zero value will limit the exe-
! 6258: cution of pcre to keep it from doing bad things like blowing the stack
! 6259: or taking an eternity to return a result. A value of 5000 is good
! 6260: enough to stop stack blowup in a 2MB thread stack. Setting match_limit
! 6261: to zero disables match limiting. Alternatively, you can call
! 6262: match_limit_recursion() which uses PCRE_EXTRA_MATCH_LIMIT_RECURSION to
! 6263: limit how much PCRE recurses. match_limit() limits the number of
! 6264: matches PCRE does; match_limit_recursion() limits the depth of internal
! 6265: recursion, and therefore the amount of stack that is used.
! 6266:
! 6267: Normally, to pass one or more modifiers to a RE class, you declare a
! 6268: RE_Options object, set the appropriate options, and pass this object to
! 6269: a RE constructor. Example:
! 6270:
! 6271: RE_options opt;
! 6272: opt.set_caseless(true);
! 6273: if (RE("HELLO", opt).PartialMatch("hello world")) ...
! 6274:
! 6275: RE_options has two constructors. The default constructor takes no argu-
! 6276: ments and creates a set of flags that are off by default. The optional
! 6277: parameter option_flags is to facilitate transfer of legacy code from C
! 6278: programs. This lets you do
! 6279:
! 6280: RE(pattern,
! 6281: RE_Options(PCRE_CASELESS|PCRE_MULTILINE)).PartialMatch(str);
! 6282:
! 6283: However, new code is better off doing
! 6284:
! 6285: RE(pattern,
! 6286: RE_Options().set_caseless(true).set_multiline(true))
! 6287: .PartialMatch(str);
! 6288:
! 6289: If you are going to pass one of the most used modifiers, there are some
! 6290: convenience functions that return a RE_Options class with the appropri-
! 6291: ate modifier already set: CASELESS(), UTF8(), MULTILINE(), DOTALL(),
! 6292: and EXTENDED().
! 6293:
! 6294: If you need to set several options at once, and you don't want to go
! 6295: through the pains of declaring a RE_Options object and setting several
! 6296: options, there is a parallel method that give you such ability on the
! 6297: fly. You can concatenate several set_xxxxx() member functions, since
! 6298: each of them returns a reference to its class object. For example, to
! 6299: pass PCRE_CASELESS, PCRE_EXTENDED, and PCRE_MULTILINE to a RE with one
! 6300: statement, you may write:
! 6301:
! 6302: RE(" ^ xyz \\s+ .* blah$",
! 6303: RE_Options()
! 6304: .set_caseless(true)
! 6305: .set_extended(true)
! 6306: .set_multiline(true)).PartialMatch(sometext);
! 6307:
! 6308:
! 6309: SCANNING TEXT INCREMENTALLY
! 6310:
! 6311: The "Consume" operation may be useful if you want to repeatedly match
! 6312: regular expressions at the front of a string and skip over them as they
! 6313: match. This requires use of the "StringPiece" type, which represents a
! 6314: sub-range of a real string. Like RE, StringPiece is defined in the
! 6315: pcrecpp namespace.
! 6316:
! 6317: Example: read lines of the form "var = value" from a string.
! 6318: string contents = ...; // Fill string somehow
! 6319: pcrecpp::StringPiece input(contents); // Wrap in a StringPiece
! 6320:
! 6321: string var;
! 6322: int value;
! 6323: pcrecpp::RE re("(\\w+) = (\\d+)\n");
! 6324: while (re.Consume(&input, &var, &value)) {
! 6325: ...;
! 6326: }
! 6327:
! 6328: Each successful call to "Consume" will set "var/value", and also
! 6329: advance "input" so it points past the matched text.
! 6330:
! 6331: The "FindAndConsume" operation is similar to "Consume" but does not
! 6332: anchor your match at the beginning of the string. For example, you
! 6333: could extract all words from a string by repeatedly calling
! 6334:
! 6335: pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word)
! 6336:
! 6337:
! 6338: PARSING HEX/OCTAL/C-RADIX NUMBERS
! 6339:
! 6340: By default, if you pass a pointer to a numeric value, the corresponding
! 6341: text is interpreted as a base-10 number. You can instead wrap the
! 6342: pointer with a call to one of the operators Hex(), Octal(), or CRadix()
! 6343: to interpret the text in another base. The CRadix operator interprets
! 6344: C-style "0" (base-8) and "0x" (base-16) prefixes, but defaults to
! 6345: base-10.
! 6346:
! 6347: Example:
! 6348: int a, b, c, d;
! 6349: pcrecpp::RE re("(.*) (.*) (.*) (.*)");
! 6350: re.FullMatch("100 40 0100 0x40",
! 6351: pcrecpp::Octal(&a), pcrecpp::Hex(&b),
! 6352: pcrecpp::CRadix(&c), pcrecpp::CRadix(&d));
! 6353:
! 6354: will leave 64 in a, b, c, and d.
! 6355:
! 6356:
! 6357: REPLACING PARTS OF STRINGS
! 6358:
! 6359: You can replace the first match of "pattern" in "str" with "rewrite".
! 6360: Within "rewrite", backslash-escaped digits (\1 to \9) can be used to
! 6361: insert text matching corresponding parenthesized group from the pat-
! 6362: tern. \0 in "rewrite" refers to the entire matching text. For example:
! 6363:
! 6364: string s = "yabba dabba doo";
! 6365: pcrecpp::RE("b+").Replace("d", &s);
! 6366:
! 6367: will leave "s" containing "yada dabba doo". The result is true if the
! 6368: pattern matches and a replacement occurs, false otherwise.
! 6369:
! 6370: GlobalReplace is like Replace except that it replaces all occurrences
! 6371: of the pattern in the string with the rewrite. Replacements are not
! 6372: subject to re-matching. For example:
! 6373:
! 6374: string s = "yabba dabba doo";
! 6375: pcrecpp::RE("b+").GlobalReplace("d", &s);
! 6376:
! 6377: will leave "s" containing "yada dada doo". It returns the number of
! 6378: replacements made.
! 6379:
! 6380: Extract is like Replace, except that if the pattern matches, "rewrite"
! 6381: is copied into "out" (an additional argument) with substitutions. The
! 6382: non-matching portions of "text" are ignored. Returns true iff a match
! 6383: occurred and the extraction happened successfully; if no match occurs,
! 6384: the string is left unaffected.
! 6385:
! 6386:
! 6387: AUTHOR
! 6388:
! 6389: The C++ wrapper was contributed by Google Inc.
! 6390: Copyright (c) 2007 Google Inc.
! 6391:
! 6392:
! 6393: REVISION
! 6394:
! 6395: Last updated: 12 November 2007
! 6396: ------------------------------------------------------------------------------
! 6397:
! 6398:
! 6399: PCRESAMPLE(3) PCRESAMPLE(3)
! 6400:
! 6401:
! 6402: NAME
! 6403: PCRE - Perl-compatible regular expressions
! 6404:
! 6405:
! 6406: PCRE SAMPLE PROGRAM
! 6407:
! 6408: A simple, complete demonstration program, to get you started with using
! 6409: PCRE, is supplied in the file pcredemo.c in the PCRE distribution.
! 6410:
! 6411: The program compiles the regular expression that is its first argument,
! 6412: and matches it against the subject string in its second argument. No
! 6413: PCRE options are set, and default character tables are used. If match-
! 6414: ing succeeds, the program outputs the portion of the subject that
! 6415: matched, together with the contents of any captured substrings.
! 6416:
! 6417: If the -g option is given on the command line, the program then goes on
! 6418: to check for further matches of the same regular expression in the same
! 6419: subject string. The logic is a little bit tricky because of the possi-
! 6420: bility of matching an empty string. Comments in the code explain what
! 6421: is going on.
! 6422:
! 6423: If PCRE is installed in the standard include and library directories
! 6424: for your system, you should be able to compile the demonstration pro-
! 6425: gram using this command:
! 6426:
! 6427: gcc -o pcredemo pcredemo.c -lpcre
! 6428:
! 6429: If PCRE is installed elsewhere, you may need to add additional options
! 6430: to the command line. For example, on a Unix-like system that has PCRE
! 6431: installed in /usr/local, you can compile the demonstration program
! 6432: using a command like this:
! 6433:
! 6434: gcc -o pcredemo -I/usr/local/include pcredemo.c \
! 6435: -L/usr/local/lib -lpcre
! 6436:
! 6437: Once you have compiled the demonstration program, you can run simple
! 6438: tests like this:
! 6439:
! 6440: ./pcredemo 'cat|dog' 'the cat sat on the mat'
! 6441: ./pcredemo -g 'cat|dog' 'the dog sat on the cat'
! 6442:
! 6443: Note that there is a much more comprehensive test program, called
! 6444: pcretest, which supports many more facilities for testing regular
! 6445: expressions and the PCRE library. The pcredemo program is provided as a
! 6446: simple coding example.
! 6447:
! 6448: On some operating systems (e.g. Solaris), when PCRE is not installed in
! 6449: the standard library directory, you may get an error like this when you
! 6450: try to run pcredemo:
! 6451:
! 6452: ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or
! 6453: directory
! 6454:
! 6455: This is caused by the way shared library support works on those sys-
! 6456: tems. You need to add
! 6457:
! 6458: -R/usr/local/lib
! 6459:
! 6460: (for example) to the compile command to get round this problem.
! 6461:
! 6462:
! 6463: AUTHOR
! 6464:
! 6465: Philip Hazel
! 6466: University Computing Service
! 6467: Cambridge CB2 3QH, England.
! 6468:
! 6469:
! 6470: REVISION
! 6471:
! 6472: Last updated: 23 January 2008
! 6473: Copyright (c) 1997-2008 University of Cambridge.
! 6474: ------------------------------------------------------------------------------
! 6475: PCRESTACK(3) PCRESTACK(3)
! 6476:
! 6477:
! 6478: NAME
! 6479: PCRE - Perl-compatible regular expressions
! 6480:
! 6481:
! 6482: PCRE DISCUSSION OF STACK USAGE
! 6483:
! 6484: When you call pcre_exec(), it makes use of an internal function called
! 6485: match(). This calls itself recursively at branch points in the pattern,
! 6486: in order to remember the state of the match so that it can back up and
! 6487: try a different alternative if the first one fails. As matching pro-
! 6488: ceeds deeper and deeper into the tree of possibilities, the recursion
! 6489: depth increases.
! 6490:
! 6491: Not all calls of match() increase the recursion depth; for an item such
! 6492: as a* it may be called several times at the same level, after matching
! 6493: different numbers of a's. Furthermore, in a number of cases where the
! 6494: result of the recursive call would immediately be passed back as the
! 6495: result of the current call (a "tail recursion"), the function is just
! 6496: restarted instead.
! 6497:
! 6498: The pcre_dfa_exec() function operates in an entirely different way, and
! 6499: hardly uses recursion at all. The limit on its complexity is the amount
! 6500: of workspace it is given. The comments that follow do NOT apply to
! 6501: pcre_dfa_exec(); they are relevant only for pcre_exec().
! 6502:
! 6503: You can set limits on the number of times that match() is called, both
! 6504: in total and recursively. If the limit is exceeded, an error occurs.
! 6505: For details, see the section on extra data for pcre_exec() in the
! 6506: pcreapi documentation.
! 6507:
! 6508: Each time that match() is actually called recursively, it uses memory
! 6509: from the process stack. For certain kinds of pattern and data, very
! 6510: large amounts of stack may be needed, despite the recognition of "tail
! 6511: recursion". You can often reduce the amount of recursion, and there-
! 6512: fore the amount of stack used, by modifying the pattern that is being
! 6513: matched. Consider, for example, this pattern:
! 6514:
! 6515: ([^<]|<(?!inet))+
! 6516:
! 6517: It matches from wherever it starts until it encounters "<inet" or the
! 6518: end of the data, and is the kind of pattern that might be used when
! 6519: processing an XML file. Each iteration of the outer parentheses matches
! 6520: either one character that is not "<" or a "<" that is not followed by
! 6521: "inet". However, each time a parenthesis is processed, a recursion
! 6522: occurs, so this formulation uses a stack frame for each matched charac-
! 6523: ter. For a long string, a lot of stack is required. Consider now this
! 6524: rewritten pattern, which matches exactly the same strings:
! 6525:
! 6526: ([^<]++|<(?!inet))+
! 6527:
! 6528: This uses very much less stack, because runs of characters that do not
! 6529: contain "<" are "swallowed" in one item inside the parentheses. Recur-
! 6530: sion happens only when a "<" character that is not followed by "inet"
! 6531: is encountered (and we assume this is relatively rare). A possessive
! 6532: quantifier is used to stop any backtracking into the runs of non-"<"
! 6533: characters, but that is not related to stack usage.
! 6534:
! 6535: This example shows that one way of avoiding stack problems when match-
! 6536: ing long subject strings is to write repeated parenthesized subpatterns
! 6537: to match more than one character whenever possible.
! 6538:
! 6539: In environments where stack memory is constrained, you might want to
! 6540: compile PCRE to use heap memory instead of stack for remembering back-
! 6541: up points. This makes it run a lot more slowly, however. Details of how
! 6542: to do this are given in the pcrebuild documentation. When built in this
! 6543: way, instead of using the stack, PCRE obtains and frees memory by call-
! 6544: ing the functions that are pointed to by the pcre_stack_malloc and
! 6545: pcre_stack_free variables. By default, these point to malloc() and
! 6546: free(), but you can replace the pointers to cause PCRE to use your own
! 6547: functions. Since the block sizes are always the same, and are always
! 6548: freed in reverse order, it may be possible to implement customized mem-
! 6549: ory handlers that are more efficient than the standard functions.
! 6550:
! 6551: In Unix-like environments, there is not often a problem with the stack
! 6552: unless very long strings are involved, though the default limit on
! 6553: stack size varies from system to system. Values from 8Mb to 64Mb are
! 6554: common. You can find your default limit by running the command:
! 6555:
! 6556: ulimit -s
! 6557:
! 6558: Unfortunately, the effect of running out of stack is often SIGSEGV,
! 6559: though sometimes a more explicit error message is given. You can nor-
! 6560: mally increase the limit on stack size by code such as this:
! 6561:
! 6562: struct rlimit rlim;
! 6563: getrlimit(RLIMIT_STACK, &rlim);
! 6564: rlim.rlim_cur = 100*1024*1024;
! 6565: setrlimit(RLIMIT_STACK, &rlim);
! 6566:
! 6567: This reads the current limits (soft and hard) using getrlimit(), then
! 6568: attempts to increase the soft limit to 100Mb using setrlimit(). You
! 6569: must do this before calling pcre_exec().
! 6570:
! 6571: PCRE has an internal counter that can be used to limit the depth of
! 6572: recursion, and thus cause pcre_exec() to give an error code before it
! 6573: runs out of stack. By default, the limit is very large, and unlikely
! 6574: ever to operate. It can be changed when PCRE is built, and it can also
! 6575: be set when pcre_exec() is called. For details of these interfaces, see
! 6576: the pcrebuild and pcreapi documentation.
! 6577:
! 6578: As a very rough rule of thumb, you should reckon on about 500 bytes per
! 6579: recursion. Thus, if you want to limit your stack usage to 8Mb, you
! 6580: should set the limit at 16000 recursions. A 64Mb stack, on the other
! 6581: hand, can support around 128000 recursions. The pcretest test program
! 6582: has a command line option (-S) that can be used to increase the size of
! 6583: its stack.
! 6584:
! 6585:
! 6586: AUTHOR
! 6587:
! 6588: Philip Hazel
! 6589: University Computing Service
! 6590: Cambridge CB2 3QH, England.
! 6591:
! 6592:
! 6593: REVISION
! 6594:
! 6595: Last updated: 05 June 2007
! 6596: Copyright (c) 1997-2007 University of Cambridge.
! 6597: ------------------------------------------------------------------------------
! 6598:
! 6599:
E-mail:
![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to pcre.txt CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [parser3project] / win32 / pcre