--- win32/pcre/README 2009/04/10 12:53:36 1.2 +++ win32/pcre/README 2012/05/25 11:44:03 1.5 @@ -24,6 +24,7 @@ The contents of this README file are: Shared libraries on Unix-like systems Cross-compiling on Unix-like systems Using HP's ANSI C++ compiler (aCC) + Using PCRE from MySQL Making new tarballs Testing PCRE Character tables @@ -33,16 +34,19 @@ The contents of this README file are: The PCRE APIs ------------- -PCRE is written in C, and it has its own API. The distribution also includes a -set of C++ wrapper functions (see the pcrecpp man page for details), courtesy -of Google Inc. - -In addition, there is a set of C wrapper functions that are based on the POSIX -regular expression API (see the pcreposix man page). These end up in the -library called libpcreposix. Note that this just provides a POSIX calling -interface to PCRE; the regular expressions themselves still follow Perl syntax -and semantics. The POSIX API is restricted, and does not give full access to -all of PCRE's facilities. +PCRE is written in C, and it has its own API. There are two sets of functions, +one for the 8-bit library, which processes strings of bytes, and one for the +16-bit library, which processes strings of 16-bit values. The distribution also +includes a set of C++ wrapper functions (see the pcrecpp man page for details), +courtesy of Google Inc., which can be used to call the 8-bit PCRE library from +C++. + +In addition, there is a set of C wrapper functions (again, just for the 8-bit +library) that are based on the POSIX regular expression API (see the pcreposix +man page). These end up in the library called libpcreposix. Note that this just +provides a POSIX calling interface to PCRE; the regular expressions themselves +still follow Perl syntax and semantics. The POSIX API is restricted, and does +not give full access to all of PCRE's facilities. The header file for the POSIX-style functions is called pcreposix.h. The official POSIX name is regex.h, but I did not want to risk possible problems @@ -85,6 +89,10 @@ documentation is supplied in two other f in various ways, and rooted in a file called index.html, is distributed in doc/html and installed in /share/doc/pcre/html. +Users of PCRE have contributed files containing the documentation for various +releases in CHM format. These can be found in the Contrib directory of the FTP +site (see next section). + Contributions by users of PCRE ------------------------------ @@ -107,8 +115,8 @@ Building PCRE on non-Unix systems For a non-Unix system, please read the comments in the file NON-UNIX-USE, though if your system supports the use of "configure" and "make" you may be able to build PCRE in the same way as for Unix-like systems. PCRE can also be -configured in many platform environments using the GUI facility of CMake's -CMakeSetup. It creates Makefiles, solution files, etc. +configured in many platform environments using the GUI facility provided by +CMake's cmake-gui command. This creates Makefiles, solution files, etc. PCRE has been compiled on many different operating systems. It should be straightforward to build PCRE on any system that has a Standard C compiler and @@ -138,9 +146,9 @@ the usual methods of changing standard d CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local -specifies that the C compiler should be run with the flags '-O2 -Wall' instead -of the default, and that "make install" should install PCRE under /opt/local -instead of the default /usr/local. +This command specifies that the C compiler should be run with the flags '-O2 +-Wall' instead of the default, and that "make install" should install PCRE +under /opt/local instead of the default /usr/local. If you want to build in a different directory, just run "configure" with that directory as current. For example, suppose you have unpacked the PCRE source @@ -154,24 +162,59 @@ possible to build it as a C++ library, t does not have any features to support this. There are some optional features that can be included or omitted from the PCRE -library. You can read more about them in the pcrebuild man page. +library. They are also documented in the pcrebuild man page. + +. By default, both shared and static libraries are built. You can change this + by adding one of these options to the "configure" command: + + --disable-shared + --disable-static -. If you want to suppress the building of the C++ wrapper library, you can add - --disable-cpp to the "configure" command. Otherwise, when "configure" is run, - it will try to find a C++ compiler and C++ header files, and if it succeeds, - it will try to build the C++ wrapper. - -. If you want to make use of the support for UTF-8 character strings in PCRE, - you must add --enable-utf8 to the "configure" command. Without it, the code - for handling UTF-8 is not included in the library. (Even when included, it - still has to be enabled by an option at run time.) - -. If, in addition to support for UTF-8 character strings, you want to include - support for the \P, \p, and \X sequences that recognize Unicode character - properties, you must add --enable-unicode-properties to the "configure" - command. This adds about 30K to the size of the library (in the form of a - property table); only the basic two-letter properties such as Lu are - supported. + (See also "Shared libraries on Unix-like systems" below.) + +. By default, only the 8-bit library is built. If you add --enable-pcre16 to + the "configure" command, the 16-bit library is also built. If you want only + the 16-bit library, use "./configure --enable-pcre16 --disable-pcre8". + +. If you are building the 8-bit library and want to suppress the building of + the C++ wrapper library, you can add --disable-cpp to the "configure" + command. Otherwise, when "configure" is run without --disable-pcre8, it will + try to find a C++ compiler and C++ header files, and if it succeeds, it will + try to build the C++ wrapper. + +. If you want to include support for just-in-time compiling, which can give + large performance improvements on certain platforms, add --enable-jit to the + "configure" command. This support is available only for certain hardware + architectures. If you try to enable it on an unsupported architecture, there + will be a compile time error. + +. When JIT support is enabled, pcregrep automatically makes use of it, unless + you add --disable-pcregrep-jit to the "configure" command. + +. If you want to make use of the support for UTF-8 Unicode character strings in + the 8-bit library, or UTF-16 Unicode character strings in the 16-bit library, + you must add --enable-utf to the "configure" command. Without it, the code + for handling UTF-8 and UTF-16 is not included in the relevant library. Even + when --enable-utf is included, the use of a UTF encoding still has to be + enabled by an option at run time. When PCRE is compiled with this option, its + input can only either be ASCII or UTF-8/16, even when running on EBCDIC + platforms. It is not possible to use both --enable-utf and --enable-ebcdic at + the same time. + +. There are no separate options for enabling UTF-8 and UTF-16 independently + because that would allow ridiculous settings such as requesting UTF-16 + support while building only the 8-bit library. However, the option + --enable-utf8 is retained for backwards compatibility with earlier releases + that did not support 16-bit character strings. It is synonymous with + --enable-utf. It is not possible to configure one library with UTF support + and the other without in the same configuration. + +. If, in addition to support for UTF-8/16 character strings, you want to + include support for the \P, \p, and \X sequences that recognize Unicode + character properties, you must add --enable-unicode-properties to the + "configure" command. This adds about 30K to the size of the library (in the + form of a property table); only the basic two-letter properties such as Lu + are supported. . You can build PCRE to recognize either CR or LF or the sequence CRLF or any of the preceding, or any of the Unicode newline sequences as indicating the @@ -224,10 +267,11 @@ library. You can read more about them in sizes in the pcrestack man page. . The default maximum compiled pattern size is around 64K. You can increase - this by adding --with-link-size=3 to the "configure" command. You can - increase it even more by setting --with-link-size=4, but this is unlikely - ever to be necessary. Increasing the internal link size will reduce - performance. + this by adding --with-link-size=3 to the "configure" command. In the 8-bit + library, PCRE then uses three bytes instead of two for offsets to different + parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is + the same as --with-link-size=4, which (in both libraries) uses four-byte + offsets. Increasing the internal link size reduces performance. . You can build PCRE so that its internal match() function that is called from pcre_exec() does not call itself recursively. Instead, it uses memory blocks @@ -239,9 +283,10 @@ library. You can read more about them in on the "configure" command. PCRE runs more slowly in this mode, but it may be necessary in environments with limited stack sizes. This applies only to the - pcre_exec() function; it does not apply to pcre_dfa_exec(), which does not - use deeply nested recursion. There is a discussion about stack sizes in the - pcrestack man page. + normal execution of the pcre_exec() function; if JIT support is being + successfully used, it is not relevant. Equally, it does not apply to + pcre_dfa_exec(), which does not use deeply nested recursion. There is a + discussion about stack sizes in the pcrestack man page. . For speed, PCRE uses four tables for manipulating and identifying characters whose code point values are less than 256. By default, it uses a set of @@ -255,20 +300,31 @@ library. You can read more about them in pcre_chartables.c.dist. See "Character tables" below for further information. . It is possible to compile PCRE for use on systems that use EBCDIC as their - default character code (as opposed to ASCII) by specifying + character code (as opposed to ASCII) by specifying --enable-ebcdic - This automatically implies --enable-rebuild-chartables (see above). - -. It is possible to compile pcregrep to use libz and/or libbz2, in order to - read .gz and .bz2 files (respectively), by specifying one or both of + This automatically implies --enable-rebuild-chartables (see above). However, + when PCRE is built this way, it always operates in EBCDIC. It cannot support + both EBCDIC and UTF-8/16. + +. The pcregrep program currently supports only 8-bit data files, and so + requires the 8-bit PCRE library. It is possible to compile pcregrep to use + libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by + specifying one or both of --enable-pcregrep-libz --enable-pcregrep-libbz2 Of course, the relevant libraries must be installed on your system. +. The default size of internal buffer used by pcregrep can be set by, for + example: + + --with-pcregrep-bufsize=50K + + The default value is 20K. + . It is possible to compile pcretest so that it links with the libreadline library, by specifying @@ -286,41 +342,50 @@ library. You can read more about them in to specify something like LIBS="-lncurses" as well. This is because, to quote the readline INSTALL, "Readline uses the termcap functions, but does not link with the termcap or curses library itself, allowing applications which link - with readline the to choose an appropriate library." + with readline the to choose an appropriate library." If you get error + messages about missing functions tgetstr, tgetent, tputs, tgetflag, or tgoto, + this is the problem, and linking with the ncurses library should fix it. The "configure" script builds the following files for the basic C library: -. Makefile is the makefile that builds the library -. config.h contains build-time configuration options for the library -. pcre.h is the public PCRE header file -. pcre-config is a script that shows the settings of "configure" options -. libpcre.pc is data for the pkg-config command -. libtool is a script that builds shared and/or static libraries -. RunTest is a script for running tests on the basic C library -. RunGrepTest is a script for running tests on the pcregrep command - -Versions of config.h and pcre.h are distributed in the PCRE tarballs under -the names config.h.generic and pcre.h.generic. These are provided for the -benefit of those who have to built PCRE without the benefit of "configure". If -you use "configure", the .generic versions are not used. - -If a C++ compiler is found, the following files are also built: - -. libpcrecpp.pc is data for the pkg-config command -. pcrecpparg.h is a header file for programs that call PCRE via the C++ wrapper -. pcre_stringpiece.h is the header for the C++ "stringpiece" functions +. Makefile the makefile that builds the library +. config.h build-time configuration options for the library +. pcre.h the public PCRE header file +. pcre-config script that shows the building settings such as CFLAGS + that were set for "configure" +. libpcre.pc ) data for the pkg-config command +. libpcre16.pc ) +. libpcreposix.pc ) +. libtool script that builds shared and/or static libraries +. RunTest script for running tests on the basic C library +. RunGrepTest script for running tests on the pcregrep command + +Versions of config.h and pcre.h are distributed in the PCRE tarballs under the +names config.h.generic and pcre.h.generic. These are provided for those who +have to built PCRE without using "configure" or CMake. If you use "configure" +or CMake, the .generic versions are not used. + +When building the 8-bit library, if a C++ compiler is found, the following +files are also built: + +. libpcrecpp.pc data for the pkg-config command +. pcrecpparg.h header file for calling PCRE via the C++ wrapper +. pcre_stringpiece.h header for the C++ "stringpiece" functions The "configure" script also creates config.status, which is an executable script that can be run to recreate the configuration, and config.log, which contains compiler output from tests that "configure" runs. -Once "configure" has run, you can run "make". It builds two libraries, called -libpcre and libpcreposix, a test program called pcretest, and the pcregrep -command. If a C++ compiler was found on your system, "make" also builds the C++ -wrapper library, which is called libpcrecpp, and some test programs called -pcrecpp_unittest, pcre_scanner_unittest, and pcre_stringpiece_unittest. -Building the C++ wrapper can be disabled by adding --disable-cpp to the -"configure" command. +Once "configure" has run, you can run "make". This builds either or both of the +libraries libpcre and libpcre16, and a test program called pcretest. If you +enabled JIT support with --enable-jit, a test program called pcre_jit_test is +built as well. + +If the 8-bit library is built, libpcreposix and the pcregrep command are also +built, and if a C++ compiler was found on your system, and you did not disable +it with --disable-cpp, "make" builds the C++ wrapper library, which is called +libpcrecpp, as well as some test programs called pcrecpp_unittest, +pcre_scanner_unittest, and pcre_stringpiece_unittest. The command "make check" runs all the appropriate tests. Details of the PCRE tests are given below in a separate section of this document. @@ -331,16 +396,19 @@ system. The following are installed (fil Commands (bin): pcretest - pcregrep + pcregrep (if 8-bit support is enabled) pcre-config Libraries (lib): - libpcre - libpcreposix - libpcrecpp (if C++ support is enabled) + libpcre16 (if 16-bit support is enabled) + libpcre (if 8-bit support is enabled) + libpcreposix (if 8-bit support is enabled) + libpcrecpp (if 8-bit and C++ support is enabled) Configuration information (lib/pkgconfig): + libpcre16.pc libpcre.pc + libpcreposix.pc libpcrecpp.pc (if C++ support is enabled) Header files (include): @@ -354,6 +422,7 @@ system. The following are installed (fil Man pages (share/man/man{1,3}): pcregrep.1 pcretest.1 + pcre-config.1 pcre.3 pcre*.3 (lots more pages, all starting "pcre") @@ -368,9 +437,10 @@ system. The following are installed (fil LICENCE NEWS README - pcre.txt (a concatenation of the man(3) pages) - pcretest.txt the pcretest man page - pcregrep.txt the pcregrep man page + pcre.txt (a concatenation of the man(3) pages) + pcretest.txt the pcretest man page + pcregrep.txt the pcregrep man page + pcre-config.txt the pcre-config man page If you want to remove PCRE from your system, you can run "make uninstall". This removes all the files that "make install" installed. However, it does not @@ -467,6 +537,26 @@ running the "configure" script: CXXLDFLAGS="-lstd_v2 -lCsup_v2" +Using Sun's compilers for Solaris +--------------------------------- + +A user reports that the following configurations work on Solaris 9 sparcv9 and +Solaris 9 x86 (32-bit): + + Solaris 9 sparcv9: ./configure --disable-cpp CC=/bin/cc CFLAGS="-m64 -g" + Solaris 9 x86: ./configure --disable-cpp CC=/bin/cc CFLAGS="-g" + + +Using PCRE from MySQL +--------------------- + +On systems where both PCRE and MySQL are installed, it is possible to make use +of PCRE from within MySQL, as an alternative to the built-in pattern matching. +There is a web page that tells you how to do this: + + http://www.mysqludf.org/lib_mysqludf_preg/index.php + + Making new tarballs ------------------- @@ -486,26 +576,41 @@ To test the basic PCRE library on a Unix created by the configuring process. There is also a script called RunGrepTest that tests the options of the pcregrep command. If the C++ wrapper library is built, three test programs called pcrecpp_unittest, pcre_scanner_unittest, and -pcre_stringpiece_unittest are also built. +pcre_stringpiece_unittest are also built. When JIT support is enabled, another +test program called pcre_jit_test is built. Both the scripts and all the program tests are run if you obey "make check" or "make test". For other systems, see the instructions in NON-UNIX-USE. The RunTest script runs the pcretest test program (which is documented in its -own man page) on each of the testinput files in the testdata directory in -turn, and compares the output with the contents of the corresponding testoutput -files. A file called testtry is used to hold the main output from pcretest -(testsavedregex is also used as a working file). To run pcretest on just one of -the test files, give its number as an argument to RunTest, for example: - - RunTest 2 - -The first test file can also be fed directly into the perltest.pl script to -check that Perl gives the same results. The only difference you should see is -in the first few lines, where the Perl version is given instead of the PCRE -version. +own man page) on each of the relevant testinput files in the testdata +directory, and compares the output with the contents of the corresponding +testoutput files. Some tests are relevant only when certain build-time options +were selected. For example, the tests for UTF-8/16 support are run only if +--enable-utf was used. RunTest outputs a comment when it skips a test. + +Many of the tests that are not skipped are run up to three times. The second +run forces pcre_study() to be called for all patterns except for a few in some +tests that are marked "never study" (see the pcretest program for how this is +done). If JIT support is available, the non-DFA tests are run a third time, +this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option. + +When both 8-bit and 16-bit support is enabled, the entire set of tests is run +twice, once for each library. If you want to run just one set of tests, call +RunTest with either the -8 or -16 option. + +RunTest uses a file called testtry to hold the main output from pcretest. +Other files whose names begin with "test" are used as working files in some +tests. To run pcretest on just one or more specific test files, give their +numbers as arguments to RunTest, for example: + + RunTest 2 7 11 + +The first test file can be fed directly into the perltest.pl script to check +that Perl gives the same results. The only difference you should see is in the +first few lines, where the Perl version is given instead of the PCRE version. -The second set of tests check pcre_fullinfo(), pcre_info(), pcre_study(), +The second set of tests check pcre_fullinfo(), pcre_study(), pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error detection, and run-time flags that are specific to PCRE, as well as the POSIX wrapper API. It also uses the debugging flags to check some of the internals of @@ -540,23 +645,32 @@ RunTest.bat. The version of RunTest.bat Windows versions of test 2. More info on using RunTest.bat is included in the document entitled NON-UNIX-USE.] -The fourth test checks the UTF-8 support. It is not run automatically unless -PCRE is built with UTF-8 support. To do this you must set --enable-utf8 when -running "configure". This file can be also fed directly to the perltest script, -provided you are running Perl 5.8 or higher. (For Perl 5.6, a small patch, -commented in the script, can be be used.) - -The fifth test checks error handling with UTF-8 encoding, and internal UTF-8 -features of PCRE that are not relevant to Perl. - -The sixth test checks the support for Unicode character properties. It it not -run automatically unless PCRE is built with Unicode property support. To to -this you must set --enable-unicode-properties when running "configure". - -The seventh, eighth, and ninth tests check the pcre_dfa_exec() alternative -matching function, in non-UTF-8 mode, UTF-8 mode, and UTF-8 mode with Unicode -property support, respectively. The eighth and ninth tests are not run -automatically unless PCRE is build with the relevant support. +The fourth and fifth tests check the UTF-8/16 support and error handling and +internal UTF features of PCRE that are not relevant to Perl, respectively. The +sixth and seventh tests do the same for Unicode character properties support. + +The eighth, ninth, and tenth tests check the pcre_dfa_exec() alternative +matching function, in non-UTF-8/16 mode, UTF-8/16 mode, and UTF-8/16 mode with +Unicode property support, respectively. + +The eleventh test checks some internal offsets and code size features; it is +run only when the default "link size" of 2 is set (in other cases the sizes +change) and when Unicode property support is enabled. + +The twelfth test is run only when JIT support is available, and the thirteenth +test is run only when JIT support is not available. They test some JIT-specific +features such as information output from pcretest about JIT compilation. + +The fourteenth, fifteenth, and sixteenth tests are run only in 8-bit mode, and +the seventeenth, eighteenth, and nineteenth tests are run only in 16-bit mode. +These are tests that generate different output in the two modes. They are for +general cases, UTF-8/16 support, and Unicode property support, respectively. + +The twentieth test is run only in 16-bit mode. It tests some specific 16-bit +features of the DFA matching engine. + +The twenty-first and twenty-second tests are run only in 16-bit mode, when the +link size is set to 2. They test reloading pre-compiled patterns. Character tables @@ -616,7 +730,9 @@ will cause PCRE to malfunction. File manifest ------------- -The distribution should contain the following files: +The distribution should contain the files listed below. Where a file name is +given as pcre[16]_xxx it means that there are two files, one with the name +pcre_xxx and the other with the name pcre16_xxx. (A) Source files of the PCRE library functions and their headers: @@ -625,33 +741,40 @@ The distribution should contain the foll pcre_chartables.c.dist a default set of character tables that assume ASCII coding; used, unless --enable-rebuild-chartables is - specified, by copying to pcre_chartables.c + specified, by copying to pcre[16]_chartables.c pcreposix.c ) - pcre_compile.c ) - pcre_config.c ) - pcre_dfa_exec.c ) - pcre_exec.c ) - pcre_fullinfo.c ) - pcre_get.c ) sources for the functions in the library, - pcre_globals.c ) and some internal functions that they use - pcre_info.c ) - pcre_maketables.c ) - pcre_newline.c ) + pcre[16]_byte_order.c ) + pcre[16]_compile.c ) + pcre[16]_config.c ) + pcre[16]_dfa_exec.c ) + pcre[16]_exec.c ) + pcre[16]_fullinfo.c ) + pcre[16]_get.c ) sources for the functions in the library, + pcre[16]_globals.c ) and some internal functions that they use + pcre[16]_jit_compile.c ) + pcre[16]_maketables.c ) + pcre[16]_newline.c ) + pcre[16]_refcount.c ) + pcre[16]_string_utils.c ) + pcre[16]_study.c ) + pcre[16]_tables.c ) + pcre[16]_ucd.c ) + pcre[16]_version.c ) + pcre[16]_xclass.c ) pcre_ord2utf8.c ) - pcre_refcount.c ) - pcre_study.c ) - pcre_tables.c ) - pcre_try_flipped.c ) - pcre_ucd.c ) pcre_valid_utf8.c ) - pcre_version.c ) - pcre_xclass.c ) - pcre_printint.src ) debugging function that is #included in pcretest, + pcre16_ord2utf16.c ) + pcre16_utf16_utils.c ) + pcre16_valid_utf16.c ) + + pcre[16]_printint.c ) debugging function that is used by pcretest, ) and can also be #included in pcre_compile() + pcre.h.in template for pcre.h when built by "configure" pcreposix.h header for the external POSIX wrapper API pcre_internal.h header for internal use + sljit/* 16 files that make up the JIT compiler ucp.h header for Unicode property handling config.h.in template for config.h, which is built by "configure" @@ -701,7 +824,7 @@ The distribution should contain the foll ) "configure" and config.h depcomp ) script to find program dependencies, generated by ) automake - doc/*.3 man page sources for the PCRE functions + doc/*.3 man page sources for PCRE doc/*.1 man page sources for pcregrep and pcretest doc/index.html.src the base HTML page doc/html/* HTML documentation @@ -709,7 +832,9 @@ The distribution should contain the foll doc/pcretest.txt plain text documentation of test program doc/perltest.txt plain text documentation of Perl test program install-sh a shell script for installing files + libpcre16.pc.in template for libpcre16.pc for pkg-config libpcre.pc.in template for libpcre.pc for pkg-config + libpcreposix.pc.in template for libpcreposix.pc for pkg-config libpcrecpp.pc.in template for libpcrecpp.pc for pkg-config ltmain.sh file used to build a libtool script missing ) common stub for a few missing GNU programs while @@ -717,12 +842,14 @@ The distribution should contain the foll mkinstalldirs script for making install directories perltest.pl Perl test program pcre-config.in source of script which retains PCRE information + pcre_jit_test.c test program for the JIT compiler pcrecpp_unittest.cc ) pcre_scanner_unittest.cc ) test programs for the C++ wrapper pcre_stringpiece_unittest.cc ) testdata/testinput* test data for main library tests testdata/testoutput* expected test results testdata/grep* input and output for pcregrep tests + testdata/* other supporting test files (D) Auxiliary files for cmake support @@ -753,4 +880,4 @@ The distribution should contain the foll Philip Hazel Email local part: ph10 Email domain: cam.ac.uk -Last updated: 05 September 2008 +Last updated: 20 January 2012