| 1 | <html> |
|---|
| 2 | |
|---|
| 3 | <head> |
|---|
| 4 | <meta http-equiv="Content-Type" |
|---|
| 5 | content="text/html; charset=iso-8859-1"> |
|---|
| 6 | <meta name="GENERATOR" content="Microsoft FrontPage Express 2.0"> |
|---|
| 7 | <title>TokenizerFunction Concept</title> |
|---|
| 8 | </head> |
|---|
| 9 | |
|---|
| 10 | <body bgcolor="#FFFFFF" text="#000000" link="#0000EE" |
|---|
| 11 | vlink="#551A8B" alink="#FF0000"> |
|---|
| 12 | |
|---|
| 13 | <p><img src="../../boost.png" alt="C++ Boost" width="277" |
|---|
| 14 | height="86"> </p> |
|---|
| 15 | |
|---|
| 16 | <h1 align="center">TokenizerFunction Concept</h1> |
|---|
| 17 | |
|---|
| 18 | <p>A TokenizerFunction is a functor whose purpose is to parse a |
|---|
| 19 | given sequence until exactly 1 token has been found or the end is |
|---|
| 20 | reached. It then updates the token, and informs the caller of the |
|---|
| 21 | location in the sequence of the next element immediately after |
|---|
| 22 | the last element of the sequence that was parsed for the current |
|---|
| 23 | token. </p> |
|---|
| 24 | |
|---|
| 25 | <h2>Refinement of</h2> |
|---|
| 26 | |
|---|
| 27 | <p>Assignable, CopyConstructable</p> |
|---|
| 28 | |
|---|
| 29 | <h2>Notation</h2> |
|---|
| 30 | |
|---|
| 31 | <table border="1"> |
|---|
| 32 | <tr> |
|---|
| 33 | <td valign="top"><tt>X</tt> </td> |
|---|
| 34 | <td valign="top">A type that is a model of |
|---|
| 35 | TokenizerFunction</td> |
|---|
| 36 | </tr> |
|---|
| 37 | <tr> |
|---|
| 38 | <td valign="top"><tt>func</tt> </td> |
|---|
| 39 | <td valign="top">Object of type <tt>X</tt> </td> |
|---|
| 40 | </tr> |
|---|
| 41 | <tr> |
|---|
| 42 | <td valign="top"><tt>tok</tt></td> |
|---|
| 43 | <td valign="top">Object of Token</td> |
|---|
| 44 | </tr> |
|---|
| 45 | <tr> |
|---|
| 46 | <td>next</td> |
|---|
| 47 | <td>iterator that points to the first unparsed element of |
|---|
| 48 | the sequence being parsed</td> |
|---|
| 49 | </tr> |
|---|
| 50 | <tr> |
|---|
| 51 | <td>end</td> |
|---|
| 52 | <td>iterator that points to the past the end of the |
|---|
| 53 | sequence being parsed</td> |
|---|
| 54 | </tr> |
|---|
| 55 | </table> |
|---|
| 56 | |
|---|
| 57 | <h2>Definitions</h2> |
|---|
| 58 | |
|---|
| 59 | <p>A token is the result of parsing a sequence.</p> |
|---|
| 60 | |
|---|
| 61 | <h2>Valid expressions</h2> |
|---|
| 62 | |
|---|
| 63 | <p>In addition to the expression in Assignable and |
|---|
| 64 | CopyConstructable the following expressions are valid</p> |
|---|
| 65 | |
|---|
| 66 | <table border="1"> |
|---|
| 67 | <tr> |
|---|
| 68 | <th>Name </th> |
|---|
| 69 | <th>Expression </th> |
|---|
| 70 | <th>Return type </th> |
|---|
| 71 | </tr> |
|---|
| 72 | <tr> |
|---|
| 73 | <td valign="top">Functor</td> |
|---|
| 74 | <td valign="top"><tt>func(next, end, tok)</tt></td> |
|---|
| 75 | <td valign="top"><tt>bool</tt></td> |
|---|
| 76 | </tr> |
|---|
| 77 | <tr> |
|---|
| 78 | <td valign="top">reset</td> |
|---|
| 79 | <td valign="top"><tt>reset()</tt></td> |
|---|
| 80 | <td valign="top"><tt>void</tt></td> |
|---|
| 81 | </tr> |
|---|
| 82 | </table> |
|---|
| 83 | |
|---|
| 84 | <h2>Expression semantics</h2> |
|---|
| 85 | |
|---|
| 86 | <p>In addition to the expression semantics in Assignable and |
|---|
| 87 | CopyConstructable, TokenizerFunction has the following expression |
|---|
| 88 | semantcs</p> |
|---|
| 89 | |
|---|
| 90 | <table border="1"> |
|---|
| 91 | <tr> |
|---|
| 92 | <th>Name </th> |
|---|
| 93 | <th>Expression </th> |
|---|
| 94 | <th>Precondition </th> |
|---|
| 95 | <th>Semantics </th> |
|---|
| 96 | <th>Postcondition </th> |
|---|
| 97 | </tr> |
|---|
| 98 | <tr> |
|---|
| 99 | <td>operator()</td> |
|---|
| 100 | <td><tt>func(next, end, tok)</tt></td> |
|---|
| 101 | <td><tt>next</tt> and <tt>end</tt> are valid iterators to |
|---|
| 102 | the same sequence. next is a reference the function is |
|---|
| 103 | free to modify. tok is constructed. </td> |
|---|
| 104 | <td>The return value indicates whether a new token was |
|---|
| 105 | found in the sequence (next,end]</td> |
|---|
| 106 | <td>If the return value is true, the new token is |
|---|
| 107 | assigned to tok. next is always updated to the position |
|---|
| 108 | where parsing should start on the subsequent call.</td> |
|---|
| 109 | </tr> |
|---|
| 110 | <tr> |
|---|
| 111 | <td>reset</td> |
|---|
| 112 | <td><tt>reset()</tt></td> |
|---|
| 113 | <td><tt>None</tt></td> |
|---|
| 114 | <td>Clears out all state variables that are used by the |
|---|
| 115 | object in parsing the current sequence.</td> |
|---|
| 116 | <td>A new sequence to parse can be given.</td> |
|---|
| 117 | </tr> |
|---|
| 118 | </table> |
|---|
| 119 | |
|---|
| 120 | <h2>Complexity guarantees</h2> |
|---|
| 121 | |
|---|
| 122 | <p>No guarantees. Models of TokenizerFunction are free to define |
|---|
| 123 | their own complexity</p> |
|---|
| 124 | |
|---|
| 125 | <h2>Models</h2> |
|---|
| 126 | |
|---|
| 127 | <p><a href="escaped_list_separator.htm">escaped_list_separator</a></p> |
|---|
| 128 | |
|---|
| 129 | <p><a href="offset_separator.htm">offset_separator</a></p> |
|---|
| 130 | |
|---|
| 131 | <p><a href="char_delimiters_separator.htm">char_delimiters_separator</a></p> |
|---|
| 132 | |
|---|
| 133 | <p> </p> |
|---|
| 134 | |
|---|
| 135 | <p> </p> |
|---|
| 136 | |
|---|
| 137 | <hr> |
|---|
| 138 | |
|---|
| 139 | <p>© Copyright John R. Bandela 2001. Permission to copy, use, |
|---|
| 140 | modify, sell and distribute this document is granted provided |
|---|
| 141 | this copyright notice appears in all copies. This document is |
|---|
| 142 | provided "as is" without express or implied warranty, |
|---|
| 143 | and with no claim as to its suitability for any purpose.</p> |
|---|
| 144 | </body> |
|---|
| 145 | </html> |
|---|
