readline ) # tokenize the string for toknum, tokval, _, _, _ in g : if toknum = NUMBER and '.' in tokval : # replace NUMBER tokens result. > exec(s) #doctest: +ELLIPSIS -3.21716034272e-0.7 Output from calculations with Decimal should be identical across all platforms. Since we're only showing 12 digits, and the 13th isn't close to 5, the rest of the output should be platform-independent. Known cases are "e-007" (Windows) and "e-07" (not Windows). > from decimal import Decimal > s = 'print(+21.3e-5*-.1234/81.7)' > decistmt(s) "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))" The format of the exponent is inherited from the platform C library. Open a file in read only mode using the encoding detected byįrom tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP from io import BytesIO def decistmt ( s ): """Substitute Decimals for floats in a string of statements. Use open() to open Python source files: it usesĭetect_encoding() to detect the file encoding. If no encoding is specified, then the default of 'utf-8' will be 'utf-8-sig' will be returned as an encoding. If both a BOM and a cookie are present,īut disagree, a Synta圎rror will be raised. It detects the encoding from the presence of a UTF-8 BOM or an encodingĬookie as specified in PEP 263. (as a string) and a list of any lines (not decoded from bytes) it has read It will call readline a maximum of twice, and return the encoding used Readline, in the same way as the tokenize() generator. Should be used to decode a Python source file. The detect_encoding() function is used to detect the encoding that Theįunction it uses to do this is available: tokenize.
Tokenize() needs to detect the encoding of source files it tokenizes. If there is noĮncoding token in the input, it returns a str instead. Is the first token sequence output by tokenize(). It returns bytes, encoded using the ENCODING token, which Token type and token string as the spacing between tokens (column Guaranteed to tokenize back to match the input so that the conversion is The reconstructed script is returned as a single string. Sequences with at least two elements, the token type and the token string.Īny additional sequence elements are ignored. untokenize ( iterable ) ¶Ĭonverts tokens back into Python source code. Useful for creating tools that tokenize a script, modify the token stream, and It does not yield an ENCODING token.Īll constants from the token module are also exported fromĪnother function is provided to reverse the tokenization process. The result is an iterator yielding named tuples, exactly like To return a str object rather than bytes.
However, generate_tokens() expects readline Like tokenize(), the readline argument is a callable returningĪ single line of input. Tokenize a source reading unicode strings instead of bytes. UTF-8 BOM or encoding cookie, according to PEP 263. Tokenize() determines the source encoding of the file by looking for a For all other token types exact_typeĬhanged in version 3.3: Added support for exact_type. The returned named tuple has an additional property namedĮxact_type that contains the exact operator type for
Ints specifying the row and column where the token ends in the source and Token string a 2-tuple (srow, scol) of ints specifying the row andĬolumn where the token begins in the source a 2-tuple (erow, ecol) of
#Logicworks 5 where to find library generator#
The generator produces 5-tuples with these members: the token type the Each call to theįunction should return one line of input as bytes. Io.IOBase.readline() method of file objects. Must be a callable object which provides the same interface as the The tokenize() generator requires one argument, readline, which The primary entry point is a generator: tokenize.
0 kommentar(er)
