size_t is the correct data type for this purpose. Our APIs (in particular
ExternalSourceStream::GetMoreData) are already using it, and there were some
static_casts to convert between them.
This CL doesn't intend to fix all of V8, just the minimal sense-making part
around scanner character streams.
BUG=
Review URL: https://codereview.chromium.org/864273005
Cr-Commit-Position: refs/heads/master@{#26449}
If we hade }` the right brace was always treated as part of the
template literal. We should only treat the right brace as part of
the literal when we continue to parse the template literal after a
placeholder.
BUG=v8:3734
LOG=Y
Review URL: https://codereview.chromium.org/778813003
Cr-Commit-Position: refs/heads/master@{#25661}
The spec explicitly forbids them. V8 never handled them properly either, just
the Scanner accepted them (it had code to add them literally to the
LiteralBuffer) and later on, Regexp constructor disallowed them.
According to the spec, unicode escapes in regexp flags should be an early error
("It is a Syntax Error if IdentifierPart contains a Unicode escape sequence.").
Note that Scanner is still more relaxed about regexp flags than the
spec. Especially, it accepts any identifier parts (not just a small set of
letters) and doesn't check for duplicates.
R=rossberg@chromium.org
Review URL: https://codereview.chromium.org/700373003
Cr-Commit-Position: refs/heads/master@{#25215}
git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@25215 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
ES5.1 section 6 ("Source Text"):
"Throughout the rest of this document, the phrase “code unit” and the
word “character” will be used to refer to a 16-bit unsigned value
used to represent a single 16-bit unit of text."
This changed in ES6 draft section 10.1 ("Source Text"):
"The ECMAScript code is expressed using Unicode, version 5.1 or later.
ECMAScript source text is a sequence of code points. All Unicode code
point values from U+0000 to U+10FFFF, including surrogate code points,
may occur in source text where permitted by the ECMAScript grammars."
This patch is to reflect this spec change.
BUG=v8:3617
LOG=Y
R=jochen@chromium.org
Review URL: https://codereview.chromium.org/640193002
git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@24510 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
Arrow functions are parsed from ParseAssignmentExpression(). Handling the
parameter list is done by letting ParseConditionalExpression() parse a comma
separated list of identifiers, and it returns a tree of BinaryOperation nodes
with VariableProxy leaves, or a single VariableProxy if there is only one
parameter. When the arrow token "=>" is found, the VariableProxy nodes are
passed to ParseArrowFunctionLiteral(), which will then skip parsing the
paramaeter list. This avoids having to rewind when the arrow is found and
restart parsing the parameter list.
Note that the empty parameter list "()" is handled directly in
ParsePrimaryExpression(): after is has consumed the opening parenthesis,
if a closing parenthesis follows, then the only valid input is an arrow
function. In this case, ParsePrimaryExpression() directly calls
ParseArrowFunctionLiteral(), to avoid needing to return a sentinel value
to signal the empty parameter list. Because it will consume the body of
the arrow function, ParseAssignmentExpression() will not see the arrow
"=>" token as next, and return the already-parser expression.
The implementation is done in ParserBase, so it was needed to do some
additions to ParserBase, ParserTraits and PreParserTraits. Some of the
glue code can be removed later on when more more functionality is moved
to ParserBase.
Additionally, this adds a runtime flag "harmony_arrow_functions"
(disabled by default); enabling "harmony" will enable it as well.
BUG=v8:2700
LOG=N
R=marja@chromium.org
Review URL: https://codereview.chromium.org/383983002
Patch from Adrián Pérez de Castro <aperez@igalia.com>.
git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@22366 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
Arrow functions are parsed from ParseAssignmentExpression(). Handling the
parameter list is done by letting ParseConditionalExpression() parse a comma
separated list of identifiers, and it returns a tree of BinaryOperation nodes
with VariableProxy leaves, or a single VariableProxy if there is only one
parameter. When the arrow token "=>" is found, the VariableProxy nodes are
passed to ParseArrowFunctionLiteral(), which will then skip parsing the
paramaeter list. This avoids having to rewind when the arrow is found and
restart parsing the parameter list.
Note that the empty parameter list "()" is handled directly in
ParsePrimaryExpression(): after is has consumed the opening parenthesis,
if a closing parenthesis follows, then the only valid input is an arrow
function. In this case, ParsePrimaryExpression() directly calls
ParseArrowFunctionLiteral(), to avoid needing to return a sentinel value
to signal the empty parameter list. Because it will consume the body of
the arrow function, ParseAssignmentExpression() will not see the arrow
"=>" token as next, and return the already-parser expression.
The implementation is done in ParserBase, so it was needed to do some
additions to ParserBase, ParserTraits and PreParserTraits. Some of the
glue code can be removed later on when more more functionality is moved
to ParserBase.
Additionally, this adds a runtime flag "harmony_arrow_functions"
(disabled by default); enabling "harmony" will enable it as well.
BUG=v8:2700
LOG=N
R=marja@chromium.org
Review URL: https://codereview.chromium.org/385553003
Patch from Adrián Pérez de Castro <aperez@igalia.com>.
git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@22320 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
Arrow functions are parsed from ParseAssignmentExpression. Handling the
parameter list is done by letting ParseConditionalExpression() parse
a comma-separated list of identifiers, and it returns a tree of
BinaryOperation nodes with VariableProxy leaves, or a single
VariableProxy if there is only one parameter. When the arrow token "=>"
is found, the VariableProxy nodes are passed to ParseFunctionLiteral(),
which will then skip parsing the paramaeter list. This avoids having
to rewind when the arrow is found and restart parsing the parameter
list. Note that ParseExpression() expects parenthesized expressions
to not be empty, so checking for a closing parenthesis is added in
handling the empty parameter list "()" will accept a right-paren and
return an empty expression, which means that the parameter list is
empty.
Additionally, this adds the following machinery:
- A runtime flag "harmony_arrow_functions" (disabled by default).
Enabling "harmony" will enable it as well.
- An IsArrow bit in SharedFunctionInfo, and accessors for it.
- An IsArrow bit in FunctionLiteral, accessorts for it, and
a constructor parameter to set its value.
- In ParserBase: allow_arrow_functions() and set_allow_arrow_functions()
- A V8 native %FunctionIsArrow(), which is used to skip adding the
"function " prefix when getting the source code for an arrow
function.
R=marja@chromium.org
Review URL: https://codereview.chromium.org/160073006
Patch from Adrián Pérez de Castro <aperez@igalia.com>.
git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@22265 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
Removing it seems to be a clear win on mobile: producing symbol data makes cold
parsing 20-30% slower, and having symbol data doesn't make warm parsing any
faster.
Notes:
- V8 used to produce symbol data, but because of a bug, it was never used until
recently. (See fix https://codereview.chromium.org/172753002 which takes the
symbol data into use again.)
- On desktop, warm parsing is faster if we have symbol data, and producing it
during cold parsing doesn't make parsing substantially slower. However, this
doesn't seem to be the case on mobile.
- The preparse data (cached data) will now contain only the positions of the
lazy functions.
BUG=
R=dcarney@chromium.org
Review URL: https://codereview.chromium.org/261273003
git-svn-id: http://v8.googlecode.com/svn/branches/bleeding_edge@21146 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
This patch refactors the parser and preparser interface to be more
readable and type-safe. It has no behavior changes.
Previously, parsers and preparsers were configured via bitfield called
parser_flags in the Parser constructor, and flags in
PreParser::PreParseProgram, ParserApi::Parse, and ParserApi::PreParse.
This was error-prone in practice: six call sites passed incorrectly
typed values to this interface (a boolean FLAG value, a boolean false
and a boolean true value). None of these errors were caught by the
compiler because it's just an "int".
The parser flags interface was also awkward because it encoded a
language mode, but the language mode was only used to turn on harmony
scoping or not -- it wasn't used to actually set the parser's language
mode.
Fundamentally these errors came in because of the desire for a
procedural parser interface, in ParserApi. Because we need to be able
to configure the parser in various ways, the flags argument got added;
but no one understood how to use the flags properly. Also they were
only used by constructors: callers packed bits, and the constructors
unpacked them into booleans on the parser or preparser.
The solution is to allow parser construction, configuration, and
invocation to be separated. This patch does that.
It passes the existing tests.
BUG=
Review URL: https://codereview.chromium.org/13450007
Patch from Andy Wingo <wingo@igalia.com>.
git-svn-id: http://v8.googlecode.com/svn/branches/bleeding_edge@14151 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
This patchset begins by adding support for "yield", which is unlike other tokens
in JS. In a generator, whether strict or classic, it is a syntactic keyword.
In classic mode it is an identifier. In strict mode it is reserved.
This patch adds YIELD as a token to the scanner, and adapts the preparser and
parser appropriately. It also parses "function*", indicating that a function is
actually a generator, for both eagerly and lazily parsed functions.
Currently "yield" just compiles as "return".
BUG=v8:2355
TEST=mjsunit/harmony/generators-parsing
Review URL: https://codereview.chromium.org/12646003
Patch from Andy Wingo <wingo@igalia.com>.
git-svn-id: http://v8.googlecode.com/svn/branches/bleeding_edge@14116 ce2b1a6d-e550-0410-aec6-3dcde31c8c00
