In macros, $($x:expr),* fragments can be used to bypass future-proofing restrictions

@pnkfelix

This should not be accepted, but it is:

macro_rules! foo {
    ( $($x:expr),* ... ) => { 1 }
}

fn main() { }

cc @pnkfelix @cmr

@bluss

cc @bluss

Note that

macro_rules! foo {
    ( $x:expr ... ) => { 1 }
}

fn main() { }

is (correctly) prohibited.

It's not just the type expr, I also tested pat.

Nooooooo! You're going to break my macro article again, aren't you? :(

(Note: I realise the following is not a good reason to not fix this problem.)

The problem for me, personally, is that the following doesn't work:

macro_rules! foo {
    ( $($x:expr),* , ... ) => { 1 }
}

fn main() {
// error: local ambiguity: multiple parsing options: built-in NTs expr ('x') or 1 other options.
    let _ = foo!(1, 2, 3, ...);
//                        ^~~
}

It would be nice if MBE preferred literal matches to captures.

@DanielKeep

@DanielKeep Shouldn't that be $($x:expr,)*? Your example won't work anyways for the reason you posted but fixing this bug shouldn't change anything. Anyways, you should be able to do this with a recursive macro:

macro_rules! foo {
    (...) => {{
        ()
    }};
    ($e:expr, $($tok:tt)+) => {{
        ($e, foo!($($tok)+))
    }}
}

fn main() {
    let tpl = foo!(1,2,3,4,...);
    println!("{:?}", tpl);
}

triage: I-nominated

@pnkfelix

cc @pnkfelix

So my feeling is that we should probably fix this, but it'd be good to evaluate the impact. It might be a good place to use a more elongated "breaking change procedure", or at least actively reach out to help people rewrite.

Is this a bug where we need to consider "FOLLOW"? I thought I saw a PR/RFC that added detailed FOLLOW checking but I could be wrong.

@DanielKeep

Would it be a middle ground to always warn on this, and then "just let it break" if expression syntax ever changes so that it does -- the macro would suddenly match differently.

@DanielKeep and I found out a parsing strategy with $x:tt $($xs:tt)* where you can split on any token you want to and reinterpret the parts on either side. That is like an explicit way to circumvent any syntax restriction, and it's not tied to any changing meaning of expr or similar macro parts.

triage: P-high

Assigning high priority as, if we are going to fix this, we need to do it soon. If we're just going to warn, urgency is somewhat lower. I'm not yet sure of best overall strategy here. A crater report might be helpful in making the determination.

OK, so, this "trailing comma" pattern jumps out: $($vs:expr),* $(,)*, and it seems like this IS a case where a knowledge of FIRST would solve the issue, right? (depending, I guess, on what comes after the $(,)*...?)

@nikomatsakis

this "trailing comma" pattern jumps out: $($vs:expr),* $(,)*, and it seems like this IS a case where a knowledge of FIRST would solve the issue, right?

@nikomatsakis yes I think that is right. I'm working on a revised version of the code that does a more precise computation of FIRST for the suffix after the cursor (and also tracks LAST for the token sequence prefix before the cursor, since you can have things like $($a:expr $( stuff; )*) $(,)*), and so when the cursor is right before the $(,)*, the LAST would hold { expr, ';' } The current code doesn't appear to be general enough to possibly be handling that correctly.

results of new crater run with a smarter (still in development) changeset:

https://gist.github.com/pnkfelix/b351caa523d45f9aa86b

I interpret this as a signal that I need to add { to the follow set for expr and probably all other NT's. But the other non-noise regressions there look legit:

expr followed by ...
ty followed by functions
ty followed by |
ty followed by varargs
expr followed by _
ty followed by [
expr followed by copy
expr followed by swap
~~ty followed by where~~ (actually, we can/should expand the follow set of ty to include where).

(Some of the regressing crates fall into more than one of the buckets above...)

I don't understand disallowing ty |, to be honest. Clearly a type can be followed by | since that is the syntax of closures.

Also, as a matter of policy, could/should crate authors be notified (assuming an email was provided) when their crate is caught regressing in a crater run related to an accepted breaking change?

I don't understand disallowing ty |, to be honest. Clearly a type can be followed by | since that is the syntax of closures.

Almost certainly an oversight. Good point!

On Wed, Nov 18, 2015 at 08:53:16AM -0800, Felix S Klock II wrote:

results of new crater run with a smarter (still in development) changeset:

https://gist.github.com/pnkfelix/b351caa523d45f9aa86b

I interpret this as a signal that I need to add { to the follow set for expr and probably all other NT's.

I still feel uncomfortable with this. I would rather than a nt for
"expr that cannot have {", which would correspond to basically "the
stuff you can put in an if". I realize this is not backwards compat
though, but it seems like having { in follow set of expr is
just...bogus. (e.g. imagine we make break take an argument.)

Perhaps though there is a compromise:

add another fragment specifier for expr-minus-{
add a lint warning that suggests people change expr to expr-minus-{ when
appropriate

@nikomatsakis

@nikomatsakis

Perhaps though there is a compromise:

add another fragment specifier for expr-minus-{

add a lint warning that suggests people change expr to expr-minus-{ when appropriate

yeah, I have come around to seeing this as a better solution (see e.g. rust-lang/rfcs#1384 (comment) )

Just need to think of a good name for the specifier. :)

now that amendment rust-lang/rfcs#1384 has been accepted, this issue should be fixed once the implementation has landed (i.e. #30450 is resolved).

I should point out that crater is still massively deficient as it still does not test Windows targets at all, and did not notice that winapi would be hit by the ty followed by [ change. Even if I fix this on my end, it will still end up breaking everyone on windows who uses an older version of winapi. Please reconsider at least this case.

C:\msys64\home\Peter\.cargo\registry\src\github.life-white.uk-88ac128001ac3a9a\winapi-0.2.5\src\macros.rs:159:46: 159:47 warning: `$fieldtype:ty` is followed by `[`, which is not allowed for `ty` fragments
C:\msys64\home\Peter\.cargo\registry\src\github.life-white.uk-88ac128001ac3a9a\winapi-0.2.5\src\macros.rs:159     ($base:ident $field:ident: $fieldtype:ty [
                                                                                                                                                 ^
C:\msys64\home\Peter\.cargo\registry\src\github.life-white.uk-88ac128001ac3a9a\winapi-0.2.5\src\macros.rs:159:46: 159:47 note: The above warning will be a hard error in the next release.
C:\msys64\home\Peter\.cargo\registry\src\github.life-white.uk-88ac128001ac3a9a\winapi-0.2.5\src\macros.rs:159     ($base:ident $field:ident: $fieldtype:ty [
                                                                                                                                                 ^

@brson

Cc @brson re above crater note

(I strongly doubt we would revert the RFC amendment or associated PR. you might find support for adding [ to FOLLOW of ty )

@retep998

@retep998 Thanks for bringing this up. I assure you the deficiencies of crater are well-known, and it'd be really great if we could improve it -- though even if we got 100% coverage for crates.io, obviously it still wouldn't be 100% coverage of all the Rust code out there. In any case though, this is part of why we try our best to do warning cycles for bug fixes that are likely to have major impact, so that we can find out about other problems before changes become permanent (and also of course to give people time to transition).

As @pnkfelix said, I don't think it makes sense to rollback this bug fix completely or anything like that. It might alleviate some short term pain but it'd be guaranteed to cause more breakage in the long-term, since basically every change to Rust's grammar would be a potential breaking change to some macro.

However, also as @pnkfelix said, adding [ to the follow set is a plausible change. I don't think I'd have added it initially, as it is not something that follows from today's grammar (I don't think). It'd be a restriction on future types, but I tend to think since we have [T], I'd be disinclined to add T[...] anyway, since it seems ripe for confusion. Worth strong consideration, certainly.

This issue (as described in the description) is fixed by #30694

Auto merge of #33982 - LeoTestard:remove-check-matcher-old, r=pnkfelix

Navigation Menu

Repository navigation

In macros, `$($x:expr),*` fragments can be used to bypass future-proofing restrictions #25658

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

In macros, $($x:expr),* fragments can be used to bypass future-proofing restrictions #25658

Description

Activity

nikomatsakis commented on May 20, 2015

nikomatsakis commented on May 20, 2015

bluss commented on May 20, 2015

DanielKeep commented on May 21, 2015

Stebalien commented on May 21, 2015

alexcrichton commented on May 26, 2015

nikomatsakis commented on Jun 2, 2015

nikomatsakis commented on Jun 2, 2015

emberian commented on Jun 2, 2015

bluss commented on Jun 2, 2015

nikomatsakis commented on Jun 3, 2015

11 remaining items

nikomatsakis commented on Nov 17, 2015

pnkfelix commented on Nov 18, 2015

pnkfelix commented on Nov 18, 2015

durka commented on Nov 24, 2015

durka commented on Nov 24, 2015

huonw commented on Nov 24, 2015

nikomatsakis commented on Dec 1, 2015

pnkfelix commented on Dec 2, 2015

pnkfelix commented on Dec 17, 2015

retep998 commented on Jan 13, 2016

pnkfelix commented on Jan 14, 2016

nikomatsakis commented on Jan 14, 2016

pnkfelix commented on Jan 14, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

In macros, `$($x:expr),*` fragments can be used to bypass future-proofing restrictions #25658