Add wildcard support for UI tests #42434

Activity

1. Mark-Simulacrum

   added

   A-testsuiteArea: The testsuite used to check the correctness of rustcArea: The testsuite used to check the correctness of rustc

   on Jun 4, 2017
2. nikomatsakis commented on Jun 6, 2017

   

   nikomatsakis

   on Jun 6, 2017

   Contributor

   More actions

   cc @petrochenkov @cengizio @kennytm -- all of whom have shown an interest in our testing infrastructure
3. nikomatsakis commented on Jun 6, 2017

   

   nikomatsakis

   on Jun 6, 2017

   Contributor

   More actions

   I agree, we need to fix this, but I'm not sure the best way.

   I'm a bit reluctant to just add random wildcards into the output, since then you can't run the "regenerate reference" script and have it work.

   One thought I had was to permit multiple references, one per architecture, but that does seem to have the same problem.

   Another thought I had was to permit you to add a custom file that indicates other search-and-replace pairs specific to a particular test.

   I guess it really depends on how often it comes up -- maybe it's ok to have to regenerate some reference files by hand, but it seems like a drag.

   👍React with 👍1Reacted by kennytm
4. Mark-Simulacrum commented on Jun 6, 2017

   

   Mark-Simulacrum

   on Jun 6, 2017

   Member

   More actions

   I think maybe the right way to fix this is to have an easy way to not run UI tests on more than one platform; most UI tests don't really need this I think. That is, we're primarily testing spans, not so much the error itself. Perhaps that's not the right way to think about them, though.
5. kennytm commented on Jun 7, 2017

   

   kennytm

   on Jun 7, 2017

   Member

   More actions

   Instead of a custom file, I think the *.rs could specify headers for TestCx::normalize_output to teach it further normalization rules, e.g. for #42304

   // normalize-stderr-x86: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g
   // normalize-stderr-arm: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g
   // normalize-stderr-emscripten: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g

   If regex is too heavy, we could use simple literals instead:

   // normalize-stderr-x86: "fn() (32 bits)" -> "fn() (64 bits)"
   // normalize-stderr-x86: "fn()> (32 bits)" -> "fn()> (64 bits)"
   // normalize-stderr-arm: "fn() (32 bits)" -> "fn() (64 bits)"
   // normalize-stderr-arm: "fn()> (32 bits)" -> "fn()> (64 bits)"
   // normalize-stderr-emscripten: "fn() (32 bits)" -> "fn() (64 bits)"
   // normalize-stderr-emscripten: "fn()> (32 bits)" -> "fn()> (64 bits)"

   BTW there should be ignore-32bit / ignore-64bit / normalize-std???-32bit / normalize-std???-64bit so you don't need to write everything three times.
6. nikomatsakis commented on Jun 7, 2017

   

   nikomatsakis

   on Jun 7, 2017

   Contributor

   More actions

   @kennytm yes, that sounds fine too. We'd have to make the "update references" script rather smarter (so it can parse those things out).
7. nikomatsakis commented on Jun 7, 2017

   

   nikomatsakis

   on Jun 7, 2017

   Contributor

   More actions

   Maybe rewrite it in Rust :)
8. kennytm commented on Jun 8, 2017

   

   kennytm

   on Jun 8, 2017

   Member

   More actions

   @nikomatsakis since the files generated by compile-test is already normalized, the "update references" script shouldn't need to be changed.
9. nikomatsakis commented on Jun 9, 2017

   

   nikomatsakis

   on Jun 9, 2017

   Contributor

   More actions

   @kennytm oh right, good point :)
10. bors

    closed this as completedin #43083on Jul 12, 2017