[ruby-cvs:67074] nobu:r59896 (trunk): ripper: add states of scanner

[nobu at ruby-lang.org]

nobu	2017-09-14 19:53:47 +0900 (Thu, 14 Sep 2017)

  New Revision: 59896

  https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=59896

  Log:
    ripper: add states of scanner
    
    * parse.y (ripper_state): add states of scanner to tokens from
      Ripper.lex and Ripper::Filter#on_*.  based on the patch by
      aycabta (Code Ahss) at [ruby-core:81789].  [Feature #13686]
    
    * ext/ripper/tools/preproc.rb (prelude, usercode): generate EXPR_*
      constants from enums.

  Modified files:
    trunk/NEWS
    trunk/ext/ripper/lib/ripper/filter.rb
    trunk/ext/ripper/lib/ripper/lexer.rb
    trunk/ext/ripper/tools/preproc.rb
    trunk/parse.y
    trunk/test/ripper/test_filter.rb
    trunk/test/ripper/test_ripper.rb
    trunk/test/ripper/test_scanner_events.rb