A new package has been installed at tug.ctan.org and should soon make its way to your favorite mirror.
Thanks again, Jim
======================================================================
The following information was provided by the package's contributor.
Name of contribution: tex2tok Author's name: Jonathan Fine Location on CTAN: /support/tex2tok Summary description: tex2tok: convert a TeX source file into token License type: gpl
Announcement text given by the package's contributor: - - ---------------------------------------------------------------------- tex2tok: convert a TeX source file into tokens
usage: $ tex '&tex2tok' 'LEX{story}' This will create a file story.tok. This file will contain one line for each token. - - ----------------------------------------------------------------------
You can have a look at the package at http://tug.ctan.org/tex-archive/support/tex2tok although you may get a better network connection by visiting a mirror of CTAN that is near to you; see http://www.ctan.org/tex-archive/README.mirrors .