Issue 3297: Make lexer more robust against unexpected EOF in main input

Issue 3297: Make lexer more robust against unexpected EOF in main input

This approach works by letting Lily_lexer::pop_state not cross the
maininput threshold.  Since Lily_lexer::pop_state is the "cooked" form
of yy_pop_state (also hiding the extratoken machinery), this keeps the
"maininput" state firmly under the lexer's control and avoids exiting
it prematurely.