Что такое код yacc


Содержание

Что такое код yacc

Анализатор Bison — это на самом деле функция на C, называющаяся yyparse . Здесь мы опишем соглашения по интерфейсу yyparse и других необходимых ей функций.

Имейте в виду, что для своих внутренних целей анализатор использует множество идентификаторов C, начинающихся с `yy’ и `YY’ . Если вы используете такой идентификатор (кроме тех, что описаны в настоящем руководстве) в действии или дополнительном коде на C в файле грамматики, вероятно, вы столкнётесь с неприятностями.

Вы вызываете функцию yyparse для запуска анализа. Эта функция читает лексемы, выполняет действия, и в конце концов завершает работу, когда встречает конец входного текста или сталкивается с невосстановимой синтаксической ошибкой. Вы можете также написать действие, которое укажет yyparse завершить работу немедленно, без продолжения чтения.

Если разбор завершён успешно (возврат вызван концом входного текста), yyparse возвращает значение 0.

Значение 1 возвращается, если разбор не удался (возврат вызван синтаксической ошибкой).

В действии вы можете потребовать немедленного возврата из yyparse , используя следующие макросы: YYACCEPT Немедленный возврат со значением 0 (сообщение об удачном разборе). YYABORT Немедленный возврат со значением 1 (сообщение об ошибке).

Функция лексического анализатора yylex распознаёт лексемы во входном потоке и передаёт их анализатору. Bison не создаёт эту функцию автоматически, вы должны написать её так, чтобы yyparse могла вызывать её. Эту функцию иногда называют лексическим сканером.

В простых программах yylex часто определяется в конце файла грамматики Bison. Если yylex определена в отдельном исходном файле, вам нужно сделать доступными там макроопределения типов лексем. Для этого используйте параметр `-d’ при запуске Bison, чтобы он записал эти макроопределения в отдельный файл заголовка ` имя .tab.h’ , который вы можете включить в другие исходные файлы, которым он нужен. См. раздел 10. Вызов Bison.

Значение, возвращаемое yylex должно быть числовым кодом типа только что встреченной лексемы, или 0 для обозначени конца входного текста.

Если в правилах грамматики на лексему ссылаются по имени, это имя становится в файле анализатора макросом C, определением которого будет числовой код, соответствующий этому типу лексемы. Таким образом, yylex может использовать для обозначения типа это имя. См. раздел 4.2 Символы, терминальные и нетерминальные.

Если в правилах грамматики на лексему ссылаются с помощью однолитерной константы, числовой код этой литеры также является кодом типа лексемы. Таким образом yylex может просто вернуть код этой литеры. Нулевая литера не должна использоваться таким образом, потому что её код — ноль, означающий конец входного текста.

Приведём пример, иллюстрирующий это:

Этот интерфейс разрабатывался так, чтобы выход утилиты lex мог быть без изменений использован как определение yylex .

Если грамматика использует строковые лексемы, есть два способа, которыми yylex может определить коды их типов лексем:

  • Если грамматика определяет символические имена лексем как псевдонимы строковых лексем, yylex может использовать эти символические имена как и все остальные. В этом случае использование строковых лексем в файле грамматики не окажет влияния на yylex .
  • yylex может найти многолитерную лексему в таблице yytname . Индекс лексемы в этой таблице — это код типа лексемы. Имя многолитерной лексемы записывается в yytname в виде: двойная кавычка, литеры лексемы, вторая двойная кавычка. Литеры лексемы никаким образом не экранируются, они дословно переносятся в содержимое строки в таблице. Приведём код поиска лексемы в yytname , полагая, что литеры лексемы находятся в массиве token_buffer . Таблица yytname создаётся только если вы используете объявление %token_table . См. раздел 4.7.8 Обзор объявлений Bison.

В обычном (не повторно входимом) анализаторе семантические значения лексем должны помещаться в глобальную переменную yylval . Если вы используете единственный тип данных для семантических значений, yylval имеет этот тип. Так, если этот тип int (по умолчанию), вы можете написать в yylex :

Если вы используете множественные типы данных, тип yylval — объединение типов, полученное из объявления %union (см. раздел 4.7.3 Набор типов значений). Так, если вы сохраняете значение лексемы, вы должны использовать правильный элемент объединения. Если объявление %union выглядит так:

то код в yylex может выглядеть так:

Если вы используете в действиях `@ n ‘ -свойства (см. раздел 4.6 Отслеживание положений) для отслеживания положений лексем и групп в тексте, ваша функция yylex должна предоставить эту информацию. Функция yyparse ожидает, что положение только что разобранной лексемы в тексте находится в глобальной переменной yylloc . Таким образом, yylex должна поместить в эту переменную правильные данные.

По умолчанию значение yylloc — это структура, и вам нужно только проинициализировать её элементы, которые вы собираетесь использовать в действиях. Эти четыре элемента называются, first_line , first_column , last_line и last_column . Отметим, что использование этих свойств делает анализатор заметно более медленным.

Тип данных yylloc называется YYLTYPE .

Если вы используете объявление Bison %pure_parser , требующее создания чистого, повторно входимого анализатора, глобальные переменные взаимодействия yylval и yylloc использовать нельзя (см. раздел 4.7.7 Чистый (повторно входимый) анализатор). В таких анализаторах эти две глобальные переменные замещаются указателями, передаваемыми в качестве аргументов функции yylex . Вы должны объявить их, как здесь показано, и передавать информацию назад, помещая её по этим указателям.

Если файл грамматики не использует конструкции `@’ для ссылок на позиции в тексте, тип YYLTYPE не будет определён. В этом случае опустите второй аргумент, yylex будет вызываться только с одним аргументом.

Если вы используете повторно входимый анализатор, вы можете (необязательно) передавать ему информацию о дополнительных параметрах повторно входимым способом. Для этого определите макрос YYPARSE_PARAM как имя переменной. Это изменит функцию yyparse чтобы она принимала один аргумент с этим именем типа void * .

При вызове yyparse передайте адрес объекта, приведя его к типу void * . Действия грамматики могут ссылаться на сожержимое объекта, приводя значение указателя обратно к его правильному типу, и затем разыменовывая его. Приведём пример. Напишите в анализаторе:

Затем вызовите анализатор следующим образом:

В действиях грамматики используйте для обращения к данным выражения наподобие следующего:

Если вы хотите передать данные дополнительных параметров функции yylex , определите макрос YYLEX_PARAM тем же способом, что и для YYPARSE_PARAM , как показано ниже:

Затем вам следует определить yylex , чтобы она принимала дополнительный аргумент — значение parm (всего будет два или три аргумента, в зависимости от того, передаётся ли аргумент типа YYLTYPE ). Вы можете объявить аргумент как указатель на правильный тип объекта, или же объявить его как void * и получать доступ к содержимому как показано выше.

Вы можете использовать `%pure_parser’ и потребовать создания повторно входимого анализатора, не используя при этом YYPARSE_PARAM . Тогда вам следует вызывать yyparse без аргументов, как обычно.

Анализатор Bison обнаруживает ошибку разбора или синтаксическую ошибку каждый раз, когда читает лексему, которая не может удовлетворять никакому синтаксическому правилу. Действие в грамматике может также явно сообщить об ошибке, используя макрос YYERROR (см. раздел 5.4 Специальные возможности, используемые в действиях).

Анализатор Bison рассчитывает сообщить об ошибке, вызывая функцию сообщения об ошибке yyerror , которую должны предоставить вы. Она вызывается функцией yyparse каждый раз при обнаружении синтаксической ошибки, и принимает один аргумент. В случае ошибки разбора это обычно строка «parse error» .

Если вы определите макрос YYERROR_VERBOSE в секции объявлений Bison (см. раздел 4.1.2 Секция объявлений Bison), Bison будет давать более подробные и обстоятельные строки сообщений об ошибках, вместо обычного «parse error» . Не имеет значения, какое определение вы используете для YYERROR_VERBOSE , только то, определили ли вы его.

Анализатор может обнаружить ещё один тип ошибки — переполнение стека. Это происходит, когда входной текст содержит конструкции слишком большой глубины вложенности. Маловероятно, что вы столкнётесь с этим, поскольку анализатор Bison расширяет свой стек автоматически до очень больших пределов. Но если переполнение всё же происходит, yyparse вызывает обычным образом yyerror , за исключением того, что аргументом будет строка «parser stack overflow» .

Следующего определения достаточно для простых программ:

После возвращения из yyerror в yyparse последняя попытается произвести восстановление после ошибки, если в грамматике вы написали подходящие правила восстановления после ошибок (см. раздел 7. Восстановление после ошибок). Если восстановление невозможно, yyparse немедленно завершит работу, вернув 1.

Переменная yynerrs содержит число обнаруженных до сих пор синтаксических ошибок. Обычно эта переменная глобальная, но если вы требуете создания чистого анализатора (см. раздел 4.7.7 Чистый (повторно входимый) анализатор), это локальная переменная, к которой могут иметь доступ только действия.

Ниже представлена таблица конструкций Bison, переменных и макросов, которые могут быть полезны в действиях. `$$’ Играет роль переменной, содержащей семантическое значение группы, собираемой текущим правилом. См. раздел 4.5.3 Действия. `$ n ‘ Играет роль переменной, содержащей семантическое значение n -го компонента текущего правила. См. раздел 4.5.3 Действия. `$ тип_альт >$’ Аналогична $$ , но задаёт альтернативу тип_альт в объединении, заданном объявлением %union . См. раздел 4.5.4 Типы данных значений в действиях. `$ тип_альт > n ‘ Аналогична n , но задаёт альтернативу тип_альт в объединении, заданном объявлением %union . См. раздел 4.5.4 Типы данных значений в действиях. `YYABORT;’ Немедленно завершает работу yyparse , сообщая об ошибке. См. раздел 5.1 Функция анализатора yyparse . `YYACCEPT;’ Немедленно завершает работу yyparse , сообщая об удачном разборе. См. раздел 5.1 Функция анализатора yyparse . `YYBACKUP ( лексема , значение );’ Отмена сдвига лексемы. Этот макрос допустим только в правилах, которые выполняют свёртку единственного значения, и только когда нет предпросмотренной лексемы. Он устанавливает для предпросмотренной лексемы тип лексема и семантическое значение значение . Затем он отбрасывает значение, которое должно быть свёрнуто по этому правилу. Если макрос используется, когда его применение недопустимо, как например, когда уже есть предпросмотренная лексема, он сообщает о синтаксической ошибке сообщением `cannot back up’ и производит обычное восстановление после ошибки. В любом случае оставшаяся часть правила не выполняется. `YYEMPTY’ Значение, помещаемое в yychar , когда там нет предпросмотренной лексемы. `YYERROR;’ Немедленно вызывает синтаксическую ошибку. Этот оператор запускает восстановление после ошибки, как если бы ошибку обнаружил сам анализатор, и не выводит никакого сообщения. Если вы хотите вывести сообщение об ошибке, перед оператором `YYERROR;’ вызовите явно yyerror . См. раздел 7. Восстановление после ошибок. `YYRECOVERING’ Этот макрос заменяет выражение, имеющее значение 1 когда анализатор выполняет восстановление после синтаксической ошибки, и 0 всё остальное время. См. раздел 7. Восстановление после ошибок. `yychar’ Переменная, содержащая текущую предпросмотренную лексему (в чистом анализаторе это на самом деле локальная для yyparse переменная). Когда предпросмотренной лексемы нет, в неё помещается значение YYEMPTY . См. раздел 6.1 Предпросмотренные лексемы. `yyclearin;’ Отбросить текущую предпросмотренную лексему. Это полезно, прежде всего, в правилах обработки ошибок. См. раздел 7. Восстановление после ошибок. `yyerrok;’ Немедленно взобновляет создание сообщений об ошибках для последующих синтаксических ошибок. Это полезно, прежде всего, в правилах обработки ошибок. См. раздел 7. Восстановление после ошибок. `@$’ Играет роль структурной переменной, содержащей информацию о позиции в тексте группы, создаваемой текущим правилом. См. раздел 4.6 Отслеживание положений. `@ n ‘ Играет роль структурной переменной, содержащей информацию о позиции в тексте n -го компонента текущего правила. См. раздел 4.6 Отслеживание положений.

Yacc: невозможно получить значение yytext от lex до yacc

Лекс печатает, но Yacc не делает!

Чтобы сделать вывод в файле Lex, добавьте между yytext и его следующей точкой с запятой. В противном случае вывод буферизуется до появления новой строки или закрытия файла в конце программы.

Код Лекса присваивается yylval.strVal , но ваша грамматика Yacc не определяет strVal как часть вашего %union . Если код компилируется, это указывает на разъединение где-нибудь в вашем использовании заголовков. Ваш код Lex должен использовать заголовок, сгенерированный Yacc ( yacc -d ).

С отключением между разрешенным соединением и подтверждением того, что добавление в код Lex показало этот вывод, вы также подумали о добавлении в код Yacc? Если нет, сделайте это! Если вы это сделали, отредактируйте код в вопросе, чтобы точно отразить то, что у вас есть; мы не можем читать ваш экран с этой стороны Интернета.

‘yytext’ — статический буфер, содержащий текущий токен. Затем вы передаете указатель в этот буфер (как yylval) в синтаксический анализатор. Это имеет довольно серьезную проблему: если на вашем входе больше токенов, эти поздние маркеры могут перезаписать один и тот же yytext-буфер, на который указывает предыдущий токен, поэтому вы, вероятно, начнете видеть случайный мусор, если вы сделаете ваш синтаксический анализатор более сложным. Тривиальный пример здесь не показывает эту проблему, так как он не пытается читать другой токен после просмотра токена идентификатора.

Создание анализаторов текста при помощи yacc и lex

В этой статье на примере создания простого калькулятора показано, как создать анализатор при помощи инструментов lex/flex и yacc/bison, а затем более подробно рассмотрено, как применить эти принципы к синтаксическому разбору текста. Синтаксический разбор текста — анализ и извлечение ключевых частей текста — важная часть многих приложений. В UNIX® многие элементы операционной системы зависят от синтаксического анализа текста: оболочка, которая используется для взаимодействия с системой, распространенные утилиты и команды типа awk или Perl, вплоть до компилятора Си, используемого для разработки приложений. Анализаторы собственной разработки можно использовать в UNIX-программах (и не только UNIX) для создания простых анализаторов конфигурации или даже для создания своего собственного языка программирования.

Re: Создание анализаторов текста при помощи yacc и lex

>В этой статье на примере создания простого *калькулятора* показано, как создать анализатор при помощи инструментов lex/flex и yacc/bison

не читал, но по ссылке ходить боюсь о_О

Re: Создание анализаторов текста при помощи yacc и lex

для этого надо пользовать lemon

Re: Создание анализаторов текста при помощи yacc и lex

Re: Создание анализаторов текста при помощи yacc и lex

Re^2: Создание анализаторов текста при помощи yacc и lex

> баян.. я ещё в институте калькулятор на бизоне делал. там у них это самый главный пример был. почему то мне кажется и единственный =))

> а вы используете бизон в своих проектах?

Допустим, не бизон, а antlr, но используем. А поцчему ви спгашиваете?

Re: Создание анализаторов текста при помощи yacc и lex

во-первых пример создания калькулятора есть в info bison. во-вторых домащняя страничка IBM_dW сасед. если смотреть её без картинок то вместо стрелки на следующую страницу написано «на предыдущую страницу». вместо стрелки на предыдущую страницу красуется та же надпись.

а вообще разработчики gcc посчитали что бизон ацтой и выкинули его из гнуса4. в гнус3 он был. хотя может счас опять посчитают что он крут?

капча lipsed говорит что lisp — наше все

Re: Создание анализаторов текста при помощи yacc и lex

О! Как ни странно, но давно хотел почитать про эти две тулзы. Как раз кстати.

Re: Создание анализаторов текста при помощи yacc и lex

> а вы используете бизон в своих проектах?
Для примера Bison используется в PostgreSQL для разбора запросов.

Re: Создание анализаторов текста при помощи yacc и lex

прочитал статью и не понял: как с помощью bison/flex построить AST? В статье просто распечатываются строковые лексемы, не строится дерево разбора (CST), не строится семантическое дерево в AST. Пример с интерпретатором слишком простой, ничего не показывает. Куда интереснее пример с транслятором исходников с одного языка в другой (желательно, не как с распечаткой строковых лексем, как тот макрос с распечаткой ассемблерной вставки в статье). На таком комплексном примере понадобилось бы и построить AST из CST, трансформировать его из одного входного языка через метаязык в другой целевой, как-то обеспечить автоматическое построение CST/разных AST из единой спецификации, какая-то обработка ошибок, тестирование, профилирование и оценка производительности парсера. Для введения в предмет, впрочем, статья неплохая. Хотя на современных средствах вроде ANTLR/LLVM/HLVM/BNFC/Happy можно найти и более жизненный пример, весь цикл целиком в рамках одного инструментария.


Re: Создание анализаторов текста при помощи yacc и lex

уууу.. помню прак на втором курсе.. мне от этой хрени свело мозг )))

но инструмент реально всеядный и крайне полезный.

Re: Создание анализаторов текста при помощи yacc и lex

> капча lipsed говорит что lisp — наше все

Эта капча говорит, что наше все — sed, иначе жепа слипнется.

Re: Создание анализаторов текста при помощи yacc и lex

> прочитал статью и не понял: как с помощью bison/flex построить AST? В статье просто распечатываются строковые лексемы, не строится дерево разбора (CST), не строится семантическое дерево в AST. Пример с интерпретатором слишком простой, ничего не показывает. Куда интереснее пример с транслятором исходников с одного языка в другой (желательно, не как с распечаткой строковых лексем, как тот макрос с распечаткой ассемблерной вставки в статье). На таком комплексном примере понадобилось бы и построить AST из CST, трансформировать его из одного входного языка через метаязык в другой целевой, как-то обеспечить автоматическое построение CST/разных AST из единой спецификации, какая-то обработка ошибок, тестирование, профилирование и оценка производительности парсера. Для введения в предмет, впрочем, статья неплохая. Хотя на современных средствах вроде ANTLR/LLVM/HLVM/BNFC/Happy можно найти и более жизненный пример, весь цикл целиком в рамках одного инструментария.

А что почитать более полноценного посоветуете по теме?

Re: Создание анализаторов текста при помощи yacc и lex

> а вы используете бизон в своих проектах? Нафиг. Есть GPB

Re: Создание анализаторов текста при помощи yacc и lex

Re: Создание анализаторов текста при помощи yacc и lex

или более современную Абеля «Modern compiler construction in Java/ML/C». Примеры на ML прикольные.

Re: Создание анализаторов текста при помощи yacc и lex

Да и нечего там по ссылке делать. Все дружно читаем первые 3-4-е главы Керниган/Пайк «Введение в UNIХ» от 19хх-лохматого года. Потом в руки берётся двухтомник Д.Грис «Создание компиляторов» и вдумчиво читается до потери пульса. Потом всё это, убирается в дальний угол, присыпается другой макулатурой, и все дружно идём работать. Т.к. жить на что-то надо.

Капча:nerwed — поди догадайся на что она намекает .

Re: Создание анализаторов текста при помощи yacc и lex

можно ссылку или расшифровку акронима?

Re: Создание анализаторов текста при помощи yacc и lex

>не читал, но по ссылке ходить боюсь о_О

Re: Создание анализаторов текста при помощи yacc и lex

а ты попробуй написать полноценный шелл с паплайнами, эндами, орами, ковычками без лекса\бисона

Re: Создание анализаторов текста при помощи yacc и lex

для программирующих на С++ рекомендую boost.spirit — удобная штука, при этом никаких зависимостей от внешних утилит

Re: Создание анализаторов текста при помощи yacc и lex

Re: Создание анализаторов текста при помощи yacc и lex

Юзаю yapp — yacc для перла. (Там есть AST, если кому надо.)

Но мощностей yacc не хватает, так как грамматика внезапно перестала быть LRn. Что можно использовать в таком случае?

Re: Создание анализаторов текста при помощи yacc и lex

> для программирующих на С++ рекомендую boost.spirit — удобная штука, при этом никаких зависимостей от внешних утилит

Ты бы видел как оно тормозит при компиляции грамматике языка большего чем язык калькулятора. Bison2.3 с шаблоном для С++ (lalr1) и ручным сканером неплохо работают, но тоже есть недостатки.

А antlr3 для С++ так никто еще и не сделал :(

Re: Создание анализаторов текста при помощи yacc и lex

> Но мощностей yacc не хватает, так как грамматика внезапно перестала быть LRn. Что можно использовать в таком случае?

А что за язык такой?

Re: Создание анализаторов текста при помощи yacc и lex

> а ты попробуй написать полноценный шелл с паплайнами, эндами, орами, ковычками без лекса\бисона

Вот как раз пазбор синтаксиса шелла замечательно реализуется при помощи ручного рекурсивного разбора.

Re: Создание анализаторов текста при помощи yacc и lex

Классно! А я еще много-много лет назад читал книжицу Кернигана и Пайка. И надо же, там тоже калькулятор на lex + yacc рассматривался. Вот оказывается, кто идеи у ИбэМе ворует! И если бы не IBM EE/A сидели бы мы тут без знания, что «UNIX — универсальная среда программирования».

Re: Создание анализаторов текста при помощи yacc и lex

а что означает «Поделиться этой статьей: забобрить» ?

Re: Создание анализаторов текста при помощи yacc и lex

это значит закладку оставить. типа бабёр. хорошо еще не писец.

Re: Создание анализаторов текста при помощи yacc и lex

> Потом всё это, убирается в дальний угол, присыпается другой макулатурой, и все дружно идём работать. Т.к. жить на что-то надо.

Re: Создание анализаторов текста при помощи yacc и lex

> А antlr3 для С++ так никто еще и не сделал :(

То есть оно только для жавы/C#, ни С, ни С++?

Re: Создание анализаторов текста при помощи yacc и lex

>> А antlr3 для С++ так никто еще и не сделал :(

> То есть оно только для жавы/C#, ни С, ни С++?

Для C есть, для C++ обещают, но пока что не сделали.

Re: Создание анализаторов текста при помощи yacc и lex

> Для C есть, для C++ обещают, но пока что не сделали.

А! То есть там именно голый С? Очень хорошо, а то в apt-cache show antlr3 что-то нехорошеее пишут про С.

Re: Создание анализаторов текста при помощи yacc и lex

Есть вроде Antlr2.7/C++ http://antlr.org/grammar/list «C++ David Wigg Wed Dec 19, 2007 03:48 This is my final update of the CPP_parser (V3.2) using ANTLR V2.7 to generate C++ parser in C++. I now wish to hand it on to someone else for maintenance and further development (eg. to use ANTLR V3.0). David Wigg, wiggjd@bcs.org.uk»

Re: Создание анализаторов текста при помощи yacc и lex

Нда, в то время, как прогрессивное человечество использует packrat и GLR, пользуется прогрессивными генераторами парсеров, такими, как Antlr, IBM решили осчастливить нас рассказом о технологии тридцатилетней давности.

Re: Создание анализаторов текста при помощи yacc и lex

> boost.spirit — удобная штука, при этом никаких зависимостей

. при этом — O(exp(n)) worst case. Парсер для терпеливых!

Re: Создание анализаторов текста при помощи yacc и lex

> А antlr3 для С++ так никто еще и не сделал :(

И не надо, есть elkhound.


Re: Создание анализаторов текста при помощи yacc и lex

> Есть вроде Antlr2.7/C++

Нет, мне именно С нужен.

Re: Создание анализаторов текста при помощи yacc и lex

> Потом всё это, убирается в дальний угол, присыпается другой макулатурой, и все дружно идём работать. Т.к. жить на что-то надо.

Намекаешь, что ты лузер, работаешь быдлокодером, и тебе всё это в работе ни разу не пригдилось? Так тут кроме тебя самого никто и не виноват.

Re: Создание анализаторов текста при помощи yacc и lex

А вот можно ли как-то обзорно рассказать о современных парсерах, средах и направлениях развития?
Вот одно направление «классический LALR парсер». В духе YACC/Golden Parser/Ragel/lemon/ANTLR/SableCC и т.п. В итоге, конечный автомат для разбора. Куда оно развивается? В сторону большей «инструментальности» — готовые грамматики, IDE вроде ANTLR Works или GoldParser Builder, для отладки грамматики, интеграция на разных этапах от грамматики до CST/AST/кодогенератора.
Вот второе направление — комбинаторные парсеры, GLR, Packrat. Куда оно развивается? Парсер на Хаскелле в принципе можно вывести неявно из единой спецификации (в которой и грамматика, и кодогенератор, и подписанные узлы для AST вроде LBNF из BNFC). Сама программа — расширяемый парсер. Куда развиваются они, в сторону лучших O(n), в сторону более широких грамматик?
В общем не очень понятно как отлаживать парсер, по производительности, в худшем/лучшем/реальном случае. Что-то вроде юниттестов. Не очень понятен процесс отладки (расширения самого парсера). Ну типа проверка на не описанные типы и использование замыканий/switch..case/системы типов для отлова этого момента. Не очень понятен процесс диагностики ошибок для функционального парсера, на лиспе, или там хаскелле.
Да, в OCaML вроде есть какой-то встроенный lex. Где его место, для чего он годится, а для чего — не очень применим?

IT блог

Современные методики разработки и тестирования ПО.

Практическое использование yacc и lex

В данной заметке я рассмотрю использование связки lex + yacc (с небольшими доработками указания справедливы и для flex + Bizon).
В свое время я перелопатил много статей по синтаксическому разбору, но большинство из них (самые частые — переводные статьи про установку температуры нагревателя) не объясняют как расправиться с рекурсией или сделать чтение из файла вместо используемых по умолчанию стандартных потоков ввода/вывода. Как только встает задача разбора какого-то языка со своими правилами, а не простого конфигурационного файла, то возникает множество вопросов.

Если вы не собираетесь менять поведение умолчанию (чтение из stdin), то запуск синтаксического анализатора будет такой:

cat text_file | ./binary

./binary \n», argv[0]); > extern FILE *yyin; yyin = fopen( argv[1], «r» );

С ним бинарник можно запускать так:

Исходники целиком в тексте заметки публиковать не буду, они всё загромоздят. Скачать исходники можно по ссылке: исходники в архиве

Для локализации ошибок в разбираемом файле используются переменные:
current_line_number — номер строки, начинается с 1
current_pos — позиция символа в строке, начинается с 1

При встрече символа перевода строки увеличивается на 1 current_line_number, а при встрече какого-то иного токена значение увеличивается на длину токена.

Это средство не претендует на абсолютную точность при указании позиции ошибки (тут много погрешностей из-за того, что обход происходит рекурсивно). Но, тем не менее, оно даст намек на место в файле, которое не понравилось анализатору. А это уже не мало, когда пытаешься понять, где же ошибка: в исходниках анализатора или файле, который подвергается разбору. Если захотите сделать свой компилятор для упрощенного языка программирования, то так можно будет сохранять место объявления переменной (идентификатор).

Разбираться будут строки вида: ; >.
Т.е. перечисления слов, чисел и перечислений, разделитель — точка с запятой. Это даст возможность поработать с рекурсией.

Как и в программном коде, тут есть смысл раскидать значения по строкам, чтоб получать указание на строку и сужать круг подозреваемых операторов:

В разбираемом файле мы ищем слова (WORD) и целые числа (NUMBER). Т.е. Очередной токен может быть и числом и словом:

Мы используем перечисление для указания типа данной Value — type, а в union используем общую память для хранения конкретного типа — целого числа или указателя. Тип из перечисления нужен для понимания того, что именно хранится в union при обращении. Если вы неправильно интерпретируете его содержимое, то получите core dump.

В файле для lex эта структура используется так:

Т.е. мы присваиваем значения в union-e yylval, а именно полям структуры с псевдонимом GenericValue. Фактический тип для GenericValue задается в union, см. ниже.

Для обработки правил нашего языка используется этакий си-шный полиморфизм — union. В этом блоке вы будете задавать псевдонимы для тех типов данных, которые назначите разным токенам и даже правилам разбора:

(см. файл sample.y) В данном блоке мы перечисляем типы данных для вершин нашего дерева синтаксического разбора, которые одновременно являются и правилами в sample.y.

Разберем описание одного из правил – enumeration. Оно может быть пустым, состоять из одного элемента или быть перечислением элементов:

$$ — это псевдоним самой вершины, и если enumeration будет использоваться как составная часть другого правила, то там будет использоваться именно присвоенное значение (указатель). $1 и $3 в последнем блоке — это части в строке «enumeration SEMICOLON element», нумерация начинается с 1.

В том же файле sample.y немного выше мы описываем вершину дерева этого типа:

И еще выше определяем, что есть enumerationType:

Далее смотрим файл datastructure.h:

typedef struct enumerationNode *enumerationNodePtr; typedef struct enumerationNode < elementNodePtr element; enumerationNodePtr next; // связный список >enumerationNode;

Для создания вершины нужна функция:

enumerationNodePtr createEnumerationNode(enumerationNodePtr next, elementNodePtr element)< enumerationNodePtr retval = (enumerationNodePtr) malloc(sizeof(struct enumerationNode)); retval->next = next; retval->element = element; return retval; >

Остальные типы вершин создаются аналогично.

Функция createEnumerationNode() формирует в итоге связный список, признак конца — у очередной вершины next == NULL. В функции вызван malloc(), значит без соответствующего free() будет утечка памяти.

Для закрепления пройденного в исходниках решается такая задача. В результате разбора файла надо сформировать строку, которая формируется по рекурсивному правилу:
Строковое значение перечисления — это перемешанные случайным образом строковые значения его элементов. Строковое значение слова — само слово. Строковое значение числа — пустая строка.

[root@dmitry lex_for_outofrange]# ./sample sample_input.txt ddddd, aaaa, ffff, rr, qqq [root@dmitry lex_for_outofrange]# ./sample sample_input.txt aaaa, ffff, ddddd, qqq, rr [root@dmitry lex_for_outofrange]# ./sample sample_input.txt ddddd, ffff, aaaa, qqq, rr [root@dmitry lex_for_outofrange]# ./sample sample_input.txt ddddd, aaaa, ffff, qqq, rr [root@dmitry lex_for_outofrange]# ./sample sample_input.txt ddddd, aaaa, ffff, rr, qqq [root@dmitry lex_for_outofrange]# ./sample sample_input.txt qqq, rr, aaaa, ffff, ddddd

Вся работа происходит в этих функциях:

Сам разбор инициируется в main() (файл sample.y):

Не забывайте, что все вызовы malloc()/realloc() без соответствующего free() приводят к утечкам памяти. В данной заметке я не касаюсь вопроса очистки памяти в дереве разбора (yacc подчищает только свои огрехи). Оставляю это пытливому читателю ��

Скачать исходники можно по ссылке (дублирую для тех, кто привык читать по диагонали): исходники в архиве

Yacc — Yacc

Yacc ( Yet Another Compiler-Compiler ) является компьютерной программой для Unix операционной системы , разработанного Стивен С. Джонсоном . Это является Look Ahead Слева направо (LALR) анализатор генератора , генерируя анализатор , то часть компилятора , который пытается сделать синтаксический смысл исходного кода , в частности в LALR анализатор , основанной на аналитической грамматике , написанной в нотации аналогичен форме Бэкуса-Наура (BNF) . Yacc поставляется в виде стандартной утилиты на BSD и AT & T Unix. GNU -На Linux дистрибутивов включают Bison , а вперед-совместимую замену Yacc.

содержание

история

В начале 1970 — х годов, Стивен К. Джонсон , ученый в Bell Labs / AT & T , разработанный Yacc , потому что он хотел вставить исключающее или оператор в языке B компилятором. Он был направлен Bell Labs коллега Аль Ахо , чтобы Дональд Кнут работы «s на LR разборе , который служил в качестве основы для Yacc. В 2008 году в интервью, Джонсон подумал , что «вклад Yacc сделал для распространения Unix и C является то , что я больше всего гордитесь из». Yacc был изначально написан на языке программирования B , но вскоре был переписан в C . Она появилась как часть версии 3 Unix , и полное описание Yacc было опубликовано в 1975 году.

Описание

Вход Yacc является грамматика с фрагментами C кода ( так называемые «действия») , прикрепленные к его правилам. Его выход является сдвиг-свертка анализатор в C, выполняющей фрагменты C , связанные с каждым правилом , как только правило признается. Типичные действия включают строительство деревьев разбора . Используя пример с Джонсона, если вызов узел (метка, слева, справа) создает бинарный узел дерева синтаксического анализа с указанным ярлыком и детьми, то правило

признает суммирование выражений и конструкций узлов для них. Специальные идентификаторы $$ , $ 1 и $ 3 относятся к пунктам парсера стека .

Yacc производит только синтаксический анализатор (фразы анализатор); для полного синтаксического анализа этого требует внешнего лексического анализатора для выполнения первого этапа токенизации (анализ слов), который затем следует стадии синтаксического анализа правильных. Лексический анализатор генераторы, такие как Lex или Flex широко доступны. IEEE POSIX стандарт определяет P1003.2 функциональные возможности и требования для обоих Lex и Yacc.

Некоторые версии AT & T Yacc стали с открытым исходным кодом . Например, исходный код доступен со стандартными распределениями Plan 9 .

Влияние

Yacc и подобные программы ( в основном Переопределённый) были очень популярны. Сам Yacc используется , чтобы быть доступны в качестве парсера по умолчанию генератор на большинстве систем Unix на, хотя с тех пор он был вытеснен более поздними, в основном совместимыми, такие программы, как Беркли Yacc , GNU Bison , MKS Yacc и Абраксас PCYACC. Обновленная версия оригинальной AT & T версии входит в состав компании Sun OpenSolaris проекта. Каждый из них предлагает незначительные улучшения и дополнительные возможности по сравнению с первоначальной Yacc, но концепция и синтаксис остались прежними.

Yacc также был переписан для других языков, в том числе OCaml , RatFor , ML , Ada , Pascal , Java , Python , Ruby , , Go ,, Common Lisp и Erlang .

  • Berkeley Yacc : Реализация Беркли Yacc быстро стал более популярным , чем сама AT & T Yacc из — за его работы и отсутствие ограничений повторного использования.
  • LALR анализатор : Основной алгоритм синтаксического анализа в Yacc-генерируемых анализаторами.
  • Бизон : версия GNU из Yacc.
  • Lex (и Flex лексический анализатор ), маркер СА обычно используется в сочетании с Yacc (и Bison).
  • BNF , является metasyntax используется для выражения контекстно-свободных грамматик : то есть формальный способ описания контекстно-свободных языков.
  • PLY (Python Lex-Yacc) является альтернативной реализацией Lex и Yacc в Python.

Что такое код yacc

YACC can parse input streams consisting of tokens with certain values. This clearly describes the relation YACC has with Lex, YACC has no idea what ‘input streams’ are, it needs preprocessed tokens. While you can write your own Tokenizer, we will leave that entirely up to Lex.

A note on grammars and parsers. When YACC saw the light of day, the tool was used to parse input files for compilers: programs. Programs written in a programming language for computers are typically *not* ambiguous — they have just one meaning. As such, YACC does not cope with ambiguity and will complain about shift/reduce or reduce/reduce conflicts. More about ambiguity and YACC «problems» can be found in ‘Conflicts’ chapter.

Let’s say we have a thermostat that we want to control using a simple language. A session with the thermostat may look like this:

The tokens we need to recognize are: heat, on/off (STATE), target, temperature, NUMBER.

The Lex tokenizer (Example 4) is:

We note two important changes. First, we include the file ‘y.tab.h’, and secondly, we no longer print stuff, we return names of tokens. This change is because we are now feeding it all to YACC, which isn’t interested in what we output to the screen. Y.tab.h has definitions for these tokens.


But where does y.tab.h come from? It is generated by YACC from the Grammar File we are about to create. As our language is very basic, so is the grammar:

The first part is what I call the ‘root’. It tells us that we have ‘commands’, and that these commands consist of individual ‘command’ parts. As you can see this rule is very recursive, because it again contains the word ‘commands’. What this means is that the program is now capable of reducing a series of commands one by one. Read the chapter ‘How do Lex and YACC work internally’ for important details on recursion.

The second rule defines what a command is. We support only two kinds of commands, the ‘heat_switch’ and the ‘target_set’. This is what the |-symbol signifies — ‘a command consists of either a heat_switch or a target_set’.

A heat_switch consists of the HEAT token, which is simply the word ‘heat’, followed by a state (which we defined in the Lex file as ‘on’ or ‘off’).

Somewhat more complicated is the target_set, which consists of the TARGET token (the word ‘target’), the TEMPERATURE token (the word ‘temperature’) and a number.

A complete YACC file

The previous section only showed the grammar part of the YACC file, but there is more. This is the header that we omitted:

The function yywrap() can be used to continue reading from another file. It is called at EOF and you can than open another file, and return 0. Or you can return 1, indicating that this is truly the end. For more about this, see the ‘How do Lex and YACC work internally’ chapter.

Then there is the main() function, that does nothing but set everything in motion.

The last line simply defines the tokens we will be using. These are output using y.tab.h if YACC is invoked with the ‘-d’ option.

Compiling & running the thermostat controller

NOTE: if you get an error about your compiler not being able to find ‘yylval’, add this to example4.l, just beneath #include : This is explained in the ‘How Lex and YACC work internally’ section.

A sample session:

This is not quite what we set out to achieve, but in the interest of keeping the learning curve manageable, not all cool stuff can be presented at once.

As we’ve seen, we now parse the thermostat commands correctly, and even flag mistakes properly. But as you might have guessed by the weasely wording, the program has no idea of what it should do, it does not get passed any of the values you enter.

Let’s start by adding the ability to read the new target temperature. In order to do so, we need to learn the NUMBER match in the Lexer to convert itself into an integer value, which can then be read in YACC.

Whenever Lex matches a target, it puts the text of the match in the character string ‘yytext’. YACC in turn expects to find a value in the variable ‘yylval’. In Example 5, we see the obvious solution:

As you can see, we run atoi() on yytext, and put the result in yylval, where YACC can see it. We do much the same for the STATE match, where we compare it to ‘on’, and set yylval to 1 if it is equal. Please note that having a separate ‘on’ and ‘off’ match in Lex would produce faster code, but I wanted to show a more complicated rule and action for a change.

Now we need to learn YACC how to deal with this. What is called ‘yylval’ in Lex has a different name in YACC. Let’s examine the rule setting the new temperature target:

To access the value of the third part of the rule (ie, NUMBER), we need to use $3. Whenever yylex() returns, the contents of yylval are attached to the terminal, the value of which can be accessed with the $-construct.

To expound on this further, let’s observe the new ‘heat_switch’ rule:

If you now run example5, it properly outputs what you entered.

Let’s repeat part of the configuration file we mentioned earlier:

Remember that we already wrote a Lexer for this file. Now all we need to do is write the YACC grammar, and modify the Lexer so it returns values in a format YACC can understand.

In the lexer from Example 6 we see:

If you look carefully, you can see that yylval has changed! We no longer expect it to be an integer, but in fact assume that it is a char *. In the interest of keeping things simple, we invoke strdup and waste a lot of memory. Please note that this may not be a problem in many areas where you only need to parse a file once, and then exit.

We want to store character strings because we are now mostly dealing with names: file names and zone names. In a later chapter we will explain how to deal with multiple types of data.

In order to tell YACC about the new type of yylval, we add this line to the header of our YACC grammar:

The grammar itself is again more complicated. We chop it in parts to make it easier to digest.

This is the intro, including the aforementioned recursive ‘root’. Please note that we specify that commands are terminated (and separated) by ;’s. We define one kind of command, the ‘zone_set’. It consists of the ZONE token (the word ‘zone’), followed by a quoted name and the ‘zonecontent’. This zonecontent starts out simple enough:

This section defines what a ‘quotedname’ is: a FILENAME between QUOTEs. Then it says something special: the value of a quotedname token is the value of the FILENAME. This means that the quotedname has as its value the filename without quotes.

This is what the magic ‘$$=$2;’ command does. It says: my value is the value of my second part. When the quotedname is now referenced in other rules, and you access its value with the $-construct, you see the value that we set here with $$=$2.

NOTE: this grammar chokes on filenames without either a ‘.’ or a ‘/’ in them.

This is a generic statement that catches all kinds of statements within the ‘zone’ block. We again see the recursiveness.

This defines a block, and ‘statements’ which may be found within.

Yacc — A parser generator

Contents

It is possible to create a simple parser using Lex alone. by making extensive use of the user-defined states (ie start-conditions). However, such a parser quickly becomes unmaintainable, as the number of user-defined states tends to explode.

Once our input file syntax contains complex structures, such as «balanced» brackets, or contains elements which are context-sensitive, we should be considering yacc.

«Context-sensitive» in this case means that a word or symbol can have different interpretations, depending on where it appears in the input language. For example in C, the ‘*’ character is used for both multiplication, and to specify indirection (ie to dereference a pointer to a piece of memory). It’s meaning is «context-sensitive».

Like lex, yacc has it’s own specification language. A yacc specification is structured along the same lines as a Lex specification.

The Yacc Specification rules are the place where you «glue» the various tokens together that lex has conviniently provided to you.

Each grammar rule defines a symbol in terms of:

  • other symbols
  • tokens (or terminal symbols) which come from the lexer.

Each rule can have an associated action, which is executed after all the component symbols of the rule have been parsed. Actions are basically C-program statements surrounded by curly braces.

Instead of going into a detailed discussion of the yacc syntax, I’ll introduce the concepts one-by-one, by building an example program.

Our example program will read an olvwm menu file, with the intent of afterwards writing out an equivalent menu, for a different window manager.

Let’s review a typical openwin-menu file:

Most entries are single-line entries, begining with a label or icon filename. We also have various keywords to take into account, plus the more complex structure of a sub-menu.

The lexer will provide us with the following tokens:

  • Each possible keyword (TITLE, MENU, etc) as a separate token
  • A LABEL token for representing labels
  • A EXEC token for executable commands
  • Icon filenames will be represented as 3 separate tokens: ‘ ‘
  • To aid in error-recovery, newline characacters will be considered significant, and will be passed up as a separate token (unless they are preceded by a ‘ \ ‘),

The attached file, olmenu.l contains a suitable Lex specification.

The key aspects of the lexer are:


    Each keyword returns a seperate, unique token

A LABEL is either an identifier (loosely speaking) or an arbitrary string in quotes, or any alternating sequence of these two things

  • An EXEC token is identified by the fact that:
    • it is not a keyword.
    • it appears after we have scanned a LABEL token on the same line.
      This is achieved by setting the start-condition ACT when we scan the LABEL token.

    We use lex’s «first match» rule to ensure that keywords get priority over the corresponding LABEL and EXEC interpretations, and that the EXEC interpretation gets priority over the LABEL interpretation in the state ACT.

    It is thus essential that, where a keyword may appear on a line, the length of the other rules (for LABEL or EXEC) be no longer than the keyword rule. Otherwise, lex’s «longest match» rule would override the «first match» rule.

    The EXEC token is constructed using the exclusive start-condition CMD together with yymore() These rules allow the command to be extended across multiple lines, if the line(s) end in a backslash. Using yymore() in this fashion ensures that our arbitrarily long command-string does not override the LABEL or keyword tokens by virtue of lex’s «longest match» rule.

    The last newline after the EXEC token is not appended to the command string, but returned separately. Note that the rules
    \\\n < . yymore(); >
    .$ < . return(EXEC); >
    are treated as being the same length, so it is important that they appear in correct order.

    Please ignore the variables yylval and yylloc for now. Their meaning will only become clear after we’ve started looking at yacc in detail.


    Likewise, the lex rules associated with the start-conditions ENV, ENV1, ENV2 are not part of the basic scanner, and will be covered later.

    At this point, it’s probably a good idea to test the lexer «stand-alone». However, there’s still one thing missing: tokens .

    Yacc will choose a suitable integer values (>=256) for lex to use. (see also: «Token Types») Bison will also let you choose a value manually, like this:

    Normally, we let the parse choose values, and that’s one less thing to worry about.

    Part of the lex-yacc integration is that yacc will generate a suitable set of token-definitions for lex to use. Yacc does so by generating a file y.tab.h (bison generates basename .tab.h ) which contains the token-definitions and looks like this:

    This also keeps the lex-yacc communications automatically in step with each other.

    We don’t need the whole yacc specification to do this. We can get by on a something like the file olmenu_tokens.y

    Now we are ready to test our lexer, stand-alone, like this:

    The -d option to lex turns on the «debug» mode. This writes a message to the stderr output for each rule which has been matched.

    We are specifically looking for:

    • Any text which has not been matched, which is printed to stdout as per lex’s default action. This means anything which isn’t part of a message like: —accepting rule at line 70 (» «)
    • Any text which is not being matched by the correct rule. For example, a number after a COLUMNS keyword should be matched by the rule for the INT token.

    See the section «Lexical» in the Bison documentation for more information.

    Yacc rules define what is a legal sequence of tokens in our specification language. In our case, lets look at the rule for a simple, executable menu-command:

    This rule defines a non-terminal symbol, menu_item in terms of the two tokens LABEL and EXEC . Tokens are also known as «terminal symbols», because the parser does not need to expand them any further. Conversely, menu_item is a «non-terminal symbol» because it can be expanded into LABEL and EXEC .

    You may notice that I’m using UPPER CASE for terminal symbols (tokens), and lower-case for non-terminal symbols. This is not a strict requirement of yacc, but just a convention that has been established. We will follow this convention throughout our discussion.

    We’ve just hit our first complication: Any given menu-item may also have the keyword DEFAULT appear between the label and the executable command. Yacc allows us to have, multiple alternate definitions of menu_item , like this:

    Note that the colon ( : ) semi-colon ( ; ) and or-symbol ( | ) are part of the yacc syntax — they are not part of our menu-file definition. All yacc rules follow the basic syntax shown above and must end in a semi-colon. We’ve put the semi-colon on the next line for clarity, so that it does not get confused with our syntax-definitions. This is not a strict requirement, either, but another convention of style that we will adhere to.

    Note also that the word DEFAULT appears litterally, not because it is a keyword in our input-language, but because we have defined a %token called DEFAULT , and the lexer returns this token when it finds a certain piece of text.

    There is a way to include litteral text within a rule, but it requires that the lexer pass the characters to the parser one-by-one, as tokens.

    For example, remember that menu-items may have an icon-file instead of a label, like this:

    When our lexer encounters a or > it returns the character as a token

    We can include litteral characters in a grammar rule, like this:

    Where the second form of the menu_item is a used when specifying an icon-file instead of a text-label.

    This explains why yacc allocates token-numbers starting at >255. Because the values 1-255 are reserved for litteral characters (remember 0 is reserved for end-of-file indication).

    So far, we’ve defined a single menu-item, whereas our menu-file may contain any number of such menu-items. Yacc handles this allowing recursive rules , like this:

    By defining menu_items in terms of itself, we now have a rule which means «one or more menu items».

    Note that we could also have written our recursive definition the other way round, as:

    but, due to the internals of yacc, this builds a less memory-efficient parser. Refer to the section «Recursion» in the Yacc/Bison documentation for the reasons behind this.

    Referring back to the rule for a single menu_item , there is another way we could accomodate the optional DEFAULT keyword; by defining an empty rule, like this:

    The comment /* empty */ is ignored by yacc, and can be omitted, but again, it is conventional to include it for any empty rules.

    Strange as it may seem, the absence of the keyword DEFAULT is also a valid rule! Yacc acknowledges the empty rule for «default» when it sees it’s current look-ahead token is EXEC , and not DEFAULT . See the section «Look-Ahead» in the Bison documentation for more information about «look-ahead».

    To understand why this 2nd approach might be considered better than our earlier one, we need to explore Yacc Actions.

    Actions within yacc rules take the form:

    So far, we have only considered the tokens LABEL and EXEC as single-valued integers which are passed from the lexer to the parser. What we really need, is access to the text-strings associated with these tokens (ie their semantic value ).

    We could do this using a global variable (like token_txt in our spam-checking program), except that yacc executes the action after it has read all the tokens up to that point. Hence the string value for EXEC would overwrite the one for LABEL before we had a chance to use it. We could use seperate global variables for the LABEL and EXEC strings, but this won’t always work, because sometimes yacc has to read a token in advance before it can decide which rule to use.

    Consider the MENU keyword, in our case. Yacc has to check whether it is followed by another string or a newline, before it can decide whether it is being used to introduce a sub-menu within the same file, or an external menu-file.

    In any case, yacc provides a formal method for dealing with the semanitic value of tokens. It begins with the lexer. Every time the lexer returns a value, it should also set the external variable yylval to the value of the token. Yacc will then retain the association between the token and the corresonding value of yylval .

    In order to accomodate a variety of different token-types, yylval is declared as a union of different types.

    Token types are declared in yacc using the yacc declaration %union , like this

    This defines yylval as being a union of the types (char*) and (int) . This is a classical C-program union, so any number of types may be defined, and the union may even contain struct types, etc. For now, we’ll just have these two types.

    We also need to tell yacc which type is associated with which token. This is done by modifying our %token declarations to include a type, like this:

    We do not need to modify any other %token declarations, because they are all for keywords, which do not require any associated value.

    Now we need to modify the lexer. This is done by including the line:

    just before the lexer returns the LABEL and EXEC tokens. We’ll also include the line:

    just before the lexer returns the INT token.

    Now that we have the token value, we want to make use of it. Yacc lets us refer to the value of a given token using a syntax similar to that of awk and perl. $1 is the value of the 1st token, $2 is the 2nd, and so on. Here is a typical example of an action:

    Where:

    • new_item(); is a function which allocates some memory for the structure itemptr .
    • itemptr->label and itemptr->command are of type (char *) and are used to store the char pointers referred to by $1 and $2.

    Let’s consider what happens when we want to accomodate the DEFAULT keyword. We’ll assume that the structure itemptr contains a itemptr->default variable for storing a simple indicator of whether the DEFAULT keyword was used or not.

    If we use our first approach, we would have:

    Which is OK, but there’s a lot of almost-identical code in the two actions. So let’s try it using the 2nd approach we used above.

    This is nicer, because we’ve removed the repetition in our actions. However, I’ve added the comment /*segv*/ , because that’s exactly what we would get. This is because of the way yacc works. Here’s what would happen:

    1. Yacc gets the token LABEL , and, since it doesn’t have the complete rule yet, it just «saves it for later» (by pushing it onto a stack).
    2. Yacc sees the non-terminal symbol default , and starts processing the rule for it.
    3. The default rule is simply: DEFAULT , or nothing. There are only two possibilties: a the next token is, say, EXEC .
      It’s not DEFAULT , but that’s OK (syntactically), because empty is a valid rule in this case. Yacc saves the EXEC token for later. Since the current rule (empty) is both valid and complete, yacc executes the action for it. b Let’s assume that the next token is DEFAULT .
      This is all that we need to complete the rule default , so yacc executes the action for it. In either case, we execute an action in the default rule.
    4. Now we get to our EXEC token. We have all the elements of our menu_item rule (including the non-terminal default rule), so we can now execute the action for that.
      But WAIT! We’re just about to allocate itemptr using the function new_item() , but we’ve already used itemptr in an action for the previous rule, default . It’s too late, we’ve already crashed.

    Yacc provides a simple yet elegant solution to this dilemma, by extending the concept of the «value» of a token to non-terminal symbols, like default .

    First, we have to declare the default in our Yacc Declarations section, like this:

    Note that this is the same approach as we used for %token definitions. We can even use types which are not used by the lexer, but we must add them to our %union declaration.

    We assign the value of the left-hand-side of the rule by assigning a value to $$ .

    Our complete rule-set for menu_item , using this approach would be:

    The storage space for our $$ is just another value attached to a token, and this is handled automatically by yacc. So no crash. And we’ve removed the unwanted duplication of code in our actions.

    Just as you might do with C-program variables, it is possible to typecast a value which is being accessed via the $ mechanism, and that by writing it as $ type >n instead of $n , where type is a member of our %union declaration. For example:

    Normally, you would not need to type-cast token-values in this fashion.

    Until now, we have assumed that if we don’t specify an action, yacc does nothing. This is not true. In the absence of and explicit action, yacc applies a default action of:

    Or, put simply, the left-hand-side inherets the value from the 1st symbol on the right-hand-side. (or inherets the «absence of a value», as the case may be).

    This can be a problem if the two symbols have different type: yacc will complain about an
    error: type clash (`num’ `’) on default action
    or similar. This means that we should be fussy about the way be assign types to symbols in yacc, and take the same care as we would when writing normal C-program code.

    In the early stages of development, you can get rid of these errors by adding an explict action, like
    < >
    which will override the default action.

    The topic of tokens and their associated semantic values is covered in the section «Semantic Values» in the Bison documentation.


    Now that we understand the hows and whys of yacc rules, we can try our hand at writing a basic parser. Before we can successfully compile it, there’s a few more things we’ll need.

    • A main() function that calls our parser, yyparse()
    • A yyerror() function
    • A yywrap() function (for the lexer).
    • Debugging output (optional? :-) )

    Yacc generates a single function called yyparse() . This function requires no parameters. and returns either a 0 on success, and 1 on failure. «Failure» in the case of the parser means «if it encounters a syntax error».

    See the section «Interface» in the Bison documentation for more information.

    In addition to the usual debugging techniques, there are a few of things we can do to assist debugging

    • Invoking yacc with the -v flag.
      This generates an additional file, whose name ends in .output . This is instrumental in resolving ambiguities in the parser.
    • Invoking yacc with the -t flag, and defining YYDEBUG . This will make the parser write debugging messages at run-time, iff you also set the variable yydebug to a non-zero value.
    • Defining the macro YYERROR_VERBOSE , which will add a little more information to the message printed by yyerror() .

    This is prototype parser because, as you may notice, it does not contain any actions. If you compile it, and run it with a suitable openwin-menu file, you get exactly: nothing.

    But that’s OK for now, as we just want to check that we have understood our input-syntax properly, and that the parser works as expected.

    We’ll build the prototype using make. I like to use something like this Makefile.

    As with any compiled source, you never get it right the first time, so you have to contend with the usual syntax errors which need fixing. In addition, you may also encounter some type clashes, which we mentioned above. Once you are over these hurdles, yacc will generate C-source code that compiles.

    It it more than likely that even though yacc has generated a working parser, you got a couple of messages:

    A Shift operation is what the parser does when it saves a token for later use. (Actually, it pushes the token onto a stack)

    A Reduce operation is what the parser does when resolves a set of tokens into a single, complete rule. (The corresponding tokens are removed from the stack and replaced with a single token representing the rule. The stack has been «reduced»).

    It is not strictly necessary to eliminate these warnings, as yacc will still generate an operational parser. However, it is important to understand these conflicts, to be sure that the parser we get we wanted, and not the parser we asked for :-) .

    Both of these warnings mean that there is an ambiguity in our ruleset.

    Our prototype parser does not contain any such situations. Refer to the section «Algorithm» in the Bison documentation for examples and detailed information.

    A shift/reduce conflict occurs when there would be enough tokens shifted (saved) to make up a complete rule, but the next token may allow a longer rule to be applied.

    In the event of a shift/reduce conflict, the parser will opt for the shift operation, and hence try to build the longer rule. You can think of this as being analogous to lex’s «longest match» rule.

    If your grammar has «optional» structures, such as an optional «else» following an «if» statement, then it may not be possible to eliminate all shift/reduce conflicts from the grammar rules.

    See the section «Shift/Reduce» in the Bison documentation for further explanation.

    A reduce/reduce conflict occurs when the same set of tokens can be used to form two different rules.

    In the event of a reduce/reduce conflict, the parser will use the first rule that appears in the grammar. You can think of this as being analogous to lex’s «first match» rule.

    Reduce/reduce conflicts are usually an indication of an error in the way the grammar rules have been defined, as the whole point of having a grammar is to avoid such blatant ambiguities. It is usually possible (and desirable) to eliminate all reduce/reduce conflicts from your grammar rules, either by rewriting some rules, or redefining the grammar (if possible).

    See the section «Reduce/Reduce» in the Bison documentation for further explanation.

    In order to find out which rules are affected by these conflicts, you will need to refer to the *.output file generated by running yacc with the -v flag, for example:

    This will generate the usual *.tab.c file, plus an additional olmenu-proto1.output file. This file will tell you which rules are causing the conflicts mentioned.

    The prototype parser, olmenu-proto1 should be invoked with the -v option.

    This will print out a lot of messages about what tokens were encountered, and which rules they were used in. This information can be instrumental in determining «where the parser went wrong», if you feed it an input file which is known to be correct, but get a «parse error» from the parser.

    See the section «Debugging» in the Bison documentation for further explanation.

    As it stands, when our parser encounters an incorrect syntax, it will simply print the message «parse error» and exit. At the very least, we would like an indication of the line-number at which the error occured. To do this, we will need the co-operation of the lexer, since the parser is often «unaware» of newline characters. More often than not, newlines are not considered significant form the point of view of the grammar.

    The lexer must set an additional variable, yylloc , every time it encounters a newline. This variable must be declared in the lexer like this:

    This variable is a structure of type YYLTYPE . We can use any of it’s members to store relevant information, but we are not required to use all of them. One is usually enough.

    We should also initialise our line-counter. To this end, we can use the lex macro YY_USER_INIT in the lex declarations section:

    Lex should increment this variable every time it encounters a newline in the input stream. Be careful of using yyless() and REJECT in your lex actions, because they can confuse your line-counter.

    Once we have set up the lexer to provide line-number information, we can use it within any yacc action. We refer to a token’s line-number by using @n , in the same way that $n is used to refer to token’s value. For example:

    This requires additional code to be generated by yacc. The appropriate source-code is generated automatically if you make use of the @n notation within a yacc action. (at least, this is true of bison — I’m not sure of how yacc deals with this).

    Once we have coaxed yacc into producing the necessary code, we can also use yylloc within yyerror() , as follows:

    See the section «Token Positions» in the Bison documentation for further explanation.

    As it stands, when our parser encounters an incorrect syntax, it will simply print the message «parse error» and exits.

    We’ve just added line-number information, but even this is a bit vague. Also, we may not want our parser to simply give up at the first syntax error, in the same way the C-compiler gives you more than one error message at a single invocation.

    Yacc provides a special symbol for handling errors. The symbol is called error and it should appear within a grammar-rule. For example, we could have:

    Now, when the parser encounters an invalid syntax while processing the rule menu_item , it will

    • Discard the current token
    • Execute the error action
    • Continue parsing as if the discarded token was really a valid menu_item

    Unfortunately, we cannot vouch for the next token that the parser gets, so typically this error rule will generate several error messages.

    Yacc is also capable of more sophisticated error-handling. For example, we can tell the parser to discard some of the subsequent tokens, too, simply by putting a token after the error token.

    In this case, the parser

    • Discards everything on the stack going back to the label token
    • Keeps reading tokens up till the next ‘\n’, and discards them, too.
    • Uses the tokens label error and ‘\n’ to complete the rule for menu_command .

    In this case, the rule

    is not much different to just having

    except that the former lets us make use of the value of label to give a more informative error message.

    This approach is often used when there a «balanced» symbols in the syntax. Consider the rule:

    If the parser encounters something other than a LABEL after a ‘ , it will discard all tokens up to the next ‘>’ . This technique can be useful for keeping brackets balanced during error-recovery.

    In our case, keep in mind that any unidentifiable text after a valid LABEL is converted to an EXEC token by the lexer, and the EXEC token would swallow any subsequent ‘>’ . So this rule in unlikely to trap any real errors, except something like keyword >

    In fact, it is likely to work against us, because we may not see another ‘>’ at all, so we will miss out on a lot of input!

    Error recovery is a tricky business, and we don’t always get the results we really wanted. It doesn’t pay to be too fussy about this aspect of the parser.

    See the sections «Error Recovery» and «Action Features» in the Bison documentation for further explanations and examples.

    This prototype contains the neccessary rules and actions for handling errors in a reasonable way. Building it should be as simple as typing:

    We want to test it with both correct input, and incorrect input. We need to check that it generate meaningful error messages and does not cause any segv errors or the like.

    At this stage, our grammar rules should be complete, so so that we can start populating them with actions. Hopefully, we will not need to change them from here on.

    Up till now, we have been concentrating purely on the grammar rules. This has been quite deliberate, because we should get the grammar right first, before we spend too much time writing (and re-writing) our actions.

    The basic goal of our parser will be create an in-memory representation of the input data. Or, in other words, a set of structures, linked lists, arrays, etc, which we can use and manipulate. Once we have our memory-representation, we can generate out new menu-file in the CDE format, using conventional techniques and statements such as printf .

    We are free to use any technique we like to build up our memory structures, however yacc lends itself particularly well to a specific style of doing so.

    Yacc gives us the ability to assign values to non-terminal symbols, as if they were yylval values supplied by the lexer. This allows us to pass values and pointers along through the grammar rules themselves. We can use this feature to build up linked-lists and other structures «on the fly». We looked at this technique once before, in the section titled «Using Token Types in Yacc Actions», where we passed along an integer value representing the presence or absence of the keyword DEFAULT .

    Our menu-file format has two basic structures:

    • A single menu, which is a linear list of menu items.
    • A menu item, which may be one of:
      • An executable /bin/sh command
      • Another menu
      • A built-in olwm command


    If we allow a menu-item to include a pointer to another menu, then the tree-like nature of the menus will be catered for, so we do not need to keep a separate structure to define the menu-tree.

    Our target format, the dtwmrc format, does not use «inline» submenus like olvwm, Instead, each menu is refered to by it’s name, and it’s contents are defined after the end of the current menu. For example:

    This means that we should keep a linear list of menus, in addition to the menu-tree structure defined by menus and items. So our structures will look like this:

    The first thing we are likely to encounter in our menu-file is a menu item. We can tell if it’s the first menu-item, because it will be processed by the rule: and not

    This makes the first menu_item rule a good place to allocate our struct menu . We do this mainly so we can get access to the variables first_item and last_item

    We can «hold onto» our struct menu without resorting to global variables by circulating it within the menu_items rule. So both alternatives of the rule return a struct menu * (though only the first version allocates it).

    When there are no more menu items in the current menu or sub-menu, the rule for menu_items is complete. We then store the struct menu * pointer returned by menu_items :

    • In the menufile rule, it is stored in the global variable top_menu .
    • In the submenu rule, it is stored in the struct menu_item which invokes the menu.

    So now our rule for menu_items liooks like this:

    The only other place we would want to call new_menu() is from within the rule for submenu . This rule contains the symbol menu_items , which allocates a struct menu and returns it complete with a linked list of menu-items. So there’s not much else to do, other than to create an item using new_item() , and use it to store the struct menu pointer which menu_items returns for us.

    Our rule for menu_items works properly, and returns the right values in all cases. However, one peculiarity does arise from the way this rule interacts with the submenu rule.

    If our input contains a submenu before the first ordinary item, we call new_menu() for the submenu before the parent-menu. This is not strictly a problem, but it would be nicer to have the menu_list in the order-of-appearance. We can fix this by re-writing the rule for menu_items as:

    The problem goes away, because we now call new_menu() before we process the first menu-item.

    Lastly, there is the issue of the options rule. These items are not really true menu items at all, just some options we can set, like title and columns. These have been defined as members within struct menu However, at the time we are parsing the options, we do not have access to the correct struct menu . We could use a global-variable to store the current menu, but it might get tricky to restore the correct menu when we get to the end of a submenu.

    It is easier to let the parser do the work, and pass down values in the manner to which we are now accustommed.

    We will use the struct menuitem as a vehicle to transfer the TITLE and COLUMNS options. We will then make the add_item() function treat these items as «special», and free the structure when done,

    We are not going to use mid-rule actions, but I’ll mention them anyway, because they can be useful.

    Consider the problem of the rule: submenu .

    Let’s say we defined the rule for menu_items , such that it returned the start of a linked-list of menu-items, instead of a struct menu pointer. We would then link the struct menu_item list to the struct menu in actions for the rules menufile and submenu . The problem arises of: how do we build the linked-list? We could:

    • Build the list backwards, and prepend each new menu-item to the start of the item-list
    • Build it forwards, by traversing the list to the end for each new menu-item, and appending to the end of the list.
    • Build it forwards, by storing the pointer to the last-item.

    Clearly, the last option is the nicest, but we cannot use a simple global variable as our last_item pointer. If we did, we would get to end of the first submenu, and we would want to restore last_item to the last item of the parent menu. But we’ve already lost this pointer, because we’ve been using the same global-variable last_item to process the submenu.

    So we need to put last_item somewhere other than a global variable. Our struct menu does nicely. Except that our existing rule for a submenu will read all the menu_items before the action for the submenu rule is executed (remembering what happened when processing our keyword DEFAULT ).

    There is a way we can allocate our struct menu before we start processing menu_items , and that is to use a mid-rule action, like this:

    Notice how.

    • the rule itself has not changed, even though there is an < action >right in the middle of it
    • The mid-rule action can also return a value
    • The value returned by the mid-rule action is not assigned to the LHS of the rule, but to $5 instead
    • The mid-rule action is treated like another token by the end-of-rule action (ie the mid-rule action becomes $5 and everything after it is moved one token to the right.

    Note that the above example is not intended to be a «working example», it is just illustrative.

    See the section «Mid-Rule Actions» in the Bison documentation for further explanations and examples.

    In order to generate a complete dtwmrc file, we should really be reading the files nominated in the rule menu_file . It is technically feasible to open a submenu file as soon as we parse the menu_file line. Due to the use of «look-ahead» in both the lexer and parser, it is not simply a matter of changing yyin .

    Flex provides a set of functions and macros to handle this, and these are described in the section «Multiple Input Buffers» in the flex man page.

    In addition, we should be aware that the parser may also be one token in front, since it uses a «look-ahead» token to decide which rule to apply.

    So, while it is possible to change input-streams on-the-fly, it adds another dimension of complexity to our program.

    In our case, a much simpler solution is to read a whole file at a time, calling yyparse() once per file. We stitch the new memory-structures in manually. The function read_menu_files() does just this. It:

    • traverses our memory structure looking for instances of menu-files,
    • opens the file, and assigns it to yyin
    • calls yyparse()
    • links in the new top-level menu at the appropriate point.

    As luck would have it, menus which are read from other files are just appended to the end of our menu_list , so we don’t need to do anything else to our menu_list

    We process the menus in the order they appear menu_list , and this list can be appended-to while we are processing the current menu. Hence, this simple solution also caters for files nested several levels deep.

    See the section «Pure Calling» in the Bison documentation for further information.

    Pure calling is not required in our case, as we are not calling the parser from within the parser. We are calling yyparse() repeatedly, but we allow it to complete before calling it again.

    The openwin-menu file syntax also allows the filenames of nested menu files to be referred to using environment variables. For example,
    /usr/openwin/lib/openwin-menu-games
    could be written
    $OPENWINHOME/lib/openwin-menu-games
    or even
    $/lib/openwin-menu-games

    Traditionally, this variable expansion would be done using C-library calls like strspn() and strtok() , or maybe using a page or so of hand-written code.

    However, this seems quite tedious after what we’ve been doing with lex. After all, this kind of thing is what lex is best at.

    The GNU lexer, flex, provides several functions to do exactly what we want: scan a string variable, just like it would a text-file.

    The details of the flex functions required to scan a string are described in the flex man page.

    Our lexer, olmenu.l contains a function expand_env() which uses this feature of lex. It takes the filename string which was parsed earlier, and processes it, using lex to break the string into several components. If the component is an environment variable, the lexer returns the result of the corresponding getenv() . If the component isn’t an environment variable, it is returned verbatim.

    The function expand_env() calls yylex() repeatedly, and concatenates the components into the desired result string.

    In order that our string-scanner does not interfere with our regular text-scanner, we will set an «exclusive» start condition, ENV , before we call yylex() .

    We need to remember that yylex() returns 0 when it reaches the end-of-input, but we can return any other integer values we like. In this case, we’ll return yytext , or the result of a getenv(yytext) . Keep in mind that getenv() returns 0 if the variable doesn’t exist, so we have to be careful not to return the getenv() result without checking it.

    The really good thing about using flex to do this, is the amount of control we have over the resulting string-scanner. For example, if we wanted to allow backslash quoting to suppress the environment-variable substitution, we could change the rule: to become:

    There’s nothing really special about the output part of our program. We just traverse our linked lists, and run a lot of fprintf() statements.

    Look at the file dtwmrc.c if you are interested.

    Finally, our menu-converter is complete. The finished result, including grammar rules, actions, and support functions like main() yyerror() and yywrap() , is the file olmenu.y.

    You can build it using the command:

    Make contains implicit rules for building programs from yacc source files. The source file must end in .y for the implicit rules to work.

    The default rule for yacc is:

    I like to call my parser files something.l and something.y Unfortunately, this introduces a conflict in make’s default rules, which would try to generate something.c from both the lex and yacc source. Hence, I like to «brew my own» implicit rules, which generate something_lex.c and something_yacc.c instead.

    See the Makefile included with the examples for the details.

    Note that there are some incompatibilities at this level between yacc and bison. Yacc likes to call its output files y.tab.c for the parser and y.tab.h for the token definitions. Bison prefers to use basename.tab.c and basename.tab.h (respectively). Bison will generate y.tab.c and y.tab.h if it is invoked with the -y flag. info2www This document contains numerous hyper-references to the Bison documentation. In order to use these, you must install the perl script info2www on a web-server on your local machine.

    The Bison documentation is in «Info» format, and the info2www gateway is arguably the most convinient way of accessing this type of documentation.

    Yacc: Yet Another Compiler-Compiler

    Computer program input generally has some structure; in fact, every computer program that does input can be thought of as defining an «input language» which it accepts. An input language may be as complex as a programming language, or as simple as a sequence of numbers. Unfortunately, usual input facilities are limited, difficult to use, and often are lax about checking their inputs for validity.

    Yacc provides a general tool for describing the input to a computer program. The Yacc user specifies the structures of his input, together with code to be invoked as each such structure is recognized. Yacc turns such a specification into a subroutine that handles the input process; frequently, it is convenient and appropriate to have most of the flow of control in the user’s application handled by this subroutine.

    The input subroutine produced by Yacc calls a user-supplied routine to return the next basic input item. Thus, the user can specify his input in terms of individual input characters, or in terms of higher level constructs such as names and numbers. The user-supplied routine may also handle idiomatic features such as comment and continuation conventions, which typically defy easy grammatical specification.

    Yacc is written in portable C. The class of specifications accepted is a very general one: LALR(1) grammars with disambiguating rules.

    In addition to compilers for C, APL, Pascal, RATFOR, etc., Yacc has also been used for less conventional languages, including a phototypesetter language, several desk calculator languages, a document retrieval system, and a Fortran debugging system.

    0: Introduction

    Yacc is written in a portable dialect of C[1] and the actions, and output subroutine, are in C as well. Moreover, many of the syntactic conventions of Yacc follow C.


    The heart of the input specification is a collection of grammar rules. Each rule describes an allowable structure and gives it a name. For example, one grammar rule might be Here, date, month_name, day, and year represent structures of interest in the input process; presumably, month_name, day, and year are defined elsewhere. The comma «,» is enclosed in single quotes; this implies that the comma is to appear literally in the input. The colon and semicolon merely serve as punctuation in the rule, and have no significance in controlling the input. Thus, with proper definitions, the input might be matched by the above rule.

    An important part of the input process is carried out by the lexical analyzer. This user routine reads the input stream, recognizing the lower level structures, and communicates these tokens to the parser. For historical reasons, a structure recognized by the lexical analyzer is called a terminal symbol, while the structure recognized by the parser is called a nonterminal symbol. To avoid confusion, terminal symbols will usually be referred to as tokens.

    There is considerable leeway in deciding whether to recognize structures using the lexical analyzer or grammar rules. For example, the rules might be used in the above example. The lexical analyzer would only need to recognize individual letters, and month_name would be a nonterminal symbol. Such low-level rules tend to waste time and space, and may complicate the specification beyond Yacc’s ability to deal with it. Usually, the lexical analyzer would recognize the month names, and return an indication that a month_name was seen; in this case, month_name would be a token.

    Literal characters such as «,» must also be passed through the lexical analyzer, and are also considered tokens.

    Specification files are very flexible. It is realively easy to add to the above example the rule allowing as a synonym for In most cases, this new rule could be «slipped in» to a working system with minimal effort, and little danger of disrupting existing input.

    The input being read may not conform to the specifications. These input errors are detected as early as is theoretically possible with a left-to-right scan; thus, not only is the chance of reading and computing with bad input data substantially reduced, but the bad data can usually be quickly found. Error handling, provided as part of the input specifications, permits the reentry of bad data, or the continuation of the input process after skipping over the bad data.

    In some cases, Yacc fails to produce a parser when given a set of specifications. For example, the specifications may be self contradictory, or they may require a more powerful recognition mechanism than that available to Yacc. The former cases represent design errors; the latter cases can often be corrected by making the lexical analyzer more powerful, or by rewriting some of the grammar rules. While Yacc cannot handle all possible specifications, its power compares favorably with similar systems; moreover, the constructions which are difficult for Yacc to

    handle are also frequently difficult for human beings to handle. Some users have reported that the discipline of formulating valid Yacc specifications for their input revealed errors of conception or design early in the program development.

    The theory underlying Yacc has been described elsewhere.[2, 3, 4] Yacc has been extensively used in numerous practical applications, including lint,[5] the Portable C Compiler,[6] and a system for typesetting mathematics.[7]

    The next several sections describe the basic process of preparing a Yacc specification; Section 1 describes the preparation of grammar rules, Section 2 the preparation of the user supplied actions associated with these rules, and Section 3 the preparation of lexical analyzers. Section 4 describes the operation of the parser. Section 5 discusses various reasons why Yacc may be unable to produce a parser from a specification, and what to do about it. Section 6 describes a simple mechanism for handling operator precedences in arithmetic expressions. Section 7 discusses error detection and recovery. Section 8 discusses the operating environment and special features of the parsers Yacc produces. Section 9 gives some suggestions which should improve the style and efficiency of the specifications. Section 10 discusses some advanced topics, and Section 11 gives acknowledgements. Appendix A has a brief example, and Appendix B gives a summary of the Yacc input syntax. Appendix C gives an example using some of the more advanced features of Yacc, and, finally, Appendix D describes mechanisms and syntax no longer actively supported, but provided for historical continuity with older versions of Yacc.

    1: Basic Specifications

    Names refer to either tokens or nonterminal symbols. Yacc requires token names to be declared as such. In addition, for reasons discussed in Section 3, it is often desirable to include the lexical analyzer as part of the specification file; it may be useful to include other programs as well. Thus, every specification file consists of three sections: the declarations, (grammar) rules, and programs. The sections are separated by double percent «%%» marks. (The percent «%» is generally used in Yacc specifications as an escape character.)

    In other words, a full specification file looks like The declaration section may be empty. Moreover, if the programs section is omitted, the second %% mark may be omitted also;

    thus, the smallest legal Yacc specification is

    Blanks, tabs, and newlines are ignored except that they may not appear in names or multi-character reserved symbols. Comments may appear wherever a name is legal; they are enclosed in /* . . . */, as in C and PL/I.

    The rules section is made up of one or more grammar rules. A grammar rule has the form: A represents a nonterminal name, and BODY represents a sequence of zero or more names and literals. The colon and the semicolon are Yacc punctuation.

    Names may be of arbitrary length, and may be made up of letters, dot «.», underscore «_», and non-initial digits. Upper and lower case letters are distinct. The names used in the body of a grammar rule may represent tokens or nonterminal symbols.

    A literal consists of a character enclosed in single quotes «»’. As in C, the backslash «\» is an escape character within literals, and all the C escapes are recognized. Thus For a number of technical reasons, the NUL character (‘\0’ or 0) should never be used in grammar rules.

    If there are several grammar rules with the same left hand side, the vertical bar «|» can be used to avoid rewriting the left hand side. In addition, the semicolon at the end of a rule can be dropped before a vertical bar. Thus the grammar rules can be given to Yacc as It is not necessary that all grammar rules with the same left side appear together in the grammar rules section, although it makes the input much more readable, and easier to change.

    If a nonterminal symbol matches the empty string, this can be indicated in the obvious way:

    Names representing tokens must be declared; this is most simply done by writing in the declarations section. (See Sections 3 , 5, and 6 for much more discussion). Every name not defined in the declarations section is assumed to represent a nonterminal symbol. Every nonterminal symbol must appear on the left side of at least one rule.

    Of all the nonterminal symbols, one, called the start symbol, has particular importance. The parser is designed to recognize the start symbol; thus, this symbol represents the largest, most general structure described by the grammar rules. By default, the start symbol is taken to be the left hand side of the first grammar rule in the rules section. It is possible, and in fact desirable, to declare the start symbol explicitly in the declarations section using the %start keyword:

    The end of the input to the parser is signaled by a special token, called the endmarker. If the tokens up to, but not including, the endmarker form a structure which matches the start symbol, the parser function returns to its caller after the endmarker is seen; it accepts the input. If the endmarker is seen in any other context, it is an error.

    It is the job of the user-supplied lexical analyzer to return the endmarker when appropriate; see section 3, below. Usually the endmarker represents some reasonably obvious I/O status, such as «end-of-file» or «end-of-record».

    With each grammar rule, the user may associate actions to be

    performed each time the rule is recognized in the input process. These actions may return values, and may obtain the values returned by previous actions. Moreover, the lexical analyzer can return values for tokens, if desired.

    An action is an arbitrary C statement, and as such can do input and output, call subprograms, and alter external vectors and variables. An action is specified by one or more statements, enclosed in curly braces «<'' and ``>». For example, and are grammar rules with actions.

    To facilitate easy communication between the actions and the parser, the action statements are altered slightly. The symbol «dollar sign» «$» is used as a signal to Yacc in this context.

    To return a value, the action normally sets the pseudovariable «$$» to some value. For example, an action that does nothing but return the value 1 is

    To obtain the values returned by previous actions and the lexical analyzer, the action may use the pseudo-variables $1, $2, . . ., which refer to the values returned by the components of the right side of a rule, reading from left to right. Thus, if the rule is for example, then $2 has the value returned by C, and $3 the value returned by D.

    As a more concrete example, consider the rule The value returned by this rule is usually the value of the expr in parentheses. This can be indicated by

    By default, the value of a rule is the value of the first element in it ($1). Thus, grammar rules of the form frequently need not have an explicit action.

    In the examples above, all the actions came at the end of their rules. Sometimes, it is desirable to get control before a rule is fully parsed. Yacc permits an action to be written in the middle of a rule as well as at the end. This rule is assumed to return a value, accessible through the usual mechanism by the actions to the right of it. In turn, it may access the values returned by the symbols to its left. Thus, in the rule the effect is to set x to 1, and y to the value returned by C.

    Actions that do not terminate a rule are actually handled by Yacc by manufacturing a new nonterminal symbol name, and a new rule matching this name to the empty string. The interior action is the action triggered off by recognizing this added rule. Yacc actually treats the above example as if it had been written:

    In many applications, output is not done directly by the actions; rather, a data structure, such as a parse tree, is constructed in memory, and transformations are applied to it before output is generated. Parse trees are particularly easy to construct, given routines to build and maintain the tree structure desired. For example, suppose there is a C function node, written so that the call creates a node with label L, and descendants n1 and n2, and returns the index of the newly created node. Then parse tree can be built by supplying actions such as: in the specification.

    The user may define other variables to be used by the actions. Declarations and definitions can appear in the declarations section, enclosed in the marks «%<'' and ``%>». These declarations and definitions have global scope, so they are known to the action statements and the lexical analyzer. For example, could be placed in the declarations section, making variable accessible to all of the actions. The Yacc parser uses only names beginning in «yy»; the user should avoid such names.

    In these examples, all the values are integers: a discussion of values of other types will be found in Section 10.

    3: Lexical Analysis

    The user must supply a lexical analyzer to read the input stream and communicate tokens (with values, if desired) to the parser. The lexical analyzer is an integer-valued function called yylex. The function returns an integer, the token number, representing the kind of token read. If there is a value associated with that token, it should be assigned to the external variable yylval.

    The parser and the lexical analyzer must agree on these token numbers in order for communication between them to take place. The numbers may be chosen by Yacc, or chosen by the user. In either case, the «# define» mechanism of C is used to allow the lexical analyzer to return these numbers symbolically. For example, suppose that the token name DIGIT has been defined in the declarations section of the Yacc specification file. The relevant portion of the lexical analyzer might look like:

    The intent is to return a token number of DIGIT, and a value equal to the numerical value of the digit. Provided that the lexical analyzer code is placed in the programs section of the specification file, the identifier DIGIT will be defined as the token number associated with the token DIGIT.

    This mechanism leads to clear, easily modified lexical analyzers; the only pitfall is the need to avoid using any token names in the grammar that are reserved or significant in C or the parser; for example, the use of token names if or while will almost certainly cause severe difficulties when the lexical analyzer is compiled. The token name error is reserved for error handling, and should not be used naively (see Section 7).

    As mentioned above, the token numbers may be chosen by Yacc or by the user. In the default situation, the numbers are chosen by Yacc. The default token number for a literal character is the numerical value of the character in the local character set. Other names are assigned token numbers starting at 257.

    To assign a token number to a token (including literals), the first appearance of the token name or literal in the declarations section can be immediately followed by a nonnegative integer. This integer is taken to be the token number of the name or literal. Names and literals not defined by this mechanism retain their default definition. It is important that all token numbers be distinct.

    For historical reasons, the endmarker must have token number 0 or negative. This token number cannot be redefined by the user; thus, all lexical analyzers should be prepared to return 0 or negative as a token number upon reaching the end of their input.

    A very useful tool for constructing lexical analyzers is the Lex program developed by Mike Lesk.[8] These lexical analyzers are designed to work in close harmony with Yacc parsers. The specifications for these lexical analyzers use regular expressions instead of grammar rules. Lex can be easily used to produce quite complicated lexical analyzers, but there remain some languages (such as FORTRAN) which do not fit any theoretical framework, and whose lexical analyzers must be crafted by hand.

    4: How the Parser Works

    Yacc turns the specification file into a C program, which parses the input according to the specification given. The algorithm used to go from the specification to the parser is complex, and will not be discussed here (see the references for more information). The parser itself, however, is relatively simple, and understanding how it works, while not strictly necessary, will nevertheless make treatment of error recovery and ambiguities much more comprehensible.

    The parser produced by Yacc consists of a finite state machine with a stack. The parser is also capable of reading and remembering the next input token (called the lookahead token). The current state is always the one on the top of the stack. The states of the finite state machine are given small integer labels; initially, the machine is in state 0, the stack contains only state 0, and no lookahead token has been read.

    The machine has only four actions available to it, called shift, reduce, accept, and error. A move of the parser is done as follows:

    1. Based on its current state, the parser decides whether it needs a lookahead token to decide what action should be done; if it needs one, and does not have one, it calls yylex to obtain the next token.

    2. Using the current state, and the lookahead token if needed, the parser decides on its next action, and carries it out. This may result in states being pushed onto the stack, or popped off of the stack, and in the lookahead token being processed or left alone.

    The shift action is the most common action the parser takes. Whenever a shift action is taken, there is always a lookahead token. For example, in state 56 there may be an action: which says, in state 56, if the lookahead token is IF, the current state (56) is pushed down on the stack, and state 34 becomes the current state (on the top of the stack). The lookahead token is cleared.

    The reduce action keeps the stack from growing without bounds. Reduce actions are appropriate when the parser has seen the right hand side of a grammar rule, and is prepared to announce that it has seen an instance of the rule, replacing the right hand side by the left hand side. It may be necessary to consult the lookahead token to decide whether to reduce, but usually it is not; in fact, the default action (represented by a «.») is often a reduce action.

    Reduce actions are associated with individual grammar rules. Grammar rules are also given small integer numbers, leading to some confusion. The action refers to grammar rule 18, while the action refers to state 34.

    Suppose the rule being reduced is

    The reduce action depends on the left hand symbol (A in this case), and the number of symbols on the right hand side (three in this case). To reduce, first pop off the top three states from the stack (In general, the number of states popped equals the number of symbols on the right side of the rule). In effect, these states were the ones put on the stack while recognizing x, y, and z, and no longer serve any useful purpose. After popping these states, a state is uncovered which was the state the parser was in before beginning to process the rule. Using this uncovered state, and the symbol on the left side of the rule, perform what is in effect a shift of A. A new state is obtained, pushed onto the stack, and parsing continues. There are significant differences between the processing of the left hand symbol and an ordinary shift of a token, however, so this action is called a goto action. In particular, the lookahead token is cleared by a shift, and is not affected by a goto. In any case, the uncovered state contains an entry such as: causing state 20 to be pushed onto the stack, and become the current state.

    In effect, the reduce action «turns back the clock» in the parse, popping the states off the stack to go back to the state where the right hand side of the rule was first seen. The parser then behaves as if it had seen the left side at that time. If the right hand side of the rule is empty, no states are popped off of the stack: the uncovered state is in fact the current state.

    The reduce action is also important in the treatment of user-supplied actions and values. When a rule is reduced, the code supplied with the rule is executed before the stack is adjusted. In addition to the stack holding the states, another stack, running in parallel with it, holds the values returned from the lexical analyzer and the actions. When a shift takes place, the external variable yylval is copied onto the value stack. After the return from the user code, the reduction is carried out. When the goto action is done, the external variable yyval is copied onto the value stack. The pseudo-variables $1, $2, etc., refer to the value stack.

    The other two parser actions are conceptually much simpler. The accept action indicates that the entire input has been seen and that it matches the specification. This action appears only when the lookahead token is the endmarker, and indicates that the parser has successfully done its job. The error action, on the other hand, represents a place where the parser can no longer continue parsing according to the specification. The input

    tokens it has seen, together with the lookahead token, cannot be followed by anything that would result in a legal input. The parser reports an error, and attempts to recover the situation and resume parsing: the error recovery (as opposed to the detection of error) will be covered in Section 7.

    It is time for an example! Consider the specification

    When Yacc is invoked with the -v option, a file called y.output is produced, with a human-readable description of the parser. The y.output file corresponding to the above grammar (with some statistics stripped off the end) is:

    Notice that, in addition to the actions for each state, there is a description of the parsing rules being processed in each state. The _ character is used to indicate what has been seen, and what is yet to come, in each rule. Suppose the input is It is instructive to follow the steps of the parser while processing this input.

    Initially, the current state is state 0. The parser needs to refer to the input in order to decide between the actions available in state 0, so the first token, DING, is read, becoming the lookahead token. The action in state 0 on DING is is «shift 3», so state 3 is pushed onto the stack, and the lookahead token is cleared. State 3 becomes the current state. The next token, DONG, is read, becoming the lookahead token. The action in state 3 on the token DONG is «shift 6», so state 6 is pushed onto the stack, and the lookahead is cleared. The stack now contains 0, 3, and 6. In state 6, without even consulting the lookahead, the parser reduces by rule 2. This rule has two symbols on the right hand side, so two states, 6 and 3, are popped off of the stack, uncovering state 0. Consulting the description of state 0, looking for a goto on sound, is obtained; thus state 2 is pushed onto the stack, becoming the current state.

    In state 2, the next token, DELL, must be read. The action is «shift 5», so state 5 is pushed onto the stack, which now has 0, 2, and 5 on it, and the lookahead token is cleared. In state 5, the only action is to reduce by rule 3. This has one symbol on the right hand side, so one state, 5, is popped off, and state 2 is uncovered. The goto in state 2 on place, the left side of rule 3, is state 4. Now, the stack contains 0, 2, and 4. In state 4, the only action is to reduce by rule 1. There are two symbols on the right, so the top two states are popped off, uncovering state 0 again. In state 0, there is a goto on rhyme causing the parser to enter state 1. In state 1, the input is read; the endmarker is obtained, indicated by «$end» in the y.output file. The action in state 1 when the endmarker is seen is to accept, successfully ending the parse.


    The reader is urged to consider how the parser works when confronted with such incorrect strings as DING DONG DONG, DING DONG, DING DONG DELL DELL, etc. A few minutes spend with this and other simple examples will probably be repaid when problems arise in more complicated contexts.

    5: Ambiguity and Conflicts

    A set of grammar rules is ambiguous if there is some input string that can be structured in two or more different ways. For example, the grammar rule is a natural way of expressing the fact that one way of forming an arithmetic expression is to put two other expressions together

    with a minus sign between them. Unfortunately, this grammar rule does not completely specify the way that all complex inputs should be structured. For example, if the input is the rule allows this input to be structured as either (The first is called left association, the second right association).

    Yacc detects such ambiguities when it is attempting to build the parser. It is instructive to consider the problem that confronts the parser when it is given an input such as When the parser has read the second expr, the input that it has seen: matches the right side of the grammar rule above. The parser could reduce the input by applying this rule; after applying the rule; the input is reduced to expr(the left side of the rule). The parser would then read the final part of the input: and again reduce. The effect of this is to take the left associative interpretation.

    Alternatively, when the parser has seen

    it could defer the immediate application of the rule, and continue reading the input until it had seen It could then apply the rule to the rightmost three symbols, reducing them to expr and leaving Now the rule can be reduced once more; the effect is to take the right associative interpretation. Thus, having read the parser can do two legal things, a shift or a reduction, and has no way of deciding between them. This is called a shift / reduce conflict. It may also happen that the parser has a choice of two legal reductions; this is called a reduce / reduce conflict. Note that there are never any «Shift/shift» conflicts.

    When there are shift/reduce or reduce/reduce conflicts, Yacc still produces a parser. It does this by selecting one of the valid steps wherever it has a choice. A rule describing which choice to make in a given situation is called a disambiguating rule.

    Yacc invokes two disambiguating rules by default:

    1. In a shift/reduce conflict, the default is to do the shift.

    2. In a reduce/reduce conflict, the default is to reduce by the earlier grammar rule (in the input sequence).

    Rule 1 implies that reductions are deferred whenever there is a choice, in favor of shifts. Rule 2 gives the user rather crude control over the behavior of the parser in this situation, but reduce/reduce conflicts should be avoided whenever possible.

    Conflicts may arise because of mistakes in input or logic, or because the grammar rules, while consistent, require a more complex parser than Yacc can construct. The use of actions within rules can also cause conflicts, if the action must be done before the parser can be sure which rule is being recognized. In these cases, the application of disambiguating rules is inappropriate, and leads to an incorrect parser. For this reason, Yacc always reports the number of shift/reduce and reduce/reduce conflicts resolved by Rule 1 and Rule 2.

    In general, whenever it is possible to apply disambiguating rules to produce a correct parser, it is also possible to rewrite the grammar rules so that the same inputs are read but there are no conflicts. For this reason, most previous parser generators have considered conflicts to be fatal errors. Our experience has suggested that this rewriting is somewhat unnatural, and produces slower parsers; thus, Yacc will produce parsers even in the presence of conflicts.

    As an example of the power of disambiguating rules, consider a fragment from a programming language involving an «if-then-else» construction:

    In these rules, IF and ELSE are tokens, cond is a nonterminal symbol describing conditional (logical) expressions, and stat is a nonterminal symbol describing statements. The first rule will be called the simple-if rule, and the second the if-else rule.

    These two rules form an ambiguous construction, since input of the form can be structured according to these rules in two ways: or The second interpretation is the one given in most programming languages having this construct. Each ELSE is associated with the last preceding «un-ELSE’d» IF. In this example, consider the situation where the parser has seen and is looking at the ELSE. It can immediately reduce by the simple-if rule to get and then read the remaining input, and reduce by the if-else rule. This leads to the first of the above groupings of the input.

    On the other hand, the ELSE may be shifted, S2 read, and then the right hand portion of can be reduced by the if-else rule to get which can be reduced by the simple-if rule. This leads to the second of the above groupings of the input, which is usually desired.

    Once again the parser can do two valid things — there is a shift/reduce conflict. The application of disambiguating rule 1 tells the parser to shift in this case, which leads to the desired grouping.

    This shift/reduce conflict arises only when there is a particular current input symbol, ELSE, and particular inputs already seen, such as In general, there may be many conflicts, and each one will be associated with an input symbol and a set of previously read inputs. The previously read inputs are characterized by the state of the parser.

    The conflict messages of Yacc are best understood by examining the verbose (-v) option output file. For example, the output corresponding to the above conflict state might be:

    23: shift/reduce conflict (shift 45, reduce 18) on ELSE

    stat : IF ( cond ) stat_ (18) stat : IF ( cond ) stat_ELSE stat

    ELSE shift 45 . reduce 18

    The first line describes the conflict, giving the state and the input symbol. The ordinary state description follows, giving the grammar rules active in the state, and the parser actions. Recall that the underline marks the portion of the grammar rules which has been seen. Thus in the example, in state 23 the parser has seen input corresponding to and the two grammar rules shown are active at this time. The parser can do two possible things. If the input symbol is ELSE, it is possible to shift into state 45. State 45 will have, as part of its description, the line since the ELSE will have been shifted in this state. Back in state 23, the alternative action, described by «.», is to be done if the input symbol is not mentioned explicitly in the above actions; thus, in this case, if the input symbol is not ELSE, the parser reduces by grammar rule 18: Once again, notice that the numbers following «shift» commands refer to other states, while the numbers following «reduce» commands refer to grammar rule numbers. In the y.output file, the rule numbers are printed after those rules which can be reduced. In most one states, there will be at most reduce action possible in the state, and this will be the default command. The user who encounters unexpected shift/reduce conflicts will probably want to look at the verbose output to decide whether the default actions are appropriate. In really tough cases, the user might need to know more about the behavior and construction of the parser than can be covered here. In this case, one of the theoretical references[2, 3, 4] might be consulted; the services of a local guru might also be appropriate.

    There is one common situation where the rules given above for resolving conflicts are not sufficient; this is in the parsing of arithmetic expressions. Most of the commonly used constructions for arithmetic expressions can be naturally described by the notion of precedence levels for operators, together with information about left or right associativity. It turns out that ambiguous grammars with appropriate disambiguating rules can be used to create parsers that are faster and easier to write than parsers constructed from unambiguous grammars. The basic notion is to write grammar rules of the form and for all binary and unary operators desired. This creates a very ambiguous grammar, with many parsing conflicts. As disambiguating rules, the user specifies the precedence, or binding strength, of all the operators, and the associativity of the binary operators. This information is sufficient to allow Yacc to resolve the parsing conflicts in accordance with these rules, and construct a parser that realizes the desired precedences and associativities.

    The precedences and associativities are attached to tokens in the declarations section. This is done by a series of lines beginning with a Yacc keyword: %left, %right, or %nonassoc, followed by a list of tokens. All of the tokens on the same line

    are assumed to have the same precedence level and associativity; the lines are listed in order of increasing precedence or binding strength. Thus, describes the precedence and associativity of the four arithmetic operators. Plus and minus are left associative, and have lower precedence than star and slash, which are also left associative. The keyword %right is used to describe right associative operators, and the keyword %nonassoc is used to describe operators, like the operator .LT. in Fortran, that may not associate with themselves; thus, is illegal in Fortran, and such an operator would be described with the keyword %nonassoc in Yacc. As an example of the behavior of these declarations, the description might be used to structure the input as follows: When this mechanism is used, unary operators must, in general, be given a precedence. Sometimes a unary operator and a binary operator have the same symbolic representation, but different precedences. An example is unary and binary ‘-‘; unary minus may be given the same strength as multiplication, or even higher, while binary minus has a lower strength than multiplication. The keyword, %prec, changes the precedence level associated with a particular grammar rule. %prec appears immediately after the body of the grammar rule, before the action or closing semicolon, and is followed by a token name or literal. It causes the precedence of the grammar rule to become that of the following token

    name or literal. For example, to make unary minus have the same precedence as multiplication the rules might resemble:

    A token declared by %left, %right, and %nonassoc need not be, but may be, declared by %token as well.

    The precedences and associativities are used by Yacc to resolve parsing conflicts; they give rise to disambiguating rules. Formally, the rules work as follows:

    1. The precedences and associativities are recorded for those tokens and literals that have them.

    2. A precedence and associativity is associated with each grammar rule; it is the precedence and associativity of the last token or literal in the body of the rule. If the %prec construction is used, it overrides this default. Some grammar rules may have no precedence and associativity associated with them.

    3. When there is a reduce/reduce conflict, or there is a shift/reduce conflict and either the input symbol or the grammar rule has no precedence and associativity, then the two disambiguating rules given at the beginning of the section are used, and the conflicts are reported.

    4. If there is a shift/reduce conflict, and both the grammar rule and the input character have precedence and associativity associated with them, then the conflict is resolved in favor of the action (shift or reduce) associated with the higher precedence. If the precedences are the same, then the associativity is used; left associative implies reduce, right associative implies shift, and nonassociating implies error.

    Conflicts resolved by precedence are not counted in the number of shift/reduce and reduce/reduce conflicts reported by Yacc. This means that mistakes in the specification of precedences may disguise errors in the input grammar; it is a good idea to be sparing with precedences, and use them in an

    essentially «cookbook» fashion, until some experience has been gained. The y.output file is very useful in deciding whether the parser is actually doing what was intended.

    7: Error Handling

    Error handling is an extremely difficult area, and many of the problems are semantic ones. When an error is found, for example, it may be necessary to reclaim parse tree storage, delete or alter symbol table entries, and, typically, set switches to avoid generating any further output.

    It is seldom acceptable to stop all processing when an error is found; it is more useful to continue scanning the input to find further syntax errors. This leads to the problem of getting the parser «restarted» after an error. A general class of algorithms to do this involves discarding a number of tokens from the input string, and attempting to adjust the parser so that input can continue.

    To allow the user some control over this process, Yacc provides a simple, but reasonably general, feature. The token name «error» is reserved for error handling. This name can be used in grammar rules; in effect, it suggests places where errors are expected, and recovery might take place. The parser pops its stack until it enters a state where the token «error» is legal. It then behaves as if the token «error» were the current lookahead token, and performs the action encountered. The lookahead token is then reset to the token that caused the error. If no special error rules have been specified, the processing halts when an error is detected.

    In order to prevent a cascade of error messages, the parser, after detecting an error, remains in error state until three tokens have been successfully read and shifted. If an error is detected when the parser is already in error state, no message is given, and the input token is quietly deleted.

    As an example, a rule of the form would, in effect, mean that on a syntax error the parser would attempt to skip over the statement in which the error was seen. More precisely, the parser will scan ahead, looking for three tokens that might legally follow a statement, and start processing at the first of these; if the beginnings of statements are not sufficiently distinctive, it may make a false start in the middle of a statement, and end up reporting a second error where there is in fact no error.

    Actions may be used with these special error rules. These actions might attempt to reinitialize tables, reclaim symbol table space, etc.

    Error rules such as the above are very general, but difficult to control. Somewhat easier are rules such as Here, when there is an error, the parser attempts to skip over the statement, but will do so by skipping to the next ‘;’. All tokens after the error and before the next ‘;’ cannot be shifted, and are discarded. When the ‘;’ is seen, this rule will be reduced, and any «cleanup» action associated with it performed.

    Another form of error rule arises in interactive applications, where it may be desirable to permit a line to be reentered after an error. A possible error rule might be There is one potential difficulty with this approach; the parser must correctly process three input tokens before it admits that it has correctly resynchronized after the error. If the reentered line contains an error in the first two tokens, the parser deletes the offending tokens, and gives no message; this is clearly unacceptable. For this reason, there is a mechanism that can be used to force the parser to believe that an error has been fully recovered from. The statement in an action resets the parser to its normal mode. The last example is better written

    As mentioned above, the token seen immediately after the «error» symbol is the input token at which the error was discovered. Sometimes, this is inappropriate; for example, an error recovery action might take upon itself the job of finding the correct place to resume input. In this case, the previous lookahead token must be cleared. The statement in an action will have this effect. For example, suppose the action after error were to call some sophisticated resynchronization routine, supplied by the user, that attempted to advance the input to the beginning of the next valid statement. After this routine was called, the next token returned by yylex would

    presumably be the first token in a legal statement; the old, illegal token must be discarded, and the error state reset. This could be done by a rule like

    These mechanisms are admittedly crude, but do allow for a simple, fairly effective recovery of the parser from many errors; moreover, the user can get control to deal with the error actions required by other portions of the program.

    8: The Yacc Environment

    When the user inputs a specification to Yacc, the output is a file of C programs, called y.tab.c on most systems (due to local file system conventions, the names may differ from installation to installation). The function produced by Yacc is called yyparse; it is an integer valued function. When it is called, it in turn repeatedly calls yylex, the lexical analyzer supplied by the user (see Section 3) to obtain input tokens. Eventually, either an error is detected, in which case (if no error recovery is possible) yyparse returns the value 1, or the lexical analyzer returns the endmarker token and the parser accepts. In this case, yyparse returns the value 0.

    The user must provide a certain amount of environment for this parser in order to obtain a working program. For example, as with every C program, a program called main must be defined, that eventually calls yyparse. In addition, a routine called yyerror prints a message when a syntax error is detected.

    These two routines must be supplied in one form or another by the user. To ease the initial effort of using Yacc, a library has been provided with default versions of main and yyerror. The name of this library is system dependent; on many systems the library is accessed by a -ly argument to the loader. To show the triviality of these default programs, the source is given below: and The argument to yyerror is a string containing an error message, usually the string «syntax error». The average application will want to do better than this. Ordinarily, the program should keep track of the input line number, and print it along with the message when a syntax error is detected. The external integer variable yychar contains the lookahead token number at the time the error was detected; this may be of some interest in giving better diagnostics. Since the main program is probably supplied by the user (to read arguments, etc.) the Yacc library is useful only in small projects, or in the earliest stages of larger ones.

    The external integer variable yydebug is normally set to 0. If it is set to a nonzero value, the parser will output a verbose description of its actions, including a discussion of which input symbols have been read, and what the parser actions are. Depending on the operating environment, it may be possible to set this variable by using a debugging system.

    9: Hints for Preparing Specifications

    This section contains miscellaneous hints on preparing efficient, easy to change, and clear specifications. The individual subsections are more or less independent.

    It is difficult to provide rules with substantial actions and still have a readable specification file. The following style hints owe much to Brian Kernighan.

    a. Use all capital letters for token names, all lower case letters for nonterminal names. This rule comes under the heading of «knowing who to blame when things go wrong.»

    b. Put grammar rules and actions on separate lines. This allows either to be changed without an automatic need to change the other.

    c. Put all rules with the same left hand side together. Put the left hand side in only once, and let all following rules begin with a vertical bar.

    d. Put a semicolon only after the last rule with a given left hand side, and put the semicolon on a separate line. This allows new rules to be easily added.

    e. Indent rule bodies by two tab stops, and action bodies by three tab stops.

    The example in Appendix A is written following this style, as are the examples in the text of this paper (where space permits). The user must make up his own mind about these stylistic questions; the central problem, however, is to make the rules visible through the morass of action code.

    The algorithm used by the Yacc parser encourages so called «left recursive» grammar rules: rules of the form These rules frequently arise when writing specifications of sequences and lists: and In each of these cases, the first rule will be reduced for the first item only, and the second rule will be reduced for the second and all succeeding items.

    With right recursive rules, such as the parser would be a bit bigger, and the items would be seen, and reduced, from right to left. More seriously, an internal stack in the parser would be in danger of overflowing if a very long sequence were read. Thus, the user should use left recursion wherever reasonable.

    It is worth considering whether a sequence with zero elements has any meaning, and if so, consider writing the sequence specification with an empty rule: Once again, the first rule would always be reduced exactly once, before the first item was read, and then the second rule would be reduced once for each item read. Permitting empty sequences often leads to increased generality. However, conflicts might arise if Yacc is asked to decide which empty sequence it has seen, when it hasn’t seen enough to know!


    Some lexical decisions depend on context. For example, the

    lexical analyzer might want to delete blanks normally, but not within quoted strings. Or names might be entered into a symbol table in declarations, but not in expressions.

    One way of handling this situation is to create a global flag that is examined by the lexical analyzer, and set by actions. For example, suppose a program consists of 0 or more declarations, followed by 0 or more statements. Consider: The flag dflag is now 0 when reading statements, and 1 when reading declarations, except for the first token in the first statement. This token must be seen by the parser before it can tell that the declaration section has ended and the statements have begun. In many cases, this single token exception does not affect the lexical scan.

    This kind of «backdoor» approach can be elaborated to a noxious degree. Nevertheless, it represents a way of doing some things that are difficult, if not impossible, to do otherwise.

    Some programming languages permit the user to use words like «if», which are normally reserved, as label or variable names, provided that such use does not conflict with the legal use of these names in the programming language. This is extremely hard to do in the framework of Yacc; it is difficult to pass information to the lexical analyzer telling it «this instance of `if’ is a keyword, and that instance is a variable». The user can make a stab at it, using the mechanism described in the last subsection, but it is difficult.

    A number of ways of making this easier are under advisement. Until then, it is better that the keywords be reserved; that is, be forbidden for use as variable names. There are powerful stylistic reasons for preferring this, anyway.

    10: Advanced Topics

    This section discusses a number of advanced features of Yacc.

    Simulating Error and Accept in Actions

    The parsing actions of error and accept can be simulated in an action by use of macros YYACCEPT and YYERROR. YYACCEPT causes yyparse to return the value 0; YYERROR causes the parser to behave as if the current input symbol had been a syntax error; yyerror is called, and error recovery takes place. These mechanisms can be used to simulate parsers with multiple endmarkers or context-sensitive syntax checking.

    Accessing Values in Enclosing Rules.

    An action may refer to values returned by actions to the left of the current rule. The mechanism is simply the same as with ordinary actions, a dollar sign followed by a digit, but in this case the digit may be 0 or negative. Consider In the action following the word CRONE, a check is made that the preceding token shifted was not YOUNG. Obviously, this is only possible when a great deal is known about what might precede the symbol noun in the input. There is also a distinctly unstructured flavor about this. Nevertheless, at times this mechanism will save a great deal of trouble, especially when a few combinations are to be excluded from an otherwise regular structure.

    Support for Arbitrary Value Types

    By default, the values returned by actions and the lexical analyzer are integers. Yacc can also support values of other types, including structures. In addition, Yacc keeps track of the types, and inserts appropriate union member names so that the resulting parser will be strictly type checked. The Yacc value stack (see Section 4) is declared to be a union of the various types of values desired. The user declares the union, and associates union member names to each token and nonterminal symbol having a value. When the value is referenced through a $$ or $n construction, Yacc will automatically insert the appropriate union name, so that no unwanted conversions will take place. In addition, type checking commands such as Lint[5] will be far more silent.

    There are three mechanisms used to provide for this typing. First, there is a way of defining the union; this must be done by the user since other programs, notably the lexical analyzer, must know about the union member names. Second, there is a way of associating a union member name with tokens and nonterminals. Finally, there is a mechanism for describing the type of those few values where Yacc can not easily determine the type.

    To declare the union, the user includes in the declaration section: This declares the Yacc value stack, and the external variables yylval and yyval, to have type equal to this union. If Yacc was invoked with the -d option, the union declaration is copied onto the y.tab.h file. Alternatively, the union may be declared in a header file, and a typedef used to define the variable YYSTYPE to represent this union. Thus, the header file might also have said: The header file must be included in the declarations section, by use of %< and %>.

    Once YYSTYPE is defined, the union member names must be associated with the various terminal and nonterminal names. The construction is used to indicate a union member name. If this follows one of the keywords %token, %left, %right, and %nonassoc, the union

    member name is associated with the tokens listed. Thus, saying will cause any reference to values returned by these two tokens to be tagged with the union member name optype. Another keyword, %type, is used similarly to associate union member names with nonterminals. Thus, one might say

    There remain a couple of cases where these mechanisms are insufficient. If there is an action within a rule, the value returned by this action has no a priori type. Similarly, reference to left context values (such as $0 — see the previous subsection) leaves Yacc with no easy way of knowing the type. In this case, a type can be imposed on the reference by inserting a union member name, between , immediately after the first $. An example of this usage is This syntax has little to recommend it, but the situation arises rarely.

    A sample specification is given in Appendix C. The facilities in this subsection are not triggered until they are used: in particular, the use of %type will turn on these mechanisms. When they are used, there is a fairly strict level of checking. For example, use of $n or $$ to refer to something with no defined type is diagnosed. If these facilities are not triggered, the Yacc value stack is used to hold int’s, as was true historically.

    Yacc owes much to a most stimulating collection of users, who have goaded me beyond my inclination, and frequently beyond my ability, in their endless search for «one more feature». Their irritating unwillingness to learn how to do things my way has usually led to my doing things their way; most of the time, they have been right. B. W. Kernighan, P. J. Plauger, S. I. Feldman, C. Imagna, M. E. Lesk, and A. Snyder will recognize some of their ideas in the current version of Yacc. C. B. Haley contributed to the error recovery algorithm. D. M. Ritchie, B. W. Kernighan, and M. O. Harris helped translate this document into English. Al Aho also deserves special credit for bringing the mountain to Mohammed, and other favors.

    1. B. W. Kernighan and D. M. Ritchie, The C Programming Language, Prentice-Hall, Englewood Cliffs, New Jersey, 1978.

    2. A. V. Aho and S. C. Johnson, «LR Parsing,» Comp. Surveys, vol. 6, no. 2, pp. 99-124, June 1974.

    3. A. V. Aho, S. C. Johnson, and J. D. Ullman, «Deterministic Parsing of Ambiguous Grammars,» Comm. Assoc. Comp. Mach., vol. 18, no. 8, pp. 441-452, August 1975.

    4. A. V. Aho and J. D. Ullman, Principles of Compiler Design, Addison-Wesley, Reading, Mass., 1977.

    5. S. C. Johnson, «Lint, a C Program Checker,» Comp. Sci. Tech. Rep. No. 65, 1978 .]. updated version TM 78-1273-3

    6. S. C. Johnson, «A Portable Compiler: Theory and Practice,» Proc. 5th ACM Symp. on Principles of Programming Languages, pp. 97-104, January 1978.

    7. B. W. Kernighan and L. L. Cherry, «A System for Typesetting Mathematics,» Comm. Assoc. Comp. Mach., vol. 18, pp. 151-157, Bell Laboratories, Murray Hill, New Jersey, March 1975 .].

    8. M. E. Lesk, «Lex — A Lexical Analyzer Generator,» Comp. Sci. Tech. Rep. No. 39, Bell Laboratories, Murray Hill, New Jersey, October 1975 .].

    Appendix A: A Simple Example

    This example gives the complete Yacc specification for a small desk calculator; the desk calculator has 26 registers, labeled «a» through «z», and accepts arithmetic expressions made up of the operators +, -, *, /, % (mod operator), & (bitwise and), | (bitwise or), and assignment. If an expression at the top level is an assignment, the value is not printed; otherwise it is. As in C, an integer that begins with 0 (zero) is assumed to be octal; otherwise, it is assumed to be decimal.

    As an example of a Yacc specification, the desk calculator does a reasonable job of showing how precedences and ambiguities are used, and demonstrating simple error recovery. The major oversimplifications are that the lexical analysis phase is much simpler than for most applications, and the output is produced immediately, line by line. Note the way that decimal and octal integers are read in by the grammar rules; This job is probably better done by the lexical analyzer.

    Appendix B: Yacc Input Syntax

    This Appendix has a description of the Yacc input syntax, as a Yacc specification. Context dependencies, etc., are not considered. Ironically, the Yacc input specification language is most naturally specified as an LR(2) grammar; the sticky part comes when an identifier is seen in a rule, immediately following an action. If this identifier is followed by a colon, it is the start of the next rule; otherwise it is a continuation of the current rule, which just happens to have an action embedded in it. As implemented, the lexical analyzer looks ahead after seeing an identifier, and decide whether the next token (skipping blanks, newlines, comments, etc.) is a colon. If so, it returns the token C_IDENTIFIER. Otherwise, it returns IDENTIFIER. Literals (quoted strings) are also returned as IDENTIFIERS, but never as part of C_IDENTIFIERs.

    Appendix C: An Advanced Example

    This Appendix gives an example of a grammar using some of the advanced features discussed in Section 10. The desk calculator example in Appendix A is modified to prov that may also be used. The usage is similar to that in Appendix A; assignments return no value, and print nothing, while expressions print the (floating or interval) value.

    This example explores a number of interesting features of Yacc and C. Intervals are represented by a structure, consisting of the left and right endpoint values, stored as double’s. This structure is given a type name, INTERVAL, by using typedef. The Yacc value stack can also contain floating point scalars, and integers (used to index into the arrays holding the variable values). Notice that this entire strategy depends strongly on being able to assign structures and unions in C. In fact, many of the actions call functions that return structures as well.

    It is also worth noting the use of YYERROR to handle error conditions: division by an interval containing 0, and an interval presented in the wrong order. In effect, the error recovery mechanism of Yacc is used to throw away the rest of the offending line.

    In addition to the mixing of types on the value stack, this grammar also demonstrates an interesting use of syntax to keep track of the type (e.g. scalar or interval) of intermediate expressions. Note that a scalar can be automatically promoted to an interval if the context demands an interval value. This causes a large number of conflicts when the grammar is run through Yacc: 18 Shift/Reduce and 26 Reduce/Reduce. The problem can be seen by looking at the two input lines: and Notice that the 2.5 is to be used in an interval valued expression in the second example, but this fact is not known until the «,» is read; by this time, 2.5 is finished, and the parser cannot go back and change its mind. More generally, it might be

    necessary to look ahead an arbitrary number of tokens to decide whether to convert a scalar to an interval. This problem is evaded by having two rules for each binary interval valued operator: one when the left operand is a scalar, and one when the left operand is an interval. In the second case, the right operand must be an interval, so the conversion will be applied automatically. Despite this evasion, there are still many cases where the conversion may be applied or not, leading to the above conflicts. They are resolved by listing the rules that yield scalars first in the specification file; in this way, the conflicts will be resolved in the direction of keeping scalar valued expressions scalar valued until they are forced to become intervals.

    This way of handling multiple types is very instructive, but not very general. If there were many kinds of expression types, instead of just two, the number of rules needed would increase dramatically, and the conflicts even more dramatically. Thus, while this example is instructive, it is better practice in a more normal programming language environment to keep the type information as part of the value, and not as part of the grammar.

    Finally, a word about the lexical analysis. The only unusual feature is the treatment of floating point constants. The C library routine atof is used to do the actual conversion from a character string to a double precision value. If the lexical analyzer detects an error, it responds by returning a token that is illegal in the grammar, provoking a syntax error in the parser, and thence error recovery. Appendix D: Old Features Supported but not Encouraged

    This Appendix mentions synonyms and features which are supported for historical continuity, but, for various reasons, are not encouraged.

    1. Literals may also be delimited by double quotes «»».

    2. Literals may be more than one character long. If all the characters are alphabetic, numeric, or _, the type number of the literal is defined, just as if the literal did not have the quotes around it. Otherwise, it is difficult to find the value for such literals.

    The use of multi-character literals is likely to mislead those unfamiliar with Yacc, since it suggests that Yacc is doing a job which must be actually done by the lexical analyzer.

    3. Most places where % is legal, backslash «\» may be used. In particular, \\ is the same as %%, \left the same as %left, etc.

    4. There are a number of other synonyms:

    5. Actions may also have the form and the curly braces can be dropped if the action is a single C statement.

    6. C code between % < and %>used to be permitted at the head of the rules section, as well as in the declaration section.

    Что такое код yacc

    yacc — компьютерная программа, служащая стандартным генератором синтаксических анализаторов (парсеров) в Unix-системах. Название является акронимом «Yet Another Compiler Compiler» («ещё один компилятор компиляторов»). Yacc генерирует парсер на основе аналитической грамматики, описанной в нотации BNF (форма Бэкуса-Наура) или контекстно-свободной грамматики. На выходе yacc выдаётся код парсера на языке программирования Си.

    Yacc был разработан Стефеном Джонсоном (Stephen C. Johnson) в AT&T для операционной системы Unix. Позже были написаны совместимые версии программы, такие как Berkeley Yacc, GNU bison, MKS yacc и Abraxas yacc (обновлённый вариант AT&T-версии с открытым исходным кодом также вошёл в проект OpenSolaris от Sun). Каждый вариант предлагал незначительные улучшения и дополнительные возможности по сравнению с оригиналом, но концепция осталась той же. Yacc также был переписан на других языках, включая Java, C#, Pascal, Ada, Ratfor, EFL, ML, Limbo, Erlang, Go и т. д.

    Поскольку парсер, генерируемый с помощью yacc, требует использования лексического анализатора, то часто он используется совместно с генератором лексических анализаторов, в большинстве случаев это lex либо flex. Стандарт IEEE POSIX P1003.2 определяет как функциональность, так и требования для lex и yacc.

    Lex и Yacc (генерация таблицы символов)

    Я новичок в ЛЕКС и Yacc и составитель дизайн. Я хотел бы знать, на каком этапе (лексические, синтаксические или любой другой фазы) и как формируется таблица символов?

    Могу ли я иметь краткое описание y.output файла, который генерируется путем предоставления -v возможность yacc.I пытался смотреть в него, но не получить много информации.

    Мог ли я знать другие приложения, где закон и Yacc используются отдельно от конструкций компилятора.

    Таблица символов представляет собой глобальную структуру данных, которые могут быть использованы на всех этапах / фаз / проходов компилятор. Это означает, что он может быть использован / доступ с обеих Lex и YACC генерироваться компонентов.

    Общепризнанно , чтобы получить доступ к записи таблицы символов из лексического анализатора , когда он находит маркер , который будет храниться в таблице, например , в качестве идентификатора, он может найти запись и обновить его информацию , доступной только лексер как номер строки и позиция символа и он также может хранить значение лексемы , если он еще не существует. Указатель таблицы символов теперь могут быть возвращены в lval лексемы.

    Некоторые люди предпочитают возвращать указатель на сам лексемы (как lval ) от лексера анализатора и сделать первоначальный доступ таблицы символов там. Это имеет то преимущество , что таблица символов не должны быть видны лексером, но имеет тот недостаток , что LeXeR информацию , как описано выше , возможно , больше не будет доступна для хранения с символом. Это часто имеет тот недостаток , что делает синтаксический анализатор действие от Yacc немного больше «занято» , как они тогда могут участвовать в управлении таблицей символов, а также деревом синтаксического разбора.

    Запись таблицы символов далее будет обновляться на более поздних этапах компилятора, такие как семантический ходьбы от синтаксического дерева, которое может аннотировать записи символов с информацией о типе и флаг необъявленных объектов и тому подобное. Таблица символов будет снова использоваться в процессе генерации целевого кода, когда целевая конкретная информация может быть сохранена или необходимо, и вновь в процессе оптимизации, когда может быть рассмотрена использование переменных или даже оптимизирована прочь.

    Таблица символов представляет собой структуру данных , которые вы писатель компилятор создать для себя. Там нет особенности Лекса или Yacc , что делает это для вас. Он генерируется , как и при любом коде , вы пишете его создает!

    Файл y.output не имеет ничего общего с таблицами символов. Это запись о том, как Yacc преобразовал бесконтекстный грамматик в таблицу синтаксического анализа. Это полезно, когда у вас есть неоднозначные грамматики и хотите знать, какие правила вызывают сдвиг / уменьшить или уменьшить / уменьшить количество ошибок при отладке грамматика.

    В последней части вопроса, то, что использует сделать это инструмент есть? закон является инструментом, который генерирует код для конечного автомата, который распознает шаблоны, заданные. Он не должен быть использован при написании компиляторов. Одно интересное применение в обработке сетевых протоколов, которые могут быть обработаны с помощью государственной машины, таких как TCP / IP-дейтаграммы и так далее. Аналогичным образом, Yacc используется в соответствии последовательности, которые описаны с помощью контекстно-свободных грамматик. Они не должны быть программы, но могут быть и другие сложные последовательности символов, полей или элементов данных. Они просто обычно куски текста, и это ортодоксальное использование инструмента.

    Эти части вашего вопроса действительно звучит как своего рода экзамен вопрос, что кто-то может написать для студентов, которые посещали курс составителей!

    Илон Маск рекомендует:  Что такое код asp @transaction
  • Понравилась статья? Поделиться с друзьями:
    Кодинг, CSS и SQL