Code Generation and Meta Programming in Pharo
In this article, I will explain how I automated the tedious process of manually overriding many visitor methods after subclassing the GtPharoStyler
class which uses The Visitor Pattern to style (syntax highlight) Pharo Smalltalk code. The subclass of this code styles Pharo code in an html export, re-using functionality.
When I realized I needed to override dozens of visitor methods after subclassing the above styler class, I didn't want to manually create each method. Glamorous Toolkit has various context menu actions like 'push down' which moves a method to a subclass, but I couldn't find one to copy a method to a subclass. Furthermore I'd have to do this manually for many methods. My first attempt to automate this toil looked like below.
(GtPharoStyler gtMethodsInClass select: #isOverriding) contents do: [ :each | GtPharoHtmlStyler compile: each sourceCode classified: each protocol ]
The above worked but moved manual toil around. After copying these methods I would need to modify all of them manually in rather similar ways. Also, there were some methods that just didn't need overriding. Finally it would be nice to know which visiting methods changed attributes like font color and weight, as those would be the most relevant methods. Think we can do better.
Let's first use some code queries to figure out which visiting methods apply styling attributes to the underlying source code.
We first want to get a Set
of all the different classes referenced in these methods.
(GtPharoStyler gtMethodsInClass select: [ :aMethod | (aMethod selector includesSubstring: 'visit')]) contents flatCollectAsSet: #gtReferencedClasses
"The result of the above" {BlFontWeightAttribute. Color. BlTextForegroundAttribute}
The above looks promising as that is what we would expect a syntax highlighter would do: change the font weight and/or color of different characters/substrings. Let's now get a list of all methods that modify these attributes
stylingMethods := GtPharoStyler gtMethodsInClass select: [ :aMethod | (aMethod selector includesSubstring: 'visit') and: [ aMethod gtReferencedClasses includesAny: {BlTextForegroundAttribute. BlFontWeightAttribute} ] ]
Now let's validate that these methods don't just directly match up the list of overriding methods in the class
overridingMethods := GtPharoStyler methods select: #isOverriding.
"Below comparison is false" (overridingMethods sorted: #selector ascending) = (stylingMethods contents sorted: #selector ascending)
This proves that the naive first attempt of copying all overriding visiting methods that were originally defined in a Trait
is not the best approach.
The general approach to automate the creation of these methods is to:
• Compile them in the subclass with just a super call to the same method in the superclass.
stylingMethods contents do: [ :aMethod | GtPharoHtmlStyler compile: (aMethod sourceCode lines first in: [ :methodSelector | methodSelector , Character cr asString , Character tab asString , 'super ' , methodSelector ]). GtPharoHtmlStyler >> aMethod selector protocol: aMethod protocol ]
• We can then use a Pharo re-write snippet to add the correct logic specific to styling html with correct variable names (that will change from method to method). Let's first limit our find/replace to the visitor methods to provide that as input to the rewrite snippet
visitorMethods := GtPharoHtmlStyler gtMethodsInClass select: [ :each | each selector numArgs = 1 and: [ each selector includesSubstring: 'visit' ] ]
super `aMessage: `args
super `aMessage: `args. self html tag: #div attributes: (self cssStylesFor: `args) with: `args source
Evaluating the above snippet creates a changeset which can be applied completely or selectively
The above changes were never applied as after evaluating all the visitor methods in GtPharoStyler
I realized that some methods 'ruined' my plans, for example look at GtPharoStyler>>#visitArray:
. In that method we are styling enclosing characters at different positions at once. How could I translate this into html tags to be written into a stream? Furthermore, how an AST is traversed is not necessarily in sequential order of the underlying text/code representation.
Basically, the styling code in GT is imperative while html is declarative. So we've reached a dead end, we need a different approach. You can see the approach I used, which relies on iterating through the already styled BlRunRopedText
directly in the companion article.
While we're talking about code generation and the power of Pharo's rewrite engine, let's use it one more time to try and visualize the order in which various parts of a method get styled. Let's create a visitor class that is purposely 'slow' in styling a method and take a screen capture of how this unfolds.
Let's create a class that uses the visitor Trait.
GtGenericPharoStyler << #SlowGtPharoStyler slots: #(editorElement delay lastStyledAstNode lasVisitedAstNode shouldStyleANode styledNodes); package: 'MyGtBlogExport'; tag: 'Example Classes for Blog'; install
Let's create some accessors and copy over all the methods from GtPharoStyler
SlowGtPharoStyler slots do: [ :aSlot | (RBCreateAccessorsForVariableRefactoring variable: aSlot name class: SlowGtPharoStyler classVariable: false) execute ]
GtPharoStyler gtMethodsInClass contents do: [ :aMethod | SlowGtPharoStyler compile: aMethod sourceCode classified: aMethod protocol ]
SlowGtPharoStyler compile: 'styledNodes ^ styledNodes ifNil: [ styledNodes := OrderedCollection new ]' classified: 'accessing'. "SlowGtPharoStyler compile: 'style: aText ast: theAst self lastStyledAstNode ifNil: [ text := aText ]. [ self accept: theAst ] on: Error do: [ :e | (GtStylerErrorSignal new signaledError: e) emit ]. self lastStyledAstNode ifNil: [ text := nil ]' classified: 'api - styling'." SlowGtPharoStyler compile: 'style: aBlText <return: #BlText> ^ BlTextStylerTelemetry rootTimeSync: [ Processor activeProcess name , '' '' , self class name , '': '' , (aBlText first: (20 min: aBlText size)) asString ] during: [ | anUnstyledText aStyledText | anUnstyledText := aBlText. aStyledText := self privateStyle: anUnstyledText characters. aStyledText ]' classified: 'api - styling'. SlowGtPharoStyler compile: 'visitSmaCCParseNode: aSmaCCParseNode aSmaCCParseNode isMethod ifTrue: [ self shouldStyleANode: true ]. super visitSmaCCParseNode: aSmaCCParseNode. aSmaCCParseNode comments ifNotNil: [ :theComments | theComments do: [ :eachComment | self visitCommentFrom: eachComment first to: eachComment second ]. aSmaCCParseNode isMethod ifTrue: [ ''Visited AST once'' logToPackageBeacon. text logToPackageBeacon. self styledNodes logToPackageBeacon. {self lasVisitedAstNode. self lastStyledAstNode} logToPackageBeacon. self lasVisitedAstNode = self lastStyledAstNode ifTrue: [ self lastStyledAstNode: nil. self styledNodes: nil. ''Styling completed'' logToPackageBeacon ] ifFalse: [ ''Triggering another style loop'' logToPackageBeacon. self editorElement enqueueTask: (BlDelayedTaskAction new delay: 300 milliseconds; action: [ self editorElement onViewModelReadyToStyle ]) ]. self lasVisitedAstNode: nil ] ]' classified: 'visiting'.
Now we use a Pharo rewrite snippet to replace all the visiting method bodies to take a brief pause before we do super calls to the methods with the actual functionality
slowVisitorMethods := SlowGtPharoStyler gtMethodsInClass & (#color: gtReferences | #foreground: gtReferences)
methodsWithNoSuperCalls := slowVisitorMethods & ('super' gtSubstringMatch not)
Rewrites for methods with no super calls and one argument
`selector: `aNode | `@temps | `@.Statements.
`selector: `aNode | `@temps | self delay wait. self lasVisitedAstNode: `aNode. (self shouldStyleANode and: [(self styledNodes includes: `aNode) not] ) ifTrue: [ self lastStyledAstNode: `aNode. self styledNodes add: `aNode. self shouldStyleANode: false. thisContext method selector logToPackageBeacon. `@.Statements ].
Rewrites for methods with no super calls and multiple arguments
multipleArgMethods := methodsWithNoSuperCalls select: [:aMethod | aMethod selector numArgs > 1]
`@selector: `@arguments | `@temps | `@.Statements.
`@selector: `@arguments | `@temps argsToCompare | argsToCompare := {`@arguments.}. self delay wait. self lasVisitedAstNode: argsToCompare. (self shouldStyleANode and: [(self styledNodes includes: argsToCompare) not] ) ifTrue: [ self lastStyledAstNode: argsToCompare. self styledNodes add: argsToCompare. self shouldStyleANode: false. thisContext method selector logToPackageBeacon. `@.Statements ].
Rewrites for methods with super call at beginning
`selector: `aNode | `@temps | super `aMessage: `args. `@.AfterStatements.
`selector: `aNode | `@temps | super `aMessage: `args. self delay wait. self lasVisitedAstNode: `aNode. (self shouldStyleANode and: [(self styledNodes includes: `aNode) not] ) ifTrue: [ self lastStyledAstNode: `aNode. self styledNodes add: `aNode. self shouldStyleANode: false. thisContext method selector logToPackageBeacon. `@.AfterStatements ].
Rewrites for methods with super call at end
`selector: `aNode | `@temps | `@.BeforeStatements. super `aMessage: `args.
`selector: `aNode | `@temps | self delay wait. self lasVisitedAstNode: `aNode. (self shouldStyleANode and: [(self styledNodes includes: `aNode) not] ) ifTrue: [ self lastStyledAstNode: `aNode. self styledNodes add: `aNode. self shouldStyleANode: false. thisContext method selector logToPackageBeacon. `@.BeforeStatements ]. super `aMessage: `args.
And now we test
coder := GtPharoMethodCoder forMethod: GtPharoStyler superclass >> #accept:. viewModel := GtPharoMethodCoderViewModel new coder: coder. editorElement := GtSourceCoderEditorElement new textualCoderViewModel: viewModel; fitContent. editorElement enqueueTask: (BlDelayedTaskAction new delay: 100 milliseconds; action: [ viewModel stylers: BlTextNullStyler new asArray ]). editorElement enqueueTask: (BlDelayedTaskAction new delay: 1000 milliseconds; action: [ viewModel stylers: {SlowGtPharoStyler new delay: 25 milliseconds; editorElement: editorElement} ])
The above worked, except that the styling all happens at once after all the delays are added up. So what is going on? Basically a styler on an editor (or a combination of stylers when using BlCompositeStyler
) works by the editor calling the styler and only when it is complete (whether the code runs sync, blocking all the UI, or async) does the underlying styled text get replaced in an atomic fashion.
I went down a rabbit hole that wasn't reasonable even though I tried to timebox it multiple times. The approach was to have a styler that would keep track of the various AST nodes that have been styled and only style 1 node at a time. When the styler traverses all nodes send a message to the editor (after some wait) and tell it to re-style the text again. Continue doing this until all the code is styled and then stop. That was the theory and in reality I got some of this to work but not everything was getting styled correctly. I learned a lot and this knowledge will be useful for when I implement Spell Checking in Glamorous Toolkit. Stay tuned and in the meantime, below is a screen capture of some 'progressive'/slow styling at work. I will update this article if/when I decide to revisit the code and go down more rabbit holes to get it fully working.
