Transformation pipelines

Important

Loki is still under active development and has not yet seen a stable release. Interfaces can change at any time, objects may be renamed, or concepts may be re-thought. Make sure to sync your work to the current release frequently by rebasing feature branches and upstreaming more general applicable work in the form of pull requests.

Transformations

Transformations are the building blocks of a transformation pipeline in Loki. They encode the workflow of converting a Sourcefile or an individual program unit (such as Module or Subroutine) to the desired output format.

A transformation can encode a single modification, combine multiple steps, or call other transformations to create complex changes. If a transformation depends on another transformation, inheritance can be used to combine them.

Every transformation in a pipeline should implement the interface defined by Transformation. It provides generic entry points for transforming different objects and thus allows for batch processing. To implement a new transformation, only one or all of the relevant methods Transformation.transform_subroutine, Transformation.transform_module, or Transformation.transform_file need to be implemented.

Example: A transformation that renames every module and subroutine by appending a given suffix:

class RenameTransformation(Transformation):

    def __init__(self, suffix):
        super().__init__()
        self._suffix = suffix

    def _rename(self, item):
        item.name += self._suffix

    def transform_subroutine(self, routine, **kwargs):
        self._rename(routine)

    def transform_module(self, module, **kwargs):
        self._rename(module)

The transformation can be applied by calling apply() with the relevant object.

source = Sourcefile(...)  # may contain modules and subroutines
transformation = RenameTransformation(suffix='_example')
transformation.apply(source)

Note that, despite only implementing logic for transforming modules and subroutines, it works also for sourcefiles. While the Sourcefile object itself is not modified (because we did not implement transform_file()), the dispatch mechanism in the Transformation base class takes care of calling the relevant method for each member of the given object.

Typically, transformations should be implemented by users to encode the transformation pipeline for their individual use-case. However, Loki comes with a few built-in transformations for common tasks and we expect this list to grow in the future:

Further transformations are defined for specific use-cases but may prove useful in a wider context. These are defined in transformations:

Additionally, a number of tools for common transformation tasks are provided as functions that can be readily used in a step of the transformation pipeline:

Bulk processing large source trees

Transformations can be applied over source trees using the Scheduler. It is a work queue manager that automatically discovers source files in a list of paths. Given the name of an entry routine, it allows to build a call graph and thus derive the dependencies within this source tree.

Calling Scheduler.process on a source tree and providing it with a Transformation applies this transformation to all modules and routines, making sure that routines with the relevant CallStatement are always processed before their target Subroutine.

When applying the transformation to an item in the source tree, the scheduler provides certain information about the item to the transformation:

  • the transformation mode (provided in the scheduler’s config),

  • the item’s role (e.g., ‘driver’ or ‘kernel’, configurable via the scheduler’s config), and

  • targets (routines that are called from the item and are included in the scheduler’s tree, i.e., will be processed afterwards).