practicing techie

tech oriented notes to self and lessons learned

Monthly Archives: April 2015

Scala def macros and Java interoperability

Most of the time Scala-Java interoperability works pretty well from a Scala application developer perspective: you can use a wealth of Java class libraries in your Scala programs with fairly little effort. Simply using JavaConversions and maybe a few custom wrappers usually gets you pretty far. Sure, there can be some friction resulting from the use of different programming paradigms and mutable data structures, but if you’re a pragmatist re-using Java code in Scala is nevertheless quite feasible. Scala version 2.10 saw the introduction of an experimental language feature called macros. Specification work on Scala macros recognizes different flavors of macros, but versions 2.10 and 2.11, as well as the future version 2.12, support only def macros. This macro variety behaves similar to methods except that def macro invocations are expanded at compile time. Here’s how EPFL Scala team member and “Scala macros guy” Eugene Burmako characterizes def macros:

If, during type-checking, the compiler encounters an application of the macro m(args), it will expand that application by invoking the corresponding macro implementation method, with the abstract-syntax trees of the argument expressions args as arguments. The result of the macro implementation is another abstract syntax tree, which will be inlined at the call site and will be type-checked in turn.

Def macros resemble C/C++ pre-preprocessor macros, the m4 macro processor and similar in that the result of macro application will be inlined at the call site. A notable difference is that Scala macros are well integrated into the language meaning e.g. that the results of macro expansion are type-checked. But let’s look at this from the code perspective. I’ve implemented a “hello, world” macro and an object called MyReusableService that defines two methods: a regular method and one implemented as a macro. Two objects, ScalaClient and JavaClient, invoke methods on MyReusableService. Here’s what happens when compiling Java code that tries to invoke a macro method on a Scala object:

~ ᐅ sbt 'runMain com.practicingtechie.gist.macros.JavaClient'
...
[error] /Users/marko/blog-gists/macros-interop/src/main/java/com/practicingtechie/gist/macros/JavaClient.java:6: cannot find symbol
[error] symbol: method macroMethod()
[error] location: class com.practicingtechie.gist.macros.MyReusableService
[error] MyReusableService.macroMethod

Method macroMethod is defined by MyReusableService Scala object, but the method is not visible when inspecting the disassembled class file:

~ ᐅ javap -cp target/scala-2.11/classes com.practicingtechie.gist.macros.MyReusableService$
Compiled from "MyReusableService.scala"
public final class com.practicingtechie.gist.macros.MyReusableService$ {
  public static final com.practicingtechie.gist.macros.MyReusableService$ MODULE$;
  public static {};
  public void regularMethod();
}

After removing the macroMethod invocation JavaClient is able to compile. ScalaClient, however, is more interesting. Here’s macro debugging output from running the code (with “-Ymacro-debug-lite” argument):

~ ᐅ sbt 'runMain com.practicingtechie.gist.macros.ScalaClient'
...
[info] Compiling 2 Scala sources and 1 Java source to /Users/marko/blog-gists/macros-interop/target/scala-2.11/classes...
performing macro expansion MyReusableService.macroMethod at source-/Users/marko/blog-gists/macros-interop/src/main/scala/com/practicingtechie/gist/macros/ScalaClient.scala,line-7,offset=158
println("Hello macro world")
Apply(Ident(TermName("println")), List(Literal(Constant("Hello macro world"))))
[info] Running com.practicingtechie.gist.macros.ScalaClient
Hello, from regular method
Hello macro world

In the above extract we can see the location where the macro was applied, as well as results of macro expansion, both as Scala code and as well as abstract-syntax tree (AST) representation.

Finally, we can see macro expansion results expanded and compiled into bytecode at the ScalaClient call site:

~ ᐅ javap -c -cp target/scala-2.11/classes com.practicingtechie.gist.macros.ScalaClient$
...
  public void main(java.lang.String[]);
    Code:
       0: getstatic     #19                 // Field com/practicingtechie/gist/macros/MyReusableService$.MODULE$:Lcom/practicingtechie/gist/macros/MyReusableService$;
       3: invokevirtual #22                 // Method com/practicingtechie/gist/macros/MyReusableService$.regularMethod:()V
       6: getstatic     #27                 // Field scala/Predef$.MODULE$:Lscala/Predef$;
       9: ldc           #29                 // String Hello macro world
      11: invokevirtual #33                 // Method scala/Predef$.println:(Ljava/lang/Object;)V
...

Scala def macros are a very interesting language feature that’s planned to be officially supported in a future Scala version. Def macros are implemented by the Scala compiler, so a function or method whose definition is macro based won’t be accessible in Java code. This is because unlike ordinary function or method invocations the result of the macro application gets expanded at the call site. Still, Scala functions or methods that simply invoke macros from within their body can nonetheless be called from Java code. Depending on how def macros are used, they can sometimes hinder reuse of Scala code from Java or other JVM-based languages.

More info

Advertisements

What’s the strangest bug you’ve squashed?

As software engineers we’re tasked with creating solutions to customer’s business problems. Being complex systems, every once in a while flaws inevitably slip in the design or implementation. And sometimes flaws creep in through use of third party software, which can can make problems all the more difficult to track down. Each bug has a story to tell and the stories about hunting the most puzzling, challenging, annoying and time-consuming bugs can sometimes live with you for a long time.

What’s the strangest bug you’ve managed to squash?

Mine was quite a few years ago when we were working on a greenfield Java EE software project for a client in the health care industry. With alpha release cycle nearing I was deploying a new build in a newly created server environment and during testing we found a bug in an isolated software feature. Initially, I thought there was something wrong with the new environment setup, but after a while I realized the bug seemed to be related with the way the new release was built. During development phase we had been building the software using Oracle JDeveloper IDE, but had moved to using Apache Ant with Sun Java JDK. So, at that point I thought – this was Java EE after all – it was a packaging issue. I carefully compared the working and broken release packages, but couldn’t find any significant differences. Though it was troublesome to reproduce, the problem was fortunately nevertheless reproducible, so I started tracking it down with remote debugging. After a while I noticed that the software was executing a weird code path I couldn’t quite explain.

Puzzled by the strange behaviour I didn’t really have a clear idea how to continue troubleshooting, but I decided to take a long shot with comparing compiled bytecode from the working and broken releases. This was my first time looking at disassembled Java bytecode, which made analysis a bit slow, and all the more interesting, but fortunately in my remote debugging sessions I had been able to identify some likely places for the bug. After staring at the disassembled bytecode for a while an initially innocuous looking bit of code started to look suspect: a mutator method was present in one class in the working build, but missing in the broken one. It turned out that when the code was built with Oracle JDeveloper it automatically generated a mutator method for a subclass, which just happened to override a buggy superclass mutator method. In the broken build such a mutator method wasn’t being generated causing the buggy superclass mutator method to execute.

This story happened years ago, and while there are lots of things I do differently nowadays, including use of different technologies, design approach, unit testing, test automation, build methods and tooling etc., for me this was one of those more memorable bug squashing sessions.

Do you have a intriguing bug hunting story to share?