scala - Integrating native system libraries with SBT

Question

What is a good way to integrate various SBT tasks with native libraries (for example, those from JOGL, LWGL, or JCuda? Specifically,

1. Is there a recommended way to include a native library in the run task? A discussion on the SBT mailing list suggests these possibilities:

   • Modify JavaOptions to include -Djava.library.path=<path to native libraries>, and then modify the run task to fork the JVM. (See this plugin for an example.)
   • Use the SBT initialize setting to run code that calls System.setProperty(...) to configure java.library.path. Again, run must fork.
   • Put the native libraries on the classpath before launching SBT.

   The last one has the advantage that run need not fork, but the disadvantage that the configuration must be done outside SBT.

2. Can I automatically include native libraries in the Eclipse project generated by the sbteclipse plugin? It's possible to rewrite the .project file in a post-processing step. Is there example code? Is there a better way?

3. Can native libraries be included in the runnable Jar that is generated by a plugin such as sbt-assembly, sbt-onejar or sbt-proguard?

I assume there is no direct SBT setting for native libraries. If something like that existed, could the above tasks handle native libraries transparently?

Answer 1

From the research I've done in the past, there are only two ways to get native libraries loaded: modifying java.library.path and using System.loadLibrary (I feel like most people do this), or using System.load with an absolute path.

As you've alluded to, messing with java.library.path can be annoying in terms of configuring SBT and Eclipse, and I do not think it's possible to do automatically for an executable jar.

So that leaves System.load. In terms of writing your own native libraries, what you can do is:

• Create an SBT task that compiles your native sources (with javah and gcc), takes the resulting .so files and any .so files it depends on, puts them in a jar (as resources) in the target directory, and adds the path to the jar to unmanagedJars in Compile.
• Create a Scala method to load a library. Instead of calling System.loadLibrary, it will use Class.getResourceAsStream to read the library, File.createTempFile to write it somewhere on the filesystem, and System.load to load it into the JVM.
• Now instead of calling System.loadLibrary as you would before, call MyClasspathJniLoader.loadLibrary.

That will work with SBT run, Eclipse, and executable jars without any extra configuration (although I don't know how proguard knows which resources to include).

Now as to third party native libraries that have already been written, some of them like jblas already use this "fat jar" approach. If they expect you to set up java.library.path and then they call System.loadLibrary when they feel like it, you'll need to do some magic to make that work.

I haven't tried this, but this solution might work:

• Use a similar SBT task to put all of the libraries on their native path into a jar as resources, and put that jar on the clsaspath.
• Create a Scala method that takes a function and a list of library names, creates a temp directory, reads those libraries from the jar's resources into files in the temp directory, adds the temp directory to java.library.path, calls the passed in function, and finally reverts the java.library.path back to what it was before.
• The first time you call into the native library (when it presumably will statically initialize and make the System.loadLibrary call), wrap that particular call in your method with the list of libraries it will load. That way, when it calls System.loadLibrary, all of its libraries will be on java.library.path and will be loaded successfully.

Obviously this is annoying since you have to manually initialize the third party library before using it, but it seems more feasible to wrap all of the initialization points (your main functions and test initializations) than it does to get all of your tools to set java.library.path correctly. And it may be even easier than that if you already have your own layer of abstraction above the third party library, so there's really only one initialization point you have to wrap.

If that seems like a realistic solution, I can add more details about the SBT task or Scala wrapper methods if you're confused.