неділя, 21 липня 2013 р.

Fragments, data readers, VerifyErrors and auto-completion

   Time is passing by like crazy. GSoC is close to the middle-line soon, and so much is still to be done. The good news are that today I have a bunch of new features I've implemented from the last time. Enjoy!

Fragments and tabs

   Fragments in Android 4.0+ are quite a wide topic to cover in one blow. I started working on it from easy use-cases, one of which are actionbar tabs. Introduced in 4.0+ as well, action bar can contain tabs which allow user to switch between different application perspectives. The insides of the tab are defined by the fragment. Boldly harnessing the power of neko.ui framework, I made it look something like this:

(neko.action-bar/setup-action-bar a
  {:title "Custom title"
   :navigation-mode :tabs
   :display-options [:show-home :show-title :home-as-up]
   :subtitle "Custom subtitle"
   :tabs [[:tab {:text "Alpha"
                 :tab-listener (simple-fragment
                                [:linear-layout {:orientation :vertical}
                                 [:text-view {:def `atext
                                              :layout-width :wrap
                                              :layout-height :fill
                                              :text "Default text"
                                              :text-size [30 :dp]}]])}]
          [:tab {:text "Beta"
                 :tab-listener (simple-fragment
                                [:linear-layout {:orientation :vertical}
                                 [:button {:layout-width :wrap
                                           :layout-height :fill
                                           :text "Press to update"}]])}]]})

   In conventional Android specifying listeners for tabs requires plenty of manual labor, generics and newInstance calls. In Clojure implementation I've hidden all this into Neko, so you don't have to think about it at all.

   simple-fragment takes either a view or UI tree and returns a fragment that just displays this view. Of course, fragments can do much more than that and functions to create more sophisticated fragments will follow.

Data readers

   Tagged literals and data readers were added to Clojure 1.4 as a relatively safe way to write reader macros. They allow to specify custom functions for some tags, and when reader should encounter those tags, it should use respective functions on the data that follows. These reader functions have to be specified in data_readers.clj file in project source root.

   It was working on Android only partly. AOT-compiling code that uses data readers worked fine, but if you were then to evaluate code that relies on readers in REPL, it failed. The reason was that we store no clj files in the application package, and Clojure relies exactly on a clj-file to be read in runtime.

   For workaround I had to patch both lein-droid and neko. Before project compilation lein-droid grabs all loaded data-readers from the project environment and saves them to res/raw folder. This resource file then makes it to the final package, and at initialization stage Neko checks for this resource file and loads data reader from it granted that it exists.

   Newest Neko comes with a few data readers for resources. I returned runtime resource-resolving functions (before that all resource resolution utilities were macros) since they provide more flexibility, albeit adding extra latency. Resource readers can help mitigate this issue. For example:

[:text-view {:text #res/string :app-name}]

   It will work without a tag too and the resource will be resolved in runtime. But this allows you pass the keyword in runtime as well, keeping it in some sort of map or computing in some other way. And by sticking a tag before the keyword you ensure that resource ID is resolved in compile time and doesn't spend additional time.

   For now there are readers for strings, drawables, IDs and layouts. Hopefully data readers will prove themselves even more handy in spicing up the look of Clojure/Android applications.

Yo dawg, I heard you like vectors

   Damn, this one was annoying. At some point after I've jumped into Neko development this year, I started getting strange VerifyErrors on different occasions. Sometimes a function was too complex so I had to factor it. Sometimes replacing macro with a function fixed it. Sometimes it was about the reflection calls. The error itself hinted that VM refuses to accept the code I've just compiled. Errors popped up only in REPL while AOT compilation was fine.

   Eventually I found a minimal reproducible case - seven nested vectors. Yes, like this [[[[[[[]]]]]]]. Six vectors were compiled without any problems, but seven were just too much for poor VM.

   Hank suggested on the mailing list that the cause might be the old version of DEXing tools embedded into Clojure by Daniel. I've tried to build Clojure with the newest DEX package, but the error persisted.

   Next two days were filled with pain, depression and reading of baksmali output. I've patched DalvikDynamicClassLoader so it would save compiled classes into cache alongside with jars and dexes. Then I would take these classes, DEX them on the machine, de-dex (undex?) it and the DEX file on the device, compare javap output. Then optimize the DEX on the machine, baksmali it and the ODEX on the device, compare their smali code. Iterate for various code.

   I noticed that DEXes were identical, but optimized DEXes were not. Baksmali'd device ODEX files contained corrupted blocks with opcodes unknown to Dalvik. I started tracing DEX optimization down to the native methods, and that was when I decided to try again the updated DEX package.

   And that was it. Apparently, at one moment Google split the project that handled DEXing into two, probably reusing some of the code. During my first try I hadn't deleted older package, but just copied the new one over. Files that were supposed to be in other places remained old and caused the disruption. After I deleted the old package and copied new, Clojure wouldn't compile anymore - obviously because some files were not there. After I tracked down the project where the necessary classes were moved to, and successfully compiled Clojure, the error went away.

   The moral of this story? Double-check an easy route before you delve into the rabbit hole. On the bright side, I now understand the magic behind Daniel's Dalvik classloader.


   This one is the tastiest for today. Mon-Ouie from Github kindly pointed at that clojure-complete library is enough to enable auto-completion in REPLy and Emacs. Even while the completion in REPLy worked decently, Emacs completion provided by ac-nrepl was obscenely slow. Curious about the roots of such big difference, I investigated both clojure-complete and ac-nrepl and found what was the issue. First, clojure-complete was full of reflection cases, which haven't hampered it much while on JVM, but really mattered on Android. Second, ac-nrepl called the backend many times for different kinds of completions (vars, namespaces, classes etc.). Again, it can be afforded in conventional REPL, but since every REPL request on Android is kind of slow, altogether it results in massive completion latency.

   Long story short, I forked both projects, threw out one half of the code, totally rewrote the other one and in the end achieved decent completion speed. You might even want to use it with `ac-auto-start t` (start completion automatically), though even the slightest lag annoys me so I turned it off. Completion by TAB is just enough for me.

   The auto-completion I ended up with by much resembles the original complete-core/ac-nrepl. You type in stuff, a menu with candidates appears, next to menu there is a documentation for the current candidate. I tinkered with the documentation more to reduce response times by introducing caching on both sides, so the docs are really snappy.

   The limitations are mostly related to classes completion. Currently only classes that are imported into the current namespace and their methods can be completed. This may change if I manage to parse Android javadocs on the computer and complete from there.

   See for yourself how it looks like.

Traditional function completion with docs
Namespaces are not a problem

...Neither namespace-qualified vars are

How about static methods and fields?

Method completion with signatures in docs?

You must be kidding me


   Notice for adventurous readers: in order to be able to reproduce all above-mentioned features, you need the following versions of libraries:
  •    org.clojure-android/clojure 1.5.1-SNAPSHOT
  •    lein-droid 0.2.0-SNAPSHOT
  •    neko 3.0.0-SNAPSHOT
  •    org.clojure-android/clojure-complete 0.3.0-SNAPSHOT
  •    My fork of ac-nrepl.el
   All of those are snapshots now, but I'm going to release kind-of-stable versions soon.

   Regarding ac-nrepl, you will have to put the file into your "~/.emacs.d/" directory and at this code to init.el:

(load "~/.emacs.d/ac-nrepl.el")
(require 'ac-nrepl)
(add-hook 'nrepl-mode-hook 'ac-nrepl-setup)
(add-hook 'nrepl-mode-hook 'ac-flyspell-workaround)
(add-hook 'nrepl-interaction-mode-hook 'ac-nrepl-setup)
(add-hook 'nrepl-connected-hook 'ac-nrepl-require-complete-core)
(eval-after-load "auto-complete" '(add-to-list 'ac-modes 'nrepl-mode))

(defun set-auto-complete-as-completion-at-point-function ()
  (setq completion-at-point-functions '(auto-complete)))

(add-hook 'auto-complete-mode-hook 'set-auto-complete-as-completion-at-point-function)
(add-hook 'nrepl-mode-hook 'set-auto-complete-as-completion-at-point-function)
(add-hook 'nrepl-interaction-mode-hook 'set-auto-complete-as-completion-at-point-function)

  I hope you found today's report interesting, and see you next time!

неділя, 7 липня 2013 р.

Dynamically-inflated UI and menus

  I finally managed to get up to speed working on the project. From the beginning I started redesigning the existing neko.ui system which to this time worked in compile time. I was quite proud of it, because it was a fast way to do UI generation compared even to native XML approach, since all transformations had happened before the application even launched.

  The method had a downside though - you had to have all UI-defining trees declared in compile-time. This means that you could not just generate a Clojure data structure and call `make-ui` on it. Apparently the lack of this feature tied hands in situations where UI element style heavily depended upon the state of the application.

  The UI system being macro-based was also harder to reason about. You couldn't have functions as attributes - you needed to quote them so they don't get evaluated too soon. The traits, which are handlers for specific attributes of the widget, didn't just do their stuff - they had to generate code for doing it and add it to other code. Then there was this inconsistency in compile order - if you wanted to extend the UI system (add new elements, traits etc.) you somehow had to make sure the extension code gets compiled before the code that actually uses these extensions.

While being absolutely easier to understand, dynamic inflation is much harder to pull off. In the old system, keywords from attributes were converted to method calls by a macro, and the compiler could work out the exact method afterwards. In the case of dynamic generation, you have either to use reflection or invent something different. My immediate idea was to generate XML from Clojure data structures, and then just offload all the work to Android's native utilities. As it turned out, you can't just make a random XML and feed it to inflater - the XML file has to be first processed by *aapt* and what not. Reusing the facilities that operate on data after it's parsed was also not an option - they proved to be exceptionally coupled and reliant on the exact classes, with no possibility to get wedged into.


   So, the reflection path then. At first, I was a bit worried about the performance of this solution because of the notoriety of reflection in Clojure world. But I did some benchmarks, and manual reflection-based system showed some decent performance, and I have some further plans to cache reflected methods. Overall, the syntax of `make-ui` didn't change after the rewrite, with some minor deviations. Now you don't have to quote your functions and objects, but you do have to syntax-quote symbols for `:def` attribute.

   But the main change is that now you can manipulate your UI tree as much as you like before calling `make-ui`. It looks like this.

(defn make-button [name visibility]
  [:button {:text name
            :text-size [20 :sp]
            :padding [5 :px]
            :visible visibility
            :on-click (fn [_] (toast (str "My name's " name)))}])

(def data {:foo true, :bar false, :baz false,
           :qux true, :quux true})


(let [layout [:linear-layout {:def `top-layout
                              :orientation :vertical}]
      buttons (for [[name visibility] data]
                (make-button name visibility))]
  (make-ui (concat layout button)))

The example above pretty much shows it all. You can have UI generation as flexible as in, say, Hiccup. You can postpone the creation of actual UI objects until the very moment you need them, and before that just pass around vectors and maps.

Note how text size and padding are specified. I like it this way better than strings which you then have to parse.


   With the introduction of action bar most Android applications started using it to host access to some secondary (and sometimes even primary) functionality. This is done in a form of menu that is right-aligned, usually contain some buttons, a search field and an overflow button (under which elements are hidden when there isn't enough room for them).

Menu on the action bar
   In conventional Android the menus are described in XML as well. They get a separate folder for the configuration files and a special inflater. The ability to fully describe menu in XML is somewhat limited though, and most of the time you have to pass some configuration action to Java code. For instance, a menu item can contain an arbitrary View, and if you want to indicate this in XML, you have to do it in layouts/ folder, so your item definition gets split. Then, there is no convenient way to set a handler, so Android people rather use centralized handlers which are not the greatest design decision as well.

   With new `make-menu` you can create menus as easily as general user interface. The implementation relies on existing neko.ui infrastructure, so it was actually pretty easy to write. The tree you pass to it resembles XML for the most part, with a few pleasant exceptions.

   First, you can specify handlers in the menu tree, like in `make-ui`. Second, you can specify custom action views by providing UI tree to the `:action-view` attribute of the item. An example to follow.

(defn MainActivity-onCreateOptionsMenu [this menu]
  (.superOnCreateOptionsMenu this menu)
   menu [[:item {:title "First"
                 :icon R$drawable/ic_launcher
                 :show-as-action [:if-room :with-text]
                 :on-click (fn [_] (toast "Menu item clicked"))}]
         [:item {:show-as-action [:always :collapse-action-view]
                 :icon android.R$drawable/ic_menu_search
                 :action-view [:search-view {:iconified-by-default true
                                             (fn [q item]
                                               (toast (str "Searched for: " q))
                                               (.collapseActionView item)
         [:menu {:title "Submenu"
                 :icon android.R$drawable/ic_menu_more
                 :show-as-action :always}
          [:item {:title "Second"}]]
         [:group {:id :not-important}
          [:item {:title "Third"
                  :icon android.R$drawable/ic_menu_camera
                  :show-as-action :if-room}]
          [:item {:title "Four"
                  :show-as-action :never}]]])

   The result of this you can see on a screenshot. Notice how we defined a search menu item in the same tree, without splitting our attention.

Action modes

Contextual menu
   Action modes (also known as contextual menus) are a method of presenting additional actions when the user selects something in the application (action modes can be used for virtually any purposes, but most of the time they deal with selected stuff). Before 4.0 the usual approach was to display a conventional dialog-like menu when the user selected something by long-clicking; this approach falls short when user wants to select a couple of items, and is generally distracting.

   Android's own implementation of action modes had one bothersome property - you can't indicate if the action mode is active unless you manually keep track of it. I decided to fight it by having a map of activities to action modes, and automatically populating and depopulating this map with current action mode. Apart from that, the implementation of contextual menus was straightforward and reliant on `make-menu`. Here is an example:
(neko.ui.menu/start-action-mode activity
 (fn [mode menu]
   (make-menu menu
    [[:item {:icon android.R$drawable/ic_menu_share
             :on-click (fn [_]
                         (toast (str "Selected items: "
                                     (string/join (get-selected adapter)))))}]
     [:item {:icon android.R$drawable/ic_menu_revert}]
     [:item {:icon android.R$drawable/ic_menu_edit}]]))
 :on-destroy (fn [mode]
               (clear-selected adapter)))

   Screenshot #2 shows how it looks like.

   That's it so far. For now I will continue porting the most useful Android features to Clojure. Stay tuned!

субота, 1 червня 2013 р.

GSoC 2013 proposal

Here is the text of my proposal for Google Summer of Code 2013. I will dwell on these points more thoroughly later on when I start working on them.

I'm willing to work on the Enhance Neko for Android project. The original lein-droid and Neko projects gave a possibility to conveniently develop Clojure on Android (further CoA) projects, but Neko as a main library still lacks the maturity. Not all necessary UI elements are wrapped to be used in neko.ui's code generation facilities, some macros (like for defining an Activity) still do not cover all the usecases. Also, new Android 4.0-specific features (like ActionBars and Fragments) were beyond the scope of last year's work, but they should be implemented in Neko for it to be usable in modern Android applications. Besides that the differences in Android versions should be somehow reflected in Neko as well, either through conditional loading, or by relocating all 4.0-specific parts to a separate sublibrary.

UI being on the most paramount sides of mobile development, it would be nice to conceptually change the way to think of it besides simply wrapping it for Clojure. On way it can be improved is leveraging Clojure's data manipulation capabilities (like destructuring or core.match) to handle the UI. There are already similar attempts in Neko, like destructuring Intents or neko.data.sqlite, and doing this for UI could really improve development experience. As an additional feature, neko.ui facilities could have a XML backend to create Android-compatible XML files.

Finally there still are issues untackled on the toolchain side. Auto-completion and go-to-definition features, which most Clojure users are used to, are currently unavailable in CoA due to platform peculiarities and need a dedicated effort to be implemented. Since this would most probably be made on the nREPL side, both nrepl.el and CCW could leverage these features thereafter. The last month of development will be devoted to resolve these issues, besides most of the documentation for the implemented features will be written during this period.
My approximated timeline for the duration of GSoC looks like this:
  • Weeks 1-2.5: Add support for Android 4.0+ and devise a way to handle API differences.
  • Weeks 2.5-3: Integrate all UI elements (including 4.0-ones) into neko.ui.
  • Week 4: Complete all utility macros for services and fragments.    
  • Weeks 5-6: Redesign neko.ui to support runtime UI generation without dropping the compile-time generation capabilities.
  • Week 7: Add XML backend for a subset of neko.ui.
  • Weeks 8-9: Devise and include a set of desctructuring and/or core.match operations on UI widgets to extract data from them.
  • Week 10-11: Implement auto-completion and jump-to-definition for nREPL front-ends (like CCW and nrepl.el).
  • Week 12-13: Fixing loose screws - contribute more to UI part, convenience macros or tooling, whichever needs more attention, start writing extensive docs.
  • Week 14-15: Documentation time, for new features and those that will change throughout the program (especially the neko.ui).

State of Clojure on Android 2013

   First of all, I'd like to thank Clojure organization and Clojure community for being invited to participate in Google Summer of Code 2013. I'd like to thank Daniel Solano Gómez, the author of Android-patched Clojure fork and Neko and my last year's mentor, and Zach Oakes, my mentor for this year, the author of Nightweb. We plan to do some awesome things together this summer. But before I start, I'd like to recap the last year and analyze the current situation of Clojure on Android (CoA).

Recapping last year

   My last year's participation was successful enough to bring some attention of the community to CoA. My work resulted in creation of lein-droid - a plugin for Leiningen to seamlessly build CoA projets in Clojure-way; and fork of Neko - a convenience library that claims to bring idiomatic Clojure development practices to Android. Lein-droid managed to get 149 watchers over the year, which I suppose is a good result considering the limited target audience of the project (for comparison, Leiningen has ~2600 watchers - so roughly 5% of the community got interested in CoA).

   Over the course of the year lein-droid was polished, curated and bug-fixed to the point that now it is pretty stable (except for the moments when Google update their build toolchain, but I try to react to those quickly). It is proven to work on both *nix and Windows. Some of last year's problems that wasn't tackled, effectively became non-issues (like long dx-ing time - on my not-so modern machine it takes less than a minute which, I think, is bearable). I plan to release the final 0.1.0 version before I start working on this year's proposal to mark a clear milestone in its development.

   Neko is in a bit afloat right now. Daniel and me haven't merged our branches yet, so officially Neko still supports and prefers building with Ant and use of some concepts deprecated in my branch. I sincerely hope that we are able to finally combine our work and deliver stable version 2.0.0, again before I resume active development.

   I was incredibly excited to know that Zach Oakes developed and released Nightweb - I2P peer-to-peer anonymous social network application, written solely in Clojure using lein-droid and Neko. This is the first serious high-profile CoA application, and I'm very proud that Zach will be my mentor for this year. Giving his experience at hands-on CoA development, he knows the capabilities and limitations of the current CoA stack and can guide me through this summer's quest for better Clojure on Android experience.

Clojure on Android today

   So what can you do with CoA today and what are the current limits?

   Well, obviously you can create complete CoA applications. Lein-droid simplifies the build process to usage of a single command, without the need to touch Ant or Eclipse along the way. You can use Clojure libraries as dependencies just like you do in your JVM Clojure projects (with the only difference that the library shouldn't use JVM-specific classes that are absent on Dalvik). Besides lein-droid (when used in conjunction with neko which is by default) injects a nrepl server into your application which allows to connect to your running application on the device from the computer and develop interactively just like you are used to. Lein-droid provides a distinction between debug build and release build, and doing the latter one removes the nrepl thing automatically, so there is no overhead (although you can deliberately enable it in release builds as well). There is a brief tutorial on getting started here.

   Neko provides a bunch of useful functions and macros to simplify your CoA development. First of all, neko.ui is a namespace where the most magic happens. It allows you to define your UI declaratively and interactively in Clojure, without touching XML, and without any overhead. neko.ui takes UI specified in Clojure data structures and transforms it into Java interop calls in compile time. Right now, the number of UI widgets and properties supported by neko.ui is limited, and extending it is the first objective of my GSoC 2013 proposal. Additionally, there are utilities to work with SQLite, notifications, shared preferences, resources, logging. Neko source code is well documented, and you can also find usage documentation on project's wiki.

   Now to the said part. The main problem and limiting factor for CoA adoption remains the size of Clojure runtime. Android devices are getting more powerful everyday, but so are user expectations in terms of performance. CoA applications take some noticeable time to load (up to 10 seconds on some old devices, up to 4-5 on the latest ones); also due to high memory usage they often become candidates for the operating system to be cleaned up after they are minimized. As an example, I once wrote a simple app in Clojure that shows active processes, and a plugin for it (as a separate application, also written in Clojure); as a result, when the main app opened the plugin it was often itself closed the OS. This is certainly a repulsive experience for Android users. Fortunately though, the talks of developing a leaner Clojure runtime are out there for quite some time, and more and more people become obsessed with this idea. I think, that now when we have quite an efficient ClojureScript compiler written in Clojure, it is feasible to adapt it to compiling native Clojure code into something more lightweight.

   You still have to know Android to be able to use CoA. Actually, it was never a purpose to completely hide Java-Android behind Neko abstractions; but more than that there is a big amount of code you have to write using interop where Clojure would be more elegant. Zach has a lot to say about this. So this year' efforts are directed towards this goal too - reduce the number of interop where it makes sense to do so. Yes, you have to know Android SDK, but you don't have to care about it all the time.

   IDE support is well non-existent right now. Since CoA uses nrepl for remote interactive development, you can use Emacs/Vim/ST/Eclipse to connect to your application. Fancy stuff like jump-to-definition is currently unavailable. Moreover Android developers are quite used to IDE-driven development, especially with the goodies presented in new Android Studio. It is quite hard to win these developers over without providing them something of equal power. So yet another intent of this summer is to research if some of these IDE features can be harnessed from Clojure. The UI side is the tastiest, and since neko.ui would most probably be rewritten (I describe the reasons in my proposal), I will do it with the IDE argument in mind.


   Clojure on Android is definitely young, untapped and has its difficulties, but it can be exciting to explore. All you have to do is to submit to it, and I will make sure the fun is there. See you next time!

четвер, 12 липня 2012 р.

On UI stuff, asynchronicity and the power of reader

Time is due for my usual report about the state of Clojure Android combo. During this time lein-droid was touched only lightly as I rewrote the version resolving part of the plugin (one that chose the latest version of the library on classpath if more than one were provided) to use aether library.

But most changes were done to neko of course. A brief rundown on new things follows.

New features in defui

After I introduced eval inside defui macro I missed one not-so-subtle bug. The macro worked as following: if the UI element is a vector - then it is considered an UI tree form and processed respectively, otherwise it is considered a language element to evaluate and pass through defui one more time. So if you add a form that doesn't eventually return a vector, the compilation will end up in the infinite loop.

Now you can add arbitrary objects created in runtime by quoting their symbol/form. For example, this code will create a LinearLayout and add a button to it:

(let [but (Button. context)]
  (defui [:linear-layout {:orientation :vertical}

One more useful feature is a new :def attribute for elements. Previously I used :id attribute, which registered an element in the global map and allowed further fetching by executing something like (by-id ::ok-button). Now you can directly provide a symbol to :def, which the element would automatically be bound to. See an example (dots mean there could be other code there):

(defui ... [:text-edit {:def price-edit ...}])
(.getText price-edit)

This approach is more convenient (don't have to call anything to get the object) and faster (no time is wasted to fetch the object from the map).

First primitives for ListView

Two-argument version of get-checked
ListView is a very big topic that has to be dealt with eventually so I started with some utility functions for multi-check ListViews first.

By default calling .getCheckedItemPositions on a ListView object returns a SparseBooleanArray instance that is not exceptionally delightful to work with. That is why I wrote get-checked which when given a ListView returns a list of integers that represent the IDs of elements being checked. If provided a second argument (a list of all values in the ListView) returns a vector of only those elements that are checked. Screenshot on the right shows this in action.

set-checked! obviously checks elements whose numbers are respective to the integers in the given list.

Adapters are currently constructed via interop but they need some kind of wrapper too, especially for defining custom views for elements (it would be nice if defui could be harnessed for that as well).

Intents and Bundles as maps

This one was fun to implement. Since Bundles (things that can store activity state or send some data to another activity) are basically maps but don't directly subclass ones I wrote a wrapper called like-map that allows to treat them as Clojure maps.

Here is how it looks:

(defactivity ...
  (fn [this bundle]
        (let [{:keys [sharks-num with-lasers?]} (like-map bundle)]

The whole power of destructuring is available to you instead of boring calls to .getInteger, .getBoolean and the like!

SharedPreferences as universal data storage

If Bundles are used to store some short-lived data while the application is running, SharedPreferences are for storing data between application restarts. like-map supports SharedPreferences as well but there is one more thing to it interesting on its own.

Initially SharedPreferences class supports saving a very limited number of primitive types - integers, booleans, floats and strings. Even for a small array of primitive type you need to use some other local storage (SQLite for instance). But having Clojure reader facilities at hand its a little hard to bear with...

So I didn't: assoc-arbitrary! when called on SharedPreferences.Editor object saves an arbitrary Clojure data structure into SP by serializing it into a string. Then you can read this data by calling get-arbitrary on the like-mapped SP object.

(-> (.edit shared-prefs)
    (assoc! :cpu-level 3)
    (assoc-arbitrary! :game-info {:sharks-num 3, 
                      :shark-health [100 30 75]})

You can still use primitive values with assoc! to avoid serializing primitives.

Keep in mind that Clojure reader is a relatively slow serialization mechanism. If you have a large amount of data to save, then DB or file should be preferred.

Doing things asynchronously

It's 25% and rising.

Mobile application should be responsive at all costs. That means staying responsive no matter what complex computations or large amount of IO you are performing at any given moment. Since Android runs on Java-like Virtual Machine it is essential to use multithreading to separate the UI code execution from some time-consuming activities.

Android SDK has this thing called AsyncTask to deal with (wait for it) asynchronous tasks. It is an abstract class that you should extend in your code and provide the following methods: onPreExecute, onPostExecute, onProgressUpdate and doInBackground. First three methods are executed on the UI thread, the last one - on the individual thread. This allows a task to be run asynchronously but at the same time communicate with UI about its progress.

For you to have a general impression how it is done in Java here's the code:

final ProgressDialog pb = new ProgressDialog(this);
pb.setText("Java is not a verbose language");

new AsyncTask<Void, Integer, String>() {

    protected void onPreExecute() {

    protected String doInBackground(Void... params) {
        for (int i = 0; i < 100; i++) {
        return "sample string";

    protected void onProgressUpdate(Integer... values) {

    protected void onPostExecute(String msg) {

So in order to deal with this pile of terrifically incidental complexity I wrote... nothing! Clojure's native capabilities and on-ui macro I wrote so far was enough to implement asynchronous tasks simply and idiomatically. The same code in Clojure:

(let [pb (defui [:progress-dialog 
                 {:progress-style :horizontal
                  :text "Clojure conquering the world..."}])]
  (.show pb)
    (dotimes [i 100]
      (Thread/sleep 100)
      (on-ui (.setProgress pb i)))
    (on-ui (.hide pb)
           (some-callback "sample string"))))

Besides being three times shorter the implementation in Clojure has one more important advantage - the whole task is in the same lexical scope, so you don't need tricky callbacks to pass the data around it - everything is available everywhere.

Dealing with resources

The initial neko by Daniel Solano Gómez already had a convenient way to resolve resources, so instead of writing your.package.name.R$id/foo you could do (get-id :foo). The problem with this approach lies in its poor performance - the time was spent to resolve a proper internal class and get a field from it using reflection.

And that got me thinking - why do it in runtime if..? You know the rest of the story. Resource-resolving utilities are now macros that simply return the field reference. It looks the same but has no performance overhead at all! Did I tell you why I love Lisp?


I wrote quite a description of nealy every neko namespace. It's not finished yet, but I'll append information on the rest namespaces soon. The docs are available here.


These two weeks involved deep thinking and some difficult decisions so the list of changes might not be that impressive. There is still a lot of stuff to do like ListViews and properly implement listeners for UI elements in defui. I hope to catch up with this in the following week. For now, thank you for your attention.

вівторок, 26 червня 2012 р.

On macros, toasts, private detectives and experimental tools

My GSoC project goes on and it's two weeks now since I've reported in the last time. Good thing is I have some news to share.

Compilation modes

Runtime arity exceptions are no
longer scary if you're safe-for-ui!
One interesting feature I've implemented in neko and lein-droid simultaneously is the support for different code ending up in different compilation modes (say, in debug and release ones). It is achieved by defining a value in the compile time  (boolean one in this case) that represents the AOT-compilation mode currently happening. While compiling lein-droid puts a special property on the JVM property map (via System/setProperty), and then neko.init/is-debug var is set to provide the mode info to other parts of the application.

What is the use of this? For example, this allows to cleanly solve the "exceptions on the UI thread" problem. Brief description: all UI-specific code in Android (including UI handlers) runs on the so-called "UI thread". The issue with it is that it is uncontrollable from the REPL, so if an exceptions is uncaught on it - your program gets forcefully closed.

It's perfectly fine if you are in production (actually, it isn't, but at least you're fully responsible for not handling that exception), but REPL-driven development system has to be more forgiving. After all, you don't want lose your runtime image for accidentally dividing by zero while developing. The solution comes with the feature described above. neko.debug/safe-for-ui macro wraps any block of code in a try/catch. If an exception occurs it's message is displayed on the device screen, and the exception itself is available by using neko.debug/ui-e function. But if compiled in the release mode all your code is on its own - with no extra security and no redundant code.


Sadly though it fails to do what
it claims to but I'm working on it.
This is rather a minor change but I've added a couple of wrappers to use Android notification subsystem (specifically Toasts and Notifications). The wrapper for the former is thin, only removing the necessity of providing a Context to create a toast. Application context is used in this case, it is captured with the call to defapplication and stored as a singleton (gosh, I hate this word).

The functions for Notifications are a bit more useful. The original Notification API seems inconsistent, with separate constructor and setter to specify different (but related) parameters. Anyway, that's how making a notification looks in neko. Notice how I provide a keyword ID instead of integer one. That's much more convenient to reference it if you need to remove the notification later. There's a stubbish declaration of the intent which I would certainly replace with something more clever. Also, icon could be specified once using neko.notify/set-default-notification-icon! so the respective field in notification function could be ignored.

(fire :mynotification
      (notification :icon R$drawable/ic_launcher
                    :ticker-text "Reporting in from ClojureHQ"
                    :content-title "Android Conquer Squad"
                    :content-text "Initiate Clojure World Domination."
                    :action [:activity Intent/ACTION_WEB_SEARCH]))

Project dependencies

Conjing some vals is never
bad if your name is Clojure
No matter how fun it is to develop apps in Clojure, you'll have to be able to use the existing Java libraries. One problem with them is that they usually are not distributed in JARs, but rather as a separate projects. This crutch comes from the limitation of JAR format - it is tedious to extract the resources from the jar to compile it into your own project.

Anyway Eclipse solves this by providing a feature called "library project". This means that you can set one project as a dependency for another so the latter can use both code and resources from the former. Now lein-droid has this ability too. You can use lein init command in the existing repository to create project.clj file in it. lein-droid would automatically conclude if the project is a library or not (based on project.properties file). Then this project can be added under the :project-dependencies in the :android map (sample/project.clj contains an example).

Using this feature I was able to build a Clojure HelloWorld application with ActionBarSherlock. On the screenshot you can see the basic SherlockActivity with some tabs.

Optimizing build time

Bad news first - I've wasted two days bashing my head against the incremental DEX-file generation. Unfortunately without any positive effect. I guess the feature is yet very experimental (that's why there is no documentation on it at all). The idea is to save some time by reusing existing DEX-files to append new classes to them. This may be very valuable since DEX generation is the most time-consuming process throughout the toolchain. But it doesn't work. Most of the time it fails while merging existing file with the new ones, sometimes it succeeds to create a DEX, but application doesn't work at the end. I've tried to debug dx tool but there's lots of byte twisting I need to have deeper knowledge in to understand what's wrong.

However, I discovered the --no-optimize option which cut the DEXing time of my project from 110 seconds down to 80. That's 28% for free which I'm very happy with.

Wrapping up

lein-droid is getting better and closer to the point to be considered a serious build tool for Android applications. neko (my fork specifically) is still very young and rough around the edges (especially the UI part), so it will get most of my attention in the upcoming days.

On a side note, lein-droid 0.0.5 is available on Clojars, as well as neko 1.1.3.

This concludes the news for today. Stay tuned, mail me for any reason you might think of and have fun!

неділя, 10 червня 2012 р.

The goods and bads of my UI struggle

Since it's time for my [to-be] weekly report I wrote down a few words about my progress.

First comes the boring stuff (not boring in a sense that it is boring for me to do, but may be rather boring for the reader). From now on lein-droid correctly handles multiple Android devices connected to the machine. If there is only one of them it directs all device-specific commands (like install or run) to it, otherwise it prompts user to choose the correct device. The user can also specify the device from the command line since all device-specific commands now accept additional arguments same as adb binary (like -d for the only connected device, -e for emulator, or -s SERIAL_NUMBER for the very particular thing).

I added a code-gen task that generates R.java files from the available resources (like images, strings and XML layouts). After the file is compiled with the compile task and goes all the way to the final application.

A minor change in sign task - now it automatically creates a debug keystore if it can't find one. The default path for the keystore is /home/username/.android/debug.keystore (the one Eclipse uses) but if you can change it by providing a value to the :profiles/:dev/:android/:keystore-path in project.clj.

So with all these changes lein-droid bumped to version 0.0.2 and is available on Clojars.

Now to the fun part. I extended and sophisticated my UI toolkit by providing a convenient way of defining "keyword to real thing" relations. Let me substantiate this. Since my initial intent was (and is) to generate in compile time as much as possible, the toolkit needs to know in compile time what the user has passed to it. Class names are not entirely great for this since they are just symbols (so we have to believe user that this symbol will be resolved into the class name when compiled). But if the class names are bound to the respective keywords beforehand then it is safer and easier to use these keywords. Furthermore we need some kind of ad-hoc hierarchy between elements (I partly mentioned it in my previous post). Some attributes (like :layout-height and :layout-width) require special treatment when generating code, and we would like to reuse these specific handlers across multiple elements.

One thing I'm not quite comfortable with is that I had to add eval into the UI-generating facility. I added it to let user manipulate his UI tree any way he wants before feeding it to the defui macro. Without eval user must provide the whole tree as-is to the macro call. In order to better explain what I mean a little demonstration follows.
(def button-attributes
  {:layout-width 0
   :layout-weight 1
   :layout-height :fill})

(defn number-button [i]
   (assoc button-attributes
          :text (str i)
          :onClickListener `(on-click
                             (add-symbol ~i)))])

   [:linear-layout {:orientation   :vertical
                    :layout-width  :fill
                    :layout-height :fill}
    [:edit {:id           ::edit
            :layout-width :fill}]]
   (map (fn [i]
           [:linear-layout {:orientation   :horizontal
                            :layout-width  :fill
                            :layout-height 0
                            :layout-weight 1}] 
           (map (fn [j]
                  (let [n (+ (* i 3) j)]
                    (number-button n)))
                (range 1 4))))
        (range 3))
   [[:linear-layout {:orientation   :horizontal
                     :layout-width  :fill
                     :layout-height 0
                     :layout-weight 1}
     [:button (assoc button-attributes
                :text            (str \u2190)
                :onClickListener '(on-click (delete-symbol)))]
     (number-button 0)
     [:button (assoc button-attributes
                :text            "Call"
                :onClickListener '(on-click (call-number)))]]]))
This code generates the UI you see on the screenshot. It's a bit messy and isn't probably what I would write in the real application. Right, it's rather a proof of concept. Let me describe it step by step.

First I define a simple map of attributes that all my buttons will share. It means that all buttons would fill the entire vertical space they are granted and adjust themselves space-equally in the horizontal dimension. An interesting point arises here. We just defined an ad-hoc style for a button to reuse in any other button (or any other View). A feature that required certain implementation in XML we received for free, just for the fact that we're using the same language for both data (UI in this case) and logic.

Then I write a function called number-button that given a number returns an element of our DSL for describing a button. As you could already see I don't write all number buttons manually, I just call this function with different arguments to generate templates for different buttons. I say "templates" because they will be turned into real buttons later by defui macro.

The final block is how the complete interface is defined. Don't mind the stuff function - it equals (comp vec concat) because I have to insert the generated buttons to the LinearLayouts and every element in the tree looks like [type attributes & inner-elements], so I need to actually append every button to the :linear-layout vector (I might change this in future, perhaps). There are some undefined functions which are not necessary for the demonstration. The whole application code is available here.

There are still a couple of sad things I discovered while working on this. The stack overflows when compiling some elaborate macro. It even overflows on the simplest usages of for macro. Fortunately the separation helps, so if an overflow happens I promote some parts to separate functions and it helps. Though it may seriously hamper productivity when devising really complex layouts.

Anyways that's it for now. Thank you for reading and I'll be happy to answer any questions that may arise.