Coding the Thosmos

Should Threading Macros Handle Lambdas?

Tue, 14 Jan 2020 00:00:00 -0800

I recently thought I ran into a bug in clojure.spec because it threw an exception when I put a lambda like (fn [x] x) or #(identity %) into the -> threading macro without wrapping it first in an extra parens like ((fn [x] x)). This is the most common repeated "mistake" I've made in Clojure over the years, and I've heard from others that it's a common misconception about the threading macros. This "mistake" in my case is based on my assumption that the lambda is a fundamental "thing" or value in the language. But, in fact, Clojure is just data, mostly macros, and each macro handles its input data in different ways.

The -> threading macro treats a lambda as a list instead of as a fundamental "function value." When one does (-> :hmm (fn [x] x)) it is expanded like: (fn :hmm [x] x) which obviously is not what is intended. However, it does have a special convenience case for a non-list thing like a symbol, keyword, etc, which it assumes is something that can behave as a function with one argument, and wraps it in parens before threading. This special case adds value because it interprets the intention of the user and avoids forcing them to add parens when they're unnecessary.

I assert that the basic lambda forms (fn) and #() should and easily can be handled similarly to a symbol, by interpreting the intention of the user. The value from doing so would make the threading macros simpler and more intuitive to use, would remove an unnecessary exception that affects numerous users, and would advance the language towards treating the lambda forms as core "values." This last point is a philosophical one, but important in my opinion for making the language "just work" the way a new or non-expert user expects when getting stuff done.

The existing special case of wrapping a non-list form already demonstrates the value of providing some interpretation of the intention of the user, but there now exists an inconsistency between this special case, and the absence of it for two of the most common forms of arguably one of the most foundational "values" in the language, the function itself.

Have you been bitten by this "mistake"?

If you'd like to try it side by side with the existing -> macro, here's a version called t->:

(defmacro t->
  "Threads the expr through the forms. Inserts x as the
  second item in the first form, making a list of it if it is a lambda or not a
  list already. If there are more forms, inserts the first form as the
  second item in second form, etc."
  {:added "1.0"}
  [x & forms]
  (loop [x x, forms forms]
    (if forms
      (let [form (first forms)
            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
                       (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                       (list form x))]
        (recur threaded (next forms)))
      x)))

I've prepared a patch for -> and ->> that handles this use case and has no breaking impact that I can conceive of, keeping the existing behavior, while adding the new. It passes all of the core Clojure tests, but I have not yet tested it on outside code beyond my own. That would obviously be necessary before it could be considered a possible change in core.

From 686831062a574486413022af31e8c7a07b78cd24 Mon Sep 17 00:00:00 2001
From: Thomas Spellman <thos37@gmail.com>
Date: Mon, 13 Jan 2020 20:39:45 -0800
Subject: [PATCH] thread macros

---
 src/clj/clojure/core.clj             | 12 ++++++------
 test/clojure/test_clojure/macros.clj | 12 ++++++++++++
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/src/clj/clojure/core.clj b/src/clj/clojure/core.clj
index 8e98e072..fe43289b 100644
--- a/src/clj/clojure/core.clj
+++ b/src/clj/clojure/core.clj
@@ -1670,42 +1670,42 @@
   (. (. System (getProperties)) (get \"os.name\"))
 
   but is easier to write, read, and understand."
   {:added "1.0"}
   ([x form] `(. ~x ~form))
   ([x form & more] `(.. (. ~x ~form) ~@more)))
 
 (defmacro ->
-  "Threads the expr through the forms. Inserts x as the
-  second item in the first form, making a list of it if it is not a
+  "Threads the expr through the forms. Inserts x as the second item
+  in the first form, making a list of it if it is a lambda or not a
   list already. If there are more forms, inserts the first form as the
   second item in second form, etc."
   {:added "1.0"}
   [x & forms]
   (loop [x x, forms forms]
     (if forms
       (let [form (first forms)
-            threaded (if (seq? form)
+            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
                        (with-meta `(~(first form) ~x ~@(next form)) (meta form))
                        (list form x))]
         (recur threaded (next forms)))
       x)))
 
 (defmacro ->>
-  "Threads the expr through the forms. Inserts x as the
-  last item in the first form, making a list of it if it is not a
+  "Threads the expr through the forms. Inserts x as the last item
+  in the first form, making a list of it if it is a lambda or not a
   list already. If there are more forms, inserts the first form as the
   last item in second form, etc."
   {:added "1.1"}
   [x & forms]
   (loop [x x, forms forms]
     (if forms
       (let [form (first forms)
-            threaded (if (seq? form)
+            threaded (if (and (seq? form) (not (#{'fn 'fn*} (first form))))
               (with-meta `(~(first form) ~@(next form)  ~x) (meta form))
               (list form x))]
         (recur threaded (next forms)))
       x)))
 
 (def map)
 
 (defn ^:private check-valid-options
diff --git a/test/clojure/test_clojure/macros.clj b/test/clojure/test_clojure/macros.clj
index ce17bb38..9fb1fa9e 100644
--- a/test/clojure/test_clojure/macros.clj
+++ b/test/clojure/test_clojure/macros.clj
@@ -106,8 +106,20 @@
   (is (nil? (loop []
               (as-> 0 x
                 (when-not (zero? x)
                   (recur))))))
   (is (nil? (loop [x nil] (some-> x recur))))
   (is (nil? (loop [x nil] (some->> x recur))))
   (is (= 0 (loop [x 0] (cond-> x false recur))))
   (is (= 0 (loop [x 0] (cond->> x false recur)))))
+
+(deftest ->lambda-test
+  (is (= 'a (-> 'a ((fn [x] x)))))
+  (is (= 'a (-> 'a (fn [x] x))))
+  (is (= 'a (-> 'a #(identity %))))
+  (is (= 'a (-> 'a (#(identity %))))))
+
+(deftest ->>lambda-test
+  (is (= 'a (->> 'a ((fn [x] x)))))
+  (is (= 'a (->> 'a (fn [x] x))))
+  (is (= 'a (->> 'a #(identity %))))
+  (is (= 'a (->> 'a (#(identity %))))))
-- 
2.21.0 (Apple Git-122.2)

Spec-Driven DB

Sat, 04 Jan 2020 00:00:00 -0800

Someone asked on ask.clojure.org about modeling SQL data in Clojure, so I answered the following:

I recently moved an app from a 13 year old java/hibernate/MySQL design to Clojure/Datomic

I wrote a library that reads the MySQL table schema, creates an abstract table spec, then uses that spec to create the Datomic schema and migrates all the data over to Datomic. You're welcome to check it out for inspiration (I attempted to write it somewhat generally, but it's fairly tightly focused on my use case) thosmos/mysql-to-datomic

The main point however is that I created my own abstract data spec to describe the tables, columns, and foreign keys, such that I had everything I needed to not only generate the Datomic schema, but also later auto-generated GraphQL and EQL APIs and front-end CRUD data entry forms with linked subforms and data validations, all from this abstract spec. I've attempted to pull the essence of this model-defining spec into a minimal library here: thosmos/domain-spec

It again tightly serves my use case, but perhaps it will inspire you to realize that in Clojure you can do your own thing quite easily and powerfully without over-relying on libraries and dependencies.

Spec-Driven UI

Wed, 04 Dec 2019 00:00:00 -0800

Background: I'm embarking on a long overdue revamp of an old environmental data management system I made 12 years ago that's used by local watershed science organizations to manage their science data. Ever since I built the first version, I've been thinking about how to improve it, but life and other projects have kept me preoccupied, and it's mainly a volunteer effort.

It has a normalized MySQL DB with a Flex frontend that mutates a tree of relational data objects created by Hibernate and sent across the wire via GraniteDS.

A few years ago when I saw Paul deGrandis' Data-Driven talk I realized Clojure would be a good fit for a rewrite. Even more years ago, I considered using XML, XML Schema, and XForms. With XForms 2.0 recently released, I can't say that would be a bad way to go... Declarative UI for the win, but "XML is messy Lisp!" and why not just use the most useful real modern Lisp -> Clojure?

The intention is to migrate the database to Datomic and keep the same basic normalized table schema with a SPA web frontend that reads and transacts a tree of relational data across the wire. Clojure(script), Datomic, EDN, and EQL are perfect for this. Datomic's pull query and tree-like transaction format makes it easy to move a whole tree of relational entities both ways across the wire relatively painlessly. MySQL + Hibernate + GraniteDS were amazing for this 12 years ago, but I like the Datomic + EQL approach more now, mainly because it's a more pure data + code approach. Clojure is great because it's the most simple and stable language I'm aware of, the same language can be used for both client and server, and EDN/Transit is a desirable data format for the wire when talking to a browser.

Progress: I wrote a library that migrated the schema and data over to Datomic from MySQL (thosmos/mysql-to-datomic). This maps tables to namespaces, and columns to attributes in that namespace. One thing I needed was a higher level notion of foreign-keys from one entity type to another, which Datomic doesn't have, as well as other SQLisms like string length for VarChar columns, etc. So I created a simple domain spec that describes this (thosmos/domain-spec), which is then used for generating the Datomic schema, Lacinia GraphQL schema, and can contain DB- or GraphQL-specific things, etc. I'm just using Datomic, so I haven't bothered with other DBs.

This GraphQL backend is already live on our 1st draft production frontend at RiverDB.org and our dev frontend. The specs are used to generate the GraphQL endpoints for all of the tables using only one resolver.

Here's an example of one domain entity from the domain spec:

{:db/id "toxtest",
 :entity/name "toxtest",
 :entity/ns :entity.ns/toxtest,
 :entity/attrs
   [{:attr/name "ComplianceCode",
     :attr/key :toxtest/ComplianceCode,
     :attr/position 8,
     :attr/cardinality :one,
     :attr/type :ref,
     :attr/ref {:entity/ns :entity.ns/compliancelevellookup},
     :attr/strlen 5}
    {:attr/name "LabSampleID",
     :attr/key :toxtest/LabSampleID,
     :attr/position 4,
     :attr/cardinality :one,
     :attr/type :string,
     :attr/strlen 20}]}

Goals:

1) Generate a basic CRUD UI from the domain specs such that the domain data can be easily edited. Generating the UI from the spec enables using the same UI app for multiple different schema shapes, or to more easily migrate the schema without having to touch the UI app. In other words: a spec-driven UI.

I currently have a dev marmelab/react-admin CRUD UI running that's generated from the spec and that provides a good default editing experience for each individual "table" or entity type. The next step is to create some custom UIs that combine a graph of related entity types into a single form. This is where I'm wishing I could be doing this in CLJS instead.

1.5) It would be possible to have some simple schema migration tooling that would provide functions for doing the most common migrations (add an attribute, add an entity type, alias a key, add but not change a join reference, etc) and then would change both the spec and the live DB schema.

2) Because the spec is just data and is itself defined using its own spec format, it's possible to use the same spec-driven CRUD UI to edit it using the same means as the domain data editor. This will be a "schema editor" that enables the user to: add an attribute to an entity type; change an attribute's validation function; add a new entity type; add a new entity reference attribute to an existing entity type, etc. These types of changes result in schema migrations under the hood, using the same tooling as described in (1.5) above. This approach enables adding new fields, and aliasing existing fields, but not renaming or removing fields, since this data might be referenced from elsewhere in the distributed system that is the internet. Also, since we're dealing with historic environmental data, we want to never lose anything, but only track changes and modifications.

This self-editing of the schema is accomplished by using the same spec-driven UI CRUD editor because the schema specs are self-defined using the same spec attrs as domain specs (very meta):

[{:entity/ns    :entity.ns/entity,
  :entity/name  "Entity"
  :entity/pks   [:entity/ns]
  :entity/doc   "The entity spec for the entity spec -> so meta"
  :entity/attrs [{:attr/key         :entity/ns,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/identity   true,
                  :attr/doc         "unique keyword for this spec like -> :entity.ns/sitevisit"}
                 {:attr/key         :entity/name,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "A display name"}
                 {:attr/key         :entity/doc,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "documentation about this entity"}
                 {:attr/key         :entity/prnFn,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "a function with the entity as its arg for printing this entity"}
                 {:attr/key         :entity/summaryKeys,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "a subset of this entity's attrs to display in a summary list"}
                 {:attr/key         :entity/pks,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "list of attribute keywords that collectively act as a unique constraint for this entity"}
                 {:attr/key         :entity/attrs,
                  :attr/type        :ref,
                  :attr/ref         {:entity/ns :entity.ns/attr}
                  :attr/cardinality :many,
                  :attr/component  true,
                  :attr/doc         "attribute specs for this entity"}]}
 {:entity/ns    :entity.ns/attr,
  :entity/name  "Attribute"
  :entity/pks   [:attr/key]
  :entity/doc   "The entity spec for the attribute spec -> very meta"
  :entity/attrs [{:attr/key         :attr/key,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/identity   true,
                  :attr/doc         "the datomic attribute keyword name -> :<entity-ns>/<attr-key> -> :sitevisit/SiteVisitDate"}
                 {:attr/key         :attr/type,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/doc         "#{:keyword :string :boolean :long :bigint :float :double :bigdec :ref :instant :uuid :uri :bytes}"}
                 {:attr/key         :attr/cardinality,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/doc         "#{:one :many}"}
                 {:attr/key         :attr/name,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "a display name"}
                 {:attr/key         :attr/ref,
                  :attr/type        :ref,
                  :attr/cardinality :one,
                  :attr/doc         "a DB ref to the entity spec for the target type of this ref attribute, i.e. Foreign Key"}
                 {:attr/key         :attr/identity,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this an identity attribute?"}
                 {:attr/key         :attr/unique,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a unique attribute?"}
                 {:attr/key         :attr/primary,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a primary key?"}
                 {:attr/key         :attr/required,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this attribute required?"}
                 {:attr/key         :attr/component,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a component attribute?"}
                 {:attr/key         :attr/noHistory,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "skip keeping history in datomic?"}
                 {:attr/key         :attr/fulltext,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "index full text?"}
                 {:attr/key         :attr/derived,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this attribute a derived value?"}
                 {:attr/key         :attr/deriveFn,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "fn code for deriving this attr value"}
                 {:attr/key         :attr/deriveAttrs,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "attrs of the parent entity that are used to derive this attr value"}
                 {:attr/key         :attr/doc,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "this attribute's doc string"}
                 {:attr/key         :attr/decimals,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "number of decimals in this floating point value"}
                 {:attr/key         :attr/strlen,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "number of chars in this string value"}
                 {:attr/key         :attr/position,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "ordinal position - optional for visual ordering"}
                 {:attr/key         :attr/nextAutoVal,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/noHistory   true,
                  :attr/doc         "next auto-increment value. optional"}
                 {:attr/key         :attr/gql,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "A GraphQL-friendly attribute name"}]}]

3) An obvious further meta meta option would be to enable editing the spec's specs! This would for example enable adding a new field to the Attr spec, which would cause it to show up in the schema editor for all schema attributes. This is probably too meta to even mention, but it's possible using the same spec-driven UI!

Realworld Clojure in a Vase on a Pedestal

Sun, 06 May 2018 00:00:00 -0700

In this exceedingly complex "modern" and ever increasingly fast paced world of internet technological development, it's amazing to see an effort like the RealWorld Example Apps, "an Exemplary fullstack Medium.com clone," offer a path of unity and coherence in the midst of the chaos of so many options. It's a thorough front- and back-end API spec for a non-trivial demo app that many different people have built both front- and back-ends for, including re-frame, Keechma, React, Elm, Vue, Scala, Rails, Node, Django, Go, and many more.

Currently there are a couple of Clojurescript frontends, but no complete Clojure backend. I've never built anything with Paul de Grandis' Vase, but I've admired its data-driven architecture for declaratively building APIs fast. I'm curious how well it would do at building a Real World API backend. There's only one way to find out ...

The First step is to create a new Vase app:

lein new vase realworld-vase

This creates a basic ready-to-run Vase app:

realworld-vase
├── Capstanfile
├── Dockerfile
├── README.md
├── boot.properties
├── build.boot
├── config
│   └── logback.xml
├── project.clj
├── resources
│   └── realworld-vase_service.edn
├── src
│   └── realworld_vase
│       ├── server.clj
│       └── service.clj
└── test
    └── realworld_vase
        ├── service_test.clj
        └── test_helper.clj

6 directories, 12 files

Most of the magic happens in the EDN config file: resources/realworld-vase_service.edn

By default, the app has a few example API endpoints which we can see after running the app:

lein run

If we now browse to http://localhost:8080/api/realworld-vase/v1/db we'll see a list of all of the installed attributes in the sample Datomic database.

Our goal is to define a set of attributes to cover all of the necessary fields in the RealWorld API description.

The first entity type is the User:

{
  "user": {
    "email": "jake@jake.jake",
    "token": "jwt.token.here",
    "username": "jake",
    "bio": "I work at statefarm",
    "image": null
  }
}

We can define this in the resources/realworld-vase_service.edn file by replacing the example user schema with the following:

:realworld-vase/user-schema
{:vase.norm/requires [:realworld-vase/base-schema]
 :vase.norm/txes [#vase/schema-tx
   [[:user/username :one :string :identity "A user's unique identifier"]
    [:user/email :one :string :unique "A user's email"]
    [:user/bio :one :string :fulltext "A short blurb about a user"]
    [:user/company :one :ref "A user's employer"]
    [:user/image :one :uri "URL of a user's photo"]
    [:user/token :one :string :index "A user's auth token"]]]}

Vase idempotently applies DB schema using conformity, so we'll need to restart the server in order to see our new attributes.

lein run

Now we should see our new attributes in the db endpoint http://localhost:8080/api/realworld-vase/v1/db. (Search for user/username and user/email for example.)

Further down in our realworld-vase_service.edn file, we can change the definition for the /users endpoint to the following:

 "/users/:username" {:get #vase/query
    {:name :realworld-vase.v1/user-id-page
     :params [username]
     :edn-coerce [username]
     :query [:find ?e
             :in $ ?username
             :where
             [?e :user/username ?username]]}}

 "/user" {:get #vase/query {:name :realworld-vase.v1/user-page
                            ;; All params are required to perform the query
                            :params [email]
                            :query [:find ?e
                                    :in $ ?email
                                    :where
                                    [?e :user/email ?email]]}
          :post #vase/transact {:name :realworld-vase.v1/user-create
                                ;; `:properties` are pulled from the parameters
                                :properties [:db/id
                                             :user/username
                                             :user/email
                                             :user/bio]}
          :delete #vase/transact {:name :realworld-vase.v1/user-delete
                                  :db-op :vase/retract-entity
                                  ;; :vase/retract-entity requires :db/id to be supplied
                                  :properties [:db/id]}}

We should now be able to transact and query some users via the API.

curl -H "Accept: application/json" -X POST -d '{"payload": [{"user/username":"thosmos","user/email":"thos37@gmail.com","user/bio":"coding the thosmos"}]}' http://localhost:8080/api/realworld-vase/v1/user

To be continued ...

Boot-clj Startup

Sun, 02 Apr 2017 00:00:00 -0700

After having a great conversation with Mike Kaplisky at ClojureWest 2017 about how to optimize a Clojure build chain to be as fast as possible (they use Buck at Ladder), I was inspired to look at Boot-clj, my preferred build tool, with some fresh and curious eyes. I'd like to understand better how Boot works under the hood, so I figure a good way to start is to time some various pieces of it in comparison with lein, clojure, and the JVM.

To start with, let's create a hello world boot-built app:

boot -d boot/new new -t app -n hello

By default, after loading deps with an initial run, it takes about 16 sec to build a jar.

time boot build

Compiling 1/1 hello.core...
Writing pom.xml and pom.properties...
Adding uberjar entries...
Writing hello-0.1.0-SNAPSHOT-standalone.jar...
Writing target dir(s)...

36.62s user 5.61s system 267% cpu 15.780 total

Let's add a super basic boot task to build.boot that does nothing:

(deftask noop
  "the simplest boot task"
  []
  identity)

time boot noop

17.88s user 0.83s system 380% cpu 4.920 total

It takes 18 sec cpu time, 5 sec real time, just to do nothing! At least it's using all my 4 cores.

Let's compare this with a hello world java app:

// Hello.java
public class Hello {
    public static void main(String[] args){
        System.out.println("Hello world");
    }
}

javac Hello.java

time java Hello

0.12s user 0.04s system 103% cpu 0.149 total

0.15 secs! That's amazing! Now let's compare that with running our Clojure hello world:

time java -jar target/hello-0.1.0-SNAPSHOT-standalone.jar

2.57s user 0.16s system 196% cpu 1.395 total

1.4 secs ... There's a big time difference there from the java hello world, but it's still much less than running an empty boot task. What causes a bare boot strap to take so long?

Data Driven

Tue, 08 Sep 2015 00:00:00 -0700

It's funny to look back on my previous post from years ago. It was written literally a day or two before I watched a talk by Paul de Grandis called, Unlocking Data-Driven Systems, in which he described a system closer to what I'd imagined than anything I've seen. It inspired me to go much further with Clojure. In fact, I've spent the past year and a half dedicated to learning and working in Clojure, even building my current project with it, Take2.org. Though I have yet to re-tackle the original project that inspired my previous post, the possibility still exists, and the inspiration returns semi-regularly.

UPDATE 2017-04-01: I just experienced my first ClojureWest and I met Paul de Grandis and a bunch of other amazing Clojurists. It was very inspiring and I'm impressed with how welcoming and friendly the community is. I'm now planning to work on the aforementioned science data management project starting after Summer Solstice, so it really seems like forces are aligning for this to happen. At ClojureWest I learned a ton of approaches and solutions that are relevant, so I'm excited to apply what I've learned.

Imagining atomatizing the App

Mon, 15 Dec 2014 00:00:00 -0800

This morning, while reflecting on a late night of working through David Nolen's (swannodette) Intro to Clojurescript and Om Basic Tutorial, I had an epiphany, an insight into a solution to a problem I've been working on for about 15 years. The goal I've been seeking is an elegant app for a non-programmer user to create a custom, elegant scientific data collection and visualization app. It requires building up a data survey from simple and complex data types, possibly based on an existing denormalized database schema, and linking that with an easily customizable data entry screen. It's kind of like Access, but more complex while also easier to use. Yes, a paradox. XRX is a major step in the right direction, but the designed by commitee feel and actually writing and reading XML and XQuery leaves something to be desired. However, I've never really given it a chance and it could be great. I like the document store approach of eXistDB, similar to MongoDB for JSON data. ODK is doing amazing things with XRX, but only by reverting to HTML and javascript for the UI.

I've taken a few runs at the solution over the years, starting with a naive PHP app in the foggy beginning of my programming career, and a couple of paid database gigs for clients for similar needs, but I have yet to attain the degree of Nirvana I know is possible. When I became familiar with Clojure, I intuited that it would probably be much more useful for the job, due to its homoiconic nature, being represented in its own data format and amenable to meta-programming, and to its immutable data types - a severely useful toolbox. But the language has been opaque to me, and simultaneously both too complex and too simple to see how to use it to its fullest ... until this morning. Suddenly, while looking through my bay windows to the trees behind my monitor, I saw the architecture I needed to use. Kind of a mix between a spreadsheet tree, data schema UML, Pure Data wiring, a Swift playground, etc, but that changes based on position within the multidimensional data abstraction context tree.

One possible step toward making this app easier to contemplate is the definition of the app itself, which Om's component-based nature aids. However, I'm wondering if there's an additional step of abstraction that could be taken to a simpler declarative definition of the app, which would be easier for a user or a builder app to construct and edit. For example, here's a prototypical Om component definition:

    (defn comment [{:keys [author text]} owner]
      (om/component
        (dom/div #js {:className "comment"}
          (dom/h2 nil author)
          (dom/span nil text))))

https://gist.github.com/fredyr/8460923

It's a very nice declarative way to define a component, and reminds me of Enyo's nested declarative javascript component model:

{kind: "onyx.Button", name:"Image Button", ontap:"buttonTapped", components: [
  {tag: "img", attributes: {src: "assets/enyo-logo-small.png"}},
  {content: "There is an image here"}
]}

https://github.com/enyojs/onyx/blob/master/samples/ButtonSample.js

Enyo is the closest tool I've found yet to enabling the workflow I need. However, it feels funky. I know, I'm picky.

Clojure's Lispiness is very amenable to representing the tag nature of XML and HTML, and is preferrable if the UI is being generated programmatically anyway. In that vein, Enyo's definition is more clear in some ways because it is simpler, relying on a convention or schema for being converted into running code. But what if the Om approach were to be similarly abstracted further? Can the following simpler version work?

{ :name :comment
  :class "comment"
  ;:kind :div ; :div is the default
  :content [
   { :kind :h2, :state :author}
   { :kind :span, :state :text}]}

It's easier for me to read and seems to contain all of the same essential info, without a lot of the repetitive code (DRY). This declaration is taking advantage of some conventions. One is that a component will pass to its children the same state that it received, unless otherwise stated (pun intended). It's unnecesary to declare its input variables, and it only defines more refined sub-cursors when handing state to its children. This declaration could be used to generate the CLJS code above. This component uses no state of its own; it's just a container DIV. This is another convention: a component is by default a DIV, unless otherwise noted. Here's another more complex component, first in Om code:

(defn comment-list [cs owner]
  (om/component
    (dom/div nil
      (dom/div #js {:className "commentList"}
        ;; Helper function build-all builds a vector of components
        (om/build-all comment cs))
      (dom/button #js
        {:onClick #(change-author cs "Fredrik")}
        "Change authors"))))

from: https://gist.github.com/fredyr/8460923

and here's a mapped abstraction:

{ :name :commentList
  :state [:comment] ; Declares that this component accepts a vector of :comment components.
  :content [
    { :class "commentList"
      :content [
        {:kind :comment, :build :all}]}
    { :kind :button
      :onClick [:changeAuthor :state "Frederik"]
      :content "Change authors"}]}

The mapping of the "build-all" macro seems a bit wonky, but I'm confident there's some reasonable way to represent it. Also, does this need a schema? A Schema might be nice, but really, it's all convention over configuration, a la Enyo, yes? Hmmm ...

Here's a full mapping with its original

My impression is that the mapped form would be much easier for me to reason around with the aim of building an app that could read and write this structure. These structures would be the blueprint from which to actually output the clojure code above. It's the equivalent of the relationship between the data in the atom and its runtime representation to the user.

Plus, as if that weren't enough, there's a huge bonus to this approach: undo and change branching (and saving, and sharing) of the app itself, FOR FREE! Non-programmers are overwhelmed enough trying to make a relatively sophisticated app, but this would make it much easier to save different versions, try something for a while, save it, go back a few steps, go in a different direction, save it, and then be able to browse around through the different versions. This might even make it possible to more easily diff different versions.

One caveat to this approach is that functions will need to be saved and versioned outside of the the app-def atom. Or they can be represented as strings and edited from within the builder app. I neglected to mention that I'm fairly new to clojure, only having worked through David Nolen's intro tutorial yesterday, so I have much to learn and discover of the Joys of Clojure.

Geometric Algebra with Clojure - Part 1

Wed, 01 May 2013 00:00:00 -0700

I've been exploring Geometric Algebra (GA) and Clojure for a while, and I'm finding some potential to combine my learning of both. Here are my first steps on a Mac:

Download and install GaiGen2.5 (http://sourceforge.net/projects/g25/) which is a GA code generator for a variety of languages, including java.
Generate some sample algebra specs, generate the code from the sample Conformal 3d Java spec file, compile, and package into a jar. I get a warning, "Warning: No operator bindings are possible for output language Java." I wonder if Gaigen would benefit from being rewritten in Clojure, to gain from its metaprogramming capabilities ...

cd ~/Develop/GA/gaigen25/
g25_test_generator -sa .
cd TestG25
g25 c3ga_java/c3ga_java.xml
javac c3ga_pkg/*.java
jar cvf c3ga.jar c3ga_pkg

Create a new clojure project, add the GA jar as a resource to the project.clj, and start a repl:

cd ..
lein new gaclj
cd gaclj
mkdir lib
cp ../TestG25/c3ga.jar lib
nano project.clj

(defproject gaclj "0.1.0-SNAPSHOT"
  :description "First try at doing geometric algebra in clojure"
  :url "https://www.thosmos.com/?p=256"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :resource-paths ["lib/c3ga.jar"]
  :dependencies [[org.clojure/clojure "1.4.0"]])

lein repl

user=> (def mv (c3ga_pkg.mv.))
#'user/mv

user=> (.set mv 25.0)
nil

user=>; mv
#<mv 25.00>

user=> (.toString mv)
"25.00"

user=> (def e1 (c3ga_pkg.c3ga/vectorE1))
#'user/e1

user=>; e1
#<vectorE3GA 1.00*e1>

user=> (def e2 (c3ga_pkg.c3ga/vectorE2))
#'user/e2

user=> e2
#<vectorE3GA 1.00*e2>

user=> (def e3 (c3ga_pkg.c3ga/vectorE3))
#'user/e3

user=> (c3ga_pkg.c3ga/add e1 e2)
#<vectorE3GA 1.00*e1 + 1.00*e2>

user=> (def e12 (c3ga_pkg.c3ga/add e1 e2))
#'user/e12

user=> e12
#<vectorE3GA 1.00*e1 + 1.00*e2>

user=> (.toString e12)
"1.00*e1 + 1.00*e2"

This looks promising. Some next steps will be to get the objects drawn on a canvas of some kind, and then to get some live updating going on, ideally in a LightTable and Bret Victor style. I've also been exploring TEALsim, used by MIT for their EM simulations. So another goal is to get the GA working with an EM simulator...