Background: I'm embarking on a long overdue revamp of an old environmental data management system I made 12 years ago that's used by local watershed science organizations to manage their science data. Ever since I built the first version, I've been thinking about how to improve it, but life and other projects have kept me preoccupied, and it's mainly a volunteer effort.

It has a normalized MySQL DB with a Flex frontend that mutates a tree of relational data objects created by Hibernate and sent across the wire via GraniteDS.

A few years ago when I saw Paul deGrandis' Data-Driven talk I realized Clojure would be a good fit for a rewrite. Even more years ago, I considered using XML, XML Schema, and XForms. With XForms 2.0 recently released, I can't say that would be a bad way to go... Declarative UI for the win, but "XML is messy Lisp!" and why not just use the most useful real modern Lisp -> Clojure?

The intention is to migrate the database to Datomic and keep the same basic normalized table schema with a SPA web frontend that reads and transacts a tree of relational data across the wire. Clojure(script), Datomic, EDN, and EQL are perfect for this. Datomic's pull query and tree-like transaction format makes it easy to move a whole tree of relational entities both ways across the wire relatively painlessly. MySQL + Hibernate + GraniteDS were amazing for this 12 years ago, but I like the Datomic + EQL approach more now, mainly because it's a more pure data + code approach. Clojure is great because it's the most simple and stable language I'm aware of, the same language can be used for both client and server, and EDN/Transit is a desirable data format for the wire when talking to a browser.

Progress: I wrote a library that migrated the schema and data over to Datomic from MySQL (thosmos/mysql-to-datomic). This maps tables to namespaces, and columns to attributes in that namespace. One thing I needed was a higher level notion of foreign-keys from one entity type to another, which Datomic doesn't have, as well as other SQLisms like string length for VarChar columns, etc. So I created a simple domain spec that describes this (thosmos/domain-spec), which is then used for generating the Datomic schema, Lacinia GraphQL schema, and can contain DB- or GraphQL-specific things, etc. I'm just using Datomic, so I haven't bothered with other DBs.

This GraphQL backend is already live on our 1st draft production frontend at RiverDB.org and our dev frontend. The specs are used to generate the GraphQL endpoints for all of the tables using only one resolver.

Here's an example of one domain entity from the domain spec:

{:db/id "toxtest",
 :entity/name "toxtest",
 :entity/ns :entity.ns/toxtest,
 :entity/attrs
   [{:attr/name "ComplianceCode",
     :attr/key :toxtest/ComplianceCode,
     :attr/position 8,
     :attr/cardinality :one,
     :attr/type :ref,
     :attr/ref {:entity/ns :entity.ns/compliancelevellookup},
     :attr/strlen 5}
    {:attr/name "LabSampleID",
     :attr/key :toxtest/LabSampleID,
     :attr/position 4,
     :attr/cardinality :one,
     :attr/type :string,
     :attr/strlen 20}]}

Goals:

1) Generate a basic CRUD UI from the domain specs such that the domain data can be easily edited. Generating the UI from the spec enables using the same UI app for multiple different schema shapes, or to more easily migrate the schema without having to touch the UI app. In other words: a spec-driven UI.

I currently have a dev marmelab/react-admin CRUD UI running that's generated from the spec and that provides a good default editing experience for each individual "table" or entity type. The next step is to create some custom UIs that combine a graph of related entity types into a single form. This is where I'm wishing I could be doing this in CLJS instead.

1.5) It would be possible to have some simple schema migration tooling that would provide functions for doing the most common migrations (add an attribute, add an entity type, alias a key, add but not change a join reference, etc) and then would change both the spec and the live DB schema.

2) Because the spec is just data and is itself defined using its own spec format, it's possible to use the same spec-driven CRUD UI to edit it using the same means as the domain data editor. This will be a "schema editor" that enables the user to: add an attribute to an entity type; change an attribute's validation function; add a new entity type; add a new entity reference attribute to an existing entity type, etc. These types of changes result in schema migrations under the hood, using the same tooling as described in (1.5) above. This approach enables adding new fields, and aliasing existing fields, but not renaming or removing fields, since this data might be referenced from elsewhere in the distributed system that is the internet. Also, since we're dealing with historic environmental data, we want to never lose anything, but only track changes and modifications.

This self-editing of the schema is accomplished by using the same spec-driven UI CRUD editor because the schema specs are self-defined using the same spec attrs as domain specs (very meta):

[{:entity/ns    :entity.ns/entity,
  :entity/name  "Entity"
  :entity/pks   [:entity/ns]
  :entity/doc   "The entity spec for the entity spec -> so meta"
  :entity/attrs [{:attr/key         :entity/ns,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/identity   true,
                  :attr/doc         "unique keyword for this spec like -> :entity.ns/sitevisit"}
                 {:attr/key         :entity/name,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "A display name"}
                 {:attr/key         :entity/doc,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "documentation about this entity"}
                 {:attr/key         :entity/prnFn,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "a function with the entity as its arg for printing this entity"}
                 {:attr/key         :entity/summaryKeys,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "a subset of this entity's attrs to display in a summary list"}
                 {:attr/key         :entity/pks,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "list of attribute keywords that collectively act as a unique constraint for this entity"}
                 {:attr/key         :entity/attrs,
                  :attr/type        :ref,
                  :attr/ref         {:entity/ns :entity.ns/attr}
                  :attr/cardinality :many,
                  :attr/component  true,
                  :attr/doc         "attribute specs for this entity"}]}
 {:entity/ns    :entity.ns/attr,
  :entity/name  "Attribute"
  :entity/pks   [:attr/key]
  :entity/doc   "The entity spec for the attribute spec -> very meta"
  :entity/attrs [{:attr/key         :attr/key,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/identity   true,
                  :attr/doc         "the datomic attribute keyword name -> :<entity-ns>/<attr-key> -> :sitevisit/SiteVisitDate"}
                 {:attr/key         :attr/type,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/doc         "#{:keyword :string :boolean :long :bigint :float :double :bigdec :ref :instant :uuid :uri :bytes}"}
                 {:attr/key         :attr/cardinality,
                  :attr/type        :keyword,
                  :attr/cardinality :one,
                  :attr/doc         "#{:one :many}"}
                 {:attr/key         :attr/name,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "a display name"}
                 {:attr/key         :attr/ref,
                  :attr/type        :ref,
                  :attr/cardinality :one,
                  :attr/doc         "a DB ref to the entity spec for the target type of this ref attribute, i.e. Foreign Key"}
                 {:attr/key         :attr/identity,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this an identity attribute?"}
                 {:attr/key         :attr/unique,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a unique attribute?"}
                 {:attr/key         :attr/primary,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a primary key?"}
                 {:attr/key         :attr/required,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this attribute required?"}
                 {:attr/key         :attr/component,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this a component attribute?"}
                 {:attr/key         :attr/noHistory,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "skip keeping history in datomic?"}
                 {:attr/key         :attr/fulltext,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "index full text?"}
                 {:attr/key         :attr/derived,
                  :attr/type        :boolean,
                  :attr/cardinality :one,
                  :attr/doc         "is this attribute a derived value?"}
                 {:attr/key         :attr/deriveFn,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "fn code for deriving this attr value"}
                 {:attr/key         :attr/deriveAttrs,
                  :attr/type        :keyword,
                  :attr/cardinality :many,
                  :attr/doc         "attrs of the parent entity that are used to derive this attr value"}
                 {:attr/key         :attr/doc,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "this attribute's doc string"}
                 {:attr/key         :attr/decimals,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "number of decimals in this floating point value"}
                 {:attr/key         :attr/strlen,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "number of chars in this string value"}
                 {:attr/key         :attr/position,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/doc         "ordinal position - optional for visual ordering"}
                 {:attr/key         :attr/nextAutoVal,
                  :attr/type        :long,
                  :attr/cardinality :one,
                  :attr/noHistory   true,
                  :attr/doc         "next auto-increment value. optional"}
                 {:attr/key         :attr/gql,
                  :attr/type        :string,
                  :attr/cardinality :one,
                  :attr/doc         "A GraphQL-friendly attribute name"}]}]

3) An obvious further meta meta option would be to enable editing the spec's specs! This would for example enable adding a new field to the Attr spec, which would cause it to show up in the schema editor for all schema attributes. This is probably too meta to even mention, but it's possible using the same spec-driven UI!