If you can buy it, it’s a shtick. If you must grow it, it’s a system.
—Venkatesh Rao
Back in May 2017, Engaging and Christian Schools of Florida completed the redesign for the CSF web site. Since then we’ve not stopped developing — thanks to a collaborative relationship, reasonable business arrangement, ambitious and faithful client, and having implemented what I believe is the fundamental architecture of an organizational system.
The fundamental architecture of an organizational system
Yes, the fundamental architecture of an organizational system! [Update 2020 July 30: for the canonical knowledge on this topic, see The EnterpriseReady SaaS Feature Guides]. This being (along with the important niceties of reports and notifications) a data structure comprised of the various item content channels specifically required by the organization, alongside the following system content channels:
- Persons
- Organizations
- Positions
- Roles
- Dashboards
- Consoles
- Ports
Together these system channels comprise a browser-based application that models an organization closely enough to handle pretty much any of its business processes, whether managing them or actually performing them if they’re informational.
It’s set up like this:
- A Person can be given any number of Roles.
- A Role is a Position at an Organization, expressed by its own unique screen featuring one or more Dashboards.
- An Organization is a grouping of people (in our case an Organization is a member school or a team within a school.)
- A Position has a set of privileges enabling access to Dashboards, Consoles and Ports.
- A Dashboard is expressed as a differentiated area on a Role screen comprised of various Consoles. Each Dashboard corresponds to a user mission, ie, a grouping of activities that this particular organization undertakes.
- A Console is expressed as a listing of a subset of entries in a particular item channel, each entry having one or more action buttons linking to one or more Ports.
- A Port is expressed as a display of elements of a single entry in a particular item channel, sometimes having one or more submit buttons to save various changes, depending on the privileges that the person has for this entry based on his or her roles’ positions.
Privileges on a port are determined not only by the logged-in person’s position but also by the status of the specific entry in port; the person may be able to merely view the entry, or go beyond this and edit its contents, and/or even change its status.
On a role screen, consoles load minimized so that their container dashboards serve as aesthetic overviews. Then, when clicked and maximized, a console’s contents are revealed, varying from a single link to something very large scrolling offscreen.
Now to operate the system! A logged-in person:
- From any page on the site, opens the Member menu
- From within the Member menu, selects a Role
- On the subsequent Role Screen, views the default or last-opened Dashboard or selects a non-default Dashboard and views its minimized Consoles
- On the dashboard, maximizes a Console to view its entry listings
- Within the console, clicks an entry’s action button to link to a Port
- Within the port, works on the entry.
Changing an entry’s status forces it to appear in other consoles that may be available to other roles; persons at the organization with that other role can in turn work on the entry via a port. This shunting along of an entry by dint of changes to its status forms a process.
To grow the system, operators can add new:
- Item channels
- Dashboards, if the new functionality is best not grouped within existing user missions
- Consoles to list the new entries
- Ports for working on entries in the new item channels
- Privileges in positions (and perhaps new Positions) to grant access to the new Dashboards, Consoles and Ports.
For example at CSF, a person with a role containing the position of School Head at a particular organization will have a role screen featuring such dashboards as Manage School and Manage CSF Membership.
Better conditions ahead
One caveat regarding this system architecture is that the method for enabling privileges is primitive, based only on an entry’s status, whereas sometimes of course a privilege must be based on more complex conditions. For example, at CSF a Peer Review Team Leader can publish a new Peer Review entry, but only once the school under review has completed its Strand Compliance entries and at least two Focus Areas, which are conditions external to the Peer Review channel. While these extra conditions happen to be rare for this particular organization to date, nonetheless handling such external conditions are clearly necessary.
Currently the system derives such occasional complex privileges from within consoles, but they should of course be stored in the data; we plan to add a Conditions column to the Privileges grid field in the Positions channel.
Such complexity is handled nicely in more developed process management software, such as executable BPMN engines, but one possible advantage of our approach is that it forces every step throughout the process to be labelled, each being a status; this helps with discussion.
The new School Improvement Plan
At the beating heart of CSF is the 5-year accreditation process. Towards the end of each cycle a member school is visited by a peer review team; meanwhile throughout the cycle the school reports on progress made since the previous review.
CSF Director of Accreditation Susan Taylor has not however been entirely happy with the schools’ reporting; they have usually only turned to it during a cycle’s final few months, so that problems which could have been addressed much earlier have been missed. To harden this vital process we recently rolled up our digital sleeves and developed the new School Improvement Plan functionality.
The new SIP shepherds schools into reporting their progress in detail throughout each of the five years. It required two new item channels, Action Steps and Recommendation Responses; two new ports to add, view and edit these; and two new consoles, one for the school head, one for the CSF Director of Accreditation.
Each school must now respond in detail every year to each of the recommendations generated by the previous peer review. The SIP console provides an overview of the school’s entire five-year period and to provide structured access to the new ports. A Recommendation Response can be addressed by a Focus Area, which is a team-based improvement project comprised of a series of team-based Action Steps.
It’s the job of CSF’s Director of Accreditation to oversee member schools’ accreditation activities. The console entitled This Year’s Submitted SIPs provides an overview of how each of the schools is progressing, summarizing the number of recommendation responses and focus areas currently in each step of the process. This is also where the Director of Accreditation can see any submitted focus areas that are ready to be reviewed and accepted.
Live search
We’ve recently (okay, belatedly) added search functionality to the site, where results are filtered depending on whether the user is logged in.
As expected nowadays, results appear not on an entirely new page but somewhere onscreen on our current page; and they appear as we type our keywords, each result separated by a colored band labelling its section. The item’s illustration appears if it has one, and the search keyword is highlighted in its preview text if available.
We settled on locating the search form in the masthead just to the bottom right of the logo, which breaks the horizontal symmetry and is thereby an emphatic spot. This placing also redresses a visual imbalance whereby the stuff to the left of the masthead seemed slightly heavier than that to the right. Unlike the rest of the header and indeed any page, the search form’s background is black, introducing the sitewide style for all input.
The search is powered client-side by Andreas Lagerkvist’s tried-and-true jQuery Livesearch addon, server-side by the exemplary Low Search addon for ExpressionEngine. (Livesearch was in fact already in use for site administrators to conveniently select users for logging in as them, so reusing it was a no-brainer.)
More new functionality
SIP and search are just a couple of new features among many others.
We’ve added a Contact Members dashboard comprised of consoles listing staff members by position, eg, all Executive Committee Members, all School Heads, all MIP Coordinators, etc. Each console has, as well as a tabular list, a condensed list of emails, clicking on which copies the list to the clipboard for pasting into an email.
On the port where an MIP Coordinator reports the participants of an MIP Activity, we’ve enabled mass participations by checkbox instead of adding each person individually.
We’ve added RSS feeds for Job Opportunities and Education News & Legislative Updates.
We’ve added a series of consoles providing insight into member schools’ current and past student enrollment numbers, breaking these down into age groups and scholarship types; and a console providing a convenient overview of which member schools have not yet paid this year’s dues.
Gone mobile
Back in 2017 we noted that the site was not yet responsive so not yet optimized for mobile screens. That’s now done.
By far the biggest challenge here was the header, with its complex build and specific shape. The mobile screen is much too small to maintain the header’s form, but we wanted to somehow retain its spirit and functionality. Accomplishing this took some pondering.
The first step was to transpose both the written version of the organization’s name and the tagline to the top of the homepage. The organization’s name nonetheless remains in the header because it’s written within the logo, which is relatively larger on mobile.
We also placed the main services behind a hamburger menu and replaced the “About Us” menu opener with an “info” icon. This gave us four icons — Menu, Map, Info and Member — echoing the full-size header’s four main services.
To reduce the header’s height, the logo juts out below, superimposing itself on the page’s illustration; and with the search form appearing up top, the logo also superimposes on it, thus suggesting a floating badge.
The resulting header succeeds in maintaining not only all the links contained in the full-sized desktop version (except the nicety of the second map linking to a filtered list of schools only with job opportunities), but also its look and feel. Job done!
At least for logged-out visitors, most pages on the site are articles. On mobile, such pages ignore the bleeding/inline choice for the hero illustration; instead, the illustration always bleeds and precedes the title, blurb, dateline and body text. Further down within the article’s body, adjustments compensate for much narrower gutters — for example, bullet lists are indented instead of outdented.
Anything two-column on the site, such as the map of Florida alongside the listings of schools, is rendered single-column, usually by wrapping the two in a div with display: flexbox and flex-direction: column or flex-direction: column-reverse, which could not have been done in CSS before the advent of flexbox.
Dashboards, being a centered series of consoles of various widths, did not need to be tweaked much, their constituent consoles wrapping naturally based on browser width. Both the horizontal dashboard tab menu and the contents of consoles themselves scroll offscreen to the right on mobile, with consoles sporting left and right borders to visually delineate their horizontal beginnings and endings.
Finally, in the footer, texts are sticky-spaced to wrap appropriately no matter the screen width.
Flight to freedom
Most recently, after many years of running on ExpressionEngine v2 at HostingMatters, we migrated to Linode and upgraded to EE v3. The need had become urgent due to performance; as functionality and database size has grown, the site has slowed, and as and the organization has come to rely increasingly on the site, that slowness has been felt more acutely. We already had Pjax and intelligent caching in place; we now needed to upgrade from PHP 5 to 7. As Derek Jones, erstwhile CEO of EllisLab, wrote back in 2016, PHP 7 is twice as performant. And while the final versions of EE v2 do work with PHP 7, many of its addons do not — only their EE3 versions were updated — so that to run PHP 7 we finally had to bite the bullet and upgrade EE to v3.
For a while I’ve been lamenting the limited server access on the low-cost shared hosting account at HostMatters, but figured that having help at hand with HM tech support more than compensated for the handicap. Recently though a number of factors tipped the balance in favor of a more modern bare-metal solution such as a Linode.
First, HM has over the years migrated the account to different servers within its organization, and each time I’ve missed their notification. When the account gets copied to a new server, the old account remains active and the domain continues to point to it, only to stop working a month later when the old account is deactivated, and we are galvanized into pointing the domain name to the new server, discovering a week or so later that the data produced in the interim few weeks has disappeared, at which point I must graft that interim data onto the new database generated from a snapshot made weeks earlier. Out of carelessness I let this happen a number of times as HM moved accounts around, and after the last time decided that this wasteful exercise must not happen again.
Second, I’ve recently regained confidence in dealing directly with a server, having set up Engaging’s new tech stack of Strapi and Nuxt hosted on Nginx with PM2, and felt less the need for the security blanket of HM’s support.
And third, to actually upgrade EE from v2 to v3 (the current version is v5), I would need full access to the server anyway; without it, any time I needed to install some tooling or reset a file permission I would need to submit a support ticket. So we took the plunge.
The steps were as follows:
- On the Linode, clone the existing HM site as Dev and get it up and running on Apache
- Clone Dev as Stag
- Upgrade Stag to EE3, replacing any addons abandoned after EE’s halcyon v2 with other solutions until Stag runs on EE3
- Clone Stag as Prod
- Freeze the live site
- Graft the live site’s content onto Prod’s database.
- Change the domain name’s DNS from HM to Linode
This way we had the safety net of a working v3 in Stag and a working v2 in Prod, as well as the ongoing previous live site over at HM. Thankfully, the data structure for content did not change between versions 2 and 3, and I was able to graft on the interim data by importing only the following ExpressionEngine database tables:
- exp_assets_files
- exp_category_posts
- exp_channel_titles
- exp_channel_data
- exp_channel_files
- exp_entry_versioning
- exp_files
- exp_matrix_data
- exp_members
- exp_members_data
- exp_playa_relationships
- exp_relationships
What a pleasure to have full access to the server! And it turned out that there was nothing to worry about regarding support on Linode; sending a support ticket prompted a detailed, helpful and scarily enthusiastic response.
And what a performance jump! The Linode is itself faster and no longer at the mercy of other accounts sharing the same server; the software itself is faster; and we have total access to the server and control over installations such as for tooling. We decided to keep email handling over at HM; the Linode article on the topic argued that it’s a complex business preferably outsourced.
Inspired by the performance improvement, we went further and retooled the consoles and dashboards to load as separate files, thereby concurrently. As such, even such complex and content-rich role screens as that of the Director of Professional Development load progressively and complete within at most a handful of seconds.
The improved pageload speed let us disable Pjax when linking to ports. The forms that constitute Ports rely on EE’s Channel Forms feature, which automagically generates JavaScript code for fancier form fields, and these often caused errors when loaded via Pjax. To improve regular pageload times we moved the Google Map functionality to a separate file that only loads when called, and moved all JS and CSS files out of EE templates and into regular files so that the browser can cache them.
All that said, the update has undeniably led to some significant setbacks due to the loss of some important 3rd-party add-ons not upgraded for EEv3 compatibility. While it was a pity to lose Henshu’s field editing directly from the cpanel entry listing, the more significant losses are MX Notify Control and MX Title Control, which respectively enable notifications (remember I mentioned notifications as one of the elements of an org system) and the programmatic setting of an entry’s required Title field. I believe these functions should have been wrapped into ExpressionEngine itself but they were not (the only add-on EllisLab ever purchased was Safecracker, which became Channel Forms).
And there are many smaller issues, eg, the new Textile add-on for EE3 parses texts slightly differently from how EllisLab’s own original Textile plugin did it (it seems that a list must now start one linebreak after a paragraph, not two; annoying).
Update 2020 May 30: In the meantime we’ve developed a role-based notification system triggered whenever a form button is clicked on a port. That way anyone currently holding a particular position within a particular school can receive a particular email message whenever a particular change is made to a particular type of entry.
Southeast roadmap
Our upgrade and migration solved for the system’s top constraint/annoyance: performance. Meanwhile the new constraint is easily identifiable: reliability. It’s our dirty little secret that development on the CSF site has been, to put it delicately, too lean and agile. The fact is, we’ve consistently failed to anticipate the knock-on effects of changes and developments on any of the myriad processes and activities possible on the site, so that the system’s users — staff members at CSF’s member schools — have often born the brunt and been unwitting testers. This of course breeds frustration with the system. Automated testing, now a standard method in software development, would address this reliability issue. We’ll probably use Selenium.
More immediately, we’d like to upgrade to the latest version of ExpressionEngine, currently v5.3.0. A significant obstacle here is that a vital add-on, CausingEffect’s CE Cache, is only stated to be compatible up to EE v3.
And longer-term we’d like to swap out the site’s back-end to be an instance of a more generally applicable organizational system powered by Engaging’s new Node-based stack consisting of Strapi for the content and Nuxt for the presentation, the two connected via GraphQL.