[OPEN-ILS-DEV] Thoughts about the Staff Client Part 1 of 4 - TT2 and AngularJS

[Liam Whalen]

On Jan 24, 2014, at 6:18 AM, Bill Erickson <berick at esilibrary.com> wrote:

> 
> 
> On Thu, Jan 23, 2014 at 1:49 PM, Liam Whalen <liam.whalen at bc.libraries.coop> wrote:
> On Jan 23, 2014, at 9:19 AM, Bill Erickson <berick at esilibrary.com> wrote:
> 
> [snip] ...
>  
>>  
>> I am concerned that combining the use of TT2 with AngularJS adds complexity to the staff client design that will inhibit participation with staff client development, adds a layer of complexity to the MVC desgin, and could end up making the app incompatible with later versions of the AnuglarJS framework.
>> 
>> Using TT does add a layer of indirection.  Personally, I detest complexity.  I would rather have fewer features than a system that cannot be maintained.  In light of this, I still chose to use TT, because it solves more problems that it creates.
>> 
> 
> I would say it solves more problems from the context of our current code base, but we have the chance to rebuild and redesign.  Assuming something can be done about templates and i18l support, I think using a pure AngularJS design will simplify the code base that developers have to work with.  A lot of AngularJS's benefits come from the use of conventions.  A standard directory structure, standard naming conventions, and standard design patterns are some examples of this.  By moving away from this we loose these benefits, which is fine from our stand point because we work with this code all the time, but if I was a pure AngualrJS developer, I would be very frustrated while trying to learn the differences.   The use of conventions is designed to allow developers to come up to speed on a project very quickly.  Without those conventions, some of AngularJS's benefits are lost.
> 
> Just to clarify, I'm not suggesting we leverage TT because it's there, I think we should use it because it's a good idea.
> 
> Standard naming conventions and Angular design patterns are all in play here.  The only difference is how the files are generated.  For the directory structure, every coder with a blog claims to have designed the best way to structure their Angular apps.  Is there a One True Way now?  How many Angular apps are out there that didn't get that message in time?  Are they doomed to failure?  
> 

The best practices suggests having a single app folder with the directory structure reflecting module structure.  In my opinion, the point of using a framework is to use that framework as the One True Way of doing things.  If I pick and choose from a framework as it pleases me, then I am working against the design of the framework.  The creators of the framework build all its pieces, so they fit together.  By excluding some pieces when I use the framework, I start to loose the cohesion of the framework.  If at a later point, the maintainers of the framework decide that a part I decided to ignore is necessary for a new feature, then my use of the framework needs to be refactored to include the part I left out if I want to use the new feature.

>  
> 
>> What's more, the way we use TT in the staff interfaces is not full-blown Template Toolkit.  We don't fetch data, we don't use loops or data structures.  99% of the TT syntax is INCLUDE and [% l('translateable string') %].  If those are too complicated, then Angular will blow a developer's mind.
> 
> I have seen a loop last night.
> 
> src/templates/staff/circ/checkin/t_checking_table.tt2 contains the following code.
> 
> <div ng-init="
> checkins.setColumns([
> [%- FOR col IN COLUMNS %]
> {label:'[% col.label %]',name:'[% col.name %]'[% IF col.display %],display:true[% END %]}[% IF !loop.last; ','; END -%]
> [% END %]
> ])">
> </div>
> 
> 
> Ah, yes, you got me there. I felt liberated to experiment pretty heavily in the prototype as I was adding extra (unplanned) features.  This is from my column-picker experiment.  It's ugly and needs to be burned with fire.
> 

There was no intention of a gotcha.  But being able to do things like this is one of the concerns I have with mixing TT2 and AngularJS.  The only way to control things like this is with code policy, which requires diligent code review.  If new developers come along and build some things using TT2 functionality that we do not accept in our policy because they are unaware of it, then it will have to be rewritten (if it is caught) and this would be a frustrating experience.

> 
>  
> This is using TT2 to populate the parameters for an AngularJS function.  This level of abstraction is confusing to me.
> 
> As well, TT2 variables are used to define  the values for ng-app.
> 
> [%- IF ctx.page_app %] ng-app="[% ctx.page_app %]"[% END -%]
> 
> 
> This, to me, is a perfectly reasonable example of avoiding repetition.  When a new app is started, set the app value, load the wrapper template, and now I don't have to copy/paste 20 lines of boilerplate from another html file.  You're going to have a difficult time convincing me this is a bad thing.
> 

I am suggesting we use ng-include to handle boilerplate instead of TT2.  If I understand some of your concerns, the overhead of network traffic required to do this with ng-include is a problem.  Which makes me question if we should be using AngularJS if we cannot reliably use its template features because they are too demanding on network connections.  But, we should be able to use AngularJS' template cacheing features to reduce network traffic.

>  
> This means, conceptually, the program is broken up into a number of separate AngularJS applications.
> 
> Most definitely.  
>  
> 
> But practically, because everything is tied to the same abstracted functions these separate applications are tightly coupled to each other, which means modifying something like web/js/ui/defaults/staff/services/list.js will modify how every application is used.  
> 
> On the contrary, they are not tightly coupled.  There is a one-way dependency, like any modular structure.  Any app that uses lists.js relies on it and must change when/if list.js changes.  (After it settles in, it should stop changing).  But the services themselves have no knowledge of the apps.  
> 

I think there should be as few dependancies, one-way or any other form, between modules.  If there is a one way dependency between two modules and a service, then when I change that service for one module I change it for the other, which means those two modules are actually dependent on each other via the service.  It seems like they are not because the service is separate, but if I cannot change the service without changing both modules then the modules are dependant on each other.  

Programatically, I can create branches within a service and add logic to make it appear as if changing the service only changes one of the modules that depend on it, but this adds complexity.  The other approach is to make the functionality in each module conform to the service, but in this case as well, the modules are now tightly coupled.  Granted it is not possible to have everything exist in complete isolation, but building dependencies between two separate modules should be carefully examined before it is done.

> FWIW, better examples of services to review would be the net.js, idl.js, pcrud.js and others.  list.js is part of my nascent column-picker experimentation. 
>  

I will take a look at the other services you mention.

>  
> And this is the point of something like list.js.  But, once you get a new View that cannot accommodate the current functionality in list.js then you need to expand the logic and branches in list.js, which results in an overly complex module.  My example of list.js may not be the best in this case because I have not yet reviewed the code.  But, my point is by coupling everything together it increases complexity and decrease our flexibility to modify particular aspects of the staff client without modifying others unless we want to make overly complex code to handle the different cases.
> 
> From what I have read of AngularJS, we should have a single ng-app and tie the various staff client functionality together via separate AngularJS modules.
> 
> Yes, that is true up to a point.  After much deliberation and experimentation, I opted to break from that particular mold.  Cramming every UI from the staff client into a single Angular app would create a memory gobbling monster of an application.

AngularJS only keeps things in the current ng-view and ng-includes active.

The following link is a section of the best practices video with comments on performance and how ng-view and ng-include are used to keep only the current view active in the application.  The section on performance is 2 minutes and 30 seconds in length.

https://www.youtube.com/watch?v=ZhfUv0spHCY#t=49m00s

As for using too much memory, if we cannot use AngularJS as it is recommended because it uses too much memory, then this is a reason not to use AngularJS.  What amount of memory usage were you seeing before you moved towards multiple ng-app tags?

>  Even if you offload most of the page-specific controller code (which would require some planning), making the client download the full list of routes for every staff UI just to get to the login page is unwise and completely unnecessary.  Then, once you start using the application, as each page loaded, controllers are instantiated, which adds to memory consumption and the memory would not clear until you manually reloaded the page or closed the tab.
> 
> My solution was to collect logically grouped application features into apps.  The use of HTML pushstate routing for building links between different pages means navigating between apps is seamless.  When a user navigates to another app, the memory from the previous app is released.  So far, it's worked like a charm.  If there are specific concerns, I'd love to hear them.
> 

Google is working on adding lazy-loading of modules in the 2.0 version of AngularJS.  This will allow AngularJS apps to load modules as they are needed rather than all in one big shot.  I doubt we will be using the current version of the AngularJS libraries by the time we are ready to release this new staff client, so we need to be aware of what our options will be.

The suggested practice is to organize modules by view/function rather than by service.  Something should not be a module if it cannot be instantiated by itself and be used.  For instance, having a service as a module is not recommended because it is unlikely that the service can be instantiated by itself with no other modules and be useful without any other services, directives, or controllers.

The link below is the specific portion of the AngularJS best practices videos that speaks about organizing modules.  The portion on modules is 3 minutes and 20 seconds in length.

https://www.youtube.com/watch?v=ZhfUv0spHCY#t=34m20s

> 
> Things like the use of server defined ng-app values diverge drastically from what I have seen in Google's examples.  
> 
> Well, sure, they don't use TT, so the HTML is going to look different.  The end product is the same thing you see in Google examples, though.  
> 
> This (in Chrome) should look about the same:  view-source:https://bill-dev2.esilibrary.com/eg/staff/circ/patron/search
> 
> We have an app, some controllers, some Angular template variables, etc.
>  

I do not think the advantages of AngularJS are in its run time compilation.  The advantages are in using html files as code.  Yes, the app is an AngularJS app by the time it gets to the browsers.  But, the code base is not an AngularJS code base.  We can factor out as much TT2 logic as possible, but even the TT2 includes add a layer of obfuscation on top of the AngularJS app.

> I am pushing conformity with Google's practices because it gives us the best chance of staying compatible with later versions of AngularJS.
> 
> Me too!  Which is why the HTML we produce via TT should conform to Angular best practices.  
> 
> I realize the case of ng-app being defined on the server is not an issue, at least while we are dealing purely with client side AngularJS, but AngularJS is changing rapidly, and I would not depend on that always being the case.
>> 
>> Finally, Evergreen UI developers that don't understand the basics (and just the basics) of TT will be a rare bread, since we use it extensively (TPAC, other staff UIs, Action/Trigger templates).
>> 
> 
> I am more concerned with expanding the number of developers we have than with our current set of developers being unfamiliar with TT2.
> 
> These developers will have to navigate a sea of new technology with Evergreen and OpenSRF.  JSON-query, PCRUD, the IDL, how the tables of the database work together, managing transactions, and a whole host of APIs that return a wide array of constructs and data types.   If they've made it that far, TT will be a welcome bit of simplicity.  
> 
> There is no scenario where a developer joins the community and only understands Angular and I think it's unwise to set that expectation.    
> 

I think there could be a number of scenarios in which someone joins the community to only work on the AngularJS app.  If the MVC is organized properly, someone should be able to modify the structure of a view without ever touching the model.  So, in a pure AngularJS app, people who understand only AngularJS could reorganize the layout of the staff client without ever having to know about anything on the server side.

Pointing to the already complex nature of Evergreen as a reason to keep things complex is going to create a state of perpetual complexity.

> 
>>  
>> 
>> By using TT2 with AngularJS, we are no longer using the AngularJS framework, we are now creating a hybrid system. In essence, we are rolling our own staff client by using multiple processes.  For developers outside of the Evergreen community who like using AngularJS, having to learn TT2 could result in those developers losing interest in assisting with the staff client.
>> 
>> I see where you're coming from here and I share this concern, but in our case, it's simply not true. We are using two frameworks within two distinct problem domains.  
>> 
>> Domain 1 is delivering HTML to the client.  Domain 2 is having the client generate rich content from the base HTML.  TT takes templates as input and produces HTML as output.  Angular takes HTML as input and produces rich content in the browser as output.  There is no cross-over here.  The fact that Angular receives HTML which was generated by some mysterious server-side process is wholly irrelevant to Angular.  The HTML could have come from a thousand monkeys behind a thousand typewriters (sorry, going for humor), so long as the final product was an HTML page Angular could drive.
>> 
> 
> Using AngularJS's javascript to run the staff client code is not the biggest benefit from adopting AngularJS.  AngularJS should provide us with a way to rapidly develop application features.  It is the code base that benefits from AngularJS' use.  In many respects, the need for the javascript library to parse the AngularJS syntax is a side effect of wanting to use AngularJS in our code base.  By mixing TT2 and AngularJS we loose things like being able to debug completely within a web browser.  The final client side code can be debugged, but an error may result from the server side interaction, which requires developers to maintain two levels of abstraction when working with the code. 
> 
> As well, if I find an error in the Angular app.  I then have to got back and work with the combination of TT2 and AngularJS instead of being able to dive right into the Angular code where the error was detected.
> 
> That's not true.  The error occurs in the client, with a line number, referring to an HTML file, sitting in the browser, that has no TT syntax within in.  
> 

I have not worked with your code yet, so forgive me if I am completely misunderstanding how the debugging process works.  But, I do not see how the line numbers returned in the browser could refer to a line number  in the code base because the files in the browser do not actually exist on the server.  They are an amalgamation of TT2 files.  This is what I mean when I say I would have to work with both TT2 and AngularJS.

If have a bug in my AngularJS app, it gives me a line number.  I then have to walk through the TT2 layout to determine where the bug actually resides in the code.  Or, if I want to change a specific portion of the app, then I cannot identify where in the code base that portion lives by looking at the code the app is using on the browser side.

> Yes, if your TT syntax is bad, it will produce errors on the server.  This file will never get to the browser and Angular / JS will never be involved.  There is no situation where you will be confused about whether the problem is with Angular or TT.  
> 

Syntax errors in TT2 are only one possible kind of error.  If I have a logic error in my TT2 code, then the problem becomes much harder to locate.

> ...
> [snip]
> 
>> 
>> Additionally, Google is talking about developing server side parsing of some AngularJS code.  While this will unlikely be mandatory for an AngularJS app in the near term, the benefits added by server side processing may be deemed important in new versions of the AngularJS framework.  In this case, a later version of the framework may require server side processing in order to use newer features.  Which would put the community in the position of either figuring out how to process something via TT2 then send it to server side AngularJS (and this may not be possible without an extreme effort), or we have to start limiting what we can use in the newer versions of the framework or are limited to using only older versions of the framework.
>> 
>> I realize this is very much a what-if scenario, but I think the possible problems are important enough for it to be considered before large scale development combining TT2 and AngularJS commences.
>> 
>> In the case of content being created via TT2, I am less concerned because that can be refactored fairly easily and without the fear of breaking the application. I am thinking of things like localization and custom messages.
>> 
>> Ironically, server-side JS-driven page generation is meant to solve some of the same problems we have already solved by leveraging TT, namely providing the ability to re-use templates without requiring the client to fetch them as separate downloads. 
>> 
>> In any event, I'm not worried about it.  If we decide to go this route, then presumably this process would live on the server, maybe as an Apache filter, or some other standalone process.  In any case, we can ask Apache first for the TT-generated HTML and then run that through the server-side-angular generator, whatever form it takes, and then pass the result off to the client.  
> 
> I am not ready to make those presumptions.  We could end up in the same position we are currently in with our inability to use the newest versions of XUL.
> 
> 
> Let me get this straight, if we don't hand-code every character of every HTML file and put every file in some specific location, we risk the same fate as, for example, XULRunner deprecating remote XUL?  If this is true, then we should not use Angular.  No one should.  
> 

The idea that perhaps no one should be using AngularJS is a concern I have.  It is unproven in a large app like the staff client.  I am not saying that no one should be using AngularJS.  I am saying we need to be cautious.

> [snip]
> 
>> 
>> In the end, by combining TT2 and AngularJS we are no longer using a framework.  At this point we are rolling our own solution via the combination of two separate systesm.  I am going to touch on Google Best practices in another email, but in general, the advice I hear from the AngularJS team communications is develop everything in AngularJS to the fullest extent possible.
>> 
>> I see this as a warning against mixing Angular with other JS frameworks, a sentiment a heartily agree with.
>> 
>> Also, let's be clear that Angular is not magical.  It's just another JS framework, one that I happen to like a lot.  The people that wrote it are smart and they have a lot of wisdom to share, which we should soak up, but they are not here, now, working on Evergreen and do not have to live with these decisions.  Let's learn from them, but take our own paths where it makes sense to.  
>> 
> 
> I disagree about taking our own path.  I think we are best served by using what Google's suggestions as close to 100% of the time as possible.
> 
> I don't discard their recommendations lightly.  I tried many times to follow the recommended patterns, and in most ways I do.  I started out not using TT, but at some point, the benefits were too great.  In each case where I do leave the path, there is some precipitating factor, like the knowledge that doing XYZ is going to slow everything down or require that we write twice as much code.
> 

In what specific ways is TT2 necessary to improve the application?  I was going under the assumption that TT2 was necessary because we needed local customizations and i18l support.  I am not sure we need to use TT2 for either of those, but if TT2 is necessary to overcome shortcomings in AngularJS, then I do not think we should be using AngularJS to build the staff client.  

Liam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20140124/0dfbf000/attachment-0001.htm>