[OPEN-ILS-DEV] Thoughts about the Staff Client Part 1 of 4 - TT2 and AngularJS

[Bill Erickson]

On Fri, Jan 24, 2014 at 12:58 PM, Liam Whalen <liam.whalen at bc.libraries.coop
> wrote:

> On Jan 24, 2014, at 6:18 AM, Bill Erickson <berick at esilibrary.com> wrote:
>
>
>
> On Thu, Jan 23, 2014 at 1:49 PM, Liam Whalen <
> liam.whalen at bc.libraries.coop> wrote:
>
>> On Jan 23, 2014, at 9:19 AM, Bill Erickson <berick at esilibrary.com> wrote:
>>
>
> [snip] ...
>
>
>
>>
>>
>>> I am concerned that combining the use of TT2 with AngularJS adds
>>> complexity to the staff client design that will inhibit participation with
>>> staff client development, adds a layer of complexity to the MVC desgin, and
>>> could end up making the app incompatible with later versions of the
>>> AnuglarJS framework.
>>>
>>
>> Using TT does add a layer of indirection.  Personally, I detest
>> complexity.  I would rather have fewer features than a system that cannot
>> be maintained.  In light of this, I still chose to use TT, because it
>> solves more problems that it creates.
>>
>>
>> I would say it solves more problems from the context of our current code
>> base, but we have the chance to rebuild and redesign.  Assuming something
>> can be done about templates and i18l support, I think using a pure
>> AngularJS design will simplify the code base that developers have to work
>> with.  A lot of AngularJS's benefits come from the use of conventions.  A
>> standard directory structure, standard naming conventions, and standard
>> design patterns are some examples of this.  By moving away from this we
>> loose these benefits, which is fine from our stand point because we work
>> with this code all the time, but if I was a pure AngualrJS developer, I
>> would be very frustrated while trying to learn the differences.   The use
>> of conventions is designed to allow developers to come up to speed on a
>> project very quickly.  Without those conventions, some of AngularJS's
>> benefits are lost.
>>
>
> Just to clarify, I'm not suggesting we leverage TT because it's there, I
> think we should use it because it's a good idea.
>
> Standard naming conventions and Angular design patterns are all in play
> here.  The only difference is how the files are generated.  For the
> directory structure, every coder with a blog claims to have designed the
> best way to structure their Angular apps.  Is there a One True Way now?
>  How many Angular apps are out there that didn't get that message in time?
>  Are they doomed to failure?
>
>
> The best practices suggests having a single app folder with the directory
> structure reflecting module structure.  In my opinion, the point of using a
> framework is to use that framework as the One True Way of doing things.  If
> I pick and choose from a framework as it pleases me, then I am working
> against the design of the framework.  The creators of the framework build
> all its pieces, so they fit together.  By excluding some pieces when I use
> the framework, I start to loose the cohesion of the framework.  If at a
> later point, the maintainers of the framework decide that a part I decided
> to ignore is necessary for a new feature, then my use of the framework
> needs to be refactored to include the part I left out if I want to use the
> new feature.
>


Agree to disagree that the location of the files on the server is an
integral part of the framework.



>
>
>
>>
>> What's more, the way we use TT in the staff interfaces is not full-blown
>> Template Toolkit.  We don't fetch data, we don't use loops or data
>> structures.  99% of the TT syntax is INCLUDE and [% l('translateable
>> string') %].  If those are too complicated, then Angular will blow a
>> developer's mind.
>>
>>
>> I have seen a loop last night.
>>
>> src/templates/staff/circ/checkin/t_checking_table.tt2 contains the
>> following code.
>>
>> *<*div* ng-init=*"
>> checkins.setColumns([
>> [%- FOR *col* IN *COLUMNS* %]
>> {label:'[% *col*.*label* %]',name:'[% *col*.*name* %]'[% IF *col*.
>> *display* %],display:true[% END %]}[% IF !*loop*.*last*; ','; END -%]
>> [% END %]
>> ])"*>*
>> *</*div*>*
>>
>>
> Ah, yes, you got me there. I felt liberated to experiment pretty heavily
> in the prototype as I was adding extra (unplanned) features.  This is from
> my column-picker experiment.  It's ugly and needs to be burned with fire.
>
>
> There was no intention of a gotcha.  But being able to do things like this
> is one of the concerns I have with mixing TT2 and AngularJS.  The only way
> to control things like this is with code policy, which requires diligent
> code review.  If new developers come along and build some things using TT2
> functionality that we do not accept in our policy because they are unaware
> of it, then it will have to be rewritten (if it is caught) and this would
> be a frustrating experience.
>

>
>
>
>>  This is using TT2 to populate the parameters for an AngularJS function.
>>  This level of abstraction is confusing to me.
>>
>> As well, TT2 variables are used to define  the values for ng-app.
>>
>> [%- IF ctx.page_app %] *ng-app*="[% ctx.page_app %]"[% END -%]
>>
>>
> This, to me, is a perfectly reasonable example of avoiding repetition.
>  When a new app is started, set the app value, load the wrapper template,
> and now I don't have to copy/paste 20 lines of boilerplate from another
> html file.  You're going to have a difficult time convincing me this is a
> bad thing.
>
>
> I am suggesting we use ng-include to handle boilerplate instead of TT2.
>  If I understand some of your concerns, the overhead of network traffic
> required to do this with ng-include is a problem.  Which makes me question
> if we should be using AngularJS if we cannot reliably use its template
> features because they are too demanding on network connections.  But, we
> should be able to use AngularJS' template cacheing features to reduce
> network traffic.
>

For loading templates as-needed, ng-include is great.  However, as each
page is segregated into modular chunks (for code re-use), many of which
need to be fetched for the page to function at all, it could become a
problem, depending on the number of chunks.  This is not a limitation of
Angular.  There are only two ways to acquire a template in the client:
within the page or as an external fetch.  Angular is perfectly happy to let
us do either.

If we learned anything from the JSPAC, it's that every network call,
particularly on page load, should be very carefully considered.  If there
are ways to reduce the number of calls, particularly without affecting
functionality, then we should take advantage of those opportunities.



>
>
>
>>  This means, conceptually, the program is broken up into a number of
>> separate AngularJS applications.
>>
>
> Most definitely.
>
>
>>
>> But practically, because everything is tied to the same abstracted
>> functions these separate applications are tightly coupled to each other,
>> which means modifying something like
>> web/js/ui/defaults/staff/services/list.js will modify how every application
>> is used.
>>
>
> On the contrary, they are not tightly coupled.  There is a one-way
> dependency, like any modular structure.  Any app that uses lists.js relies
> on it and must change when/if list.js changes.  (After it settles in, it
> should stop changing).  But the services themselves have no knowledge of
> the apps.
>
>
> I think there should be as few dependancies, one-way or any other form,
> between modules.  If there is a one way dependency between two modules and
> a service, then when I change that service for one module I change it for
> the other, which means those two modules are actually dependent on each
> other via the service.  It seems like they are not because the service is
> separate, but if I cannot change the service without changing both modules
> then the modules are dependant on each other.
>

I think we're talking past each other here.  A service publishes an API.
 As with any API, if it changes, modules depending on that API will have to
change, too.  This does not mean the modules themselves depend on each
other.  That would be like saying the OPAC depends on the self-check
interface, because they both use the same server API call for renewing
items.  I don't *think* that's what you're saying.


>
> Programatically, I can create branches within a service and add logic to
> make it appear as if changing the service only changes one of the modules
> that depend on it, but this adds complexity.  The other approach is to make
> the functionality in each module conform to the service, but in this case
> as well, the modules are now tightly coupled.  Granted it is not possible
> to have everything exist in complete isolation, but building dependencies
> between two separate modules should be carefully examined before it is
> done.
>

> FWIW, better examples of services to review would be the net.js, idl.js,
> pcrud.js and others.  list.js is part of my nascent column-picker
> experimentation.
>
>
>
> I will take a look at the other services you mention.
>
>
>
>> And this is the point of something like list.js.  But, once you get a new
>> View that cannot accommodate the current functionality in list.js then you
>> need to expand the logic and branches in list.js, which results in an
>> overly complex module.  My example of list.js may not be the best in this
>> case because I have not yet reviewed the code.  But, my point is by
>> coupling everything together it increases complexity and decrease our
>> flexibility to modify particular aspects of the staff client without
>> modifying others unless we want to make overly complex code to handle the
>> different cases.
>>
>> From what I have read of AngularJS, we should have a single ng-app and
>> tie the various staff client functionality together via separate AngularJS
>> modules.
>>
>
>  Yes, that is true up to a point.  After much deliberation and
> experimentation, I opted to break from that particular mold.  Cramming
> every UI from the staff client into a single Angular app would create a
> memory gobbling monster of an application.
>
>
> AngularJS only keeps things in the current ng-view and ng-includes active.
>
> The following link is a section of the best practices video with comments
> on performance and how ng-view and ng-include are used to keep only the
> current view active in the application.  The section on performance is 2
> minutes and 30 seconds in length.
>
> https://www.youtube.com/watch?v=ZhfUv0spHCY#t=49m00s
>
> As for using too much memory, if we cannot use AngularJS as it is
> recommended because it uses too much memory, then this is a reason not to
> use AngularJS.  What amount of memory usage were you seeing before you
> moved towards multiple ng-app tags?
>

It never occurred to me that it would be viable nor did I see the point.

Even if the runtime memory for a controller is released (the $scope, the
DOM, etc.), the code to execute every controller which has been fetched
from the server thus far still lives in a parsed state within the browser.
 Since we are not changing browser pages, the browser cannot simply toss
that information aside.  Am I crazy?

It has also been my experience that DHTML, even with the finest coders and
leanest browsers, is bound to leak some memory somewhere, often beyond the
control of the developer.  A single web page that supports >100 top-level
interfaces seems like a recipe for trouble.


>
>  Even if you offload most of the page-specific controller code (which
> would require some planning), making the client download the full list of
> routes for every staff UI just to get to the login page is unwise and
> completely unnecessary.  Then, once you start using the application, as
> each page loaded, controllers are instantiated, which adds to memory
> consumption and the memory would not clear until you manually reloaded the
> page or closed the tab.
>
> My solution was to collect logically grouped application features into
> apps.  The use of HTML pushstate routing for building links between
> different pages means navigating between apps is seamless.  When a user
> navigates to another app, the memory from the previous app is released.  So
> far, it's worked like a charm.  If there are specific concerns, I'd love to
> hear them.
>
>
> Google is working on adding lazy-loading of modules in the 2.0 version of
> AngularJS.  This will allow AngularJS apps to load modules as they are
> needed rather than all in one big shot.  I doubt we will be using the
> current version of the AngularJS libraries by the time we are ready to
> release this new staff client, so we need to be aware of what our options
> will be.
>

Glad to hear it.


>
> The suggested practice is to organize modules by view/function rather than
> by service.  Something should not be a module if it cannot be instantiated
> by itself and be used.  For instance, having a service as a module is not
> recommended because it is unlikely that the service can be instantiated by
> itself with no other modules and be useful without any other services,
> directives, or controllers.
>

Yep, totally agree.



>
> The link below is the specific portion of the AngularJS best practices
> videos that speaks about organizing modules.  The portion on modules is 3
> minutes and 20 seconds in length.
>
> https://www.youtube.com/watch?v=ZhfUv0spHCY#t=34m20s
>
>
>> Things like the use of server defined ng-app values diverge drastically
>> from what I have seen in Google's examples.
>>
>
> Well, sure, they don't use TT, so the HTML is going to look different.
>  The end product is the same thing you see in Google examples, though.
>
> This (in Chrome) should look about the same:  view-source:
> https://bill-dev2.esilibrary.com/eg/staff/circ/patron/search
>
> We have an app, some controllers, some Angular template variables, etc.
>
>
>
> I do not think the advantages of AngularJS are in its run time
> compilation.  The advantages are in using html files as code.  Yes, the app
> is an AngularJS app by the time it gets to the browsers.  But, the code
> base is not an AngularJS code base.  We can factor out as much TT2 logic as
> possible, but even the TT2 includes add a layer of obfuscation on top of
> the AngularJS app.
>

Agree to disagree on this one.



>
> I am pushing conformity with Google's practices because it gives us the
>> best chance of staying compatible with later versions of AngularJS.
>>
>
> Me too!  Which is why the HTML we produce via TT should conform to Angular
> best practices.
>
> I realize the case of ng-app being defined on the server is not an issue,
>> at least while we are dealing purely with client side AngularJS, but
>> AngularJS is changing rapidly, and I would not depend on that always being
>> the case.
>>
>
>> Finally, Evergreen UI developers that don't understand the basics (and
>> just the basics) of TT will be a rare bread, since we use it extensively
>> (TPAC, other staff UIs, Action/Trigger templates).
>>
>>
>> I am more concerned with expanding the number of developers we have than
>> with our current set of developers being unfamiliar with TT2.
>>
>
> These developers will have to navigate a sea of new technology with
> Evergreen and OpenSRF.  JSON-query, PCRUD, the IDL, how the tables of the
> database work together, managing transactions, and a whole host of APIs
> that return a wide array of constructs and data types.   If they've made it
> that far, TT will be a welcome bit of simplicity.
>
> There is no scenario where a developer joins the community and only
> understands Angular and I think it's unwise to set that expectation.
>
>
> I think there could be a number of scenarios in which someone joins the
> community to only work on the AngularJS app.  If the MVC is organized
> properly, someone should be able to modify the structure of a view without
> ever touching the model.  So, in a pure AngularJS app, people who
> understand only AngularJS could reorganize the layout of the staff client
> without ever having to know about anything on the server side.
>

Agree to disagree here, as well.  I don't think organizing files via TT
adds that much burden for someone wishing to edit the HTML.



>
> Pointing to the already complex nature of Evergreen as a reason to keep
> things complex is going to create a state of perpetual complexity.
>

>
>>
>>
>>>
>>> By using TT2 with AngularJS, we are no longer using the AngularJS
>>> framework, we are now creating a hybrid system. In essence, we are rolling
>>> our own staff client by using multiple processes.  For developers outside
>>> of the Evergreen community who like using AngularJS, having to learn TT2
>>> could result in those developers losing interest in assisting with the
>>> staff client.
>>>
>>
>> I see where you're coming from here and I share this concern, but in our
>> case, it's simply not true. We are using two frameworks within two distinct
>> problem domains.
>>
>> Domain 1 is delivering HTML to the client.  Domain 2 is having the client
>> generate rich content from the base HTML.  TT takes templates as input and
>> produces HTML as output.  Angular takes HTML as input and produces rich
>> content in the browser as output.  There is no cross-over here.  The fact
>> that Angular receives HTML which was generated by some mysterious
>> server-side process is wholly irrelevant to Angular.  The HTML could have
>> come from a thousand monkeys behind a thousand typewriters (sorry, going
>> for humor), so long as the final product was an HTML page Angular could
>> drive.
>>
>>
>> Using AngularJS's javascript to run the staff client code is not the
>> biggest benefit from adopting AngularJS.  AngularJS should provide us with
>> a way to rapidly develop application features.  It is the code base that
>> benefits from AngularJS' use.  In many respects, the need for the
>> javascript library to parse the AngularJS syntax is a side effect of
>> wanting to use AngularJS in our code base.  By mixing TT2 and AngularJS we
>> loose things like being able to debug completely within a web browser.  The
>> final client side code can be debugged, but an error may result from the
>> server side interaction, which requires developers to maintain two levels
>> of abstraction when working with the code.
>>
>
>> As well, if I find an error in the Angular app.  I then have to got back
>> and work with the combination of TT2 and AngularJS instead of being able to
>> dive right into the Angular code where the error was detected.
>>
>
> That's not true.  The error occurs in the client, with a line number,
> referring to an HTML file, sitting in the browser, that has no TT syntax
> within in.
>
>
> I have not worked with your code yet, so forgive me if I am completely
> misunderstanding how the debugging process works.  But, I do not see how
> the line numbers returned in the browser could refer to a line number  in
> the code base because the files in the browser do not actually exist on the
> server.  They are an amalgamation of TT2 files.  This is what I mean when I
> say I would have to work with both TT2 and AngularJS.
>

I left out a step...  Line numbers coming from the browser console would
refer to locations within JS files, which do match the files in the
repository (when they are not minfied, of course).

Even if they did refer the HTML line numbers, they would not match line
numbers for the files in the repository in a non-TT world, because the
final document would be an amalgamation of ng-included files.




>
> If have a bug in my AngularJS app, it gives me a line number.  I then have
> to walk through the TT2 layout to determine where the bug actually resides
> in the code.  Or, if I want to change a specific portion of the app, then I
> cannot identify where in the code base that portion lives by looking at the
> code the app is using on the browser side.
>

As far as finding the problem HTML, how is this any different than finding
the correct file which is loaded via ng-include?  There will never be one
HTML file that defines the whole page.


>
>

> Yes, if your TT syntax is bad, it will produce errors on the server.  This
> file will never get to the browser and Angular / JS will never be involved.
>  There is no situation where you will be confused about whether the problem
> is with Angular or TT.
>
>
> Syntax errors in TT2 are only one possible kind of error.  If I have a
> logic error in my TT2 code, then the problem becomes much harder to locate.
>

Sure.


>
> ...
>
> [snip]
>
>
>>> Additionally, Google is talking about developing server side parsing of
>>> some AngularJS code.  While this will unlikely be mandatory for an
>>> AngularJS app in the near term, the benefits added by server side
>>> processing may be deemed important in new versions of the AngularJS
>>> framework.  In this case, a later version of the framework may require
>>> server side processing in order to use newer features.  Which would put the
>>> community in the position of either figuring out how to process something
>>> via TT2 then send it to server side AngularJS (and this may not be possible
>>> without an extreme effort), or we have to start limiting what we can use in
>>> the newer versions of the framework or are limited to using only older
>>> versions of the framework.
>>>
>>> I realize this is very much a what-if scenario, but I think the possible
>>> problems are important enough for it to be considered before large scale
>>> development combining TT2 and AngularJS commences.
>>>
>>
>>> In the case of content being created via TT2, I am less concerned
>>> because that can be refactored fairly easily and without the fear of
>>> breaking the application. I am thinking of things like localization and
>>> custom messages.
>>>
>>
>> Ironically, server-side JS-driven page generation is meant to solve some
>> of the same problems we have already solved by leveraging TT, namely
>> providing the ability to re-use templates without requiring the client to
>> fetch them as separate downloads.
>>
>> In any event, I'm not worried about it.  If we decide to go this route,
>> then presumably this process would live on the server, maybe as an Apache
>> filter, or some other standalone process.  In any case, we can ask Apache
>> first for the TT-generated HTML and then run that through the
>> server-side-angular generator, whatever form it takes, and then pass the
>> result off to the client.
>>
>>
>> I am not ready to make those presumptions.  We could end up in the same
>> position we are currently in with our inability to use the newest versions
>> of XUL.
>>
>
>
> Let me get this straight, if we don't hand-code every character of every
> HTML file and put every file in some specific location, we risk the same
> fate as, for example, XULRunner deprecating remote XUL?  If this is true,
> then we should not use Angular.  No one should.
>
>
> The idea that perhaps no one should be using AngularJS is a concern I
> have.  It is unproven in a large app like the staff client.  I am not
> saying that no one should be using AngularJS.  I am saying we need to be
> cautious.
>

Another reason I opted not to build it as one monolithic application.  I
was also unable to find similar examples of such large applications in
Angular.  Breaking the the staff client into smaller apps, which happen to
share services and some HTML boilerplate, means our app looks a lot more
(size wise) than most of the other Angular stuff I've seen.


>
> [snip]
>
>
>>> In the end, by combining TT2 and AngularJS we are no longer using a
>>> framework.  At this point we are rolling our own solution via the
>>> combination of two separate systesm.  I am going to touch on Google Best
>>> practices in another email, but in general, the advice I hear from the
>>> AngularJS team communications is develop everything in AngularJS to the
>>> fullest extent possible.
>>>
>>
>> I see this as a warning against mixing Angular with other JS frameworks,
>> a sentiment a heartily agree with.
>>
>> Also, let's be clear that Angular is not magical.  It's just another JS
>> framework, one that I happen to like a lot.  The people that wrote it are
>> smart and they have a lot of wisdom to share, which we should soak up, but
>> they are not here, now, working on Evergreen and do not have to live with
>> these decisions.  Let's learn from them, but take our own paths where it
>> makes sense to.
>>
>>
>> I disagree about taking our own path.  I think we are best served by
>> using what Google's suggestions as close to 100% of the time as possible.
>>
>
> I don't discard their recommendations lightly.  I tried many times to
> follow the recommended patterns, and in most ways I do.  I started out not
> using TT, but at some point, the benefits were too great.  In each case
> where I do leave the path, there is some precipitating factor, like the
> knowledge that doing XYZ is going to slow everything down or require that
> we write twice as much code.
>
>
> In what specific ways is TT2 necessary to improve the application?  I was
> going under the assumption that TT2 was necessary because we needed local
> customizations and i18l support.  I am not sure we need to use TT2 for
> either of those, but if TT2 is necessary to overcome shortcomings in
> AngularJS, then I do not think we should be using AngularJS to build the
> staff client.
>

TT is not needed for i18n.  We could use a 3rd-party service for
lazy-loading, etc., like we do with Dojo's i18n.  However, using TT means
fewer network calls, hence a faster app.

TT is not needed for templating.  We could do it all with ng-include.
 However, doing (some of) it with TT, means fewer network calls, hence a
faster app.

TT is needed, pending a non-TT solution, for local, upgrade safe,
org-unit-specific template overlays.

-b

-- 
Bill Erickson
| Senior Software Developer
| phone: 877-OPEN-ILS (673-6457)
| email: berick at esilibrary.com
| web: http://esilibrary.com
| Equinox Software, Inc. / The Open Source Experts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libmail.georgialibraries.org/pipermail/open-ils-dev/attachments/20140127/b43ed5b6/attachment-0001.htm>