Friday, October 30, 2015

an interesting java architecture

At work, a guy named Scott built up a kind of neat Java backend for an MVP Proof of Concept we were doing, a bunch of RESTful APIs called "Legion". This project has some interesting ideas and best practices developed from Scott's experience and a greenfield chance to do things right, so I thought I'd share some of it here.

Some of the stuff may be old hat to people who have been more in the Java loop than I have; my last big Java projects were when Generics and Annotations were still relative new and fresh.


At the highest level, Legion's job is to provide RESTful endpoints for UIs etc to use to make and edit advertising related things like Campaigns/Flights/etc. It uses Spring for endpoint wiring and dependency injection. It is very self-contained, and Spring Boot means it doesn't need an Apache container.

Scott uses Spring's recommend terms for its layers: parallel to but not exactly the same as MVC.

The layers are:

  • CONTROLLER (confusingly, this is most similar to the MVC "View")
  • SERVICE (akin to MVC "Controller")
  • REPOSITORY (akin to MVC "Model")

We'll get more into these layers later. One important note is: it might be tempting to access stuff in the Repository Layer directly in the controller, since so often the Service layer (like in CampaignService) looks like simplistic one-liner plumbing, but this should absolutely be avoided because if the Service layer provides critical transaction functionality via the @Transactional annotation.


Spring is great for dependency injection, which is great for stuff like unit testing and what not, and otherwise loosely coupling your various components. The modern preference is for lots of singleton classes (vs, say, lots of static classes).

When you see a function annotated @Autowired, its arguments are managed /injected by Spring, generally at boot.( In fact, Spring can autowire private variable members, but it's considered better to keep it to the function level.)

Spring's injection model seems to have been influenced by Guice, so a few places I'll mention the equivalent name Guice uses. (Scott preferred Spring because it's larger age/scope means stuff like Jersey connectors are made for it.)

If you're writing your own class, you can tell Spring you want it to manage it via @Component (or one of the "subclasses" of @Component) but you can also use the @Bean annotation on a function (@Inject in guice) to have Spring manage a singleton instance of an arbitrary class, the one returned by that function. (In general its the return type class name (not the variable instance name) that's important to get the right class to where it is needed.) The @Bean trick requires @Configuration in the containing class. (In general, when Spring does its massive scan at bootup, it wants to scan every class, not  every function, since that would be inefficient.)


Charmingly LegionApplication contains a good old fashioned public static void main() class.

There's a line commented out,
which is super-useful to put back in when it comes time to see what mybatis/ibatis is doing against the database.

Log4J is also setup here, and then it hands off to Spring:, args);


Controller classes are annotated with "@RestController" - which means they are a @Controller which means they are a @Component, which means they are a bean and can be managed by Spring. (Guice uses @Bean).


As previously mentioned, wrapping stuff as a transaction might be the most crucial thing being done at this layer. Spring does its @Transactional magic by generating subclasses (on a properly configured IDE like IntelliJ, the subclasses will show up highlighted differently in the stack trace) "Obviously", if you put a breakpoint in the middle a Transactional call, changes will not show up in the database until the call is completed.

Transactional calls are re-entrant, and so one transactional function can call another and the transaction moves up to the outer layer, so to speak.


Most of the meat of the MVP was here. If we were using Hibernate, this would be a trivial layer, but in Scott's experience Hibernate didn't scale very well, and often made upgrading extremely hard. iBATIS (or it's current flavor MyBatis) seems like a better bet (personally I like that it is rather transparent and lets you see the SQL sausage being made.

MyBatis looks a bit like a templating language, with tags providing flow control around the SQL query meat.

MyBatis queries can be done via annotations, but Scott prefers the XML approach, as the annotation syntax gets wonky (also since SQL is kind of its own little discipline, I think it makes sense to have it all gathered in one area. In Legion's case, it provides an @Bean SqlSessionFactoryBean where the location of the mappers are "hardwired" in -
bean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mappers/*.xml"));
Roughly speaking, all the mapper XML is lurped together; having different query groups in individual .xml files is just  for human reading convenience.

So in repository java code, you see stuff like
which refers to
 <select id="selectCampaign" parameterType="long" resultMap="campaignResult">
in the XML.

MyBatis query bodies uses #{fieldname} style insertions of parameters. These are context dependent; for a single typed parameter, the name is essentially ignored, POJO beans use field names and Maps use keys.

MyBatis then can build parts of queries using tags like <where> and <if>. It's actually super clever so if you had a clause like
            <if test="nameQuery != null">
                name LIKE #{nameQuery}
            <if test="idQuery != null">
                OR id = #{idQuery}
If nameQuery was null the query would still build correctly without the "OR" that would otherwise mess the syntax up.
In general #{} is escaped and ${} is unescaped. (Meta-stuff like column names can't be escaped, for instance)

In theory MyBatis can return complex datatypes, but our version was getting cranky about nested objects, and so there is sometimes when the Java code handles additional glomming of stuff.

Another note was our MyBatis config did stuff like
<setting name="mapUnderscoreToCamelCase" value="true"/>
to hand external_id = externalId, that kind of thing.

Sometimes the return value (set as resultMap) referred to campaignResult, which was defined earlier in the file. It helped juggle the campaign type and status foreign keys  so that the Java code could do a lookup - for certain long lived data (e.g. a list of countries: content that rarely changes) Scott wanted to avoid always the expenses of doing joins and of shipping extra data over the wire, so he made a LKPCachedRepository (LKP = lookup) that will keep an in-memory lookup table.


One cross-layer thing we looked at was exception; with the use of the @ResponseStatus annotation on "NoSuchThingException", the repository layer could throw an exception that indicated what kind of HTTP response code message should be sent at the the controller layer. In general, the plan would be to see if we could get the UI to make sense of the code and payload, and only fiddle further if necessary, since the @ResponseStatus defaults might very well be the "right thing" in this case.


So back to LegionApplication - we see 3 critical annotations:
@SpringBootApplication(exclude = DataSourceAutoConfiguration.class) //lets us specify our own damn datasource
@EnableTransactionManagement //lets us use @Transactional to great effect
@Import({SecurityConfig.class}) //turns on security

The @SpringBootApplication is of course implying @Configuration (i.e. having Spring manage the Beans), and @ComponentScan, where it can look for @Component annotations on all classes in the project.)

I would say looking at this helped me get my own head wrapped around Annotations; they're kind of funky in how they are almost like bits of source code that get preserved in the generated byte code to provide instructions that can be done at boot time. includes some values like spring.datasource.username... these are injected into LegionApplication via @Value (this would be @Named in Guice) - e.g.

perhaps the most amazing one Scott made was ${use.embedded.mysql} - if true, Legion will make its own internal SQL server, running create.sql and create-static.sql - this is a technique outlined in
and is super great for unit testing. (The one downside is because of lack of support my MySql for their mxi (which is the core of this technique) the embedded mysql instance is stuck at version of 5.5.9)


Scott made an AbstractIntegrationTest class that takes care of much of the boilerplate, so the subclasses can call the REST endpoints and check the results.
It has some cool annotations:
@RunWith(SpringJUnit4ClassRunner.class) //run with jUnit
@SpringApplicationConfiguration(classes = LegionApplication.class) //Here's the application we want to test
@WebIntegrationTest(value = {"server.port=0"}, randomPort = true) //actually boot this application on a random port, so we can run calls against it

Individual tests can describe if they clean up after themselves or if the system should do a tear down and rebuild via @DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)


Security was fairly minimal for the MVP, but still there is SimpleCORSFilter to help deal with the cross domain issue / connection with the front end.  Even though the base class is generically "Filter" they are all HTTP and so do a lot of stuff with HttpServletResponse and headers etc.

There is also a SecurityConfig class extending WebSecurityConfigurerAdapter. Its configureGlobal() function set itself as the UserDetailsService(), and there's a shell implementation of loadUserByUsername... it is making an instance of DaoAuthenticationProvider. This class also shows us using BCrypt for some basic username stuff. (Some of the Bcrypt stuff is setup in LegionApplication)

The main configure() block looked like this, and the comments are at least as good as my current understanding:
 protected void configure(HttpSecurity http) throws Exception {
 http.csrf().disable() //disable CSRF to allow POST requests to our endpoints
.headers().httpStrictTransportSecurity().disable() //disable HSTS headers so we don't override local http servers on 8080 for a year
.authorizeRequests() //begin specifying security configurations
.antMatchers("/login").authenticated() //any user can login, if they're in our system
.anyRequest().permitAll() //everything else is open to everyone
.and().httpBasic(); //and use http basicAuth
So antMatchers was the interesting bit, and at this point in configure() you can set these wildcard style things up to have specify certain user/role requirements

That was kind of it, with the additional mention of "DHC" being a great little tool in chrome for running Ajax stuff.

Monday, October 26, 2015

something of a jquery cookbook...

Starting with the "Perl Cookbook", I've always liked the cookbook concept of simple "recipes". This page of jQuery Tips Everyone Should Know falls into that category.

Thursday, October 22, 2015

ember 101: pausing an acceptance test to peek at the DOM

Ember has a powerful built-in testing utility but its native mode is to zip through a test as quickly as possible and then close its little fake window... based on my experience with Robot, I know sometimes it's very useful to be able to "pause" a test, to take a look around the DOM's current state.

Anyway, there's a pauseTest() helper for just this event.

You may want to bump up the "test timeout value" in tests/test-helper.js:

QUnit.config.testTimeout = 120000; //two minutes in millis

Tuesday, October 20, 2015

ember 101: steps to add a search widget component

So, usual disclaimers apply: I'm still very new to Ember (especially since the tutorial I used set things up as POD rather than the more traditional MVC my project is leaning towards) and so everything there can be taken with a grain of salt; still I thought it might be useful for me to document these steps for reference for my future self or others, and maybe run what I did by more experience Emberites.

I was starting with a code base my colleagues made, that had a "campaigns" route and template. I used ember-cli to generate the basic parts:

$ ember g component campaign-search

Then I filled in the basic template in campaign-search.hbs: an input widget and a button to trigger the "search" action:

{{input placeholder="search" value=searchterms}}
<button {{action 'search'}}>search</button>

The campaign-search.js was pretty trivial:

import Ember from 'ember';

export default Ember.Component.extend({
var searchTerms = this.get("searchterms");

So here I'll take a second to point out I'm glossing over some thinking I had to do about how to structure this component relative to its parent (campiagns.hbs/capmaigns.js which is holding the model that the search would need to adjust.) The search component didn't modify the model directly; instead it did a "sendAction" to the parent. It's obvious in retrospect, but this guy's sole contracted role is to simply send these search messages, it doesn't care about the underlying actions that must happen next to update the model.

I then placed a reference to the widget in the parent template campaigns.hbs...

{{campaign-search search="search"}}

Now what might be confusing is this... the search on the left of search="search" is the name of the action to use, and the "search" it's set to refers to the action defined in campaigns.js. The convention seems to be to use the same term at various levels, though search="search" requires a lot of mental scoping... (especially since "search" is also the action the button triggers!) At one point, I called the campaigns.js action (that actually did the work) updateSearch, and I told the main search action to call .sendAction("widgetsearch"), and so at that point in development the tag was

{{campaign-search widgetsearch="updateSearch"}}

which made it clearer to me, that we were linking a sendable "widgetsearch" key to be bubbled up to "updateSearch" on the parent template/route. (Still not sure I dig the naming convention, but we'll see)

Anyway, the other mental breakthrough I needed was: Ember really embraces the URL driving the current state.  Therefore, the campaign.js route shouldn't modify the current model directly, but rather change the URL which in turn will adjust the model to the correct thing. So search() as defined in campaigns.js can simply be
  search(searchTerms) {
this.transitionTo({queryParams: {search: searchTerms}});
I then had to tell the route about its parameter, so the top became:
import Ember from 'ember';

export default Ember.Route.extend({
queryParams: {
search: {
refreshModel: true
},  //[...]
This said that when the search parameter got updated, it was time to refresh the model.

The model function (we're experimenting with using the new ES6 syntax, so if you don't know the first line below is equivalent to model: function(params){   ) was as follows before I messed with it:
model(params) {
  return Ember.RSVP.hash({

The "search aware version" of that is
model(params) {
return Ember.RSVP.hash({
  } else {
return Ember.RSVP.hash({
camps:'campaign',{q :})

With that done I wanted to update mirage (the part doing or RESTful mockup) so that /api/campaigns?q= would work. This is just a quick dirty filter that, when there's a "q" parameter, filters through all campaigns and only returns ones where a conglomerated value of the searchable fields matches the search term...
    this.get('campaigns', function(db,req) {
        var allCampaigns = db.campaigns;
        if(! req.queryParams.q) {
          return { campaigns : allCampaigns };
        else {
          var myFilter = req.queryParams.q.toLowerCase();
          return { campaigns : allCampaigns.filter(function(campaign){
            var searchable = + " " +;
            return searchable.indexOf(myFilter) !== -1;
So besides writing testing, there was just one more problem: when you entered a search term, the input box was cleared out. For that it seemed expedient to add the "current search terms" as part of the model, so that part of the model function above became 
return Ember.RSVP.hash({
camps:'campaign',{q :}),

and in the template, I had to pass it to component:
{{campaign-search search="search" searchterms=model.searchterms}}

So there are parts of this that still seem like wonky black magic to me... the way the search action that the button in the widget calls is distinct from the "search" (linked to the parents function) that bubbles up when it calls sendAction(), and the automagic way setting a value where the component is embedded in the parent template "Does What I Mean" in filling the appropriately named input field. Still, as I get more fluent in Ember, this stuff will feel less odd to me, I'm sure.

Monday, October 19, 2015

streaming music

David Byrne on Internet Music and how it hurts artists.

Personally I don't really "get" the appeal of streaming music. If you had told me fifteen years ago: "here in the future people can buy any song they want - as a single even! - for around a buck, and have their whole music collection on a lil' walkman-like gadget!" I would have been even more surprised by the follow-up: "But the trend is to use those same gadgets as a fancy, heavily-customized-station radio that you have to pay for on a monthly basis" The latter sounds even more nuts than the former.

(Good thing I don't try to explain to my 2000-era self about Shazam and SoundHound; that stuff just feels like black magic.)

Thursday, October 15, 2015

ember vs angular

My group at work launched a new Ember project, starting with a lot of group self-education.

We put a lot of thought into what framework we wanted, but there are some indications that following a corporate restructure, in the future we may be directed to align with the company in Angular. (But it's not certain yet.)

Up 'til now, Angular hasn't chimed for me. Ember sort of has. I'm not certain if that's an objective rating of the frameworks themselves, an utterly subjective view of what fits with my head better, or a matter of circumstance of why and how they came into my life: what learning resources I tried, and what kind of work I was asked to do in them. (For example, hacking datatables with a homebrew shim (for some custom "infinite scroll" behavior with irregular-height rows) in Angular was extremely painful; and given how good Angular is at looping and constructing tables on its own, kind of quixotic.)

I'm kind of hoping having done a deep dive in to Ember, I might better get my head around the whys and wherefores of Angular if I'm asked to get my mojo up in it. But off the top of my had, a few things I find more pleasing in Ember:
  • Angular occupies a weird space on the spectrum of "no infrastructure, but you have to get libraries or build everything you want to do" to "heavy infrastructure, little transparency, but we make it easy to do the things you'll likely want to do". Angular seems heavy to me: a lot to learn, many syntaxes, an overabundance of conceptual structures, etc. And based on the diversity shown in this angular 2 survey, you still have to pick plugins and libraries and learn there ins and outs as well. 
  • Ember prefers templating that uses different syntaxes for control structures and DOM markup. Angular prefers xml-ish "tags for everything!" - a similar issue I ran into with the JSTL in the JSP days. I really like there to be a clean distinction in syntax for things that are consumed at different times of processing.
  • Embers documentation seems cleaner and more direct than Angular's, and the ember-cli more comprehensive than Angular's dependency on external tools.
  • Ember has its act together with testing frameworks.
  • Both Ember and Angular are suffering from Version 2.0 growing pains but it seems a lot more painful on the Angular side.
  • Ember seemed to have a cleaner one stop solution for routing. I really think routing is one of THE biggest things frameworks might carry over frameworks, and the fact that it's optionalish in Angular seems odd to me now...  (I've done some one offs in jQuery that used hashtag navigation, but it's a huge pain to bolt-on if you don't start with a sane plan. That said, I still think handlebars plus hashtag navigation in jQuery gets you 2/3 of what people want when they start looking for a framework vs a library.)
  • Ember seems better at heading towards React-like one way data binding. I think Angular's love of shared global scope and automagic two way view/model syncing is known to be a bit inefficient at large scales and is weirdly jumpy-feeling compared to event based models - and also requires a particularly deep thinking about javascript's object paradigms when scoping and shadowing issues arise.
So, I want to be a good learner for whatever I'm asked to do. Angular is still pretty dominant in the field, and the libraries can be better than frameworks meme has not gotten enough traction for me to feel safe as its champion, or to stop second-guessing my fear that I prefer libraries because I've done so much small and focused work over my career. (While I think people get misled by thinking that all the work in jQuery has to be done via messy DOM manipulation, I also need to accept that there's a style of programming, not unlike VB back in the day, that just "happens" to use the browser as the output target, so I can let go of what I do know about updating things directly.) I guess my goal is that fabled day when someone will ask me to do a medium size multipage app, and it will obviously be much faster to churn out with an Ember or Angular or whatever I've been working on... I haven't gotten there yet.

Monday, October 12, 2015

the placebo effect and videogames

The Placebo Effect and Videogames Interesting. I've been thinking about the imperfect information you get in videogames. Especially for FPS-style games for casual players: it's hard to tell if you're killed because of a smart computer opponent vs your inexperience or just because of a weapon imbalance in the game or what not- I think that tends to put a damper on the need for good AI in games. (I remember reading a book behind the scenes in "Wing Commander", that mentioned when the enemy spacecraft were visible they swoop and loop and try and look cool, but when you couldn't see them they'd just try to head straight for your tail.)

Tuesday, October 6, 2015

talk about hidden features

I've relied on a "Todo"-app for almost two decades, starting with the one built into the PalmPilot PDA, amazingly good for its time. (But not perfect: here I am in 2005 geeking out about what my "ideal" app would look like.) After Apple opened up the appstore, Appigo's "Todo" actually met most of my 2005 requirements, and has proven reliable for all these years. Over that time, I've realized what really sets apart a decent Todo app is flexibility in recurring events: every couple weeks I want to be nudged to pay off my credit card bill, every couple months I want to be nudged to get a haircut, every day I want the double check that I've video recorded a "second of the day", etc. (Heck, Todo Apps are even the "Hello, World" for JS frameworks, but they skip over the recurrence issue, because it's not trivial from a UI or implementation standpoint. Many Todo apps out there make the same shortcut.)

On Shaun McGill's blog, he wrote about starting to dig Apple's builtin apps and services, leading off with Reminders, thanks to the convenience of its Siri integration. I wrote back to complain that iOS "Reminders" will always feel like a baby app because it doesn't do recurring reminders. He said he preferred the simplicity in not having the UI ever-cluttered with future tasks, and that there were hundreds of other 3rd party task apps for my needs. I was about to continue our civil disagreement by pointing out none of those apps will be usable via Siri (which is a whole argument) but then I realized the joke was kind of on both of us:  you can create repeating Reminders via Siri.

And, I thought, only by Siri. But I was mistaken: the functionality is hidden, so that this screen

becomes the following once you click on "Remind me on a day":

(Palm had the same kind of hiding, where the recurrence details were hidden until you set a date.)

It was interesting that Shaun and I both made the same misassumption. Out of sight, out of mind!

Anyway, I'm not sure if Siri integration is enough to make switch over to Reminders. For one thing Reminders can't add a date to a task without a time as well... a very datebook way of thinking that goes in hand with "when should I send a notification" thinking, but doesn't match how I cope with my load of tasks. Similarly, the badge icon task count is underbaked, or at least not updated in a timely fashion. UPDATE: Thinking on it further, I realized that Reminders' repeating notifications lack a feature I find critical: repeat a certain time after completion vs repeat based strictly on start time. This reflects a critical conceptual difference, Reminders really consists of reminders of tasks, some of which might have a date/time or location approached, while Appigo is a bit more like Getting Things Done.

One thing I like about Reminders is that you're free to edit the order of the list, while Appigo assumes Due Date and priority sorting. Appigo also makes engineery-smart and UX-y dumb assumption about ordering... its due date sorting is strictly chronological (to the day level, and then alphabetical) - time sorting makes some sense because older items are "more overdue" and, presumably, a higher urgency. In reality, an item that has drifted a day or two overdue is probably ok there and demonstrably not a 100% priority, but the stuff that came up today has a chance of being absolutely critical. My ideal, then, would be a reverse time sort option. This might feel strange since it means everything due or overdue is sorted in reverse chronological order and everything upcoming is sorted in more normally, but from a workflow sense it's a viable option.

UPDATE 2: I realized another stupidity of Appigo... any user with a moderately full plate of tasks - SOME of which with deadlines, must in effect put a (sometimes arbitrary) deadline on ALL their tasks, because the "No Due Date" section occurs after all tasks with dates, even those far in the future. So after all my "today" events (and again, I'd prefer to see those listed earlier than things I've already allowed to 'slip' especially since I now realize I'm slapping utterly arbitrary dates on EVERYTHING) I have the 8 or so things I have pending tomorrow, and then the 12 things I have under "Next 7 Days" and then the 24 things of "Future". And THEN the 2 items I humbly suggested don't have a due date, I'd just like to do them at some point. That is really bad UI; I think the app really needs an option to fiddle with the ordering of its sections. My ideal would be "things due today" "things overdue" "things without a date" "things in the future"... actually, probably things overdue showing up first would finally make sense.

Sigh. Maybe I SHOULD make my own damn iOS app for this.

Friday, October 2, 2015

quickie: viewing JSON in chrome

JSON formatter is a nice module for chrome that makes raw JSON a bit less raw.

(And is still my favorite way of editing errant JSON)

Thursday, October 1, 2015

bad ux is a misdemeanor against humanity - google inbox "speed dial" is a joke

There on the right is a snippet from Google Inbox. Those top 5 circles fly up when you hover over the red circle (which is a plus sign at rest)

Three of them are Inbox's best guess about whom I want to mail next. (I think Google calls this "speed dial") The algorithm powering this is terrible. For a while the guesses were ludicrously out of date and arbitrary - people I was writing with regularly but not for months.  Lately they seem more chronologically correct but the relevance just isn't there. For instance, I wrote back to "R" today, in reply to a joke she sent me, and that was the only mail we've exchanged in probably a year. Sending her a new email is clearly not a priority in my online life.

In May I griped about this in the Gmail Forum where I was told they're "based on who I interact with most". The problem is easy to spot: the people I'm interacting with, I'm interacting with, replying-to, in pre-existing, long-lived email threads. A different criteria, say, "accounts I initiate email to", would be so much simpler and actually useful. (For example, I keep having to carefully retype my band's mailing list address, since the threading model wiggio uses hacks the "Reply-To" with a little postfix tag on the username... but I'm sure the no-tag version is the single most common email address I've sent to over the past few weeks, and having that one click away would save me having to wrestle with the recipient autocomplete.)

Then there are two other circles. The gold ticket is the "Invite to Inbox"- oddly overeager self-promotion. The blue finger "Reminder" is equally useless to me. I'm sure some people might try and use their email client for their general Todo management, but I'm certainly not one of them- it's a weird confusion of purpose, a reflection of that eternal goal of being "the program to end all programs" these things have. Zawinski's Law states
Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Same thing here, I guess, but going the other direction.

All this stupidity would be easy to ignore, except for one thing... see how the "I" icon is next to the "close this email"? That icon is blocking the checkbox underneath. (This happens at certain browser widths.) No problem, right? Move the mouse away, the column of circles will collapse to the single red circle, and I can click? Afraid not... somehow, the entire column where the icons would be triggers the icons "helpfully" sliding back into place... the area over the checkbox is blocked, even though it looks clear. Here is a video showing it in action:

So, the 3 poorly chosen icon plus the "let us be your todo!" useless icon plus the self-promotional icon plus poorly chosen size defaults equals a terrible user experience.

I like the Inbox concept... its grouping by category like "Finance" "Purchases" and "Low Priority" is generally good, and it's cool to be able to sweep away a whole chunk of fluff mail with a single click. Still, sometimes I think its "expand in place" paradigm (vs Gmail's "new subscreen", modal approach) can be problematic as seen here. (Also, Gmail's classic approach is pleasant in a "now you're focused on this one thing" kind of way.)

UPDATE: the next day I wasn't reliably able to get the "hidden hover column" to recur; whether that's a fix or just an intermittent bug, I'm not sure. Without that bug, the gratuitous icons are much less of a practical problem