Along the road to enlightenment, I’ve had some superlative teachers, professors, and friends instruct me on the best ways they knew to implement everything from Levenshtein Distance to Diffie-Helman compression, from polygon intersection to Relational Division. These lessons weren’t always apparent, and hardly ever enjoyable, but the knowledge and satisfactoin at the end of the project often put me years ahead of self-study.

Several times, I’ve been asked questions to which I’ve had absolutely no idea as to the solution. Yet, right then and there, an answer was demanded of me. It’s difficult and embarrassing, but situations like these are more about the reasoning than an off-the-cuff answer. This is an important concept, since it is a large step on the road to Doing Big Things.

Such occurrences were relatively commonplace, and for a long time I couldn’t figure out why someone looking to help me was asking a question that I either obviously had no business answering, or that was completely over my head. Then, just a few days ago, I stumbled onto the concept of zen. Zen, for want of a clearer definition, is the internal feeling and state of mind which is, in rudimentary forms, omniscience about a subject. It is the absence of bounds or form of knowledge in an area of concentration or meditation.

Zen and the art of programming was an interesting thought experiment, and until then I had no idea zen was being taught to me the entire time.

A mondo is a component of zen scripture where the roshi (`roh-shee) [teacher] asks the student a question, demanding an immediate answer. Since this question is often complex or multi-part, the process of developing the answer is more critical than the answer itself, akin to the Biblical scripture of teaching a man to fish. Had the pupil regurgitated some decade-old algorithm learned in Intro to VisualBasic doesn’t further the understanding of the problem at hand. Just because you may have learned bubble sort first doesn’t mean it’s the best choice for your production code base. A mondo would solve whether insertion sort would suffice instead of spending two days of company time implementing heap sort; and, more than that, it should help you realize the patterns in the future to prevent the same questions from needing attention again.

This insight, the ability to anticipate or at least keep pace with the business’ needs, is one of the most efficient ways to show off one’s skill in a profession. Especially now, when employers need to “cut off the fat” out of their budgets and reign in on ambitious funding of pipe-dream projects to stay competitive.

And there’s no reason you shouldn’t want to continue to learn. If nothing else – and trust me, there’s plenty else – you will bring those skills back to the workplace every day, inspiring those around you and impressing your superiors. That’s not even counting the personal satisfaction of reasoning through an enigma and coming out the upper side with a better solution or clearer understanding of the problem.

I am going to argue against myself here. I realize that in a prior post I praised PHP dereferencing, longing for the direct pointer manipulation of Perl, by using $value=true; $name='value'; ${$name}==true. However, for the next developer, he or she must understand prior to inheriting the code, how PHP handles inline eval() statements.

In situations where the use of these is unavoidable – usually in dynamically-created variables, LISP anyone? – a detailed comment is assuredly in order. Even if it is simple, an example, the next programmer to stumble across your code should be able to reasonably expect an outcome from the procedure.

In JavaScript I’ve been writing lately, I’ve become very cautious about the nature of functions and closures. Frequently, I’ve adopted the practice of namespacing everything I write. Because of that, grouping functions down to the program area can be tedious. No programmer wants to see com.acme.financial.payment.forms.construct(). With JavaScript, we can fake a with() statement. var pay_form = com.acme.financial.payment.forms; additionally com.acme.financial[params['object_type']].forms[action](params_hash) can save a lot of code later in the process.

As an engineer, we should always be looking for ways to perfect our trade. Often this means coding cleanly, and using conventions. Sometimes this means using quirks to their biggest advantage: shrinking the codebase. Remember, one bug for every ten lines.

Our RDB now contains two paths from Baby (B) to Deceased (D), one including the Person (P) entity. We also presume, though our system does not outright, that there are roughly 10:1 paths from B->P->D than B->D. We’ve been trying to solve one of the old problems of a User-generated SQL query: power and risk versus ease and safety. These are not necessarily exclusive, and I have argued why. We have the technology, now lets put it to good use!

The semantics, as stated, is the easy part: We will store them in our foreign key properties in the code. When the user requests a path be made, we’ll retrieve and display the semantics that relate to the links we’ve chosen.

Now comes the tricky part: choosing which links are best. By default, “average” queries follow a consistent path. Trying to determine if Alice and Bob work for Acme generally means we’re looking for the same link. As is the case if we’re looking for anyone who works for Acme and Borax. In our second query, we’d presumably cross the same link twice. This isn’t a coincidence, it’s a natural state of the human expectation.

Let’s put this into practice. Suppose we cache a rough estimate as to how many unique foreign keys are in one table for its neighbors. (This can be achieved through indexing and is a very fast query in modern RDBMS’s.) Traditionally, when attempting to discern a path from one entity to another, shortest-distance was applied. However, given our RDB situation above, this is probably flawed: What we really wanted was from B->P->D, two hops instead of one. Given that we have a fast lookup that informs us we have an alternate path out of B, let’s try that.

Graph theorists recognize this as a max-flow/min-cut problem. We are looking to maximize our probability that the path chosen is correct; likely the path with the most foreign keys. If I had the time and energy, (maybe I will later), I’d make up a diagram or code snippet to illustrate; but you get my point. Some times the shortest path isn’t always the right one, and with the proper tool set in place we can offer a much more elegant – and noticeably more user-friendly – interface.

In the next part, (which I don’t expect to have finished anytime soon), I’ll be addressing some caveats I think would hinder our progress – most notably worst-case scenarios and impossible Source-Sink choices. It’s the conclusion to a thought constructed more than a month ago, and possibly one of the funnest thought experiments I’ve had in quite some time. I hope you enjoy thinking outside the box!

Using our previous example, we can already see how rapidly a RDb can gain entities and relationships as correctly storing information can require many additional entities. For our “intelligent SQL constructor,” the system will need to weed through the myriad of tables to get to find the fastest path from start to finish.

Now suppose we expand our example yet again and include a second path between Baby and Deceased: “A Baby can be born and soon after become Deceased.” This new relationship throws a monkey wrench in some interfaces, as the most direct path from Baby to Deceased is now a single hop. For some users, that we may abstract the fact a Person will become Deceased can lead to erroneous expectations either from the developer or the user; in either event, this is a discrepancy which will cost patrons and development costs in the end.

So how can we expose this fact? My personal inclination is to be explicit is possible. By describing to users what you, as the developer, were intending, harmful reactions should realistically be cut to a minimum. To do this, the system will need to be mathematically aware of how the RDb is structured and what the relationships imply. As proposed in the exposition, knowledge of existing information, its meaning, and a relation to other algorithms should help us greatly.

The first tool we’ll need is a mathematical expectation of how to go from Baby to Deceased. Because of the wonders of modern medicine, users expect that life for a Baby does not cease until having consumed a significant time on Earth. From the data-modeling standpoint, this is to trace the relationship from Baby to Deceased through the Person entity. However, some UIs may not account for this and choose the direct, 1-hop, relationship.

From a user’s standpoint, this is misleading. “I’m calculating social security checks for the elderly; why do the results include a dozen records which are extraneous?” The system’s presumption that it must follow the most direct path from Baby to Deceased seems flawed, though – for the criteria – its data may be perfectly correct. Shouldn’t we instead initially propose to follow the path with the highest probability of success?

In such an example, based on our knowledge of the use of the system, we estimate that there are ten times the number of records for paths between Baby to Deceased using the Person table as without. Taking this into account, we almost assuredly are retrieving the right result set for social security calculations. Then, for the remaining ten percent, we offer the option to “trim” the Person entity from our query.

Additionally, what the system hasn’t exposed here is the semantic relationship the surmised from the entities provided. Perhaps the results were appropriate for what was provided, just not what was expected. This is the second tool we must produce: a means of exposing to the user our presumption of the semantic meaning of their query. This is, admittedly, less for our system to act upon than to influence what the user will input next, or perhaps withdraw and try again.

By now it might sound familiar to problems faced in graph theory. At the moment, there are two that I can think of which aptly fit into this scenario: maximal flow and minimal cut. Both are incredibly annoying problems with surprisingly low-tech solutions. They also both make use of our mathematical model tool we prescribed earlier. An improvement to this theory which deserves its own article will be proposed at the end of the article.

Luckily, there are means to do all of these things when given a well thought-out framework. Semantic ties can be stored within the framework and the mathematical model can be derived quickly from the data via a few SQL queries. (I already anticipate the cries of outrage over additional SQL queries, but those drawbacks will be addressed in the conclusion; so please, save your flames until the end of the series.)

But, no matter how dumb you play, something seeps in. That’s how you weed good teachers from the mediocre ones: they can disseminate knowledge without you even knowing it. They’re kind of like God, in a way; sometimes the vague answers and delayed responses aren’t intended as insults or shirking, it’s a device to make you think and learn for yourself. Sometimes, when you think you need a helicopter to save you, a properly thrown life vest will do just fine.

While at Pittsburgh, I had a tough-as-nails professor for my algorithms class. I spent so many nights compiling Java code until 3AM when my throat was sore from too much tea and my eyes bled zeros and ones, but I got the programs working to spec and in on time. It was, for all intents and purposes, a trial by fire¹. At the end of it, I thought I would forget everything and never use an iota of what it tought me for anything practical in my career. This view was incredibly, humerously, short-minded.

For a while, it was correct. Still, something lingered, and I should be glad it did. What remained was a desire to apply what I’d learned to what already existed. Solving old problems with more efficient solutions is always a challenge. It’s why neural networks and packet routing have evolved. But with these challenges comes personal and professional rewards, not the least of which is the ability to re-use the mental framework for the problem solved.

One of my least-favorite problems was graph search. (There’s a reason no one but mathematicians specialize in developing, proving, or otherwise entrenching themselves in graph theory.) Yet, graph searching is one of the most widely-applicable skills a software architect can develop. Case in point: dynamic SQL construction.

As a mental exercise in theory and practice, there are several parts to this post, interleaved with time for mental stretching, retractions, and analyses & comments. (This is also no coincidence, as for those literary-educated readers the scaffolding is of the classical drama; enjoy.) This is Act I: Exposition.

In my experiment, there are three factors:

• tree complexity,
• semantic-path evaluation, and
• reasonable application to graph theory.

Tree complexity is the time and effort required to maneuver ‘blindly’ from one entity to another, 2-hop-away, entity in a database. Semantic-path evaluation is the real-world implication of the statement’s conjoined entities. For instance, using the tables Painter, Plumber, and Person, a Person can become a Painter or a Plumber but each has a specific role and though both may use water they may not be what we intended to achieve. Without predefining rules, the system will have to build an assumption based on the entities provided. Lastly, while possibly obvious, the application to graph theory may need to be spelled out; this will probably be spent the least time developing.

If you can think of anything I’ve missed or misunderstood, constructive criticizm is encouraged! Act II: WYSIWYG will hopefully follow soon.

¹Taking the class alone isn’t the Pitt CS Trial by Fire. Taking 1520 and 1550 in the same semester is. To exit both classes with B’s was a personal accomplishment, as well as bragging rights.

This is not to say, however, that they are insurmountable. Often, in fact, when we stop trying to over-engineer our system and evaluate the situation as it is, we can make optimizations beyond our expectations. As a simple “for instance,” presuming we have a determinant for a conditional that is initially a free-text entry. According to this pretense, Boolean evaluation can take as little as the first byte in the string or as much as the full text of the string; thus, for sake of computation time, memory, and most of all maintainability, it should be avoided at high cost. Compare this with replacing this free-text with a numerically-indexed list of choices and the Boolean evaluation drops to a static value. (Even better, most implementations of bit-wise integer comparison are executed as close to machine code as possible.)

But, inevitably, there are cases which solutions such as these are not so prevalent or obvious. For instance, when implementing join-context SQL conditionals as PHP, it can take a tremendous amount of derived and out of band knowledge to perform the translation. Combine this with already complex logic, (nested XORs, negated string comparisons, and cross-table lookups), and the code can get slow and ugly very fast. So what do you do as an engineer with an email, the SQL, and a pat on the back? Get a game plan.

It’s more important to pencil in a strategy for defining your criteria than going off of pre-existing assumptions. For one measure, repeated calculations in the original may be consolidated as a single conditional whose value is stored and later addressed; the processing efficiency alone is worth the change when scaling up to enterprise-level datasets. Additionally, as I stated above, changing data types of pre-existing conditionals can be a great boon as well, and performing integer arithmetic is always faster than strcmp.

Once the parameters are defined, we zoom out to observe the algorithm. Keeping in mind the practice degrades maintainability, nesting some repeated conditionals can save processing power down the line. But don’t get nesting-happy; (especially with code you’re simply translating and didn’t have the mental inception for) you’ll forget why two conditionals had been nested or why their order mattered so much. This is where your formal logic class enters the picture. Class, what is {A, B, C, D} & {A, B, C}? Of course it’s {A, B, C}; it’s also ¬{D}. The optimization, however slight, of changing

if($var == A OR $var == B OR $var == C) { something_true(); }

to

if($var != D) { something_true(); }

may not increase performance by leaps and bounds, but if you’re deep in the code anyway why wouldn’t you shave off the time?

Now, after the code is written and tested, there is much rejoicing. The only thing we now have to contend with is if the translation needs to change. And while we may not, (and hopefully will not), go back to square-one with our evaluation, we should at least take the time to consider where we may have gone wrong in our previous implementations and where we can continue to go right.