How to kill a server without really trying

When you work with code that runs on a high traffic server it is very important to manage the state of critical shared resources.

The JAVA language provides the try/catch/finally construct as a way to ensure that an area of code safely cleans up it’s resources.   Often this language construct is not properly used and in an extreme case misuse of a try block can cause a server to run out of resources.  This post describes a simple technique you can ensure that your try blocks are safe.

The key is that you must always perform any work that requires clean up BEFORE you enter the try block.  This is where most developer’s make a mistake.   When you don’t follow this rule, you may encounter problems that make the finally block unsafe.   If your finally block depends on state that does not exist BEFORE the start of the try block you are in danger.  The following example is unfortunately very typical.

When you review code, be on the look out for complex logic in a finally block because often it means the developers are not following this rule.  Also you should be nervous about any method that allocates changes a critical resource state and does not immediately FOLLOW that allocation with the start of a try construct.

The following example is unfortunately very typical.  In this example a *minor* problem in the database such as a column rename could lead to a collapse of the application server.  For example, if the DB renamed the column from “USER_NAME” to “u_name” the result would be that each time the block of code is invoked we would have a SQL Exception thrown at line 91.  The server will exit the block without closing the connection and as such may run out of connections since until the GC occurs the connections that are left open but no longer owned will not be closed.

Example #1

With this example there are a number of problems.  For example, an exception on line 88 can occur if the system is unable to get a connection.  The code will reach line 98 and then throw a NullPointerException because we try to issue close on a recordset that does not exist.  Similar problems can occur with failure to prepare the statement or execute the query.   Basically until the code reaches line 92 any attempt to run the finally block will result in a NullPointerException.

On a server this type of issue has the potential to bring down the entire environment.   With high traffic a minor issue can cause a problem where a large number of connections are acquired and never released leading to an complete database failure.  Code like this can encounter “minor” problems and then cascade to exhaust resources.   This code can cause the connection pool to “run dry” in extreme cases.

The following shows a re-factored version of the code where we acquire resources before we try to use them and then handle the cleanup of each resource in an independent finally block.  Code in this style has higher level of indentation but since each concern is handled independently it is much safer and easier to review.  The style also makes it much easier to later split up a large method and/or make other changes.

Example #2

The code would still benefit from additional changes but with example #2 we can now be sure that the code never leaves a connection open.  With this key change the code is much safer then the first example.  The code is also now structured in such a way that we can split up this large method into a number of smaller methods and move towards a generic solution.

An alternate to example #2 is the use of a single master try block and a series of “if(x==null) x.close();” statement constructs in a single finally block.  This approach is valid and I occasionally use it but be careful.   When using a master try block it becomes harder to do future re-factoring changes like splitting up a large chunk of code into multiple smaller methods.   In addition, the single master try block style often results in code that is more complex and requires more time to audit because the scope for the various elements is larger and order of operations in the allocation and cleanup becomes more of a concern (i.e. you don’t want to close y before you close x).

In this example I did not alter the external behavior of the code.  In particular I do not like the way the code “swallow’s” the exception and returns false on database failure.   changing this behavior requires more careful analysis and is an issue for another day.

Looking at NetBeans

I spent a few hours yesterday with NetBeans.

My primary IDE for Java development has been Eclipse and prior to that I worked with Eclipse’s predecessor (IBM Visual Age). When I first started working with Visual Age in 1998 or 1999 I was very impressed and each release has been better then the one before. Visual Age was rebranded as Eclipse; the pace of improvement has increased in recent years via the power of the Open Source community. Eclipse continues to be the leader in Java IDEs but at the same time the other players have been improving their offerings. InteliJ IDEA, JDeveloper, and NetBeans are well designed systems and I am sure that in some usage examples thes environments are better then the status quo functionality provided by Eclipse.

The last time I looked at NetBeans was several years ago. A lot has happened since my previous look. Spending time yesterday with NetBeans showed me that the project is alive and well and includes many innovative features. I enjoyed seeing how the tool has grown and see a lot of potential. Compared with my last look the tool is much more polished and given the improved speed of modern hardware performance was not an issue. The UI for NetBeans is still not quite up to the Eclipse standard but it is much better then it was in prior releases. There are pros and cons to the Swing vs. SWT approach and for the most part the Eclipse environment rendering is better due to the native widget approach that SWT uses but the gap is much more narrow then it was in the past.

I configured several of my projects for development in the NetBeans tool and created a few test projects from scratch to get a feel for the process. In addition I installed plug-ins for working with GWT, JIndent, Jython, and a couple of other community features. Overall I found that NetBeans continues to be a strong player in the IDE marketplace. Right now I still prefer Eclipse but over the next few weeks, I plan to play with NetBeans a bit more in order to fully evaluate the feature set.

Since my GarinDriver project works with any IDE I also decided it was a good time to put together a quick slide show to provide an overview of using GarinDriver with NetBeans.

I am not sure how much I will use NetBeans in the future. I may decide to stick with Eclipse but even if that is the case spending a little bit of time every few months to look at other options is a good thing.

Playing with GWT

My wife and I find it is much easier to remember a phone number if you create words out of the letters.

A while ago, she asked me to create an application to help with the process of figuring out what words you can make out of a given phone number.

Since I had some free time today, I created the Phone Words App using GWT.

Web Frameworks, AJAX, and GWT

Over the years, I have worked with ALOT of web frameworks in various environments.

The web was origionally designed around a document sharing paradigm.  User intractivity in the web started with CGI requests and basic HMTL forms.  The early CGI mode was very lacking in terms of application interactivity due to the long request and respone cycle and the lack of state management.   Basically CGI is not much more then what you have with a command line text interaction and in some ways the CGI model is more restrictive.  Compared to the C++ GUI frameworks I worked with in the early 90’s the move to web based CGI was a big step backwards as far as design goes but the push to the web required a move away from abstract widget based approaches. Java Servlets improved the CGI by making the interactions faster and providing a crude session management system but at the it’s core the Servlet model of request response leaves the same fundemental problem present in CGI.   The session is also bad since state management is on the server and as such can be lost and can come out of sync with the user’s activity fairly easy.   This lack of a true component and widget oriented model is a big problem for complex applications.

The solution to the request/response problem is AJAX. Using AJAX lets applications in browsers act like traditional client server systems. With AJAX you can keep per user state on the client, perform background server communications with overlapping transactions, control the user’s experiance and generally produce a better applicaiton experiance. That said, it is not magic and the cost is that the browsers for AJAXs must do more work vs. the dumb term type interactions of the pure HTML3.2 model. With AJAX you need at minimum a browser with script support and the ability to perform background communications (i.e. hidden iframes). The browser also must be able to dynamically control all rendered content (i.e. DOM/DHMTL). The requirement for a rich browser makes AJAX heavier then the old school flat html model. It also makes development of AJAX more complex but thankfully frameworks like GWT go a long way towards containing the complexity.

The GWT framework addresses the issue by packaging features as widgets and using AJAX RPC behind the scenes for background comunications. With GWT you get support for the various browsers for free and if properly designed a GWT application. State for the application can be properly maintained at the browser level in the widget context and as such the server can be a true stateless environment. GWT and systems like it with a AJAX and/or future RPC techniques allow for much richer web applications.

I recently put out a GWT version of my SQL color coding demo application. With the GWT framework adding support for a new features is much easier and the end user has a much better experiance. The SQL color coding demo is very small and simple but it is fun to use and shows the basics of the GWT model. You can try out the GWT version of the SQL Format app here. This is a very simple bit of GWT but it shows how AJAX interactions work and provides a taste of the programming model for GWT. The source code is available here.

A slightly more complex example of GWT is something I put together to help my wife and I make words out of phone numbers.
http://markfarnsworth-dev.appspot.com/phoneWords.html

SQL Formatter and Pretty Printer

A couple of weeks ago I started work to build a universal parser for SQL.  The work started with a basic need I had for minimal parsing of SQL statements in the GarinDriver JDBC project.  I was unable to find any open source parser code that would fully support the basic structure used by MySQL, PostgreSQL, Oracle, and other similar systems.   In particular, I wanted support for q’<>’ used by Oracle in addition to the SQL standard string encoding, MySQL’s use of C style string escaping, and $tag$ style strings for PostgreSQL and H2.  Conceptually I wanted to build ONE parser that could be configured to properly parse ANY dialect and that would support a framework for adding additional exceptions in the future.

The basics of my parser framework is now complete. It is smart enough to parse MySQL, H2, HSQLDB, Oracle, PostgreSQL, and ISO SQL2003 variants. The shallow parsing framework uses a base Token class and a range of subclasses to describe the most basic elements of the SQL grammar. The parser identifies the core keywords, statement boundaries, string, comment, and identifier limits but does not look into deeper language issues like statement structure.   One benifit of the shallow parsing is that it allows for flexibiltiy and does not require a complete BNF style grammar.  Even with only this basic level, it is possible to leverage this code for useful stuff beyond the GarinDriver, LiquiBase data migration solution.

The following bare bones test page demonstrates how the parser can be used to format and add color to SQL batches.  The code would also be useful for someone building a universal SQL editor.
http://markfarnsworth-dev.appspot.com/RenderSql

Overall, I am happy with the approach and while I plan to use a more established framework like ANTLR or XTEXT for future deep parsing efforts I feel my homegrown framework provides a better approach for shallow parsing vs. the larger and more complex language tools that I have reviewed.

A Universal Parser for SQL

I recently completed some work on the GarinDriver to research a more flexible model for database schema change management. This work gave me a chance to dig a little deeper into the nuts and bolts of Java database interactions. Working to track schema changes and support a wide range of databases increased my appreciating for powerful open source database platforms available today. In particular, working with H2 Database gave me a chance to explore what I now think is one of the hidden treasures of the open source DB marketplace. Overall it was a fun bit of hobby coding and I believe it will be useful in future real world projects.

Working on the driver, one challenge that went beyond my expectation was parsing the various dialects of SQL. In a review of current open source projects, I did not find any public license parsers that can parse statements from ALL database systems. In a sense, we are supposed to have a standard but it is really well followed by the vendors and as such writing a universal parse is a bit difficult. I had some free time this weekend so I took on the challenge since it is an area where the open source community really does not currently have a good solution. For example the tools that ship with Eclipse can not parse the $tag$ style strings from PostgreSQL or the q’[O'Brian]‘ style supported by Oracle. In any case,  I had some free time and interest so I decided to build my own parser.  Without such a parser it would be impossible to provide full support for the statement execution model needed to support the GarinDriver desgin.  Workarounds were possible but a proper parser felt like the best approach.

Building a parser is complex and often involves specialized tools (i.e. ANTLR, BISON, JAVACC, FLEX, LEX, YACC, etc). There are benefits and drawbacks to the traditional grammar definition and parser generator approach.   My initial review it seemed that supporting the often contradictory approaches used by different systems would be quite difficult with these tools.

Since all that I needed for the GarinDriver/LiquiBase project was “shallow parsing” my approach was to use a small framework of Java classes. The use of my own framework allows me to share logic across the SQL dialects and provided a chance to explore idea for a more dynamic approach to “shallow parsing”. So far the approach seems to be working out nicely although I am considering building a deep parser at some point later on with the Eclipse XTEXT project but for now the flexiiblity of a home grown hand coded parser has proved to be a viable approach for “shallow parsing”.

Parser Fun:

• Standard SQL comments have EOL style (–) and the block style (/* */).
• Block style comments nest within each other so parser must count the nesting levels.
• MySQL supports pound sign comments in addition to the standard forms.
• HSQLDB supports // style comments in addition to the standard forms.
• Standard SQL uses double quotes for identifiers and single quotes for text strings.
• The parser must support both including quote symbol doubling for escapes (i.e. O”Brian or My “”big”" table.
• Oracle supports q’[O'Brian]‘ in addition to the standard style.
• MySQL uses C style string encoding (i.e. ‘O\’Brian’ ).
• PostgreSQL supports $tag$O’Brian$tag$ style in addition to the standard forms AND the mysql format.

My parser uses object oriented techniques to define a base token concept, extend the base concept for SqlStatements, and then extend the SqlStatements to define the dialect variants in what I hope will be an extendable framework. This approach provides more flexibility for future growth vs. more static models and parser generator tools. In any case the approach seems to work and the ability to use object orientation to extend the parser in new directions seems like a good thing.

As with other hobby projects, my work in this area is EPL and hosted on Google code. The documentation is sparse but if you are looking for SQL parser code you may find that this code can help you develop new and interesting tools for working with SQL systems.

http://code.google.com/p/garinparser/

If you decide to use the parser please let me know. Also, if you can define a legal SQL statement that does not parse correctly with my parser let me know since I like to eat tasty bugs.

GarinDriver plus LiquiBase

LiquiBase Schema Manager GarinDriver with Enhanced Schema Manager Distributed Locking Model YES YES Multiple Context Streams YES YES MySQL DB YES YES Oracle DB YES YES PostgreSQL YES YES HSQLDB YES YES H2 YES YES Liquibase XML Change Set YES YES Vendor SQL (i.e. ENGINE=InnoDB) PARTIAL YES Complex SQL batches (i.e. from mysqldump style tools) NO YES SQL/DML Recording (record from any JDBC tool) NO YES Manual, FAQ, etc. YES PENDING

GarinDriver is an JDBC driver and schema manager that I created to supports a schema deployment. The driver wraps your backend driver and provides a framework for SQL/DML script recording and playback to streamline distribution of changes across environments. The driver can also be configured to run with the enhanced schema manager or use the standard LiquiBase SchemaManager via an option in it’s JDBC URL.  With both models, GarinDriver ensures that schema changes are automatically deployed BEFORE client code connects to the system. With the enhanced manager, the schema change stream is defined as a “stack” of numbered SQL and/or XML files. This model works well with mysqldump, Toad, SQLPlus, and similar tools. Tools like mysqldump are especially useful for creating a complete system script in one simple step. Subsequent scripts can also be written as general SQL standard files and/or use the full flexibility and power of the native dialect. The driver level DML recording feature provides an option to record DML statements you issue in your JDBC based tools and automatically add them to the schema stack. This option means that you can configure ANY schema tool that uses JDBC and automatically captured the numbered series of scripts to build your schema distribution table. In addition at any point you can place LiquiBase XML files in the stack to provide flexibility for vendor neutrality while also having an option for running the vendor specific scripts when and where they make the most sense. The combination of GarinDriver and LiquiBase provides a lot of power and flexibility for complex envirionments. For more information about GarinDriver please visit the main project site. http://code.google.com/p/garindriver/ We will be providing additional documentation in our wiki in the coming weeks.

GarinDriver

I recently released the GarinDriver as an open source project published on google hosting.
http://code.google.com/p/garindriver/

The GarinDriver is a JDBC driver that includes the ability to run a stack of scripts for initial creation of a database schema and incremental updates to that schema.

By working this logic into the JDBC driver layer, it is really very easy to add the capability to exiting projects.

Working on my ANT patch

I did some work on my ANT patch today.

My goal is to have a well enginered patch ready for review after next weekend. I can really only work on it in little bits of time since it is just a hobby project.

In any case the basic plan is to add support for something similar to the date features for the Touch task integrated into Zip task.  Since Jar, War, and Ear tasks decend from Zip they will automatically inherit the feature. This patch will allow the ant build system to control the content inside the archive.

The Touch task supports two default syntax formats for setting the date and the ability to create a custom pattern using the syntax of the simple date format.

  <touch datetime="09/10/1974 4:30 pm">
    <fileset dir="src_dir"/>
  </touch>
  <touch datetime="09/10/2008 4:30:20 pm">
     <fileset dir="src_dir"/>
  </touch>
  <touch datetime="2008-09-10 16:30:20" pattern="yyyy-MM-dd kk:mm:ss">
    <fileset dir="src_dir"/>
  </touch>

My patch will support the same syntax for <zip/>, <jar/>, <war/>, <ear/>, and <tar/>. This will act like touch in that when files are stored in the archive the modification dates of the files in the archive will be set based on the value supplied in the script.  This is importaint because without such a feature the MD5 fingerprint of your archives does not reflect the source code tree.   This feature or a similar workaround allows the build to produce binaries that have MD5 hash values relecting the sources and as such you can run the build multiple times with confidence that unless the source tree has changed in a way that effects the compile your MD5 fingerprint will come out the same each time.

  <war datetime="09/10/1974 4:30 pm" destfile="ROOT.war" basedir="webapps/ROOT">
    <fileset dir="src_dir"/>
  </war>
  <jar datetime="09/10/2008 4:30:20 pm" destfile="stuff.jar" basedir="build/stuff">
    <fileset dir="src_dir"/>
  </jar>
  <ear datetime="2008-09-10 16:30:20" pattern="yyyy-MM-dd kk:mm:ss" destfile="stuff.ear" basedir="build/stuff">
    <fileset dir="src_dir"/>
  </ear>

A JDBC Driver frontend schema deployment integrated as a feature

I recently had an an idea for dealing with an old problem in a new way.

Our team has several members who work remotly including developers who live outside of the US.  We also have many servers running our code and as such when a database schema change is required it is a challenge to ensure that it is run on all databases at the proper time.

My idea was to solve this problem with a smarter JDBC driver.  The driver is the first part of the system that touches the database and as such the perfect place to check DB state vs. client schema requirements.

With typical usage, deployment to the integration, test, staging, training, and servers use the GarinDriver.   At each location the scripts needed by the client are automaticaly run before any application code connects.  This makes it easy to deploy application and schema changes.  Also since this is done at the driver level it works without any application level hooks beyond simply configuring the application to use our JDBC driver.  On the production and UAT severs the team currently standard driver and manual deploy schema changes to these environments.

Code for this is now shared:
http://code.google.com/p/garindriver