by Jason Williscroft
Last Thursday morning, Markit held a webinar announcing the release of their Markit EDM Test Harness. Two HotQuant partners were among the hundreds of attendees on the call. Here are their reactions:
Today, Markit held a WebEx meeting to announce the release of their Markit EDM Test Harness to their client base. Primarily the tool has been used internally by the Markit EDM development team and was unavailable to the client base. During today’s WebEx, they announced plans to release an open source version of that same tool for client use.
At face value, the tool seems to address some major pain points experienced by those of us who have developed or performed quality testing on the MEDM platform. The tool allows for the execution of processes from the command line, running on the Process Agent. It also allows for comparison of current process output to a previous run’s output. The comparison functionality can be used to verify most output; whether that output is file based or the output is a snapshot of data in a SQL table/view. In addition, the application writes useful information to log files, which the user can access either in file form or access directly via the Markit UI.
Overall the application interface appears to be bare bones, but solid. It appeared simple & intuitive and did not appear to have any glaring weaknesses that I could see during the short presentation. It will be interesting to see how much off page configuration may be necessary for some of the more complicated tests cases that we all come across. The application doesn’t appear to have an interface that allows for the in application review of the compared files. Although, if you prefer to have this type of functionality, there are plenty of third party applications out there that can be leveraged.
In addition, the tool did not include any fancy dashboards for reporting statuses or any macro level analytics. Given that the demonstration was more functional than technical, it is unclear if the architecture will support custom development of this type of thing.
In my book, where the application really nailed it was in the log file messaging produced by the test runs. The log files are easily readable and contained a TON of useful information regarding the run. They exposed a quick glance at the run overall metrics as a whole and the detail to allow the user the opportunity to quickly ascertain where there may be issues. The information appeared to be just the right amount; not too little to be effective, and not too much to produce overload.
In summary, from a user perspective, the application looks like a HUGE step in the right direction. The application increases the testability of the platform immensely. I, for one, can’t wait to get behind the wheel and take this thing for a test drive!
Finally, an automated testing tool worth talking about for Markit EDM! Yesterday Markit presented a live webinar demonstrating an internal test harness they use to test EDM solutions. Markit committed to releasing the tool immediately and releasing source code sometime this quarter.
This test harness has been under development and internal use for years at Markit. Although admittedly never intended for use by a wide audience, it is quite capable, and it will be valuable for any data management teams looking to strengthen their testing.
Basic testing procedures—manual or automated—include setting up the test scenario, performing the action to be tested, and validating the results against a set of known, good results. This is just plain hard to do in enterprise data management and data pipeline scenarios, Markit EDM or otherwise.
Key benefits, as listed by the presenter, include an ability to test any executable process, such as data porters, solutions, and constructors; an ability to manage a library of repeatable tests; and automating test execution. Of particular interest to some organizations will be an ability to measure execution time and compare it against predefined thresholds, as a way to measure performance.
Essentially this is a wrapper around the command line executable that ships with Markit EDM. Tests can be organized in groups and executed individually or as a set within a group. Because the test harness itself can be executed from command line, there is the potential of integrating it into continuous integration setups.
One thing that Markit has done quite well with this is the extensive logging and status around test executions. They demonstrated a UI that functioned as a test status dashboard. This does not come with the test harness, but requires some process to load the test harness log (an XML file) into a database. A UI must be developed that can display the data, but the sample UI seems in line with typical Markit EDM user interfaces.
One requirement that was a bit confusing was the need to start with a package import. Rather than executing tests as the first step, the first step of any execution is importing a package. Based on statements about the test harness wrapping Markit’s command line tools, this is probably based around the Import/Export command line tool. This seems unnecessary and introduces difficulty around dependency management of Markit EDM packages for test environments.
Another soft spot is around configuring tests and managing test executions across environments. This was not included in the demo and was glossed over. Configuration is a mix of XML configuration files and a database, both of which require maintaining by hand. The presenters did not discuss running tests on multiple environments (development, QA, stage, etc). This appears to be handled via the configuration files, which will become difficult for a test library of any size.
To address issues such as these, Markit announced releasing the source code “in an open source fashion” sometime this quarter. This generated much enthusiasm. I hope and urge Markit to embrace open source philosophy and allow all their customers to collaborate and contribute. If they do, this tool could become very powerful in a short amount of time. However, if all they do is expose the source code, then progress will be slow, and this test harness may never be enterprise capable.
So props to Markit for listening to their customer base and releasing this tool. The webinar had hundreds of attendees, so clearly this is a pain point for many data management teams and data governance organizations. The test harness is a valuable addition to an already rich product, and I hope it lives up to its potential.
There are some pros and cons here.
On the plus side, Markit has delivered a game-changing capability. There are a small number of third-party tools (hqTest among them) that enable rich unit testing of MEDM data pipelines. Those of us who have used them will attest to the acceleration of delivery and improvements in code quality made possible by true Test-Driven Development. Also, if Markit delivers on their open-source promise, we can expect rapid and significant improvements in the Test Harness feature set as MEDM clients pool their efforts and begin hacking up the source code. Preliminary efforts in this direction were already well underway as of Friday morning. And you can’t beat the price.
Even so: a note of caution. An MEDM data pipeline is useless until it is tied into downstream systems. Consequently, the testing effort around an MEDM project must take into account these integration points as well as the target systems themselves, which are generally ALSO hard to test. Markit has taken a giant step toward solving the testing problem as it exists within the MEDM system boundary, but my sense of the tool is that it is so deeply tied to the MEDM product that no amount of open-source hackery will coax it so test beyond that bright line. So for the rest of the system diagram, the hunt for an effective testing tool continues.
The Holy Grail, of course, is a single dashboard that will provide developers, testers, and project management with status at a glance, along with the ability to invoke tests at any level, against any related system or component, by whatever method makes the best contextual sense, whether that means punching a button on a UI or running a command as part of a continuous integration script. Its narrow focus on MEDM will likely prevent Markit’s Test Harness from becoming that tool, but it is highly likely that the Test Harness in some future state will be comfortably and efficiently invoked BY that tool, whether it is hqTest or some other product.
So a giant step in the right direction, but still a few more to go.