by Jason Williscroft
Imagine an institution several years into a large Data Management implementation. Their architecture is fairly standard, comprising the following layers:
- An Application Layer hosts the Data Pipeline, which ingests data from multiple sources; cleanses, matches, and masters it; and sends it on to target systems.
- A Storage Layer maintains historical data to support Application Layer logic and audit.
- A Data Access Layer (DAL) serves as an interface between the Application Layer and the rest of the enterprise. In this case, the DAL is the Application Layer’s only target.
Various Data Quality (DQ) issues cause exceptions to be thrown in the Application Layer, which generally must be handled by human operators. Business requirements state that when certain kinds of exceptions exist on a mastered entity, the entire entity—even the good parts—must be masked from visibility in the DAL.
To support this requirement, developers create an extension to the entity master table: a new table, with a 1/0:1 relationship to the master table, whose purpose is to hold DQ measures related to the mastered entity. Developers choose to use an extension table, rather than expand the entity master table, because DQ measures about a mastered entity are semantically distinct from the mastered entity they describe. Also, they want to be able to add new DQ measures without touching mastered data: both great applications of the Open-Closed Principle. But that’s for another article.
Entity data is mastered in the Application Layer and then delivered into the DAL. DQ measures are calculated in the Application Layer… but since this is new functionality, the only use case for exploiting them is the suppression described above, and the logic driving this suppression executes in the DAL. Whereas mastered data changes slowly, DQ measures can change very quickly as operators address and close exceptions. Thus, a row of mastered data in the DAL might be suppressed during the morning DAL update, but by noon the status of exceptions in the Application Layer might have changed such that—if updated DQ measures were available in the DAL—suppression on the mastered data might be lifted.
So the development team has some choices:
- Allow the DAL to query the Application Layer database directly. This would make DQ changes available to the DAL in real time.
- Use database replication to mirror changes in the Application Layer extension table to its DAL equivalent. This would create a small but manageable delay in the availability of DQ changes to the DAL.
- Move the contents of the DQ extension table into the DAL using the same machinery that transforms and moves mastered data. This is the option with by far the highest intrinsic latency.
There are good arguments in favor of each option… so how to break the stalemate?
Our analysis is rooted in the Dependency Inversion Principle, which states that it is better to depend on abstractions than on concretions. In plainer language, if there is a logical dependency between two subsystems A and B, it is usually better to decouple them by making both depend on some abstract interface C rather than on each other.
In our example, downstream systems query mastered data from the DAL. If the DAL acquired this mastered data via method 1 or 2 above, then any change in mastered data structure or content in the Application Layer would immediately be reflected in the DAL, with potentially disastrous consequences for downstream systems expecting to see mastered data in a specific format. To avoid this kind of coupling, there is a set of processes that move mastered data from the Application Layer to the DAL while guaranteeing compatibility. This Translation Layer serves as a layer of abstraction—an interface—that isolates the Application Layer from the DAL and protects each from the consequences of changes or breakage in the other.
The Translation Layer is an application of the Dependency Inversion Principle. In the case of these DQ measures, the argument on the development team is whether the benefits of bypassing the translation layer outweigh the risks.
Our answer: no way!
First there is the argument from general principle. Modular systems are more robust than tightly coupled ones. While the proposed coupling schemes presented a number of specific risks (described below), violating Dependency Inversion also makes a system less able to respond to future conditions that nobody has even anticipated. Meanwhile, doing the extra work to maintain Dependency Inversion across the board maximizes the extensibility and maintainability of any complex system. Given that changing requirements is the norm in Data Management, these are desirable characteristics.
Then there are specific objections:
- If the DAL queries the Application Layer directly, then any casualty to the Application Layer database will effectively take the DAL offline as its queries fail.
- If the DAL queries the Application Layer directly, then performance in the DAL will be subject to query latency between the two databases. Currently both databases reside on the same server. Let them wind up on separate servers—or separate continents—and performance will suffer.
- In both the direct query and the database replication scenario, any change to the schema of the Application Layer DQ table will very likely cause breakage in the DAL and its downstream systems.
Each of these objections is typical of what happens when Dependency Inversion is violated: deep dependency of one subsystem on the internal workings of another produces breakage when anything changes on either side. Development teams that routinely ignore Dependency Inversion quickly create such a tangled web of dependency that it can become nearly impossible to change anything without breaking everything, forcing system-wide refactors to accommodate even minor changes in code.
In the end, our recommendation would be to leverage the existing Translation Layer pattern in order to move DQ data from the Application Layer to be exploited in the DAL: option 3 above. Latency issues are mitigated by running the DQ upload several times during the course of the day, and the Application Layer and the DAL are each insulated from breakage, defects, or changes in the other.