Back to Volume
Paper: A Rule-Based Data Quality Startup Using PyCLIPS
Volume: 394, Astronomical Data Analysis Software and Systems (ADASS) XVII
Page: 723
Authors: DuPlain, R.F.; Radziwill, N.M.; Shelton, A.L.
Abstract: A rule-based approach to data quality provides for efficient and extensible solutions in validating data sets. A working prototype proves that CLIPS, a trusted rules engine, can integrate with existing data processing libraries through PyCLIPS, resulting in a system which isolates data quality rules from programming logic to allow for parallel development and maintenance of rules and applications. The prototype demonstrates several benefits: developers can treat rules independently from application source code, a ruleset can accept new rules without modification of existing rules in the set, rules can provide value through conditional assertions which call on external tools, and even a very small rule set can point to real errors in telescope data. A rules engine which automates data quality validation with existing tool libraries could potentially lead to a substantial increase in operational availability even when faced with resource scarcity.
Back to Volume