同步操作将从 WeBank/Qualitis 强制同步,此操作会覆盖自 Fork 仓库以来所做的任何修改,且无法恢复!!!
确定后同步将在后台操作,完成时将刷新页面,请耐心等待。
Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing.
Based on Spring Boot, Qualitis analysis quality model by Linkis. It provides functions such as data quality model construction, timing execution, trigger execution, data quality verification, reports of data quality generation and so on.
At the same time, Qualitis provides enterprise-level features of financial-level resource isolation, management, access control and collaborative edition. It is also guaranteed working well under high-concurrency, high-performance and high-availability scenarios.
Type in username and password.
After login, click on the "Rule Configuration" button on the left menu.
In the secondary menu below, click on "Project" button to enter the project module.
Then, click on the "New Project" button on the upper left corner, and you will enter the "New Project Page".
Enter the information below
The name of project, unique.
You can create a new project by clicking "ok".
After creating project, you should then create rules. The buttons in the lower left corner, can create single table rules, custom rules and multi-table rules.
First, you can click on "New Single Table Rule" button to create a new rule.
Then, type in the information below:
The name of rule.
Qualitis System provides various template to simplify rule configuration.
Select the template you want to verify.
Here you can select the data source you want to verify. When you select the data source, it will replace the placeholder in the SQL below.
Tips:
You can type in the "system expression" in input box of "Filter". When executing task, it will be replaced by the actual date.
Now we provide the following system expressions:
Expression | Execute Time | Replace Value |
---|---|---|
${yyyyMMdd}[-N(Option)] | Dec 17 18:00:00 2018 | 20181217(N day before) |
${yyyy-MM-dd}[-N(Option)] | Dec 17 18:00:00 2018 | 2018-12-17(N day before) |
${yyyyMMddHH}[-N(Option)] | Dec 17 18:00:00 2018 | 2018121715(N day before) only minus by day |
${yyyy/MM/dd}[-N(Option)] | Dec 17 18:00:00 2018 | 2018/12/17(N day before) |
If not checked, Qualitis will not monitor the application result, which means that Qualitis don't know if the result is pass of not.
If checked, you should select the result to monitor and set the threshold to make the verification failed. When the result exceed the threshold, the status of this application will become "Not Pass".
Qualitis system provides the following four methods to compare between result and threshold.
We assume the threshold you set is 'x' and the result of this application is 'r'.
1.Monthly Fluctuation:Compare this application result with the average value of this rule this month, the application will pass the verification if and only if the expression become true: (1-x)*y<=r<=(1+x)*y, and failed if false.
2.Weekly Fluctuation:The same as the Monthly Fluctuations, the value to compare is the average value of this week.
3.Daily Fluctuation:The same as the Monthly Fluctuations, the value to compare is the average value of this day.
4.Fixed Value:Compare this application result with the fixed value. You can choose "Equal", "Greater than" and other compare method to compare. The application will failed when x = r if the compare type you choose is "Equal".
As shown in the picture above, it is a rule of null value verification.
When the result of application is 0, the status of application will become "Not Pass".
Click on the button of "New Custom Rule" to create custom rules.
Type in the following information below:
The name of rule.
The alias name to check
Select function of SQL
The cluster to submit applications
If checked, it will take the data that not pass verification out and save it.Otherwise, it will not be saved.
The SQL to execute will be shown below.
Click on the button of "New Multi-table Rule" to create multi-table rules.
You can select two tables in the same cluster in multi-table rule to compare the differences between them.
Now Qualitis provides two template below:
Here is an example of accuracy verification:
Compose there is a table A, and a table B. The data in table is shown below:
Table A
ColumnA1 | ColumnA2 |
---|---|
a | 1 |
b | 2 |
ColumnB1 | ColumnB2 |
---|---|
a | 1 |
b | 3 |
Here is the step:
1.Type in basic information
2.Select the table to compare
3.Select the column to compare
Application can be run in two way, in the way of project or rule.
In the way of project to run the application, it will collect all the rules in the project. Here is an example.
And you can also run application in the way of rule. As the picture given below:
Click on the "Application" button in the left menu to enter the application page.
You can look up the application in this page, and click on one application to view the detail of application.
And you can also click on the status of application to get the verification result of application.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。