1 Star 0 Fork 9

Roc / dal-job

forked from 快鸟 / dal-job 
Create your Gitee Account
Explore and code with more than 12 million developers,Free private repositories !:)
Sign up
Clone or Download
contribute
Sync branch
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README
Apache-2.0

dal-job(Distributed As Local Job)

简介

dal-job是一个去中心化的轻量的分布式Job框架。它没有master结点,代码是在各个模块上运行的。
帮助开发人员在分布式环境下开发job时,只用关注业务,而不用去关心job被重复执行的问题。 它主要提供如下功能:

    1. 分布式环境下,同一时刻只有一台机器执行job。
    1. 记录job的运行日志,包括运行记录,耗时,异常日志等。(job_log)
    1. 记录系统下面的job信息。(job_info)
    1. 提供内置的重试job,对于需要重试的场景使用。包含:事前重试 和 事后重试
    1. dal-job也可以指定多台实例同时运行,但是需要在job的逻辑中保证数据处理不重不漏。
      参考方案:取数时可以使用随机生成的全局唯一key,去取数进行处理

问题思考
在分布式环境下,本地去中心化的分布式job需要解决的问题:

    1. 我们面对的是可能是一台实例上多线程的环境,需要保证在多线程环境下只有一个线程能执行job。
      (这种情况比较少,job是受自己模块管理的,一般一台机器一个job只会有一个线程在拉起执行)
    1. 我们面对的一定是多进程环境,需要保证在同一时刻多个机器的实例的进程,只有一个进程能执行job
    1. 分布式环境下,多个机器实例的系统时间可能不一致,导致job的运行不在同一时刻,我们需要保证这种情况下,只有一台机器对正确执行job

原理

使用数据库的行级锁来保证同一时刻只有一台机在执行任务。
具体:使用 【悲观锁 + JobStatusCheck + TimeLimit】 实现在多线程与多进程(主要是多进程)环境下,一个job在运行过程中,只会有一台机在执行job

技术

使用Quartz + MySQL。同时与Spring友好融合。提供注解(@TimedTask)形式的job配置

Quick Start

定时调度

dal-job支持分布式环境下单台启动 和 多台启动。其中多台启动会在所有的实例上运行,需要自己解决取数问题。
dal-job提供了注解形式的job配置,具体可以参考com.kvn.dal.core.single_node.SingleNodeJob.java

@Target({ TYPE })
@Retention(RUNTIME)
public @interface TimedTask {
    String corn();
    boolean isGlobalSingle() default true; // 分布式环境下,是否单台启动
    String desc() default "";
}

定时调度例子:

@TimedTask(corn = "0 0/1 * * * ?", desc = "测试job222")
@Service
public class MyTestJob2 implements ExecutableTask {
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        System.out.println(DateTime.now() + "--" + Thread.currentThread().getName() + "---------------doBizJob2222--------");
        try {
			Thread.sleep(3000L);
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
        if(new Random().nextInt() % 2 == 0){
        	throw new RuntimeException("biz执行MyTestJob2异常,xxxxxxxxx");
        }
    }
}

重试调度

dal-job提供了内置的重试调度实现,可以方便的对异常数据进行定时重试。
重试分为两种:一是,事前重试;二是,事后重试

事前重试

事前重试,即不管业务是否执行成功,都去记录执行日志(表:job_beforehand_retry),如果出现指定的异常,则标记记录为需要重试。待重试job执行时,就分发至相应的重试方法去执行。
**原理:**使用aop的方式,对需要重试的方法(含有@BeforehandRetry的方法)进行拦截
@BeforehandRetry:

/**
 * 事前补偿,确保每次执行业务时都有留底。会牺牲一性的性能。
 * @author wzy
 * @date 2017年7月14日 下午5:03:45
 */
@Target({ METHOD })
@Retention(RUNTIME)
@Inherited
public @interface BeforehandRetry {
	/**
	 * 执行重试的异常,默认是对BizRetryNeedException才去执行重试逻辑。业务异常是不需要重试的!!!
	 */
	Class<? extends Throwable> retryFor() default BizRetryNeedException.class;
	/**
	 * 最大重试次数
	 */
	int maxRetryCount() default 3;
}

**例子参考:**com.kvn.dal.core.beforehand_retry.BeforehandRetryBizService.java

@Service
public class BeforehandRetryBizService {
	@BeforehandRetry
	public String doBiz(Foo foo, String param){
		System.out.println("--->isRetryThread:" + ThreadContext.getContext().isRetryThread());
		System.out.println("参数:Foo=" + JSON.toJSONString(foo) + ", param=" + param);
		System.out.println("执行业务失败>>>>>>>>");
		throw new BizRetryNeedException("业务失败,需要重试!!!");
	}
}

事后重试

事后重试,即执行业务出现异常后,对于我们需要重试的异常,将重试参数持久化到DB(表:job_retry),然后通过事后重试调度定时发起重试。
对于需要重试的类,可以通过实现 IRetrySupport 接口,或者继承 AbstractRetrySupport 类。
IRetrySupport.java

public interface IRetrySupport {
	/**
	 * 重试
	 * @param retryContext 重试上下文
	 * @return 返回重试结果:true | false
	 */
	Boolean retry(AfterwardRetryContext retryContext);
}

例子参考:实现接口方式com.kvn.dal.core.afterward_retry.AfterwardRetryBizService.java

@Service
public class AfterwardRetryBizService implements IRetrySupport {
    @Resource
    IJobRetryDao jobRetryDao;
    public void executeBiz() {
        System.out.println(DateTime.now() + "--" + Thread.currentThread().getName() + "---------------doBizJob2222--------");
        try {
            Thread.sleep(3000L);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        Foo foo = new Foo(1001, "xxx");
        try {
            throw new RuntimeException("executeBiz异常,xxxxxxxxx");
        } catch (Exception e) {
            ArrayList<RetryParam> retryLs = new RetryParamListWrapper().buildRetryParam(foo).buildRetryParam("xxx").buildRetryParam("hehehe").toArrayList();
            JobRetry retry = JobRetry.createJobRetry(this.getClass(), "key001", retryLs);
            jobRetryDao.add(retry);
            throw e;
        }
    }
    @Override
    public Boolean retry(AfterwardRetryContext retryContext) {
    	/** 实现重试逻辑 */
        String retryDataKey = retryContext.getRetryDataKey();
        List<RetryParam> paramLs = retryContext.getRetryParamLs();
        Foo foo = paramLs.get(0).retoreParam(Foo.class);
        String originParam1 = paramLs.get(1).retoreParam(String.class);
        String originParam2 = paramLs.get(2).retoreParam(String.class);
        // 或者
        Foo foo2 = retryContext.getRetryParamValueMap().get(Foo.class).get(0);
        String originParam_1 = retryContext.getRetryParamValueMap().get(String.class).get(0);
        String originParam_2 = retryContext.getRetryParamValueMap().get(String.class).get(1);
        return true;
    }
}

例子参考:继承类的方式com.kvn.dal.core.afterward_retry.AfterwardRetryBestPracticeService.java

@Service
public class AfterwardRetryBestPracticeService extends AbstractRetrySupport {
	public void executeBiz() {
		System.out.println(DateTime.now() + "--" + Thread.currentThread().getName() + "---------------doBizJob2222--------");
		try {
			Thread.sleep(3000L);
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
		Foo foo = new Foo(1001, "xxx");
		try {
			throw new RuntimeException("executeBiz异常,xxxxxxxxx");
		} catch (Exception e) {
			this.retryEnqueue("key001", foo, "hehe", "morning");
			throw e; // 出异常后,终止业务
		}
	}
	@Override
	public Boolean retry(AfterwardRetryContext retryContext) {
		// 实现重试逻辑
		return true;
	}
}

计划

    1. 后面可以支持job管理界面,对job进行实时监控和管理
    1. 重试数据序列化的方式进行扩展,支持JSON、Protostuff、Kryo等
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2017 Kevin-Wang001 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

distributed-as-local job,轻量分布式调度框架 expand collapse
Java
Apache-2.0
Cancel

Releases

No release

Contributors

All

Activities

Load More
can not load any more
Java
1
https://gitee.com/dllen/dal-job.git
git@gitee.com:dllen/dal-job.git
dllen
dal-job
dal-job
master

Search