|
|
@@ -1,4 +1,4 @@
|
|
1
|
|
-# 《A lightweight distributed task scheduling framework. "XXL-JOB"》
|
|
|
1
|
+# 《Distributed task scheduling framework XXL-JOB》
|
|
2
|
2
|
|
|
3
|
3
|
[](https://travis-ci.org/xuxueli/xxl-job)
|
|
4
|
4
|
[](https://maven-badges.herokuapp.com/maven-central/com.xuxueli/xxl-job/)
|
|
|
@@ -41,8 +41,9 @@ In 2015 - November, XXL-JOB finally RELEASE the first big version of V1.0, then
|
|
41
|
41
|
In 2015 - December, I will XXL-JOB published to our internal knowledge base, and get internal colleagues recognized.
|
|
42
|
42
|
In 2016 - 01 months, my company started XXL-JOB internal access and custom work, in this thank Yuan and Yin two colleagues contribution, but also to thank the internal other attention and support colleagues.
|
|
43
|
43
|
In 2017-05-13, the link of "let the code run" in "[the 62nd source of open source China Genesis](https://www.oschina.net/event/2236961)" held in Shanghai,, I stepped on and made a speech about the XXL-JOB, five hundred spectators in the audience reacted enthusiastically ([pictorial review](https://www.oschina.net/question/2686220_2242120)).
|
|
44
|
|
-#### My company have access to XXL-JOB, internal alias "Ferrari" (Ferrari based on XXL-JOB V1.1 version customization, new access application recommended to upgrade the latest version).
|
|
|
44
|
+> Our company have access to XXL-JOB, internal alias "Ferrari" (Ferrari based on XXL-JOB V1.1 version customization, new access application recommended to upgrade the latest version).
|
|
45
|
45
|
According to the latest statistics, from 2016-01-21 to 2017-07-07 period, the system has been scheduled about 600,000 times, outstanding performance. New access applications recommend the latest version, because after several major updates, the system's task model, UI interaction model and the underlying scheduling communication model has a greater optimization and upgrading, the core function more stable and efficient.
|
|
|
46
|
+
|
|
46
|
47
|
So far, XXL-JOB has access to a number of companies online product line, access to scenes such as electronic commerce, O2O business and large data operations, as of 2016-07-19, XXL-JOB has access to the company But not limited to:
|
|
47
|
48
|
|
|
48
|
49
|
- 1、大众点评；
|
|
|
@@ -82,11 +83,17 @@ So far, XXL-JOB has access to a number of companies online product line, access
|
|
82
|
83
|
- 35、北京诺亦腾科技有限公司
|
|
83
|
84
|
- 36、增长引擎(北京)信息技术有限公司
|
|
84
|
85
|
- ……
|
|
85
|
|
-
|
|
|
86
|
+
|
|
|
87
|
+> The company that access and use this product is welcome to register at the [address](https://github.com/xuxueli/xxl-job/issues/1 ), only for product promotion.
|
|
|
88
|
+
|
|
86
|
89
|
Welcome everyone's attention and use, XXL-JOB will also embrace changes, sustainable development.
|
|
87
|
90
|
|
|
88
|
91
|
### 1.4 Download
|
|
89
|
92
|
|
|
|
93
|
+#### Documentation
|
|
|
94
|
+- [中文文档](http://www.xuxueli.com/xxl-job/)
|
|
|
95
|
+- [English Documentation](http://www.xuxueli.com/xxl-job/en/)
|
|
|
96
|
+
|
|
90
|
97
|
#### Source repository address (The latest code will be released in the two git warehouse in the same time)
|
|
91
|
98
|
|
|
92
|
99
|
Source repository address | Release Download
|
|
|
@@ -104,19 +111,15 @@ Source repository address | Release Download
|
|
104
|
111
|
</dependency>
|
|
105
|
112
|
```
|
|
106
|
113
|
|
|
107
|
|
-#### Blog address
|
|
108
|
|
-
|
|
109
|
|
-- [oschina address](http://my.oschina.net/xuxueli/blog/690978)
|
|
110
|
|
-- [cnblogs address](http://www.cnblogs.com/xuxueli/p/5021979.html)
|
|
111
|
|
-
|
|
112
|
114
|
#### Technical exchange group (technical exchange only)
|
|
113
|
115
|
|
|
114
|
|
-- Group 6：399758605
|
|
115
|
|
-- Group 5：138274130 （Group is full, please add group 6）
|
|
116
|
|
-- Group 4：464762661 （Group is full, please add group 6）
|
|
117
|
|
-- Group 3：242151780 （Group is full, please add group 6）
|
|
118
|
|
-- Group 2：438249535 （Group is full, please add group 6）
|
|
119
|
|
-- Group 1：367260654 （Group is full, please add group 6）
|
|
|
116
|
+- Tecent QQ Group 6：399758605
|
|
|
117
|
+- Tecent QQ Group 5：138274130
|
|
|
118
|
+- Tecent QQ Group 4：464762661
|
|
|
119
|
+- Tecent QQ Group 3：242151780
|
|
|
120
|
+- Tecent QQ Group 2：438249535
|
|
|
121
|
+- Tecent QQ Group 1：367260654
|
|
|
122
|
+- [Gitter](https://gitter.im/xuxueli/xxl-job)
|
|
120
|
123
|
|
|
121
|
124
|
### 1.5 Environment
|
|
122
|
125
|
- JDK：1.7+
|
|
|
@@ -399,7 +402,7 @@ Actually it is a python script fragment.
|
|
399
|
402
|
|
|
400
|
403
|
|
|
401
|
404
|
## 4. Task Management
|
|
402
|
|
-#### 4.0 configure executor
|
|
|
405
|
+### 4.0 configure executor
|
|
403
|
406
|
click"执行器管理" on the left menu,it will go to the page as shown below:
|
|
404
|
407
|
|
|
405
|
408
|
|
|
|
@@ -409,7 +412,7 @@ click"执行器管理" on the left menu,it will go to the page as shown below:
|
|
409
|
412
|
If you want to create a new executor,please click "+新增执行器" button:
|
|
410
|
413
|
|
|
411
|
414
|
|
|
412
|
|
-#### Description of executor attributes
|
|
|
415
|
+### Description of executor attributes
|
|
413
|
416
|
|
|
414
|
417
|
AppName: the unique identity of the executor cluster,executor will registe automatically and periodically by appName so that it can be scheduled.
|
|
415
|
418
|
名称: the name of ther executor,it is used to describe the executor.
|
|
|
@@ -419,29 +422,29 @@ If you want to create a new executor,please click "+新增执行器" button:
|
|
419
|
422
|
手动录入:fill in executor address manually and it will be used by schedule center, multiple address separated by commas.
|
|
420
|
423
|
机器地址:only effective when "注册方式" is "手动录入",support fill in executor address manually.
|
|
421
|
424
|
|
|
422
|
|
-#### 4.1 create new task
|
|
|
425
|
+### 4.1 create new task
|
|
423
|
426
|
Go to task management list page,click “新增任务” button on the upper right corner，on the pop-up window“新增任务”page configure task property and save.learn more info please go and see "3,task details".
|
|
424
|
427
|
|
|
425
|
|
-#### 4.2 edit task
|
|
|
428
|
+### 4.2 edit task
|
|
426
|
429
|
Go to task management list page and choose the task you want to edit ,click”编辑”button on the right side of the task,on the pop-up window “编辑任务”page edit task property and save.
|
|
427
|
430
|
|
|
428
|
|
-#### 4.3 edit GLUE source code
|
|
|
431
|
+### 4.3 edit GLUE source code
|
|
429
|
432
|
|
|
430
|
433
|
Only fit to GLUE task.
|
|
431
|
434
|
|
|
432
|
435
|
choose the task you want to edit and click” GLUE”button on the right side of the task, it will go to the Web IDE page of GLUE task,then you can edit task source code on this page.you can read "3.2 GLUE模式(Java)" for more info.
|
|
433
|
436
|
|
|
434
|
|
-#### 4.4 pause/recover task
|
|
|
437
|
+### 4.4 pause/recover task
|
|
435
|
438
|
You can pause or recover task but it just fit to follow up schedule trigger and won’t affect scheduled tasks,if you want to stop tasks which has been triggered,please go and see “4.8 stop the running task”
|
|
436
|
439
|
|
|
437
|
440
|
|
|
438
|
441
|
|
|
439
|
|
-#### 4.5 manually trigger
|
|
|
442
|
+### 4.5 manually trigger
|
|
440
|
443
|
You can trigger a task manually by Click “执行”button,it won’t affect original scheduling rules.
|
|
441
|
444
|
|
|
442
|
445
|
|
|
443
|
446
|
|
|
444
|
|
-#### 4.6 view schedule log
|
|
|
447
|
+### 4.6 view schedule log
|
|
445
|
448
|
You can view task’s history schedule log by click “日志” button,on the history schedule log list page you can view every time of task’s schedule result,execution result and so on,click “执行日志” button can view the task’s full execute log.
|
|
446
|
449
|
|
|
447
|
450
|
|
|
|
@@ -461,12 +464,12 @@ You can view task’s history schedule log by click “日志” button,on the h
|
|
461
|
464
|
"执行日志"button:click this button you can view task’s execution detail log,go and see chapter 4.7 “view execution log” for more info;
|
|
462
|
465
|
"终止任务"button:click this button you can stop the task’s execution thread on this executor,include bloked task instance which didn’t has started;
|
|
463
|
466
|
|
|
464
|
|
-#### 4.7 view execution log
|
|
|
467
|
+### 4.7 view execution log
|
|
465
|
468
|
Click the “执行日志” button on the right side of the record,you can go to the execution log page,you can view the full execution log of the logic business code, shown as below:
|
|
466
|
469
|
|
|
467
|
470
|
|
|
468
|
471
|
|
|
469
|
|
-#### 4.8 stop running tasks
|
|
|
472
|
+### 4.8 stop running tasks
|
|
470
|
473
|
Just fit to running tasks,on the task log list page,click “终止任务” button on the right side of the record, it will send stop command to the executor where the task was executed,finally the task was killed and the task instance execute queue of this task will be clear.
|
|
471
|
474
|
|
|
472
|
475
|
|
|
|
@@ -488,26 +491,26 @@ try{
|
|
488
|
491
|
If JobHandler start child thread,child thread also must not catch InterruptedException,and it should throw exception.
|
|
489
|
492
|
|
|
490
|
493
|
|
|
491
|
|
-#### 4.9 delete execution log
|
|
|
494
|
+### 4.9 delete execution log
|
|
492
|
495
|
On the task log list page, after you select executor and task, you can click"删除" button on the right side and it will pop-up "日志清理" window,on the pop-up window you can choose different log delete policy,choose the policy you want to execute and click "确定" button it will delele relative logs:
|
|
493
|
496
|
|
|
494
|
497
|
|
|
495
|
498
|
|
|
496
|
499
|
|
|
497
|
|
-#### 4.10 delete task
|
|
|
500
|
+### 4.10 delete task
|
|
498
|
501
|
Click the delete button on the right side of the task,the task will be deteted.
|
|
499
|
502
|
|
|
500
|
503
|
|
|
501
|
504
|
|
|
502
|
505
|
## 5. Overall design
|
|
503
|
|
-#### 5.1 Source directory introduction
|
|
|
506
|
+### 5.1 Source directory introduction
|
|
504
|
507
|
- /doc :documentation and material
|
|
505
|
508
|
- /db :db scripts
|
|
506
|
509
|
- /xxl-job-admin :schedule and admin center
|
|
507
|
510
|
- /xxl-job-core :common core Jar
|
|
508
|
511
|
- /xxl-job-executor-samples :executor，Demo project（you can develop on this demo project or adjust your own exist project to executor project)
|
|
509
|
512
|
|
|
510
|
|
-#### 5.2 configure database
|
|
|
513
|
+### 5.2 configure database
|
|
511
|
514
|
XXL-JOB schedule module is implemented based on Quartz cluster,it’s “database” is extended based on Quartz’s 11 mysql tables.
|
|
512
|
515
|
|
|
513
|
516
|
XXL-JOB custom Quartz table structure prefix(XXL_JOB_QRTZ_).
|
|
|
@@ -523,15 +526,15 @@ The added tables as shown below:
|
|
523
|
526
|
|
|
524
|
527
|
So XXL-JOB database total has 16 tables.
|
|
525
|
528
|
|
|
526
|
|
-#### 5.3 Architecture design
|
|
527
|
|
-##### 5.3.1 Design target
|
|
|
529
|
+### 5.3 Architecture design
|
|
|
530
|
+#### 5.3.1 Design target
|
|
528
|
531
|
All schedule behavior has been abstracted into “schedule center” common platform , it dosen’t include business logic and just responsible for starting schedule requests.
|
|
529
|
532
|
|
|
530
|
533
|
All tasks was abstracted into separate JobHandler and was managed by executors, executor is responsible for receiving schedule request and execute the relative JobHandler business.
|
|
531
|
534
|
|
|
532
|
535
|
So schedule and task can be decoupled from each other, by the way it can improve the overall stability and scalability of the system.
|
|
533
|
536
|
|
|
534
|
|
-##### 5.3.2 System composition
|
|
|
537
|
+#### 5.3.2 System composition
|
|
535
|
538
|
- **Schedule module（schedule center）**:
|
|
536
|
539
|
it is responsible for manage schedule info,send schedule request accord task configuration and it is not include an business code.schedule system decouple with the task, improve the overall stability and scalability of the system, at the same time schedule system performance is no longer limited to task modules.
|
|
537
|
540
|
Support visualization, simple and dynamic management schedule information, include create,update,delete, GLUE develop and task alarm and so on, All of the above operations will take effect in real time，support monitor schedule result and execution log and executor failover.
|
|
|
@@ -539,24 +542,24 @@ So schedule and task can be decoupled from each other, by the way it can improve
|
|
539
|
542
|
it is responsible for receive schedule request and execute task logic,task module focuses on the execution of the task, Development and maintenance is simpler and more efficient.
|
|
540
|
543
|
Receive execution request, end request and log request from schedule center.
|
|
541
|
544
|
|
|
542
|
|
-##### 5.3.3 Architecture diagram
|
|
|
545
|
+#### 5.3.3 Architecture diagram
|
|
543
|
546
|
|
|
544
|
547
|
|
|
545
|
548
|
|
|
546
|
|
-#### 5.4 Schedule module analysis
|
|
547
|
|
-##### 5.4.1 Disadvantage of quartz
|
|
|
549
|
+### 5.4 Schedule module analysis
|
|
|
550
|
+#### 5.4.1 Disadvantage of quartz
|
|
548
|
551
|
Quartz is a good open source project and was often as the first choice for job schedule.Tasks was managed by api in quartz cluster so it can avoid some disadvantages of single quartz instance,but it also has some disadvantage as shown below:
|
|
549
|
552
|
- problem 1:it is not humane while operate task by call apill.
|
|
550
|
553
|
- problem 2:it is need to store business QuartzJobBean into database, System Invasion is quite serious.
|
|
551
|
554
|
- problem 3:schedule logic and couple with QuartzJobBean in the same project,it will lead a problem in case that if schedule tasks gradually increased and task logic gradually increased,under this situation the performance of the schedule system will be greatly limited by business.
|
|
552
|
555
|
XXL-JOB solve above problems of quartz.
|
|
553
|
556
|
|
|
554
|
|
-##### 5.4.2 RemoteHttpJobBean
|
|
|
557
|
+#### 5.4.2 RemoteHttpJobBean
|
|
555
|
558
|
Under Quartz develop,task logic often was maintained by QuartzJobBean, couple is very serious.in XXL-JOB"Schedule module" and "task module" are completely decoupled,all scheduled tasks in schedule module use the same QuartzJobBean called RemoteHttpJobBean.the params of the tasks was maintained in the extended tables,when trigger RemoteHttpJobBean,it will parse different params and start remote cal l and it wil call relative remote executor.
|
|
556
|
559
|
|
|
557
|
560
|
This call module is like RPC,RemoteHttpJobBean provide call proxy functionality,the executor is provided as remote service.
|
|
558
|
561
|
|
|
559
|
|
-##### 5.4.3 Schedule Center HA（Cluster）
|
|
|
562
|
+#### 5.4.3 Schedule Center HA（Cluster）
|
|
560
|
563
|
It is based on Quartz cluster，databse use Mysql；while QUARTZ task schedule is used in Clustered Distributed Concurrent Environment,all nodes will report task info and store into database.it will fetch trigger from database while execute task,if trigger name and execute time is the same only one node will execute the task.
|
|
561
|
564
|
|
|
562
|
565
|
```
|
|
|
@@ -568,7 +571,7 @@ org.quartz.jobStore.isClustered: true
|
|
568
|
571
|
org.quartz.jobStore.clusterCheckinInterval: 1000
|
|
569
|
572
|
```
|
|
570
|
573
|
|
|
571
|
|
-##### 5.4.4 Schedule threadpool
|
|
|
574
|
+#### 5.4.4 Schedule threadpool
|
|
572
|
575
|
Default threads in the threadpool is 10 so it can avoid task schedule delay because of single thread block.
|
|
573
|
576
|
|
|
574
|
577
|
```
|
|
|
@@ -582,12 +585,12 @@ business logic was executed on remote executor in XXL-JOB,schedule center just s
|
|
582
|
585
|
|
|
583
|
586
|
the logic of task in XXL-JOB schedule center is very light and single job average run time alaways under 100ms,（most is network time consume）.so it can use limited threads to support a large mount of job run concurrently, 10 threads configured above can support at least 100 JOB normal execution.
|
|
584
|
587
|
|
|
585
|
|
-##### 5.4.5 @DisallowConcurrentExecution
|
|
|
588
|
+#### 5.4.5 @DisallowConcurrentExecution
|
|
586
|
589
|
This annotation is not used default by the schedule center of XXL-JOB schedule module, it use concurrent policy default,because RemoteHttpJobBean is common QuartzJobBean,so it greatly improve the capacity of schedule system and decrease the blocked chance of schedule module in the case of multi-threaded schedule.
|
|
587
|
590
|
|
|
588
|
591
|
Every schedule module was scheduled and executed parallel in XXL-JOB,but tasks in executor is executed serially and support stop task.
|
|
589
|
592
|
|
|
590
|
|
-##### 5.4.6 misfire
|
|
|
593
|
+#### 5.4.6 misfire
|
|
591
|
594
|
The handle policy when miss the job’s trigger time.
|
|
592
|
595
|
he reason may be:restart service,schedule thread was blocked by QuartzJobBean, threads was exhausted,some task enable @DisallowConcurrentExecution，the last schedule was blocked and next schedule was missed.
|
|
593
|
596
|
|
|
|
@@ -608,7 +611,7 @@ CronScheduleBuilder cronScheduleBuilder = CronScheduleBuilder.cronSchedule(jobIn
|
|
608
|
611
|
CronTrigger cronTrigger = TriggerBuilder.newTrigger().withIdentity(triggerKey).withSchedule(cronScheduleBuilder).build();
|
|
609
|
612
|
```
|
|
610
|
613
|
|
|
611
|
|
-##### 5.4.7 log callback service
|
|
|
614
|
+#### 5.4.7 log callback service
|
|
612
|
615
|
When schedule center of the schedule module was deployed as web service, on one side it play as schedule center, on the other side it also provide api service for executor.
|
|
613
|
616
|
|
|
614
|
617
|
The source code location of schedule center’s “log callback api service” as shown below:
|
|
|
@@ -618,7 +621,7 @@ xxl-job-admin#com.xxl.job.admin.controller.JobApiController.callback
|
|
618
|
621
|
|
|
619
|
622
|
Executor will execute task when it receive task execute request.it will notify the task execute result to schedule center when the task is done.
|
|
620
|
623
|
|
|
621
|
|
-##### 5.4.8 task HA（Failover）
|
|
|
624
|
+#### 5.4.8 task HA（Failover）
|
|
622
|
625
|
If executor project was deployed as cluster schedule center will known all online executor nodes,such as:“127.0.0.1:9997, 127.0.0.1:9998, 127.0.0.1:9999”.
|
|
623
|
626
|
|
|
624
|
627
|
When "路由策略" select "故障转移(FAILOVER)",it will send heart beat check request in order while schedule center start schedule request. The first alive checked executor node will be selected and send schedule request to it.
|
|
|
@@ -629,7 +632,7 @@ When "路由策略" select "故障转移(FAILOVER)",it will send heart beat chec
|
|
629
|
632
|
|
|
630
|
633
|
“调度备注” will display local schedule route path、executor’s "注册方式"、"地址列表" and task’s "路由策略"。Under "故障转移(FAILOVER)" policy, schedule center take first address to do heartbeat detection, heat beat fail will automatically skip, the second address heart beat fail…… until the third address “127.0.0.1:9999” heart beat success, it was selected as target executor, then send schedule request to target executor, now the schedule process is end wait for the executor’s callback execution result.
|
|
631
|
634
|
|
|
632
|
|
-##### 5.4.9 schedule log
|
|
|
635
|
+#### 5.4.9 schedule log
|
|
633
|
636
|
Every time when task was scheduled in the schedule center it will record a task log, the task log include three part as shown below:
|
|
634
|
637
|
|
|
635
|
638
|
- 任务信息:include executor address、JobHandler and executor params，accord these parameters it can locate specific machine and task code that the task will be executed.
|
|
|
@@ -650,7 +653,7 @@ Schedule log stands fo single task schedule, attribute description is as follows
|
|
650
|
653
|
- 执行备注:task execute remark info in the executor,such as exception log.
|
|
651
|
654
|
- 执行日志:full execution log of the business code during execution of the task,go and see “4.7 view execution log”.
|
|
652
|
655
|
|
|
653
|
|
-##### 5.4.10 Task dependency
|
|
|
656
|
+#### 5.4.10 Task dependency
|
|
654
|
657
|
principle:every task has a task key in XXL-JOB, every task can configure property “child task Key”,it can match task dependency relationship through task key.
|
|
655
|
658
|
|
|
656
|
659
|
When parent task end execute and success, it will match child task dependency accord child task key, it will trigger child task execute once if it matched child task.
|
|
|
@@ -661,12 +664,12 @@ On the task log page ,you can see matched child task and triggered child task’
|
|
661
|
664
|
|
|
662
|
665
|
|
|
663
|
666
|
|
|
664
|
|
-#### 5.5 Task "run mode" analysis
|
|
665
|
|
-##### 5.5.1 "Bean模式" task
|
|
|
667
|
+### 5.5 Task "run mode" analysis
|
|
|
668
|
+#### 5.5.1 "Bean模式" task
|
|
666
|
669
|
Development steps:go and see "chapter 3" .
|
|
667
|
670
|
principle: every Bean mode task is a Spring Bean instance and it is maintained in executor project’s Spring container. task class nedd to add “@JobHander(value="name")” annotation, because executor identify task bean instance in spring container through annotation. Task class nedd to implements interface IJobHandler, task logic code in method execute(), the task logic in execute() method will be executed when executor received a schedule request from schedule center.
|
|
668
|
671
|
|
|
669
|
|
-##### 5.5.2 "GLUE模式(Java)" task
|
|
|
672
|
+#### 5.5.2 "GLUE模式(Java)" task
|
|
670
|
673
|
Development steps:go and see "chapter 3" .
|
|
671
|
674
|
Principle : every "GLUE模式(Java)" task code is a class implemets interface IJobHandler, when executor received schedule request from schedule center these code will be loaded by Groovy classloader and instantiate into a Java object and inject spring bean service declared in this code at the same time（please confirm service and class reference in Glue code exist in executor project）, then call the object’s execute() method and execute task logic.
|
|
672
|
675
|
|
|
|
@@ -680,7 +683,7 @@ All supported types of scripts as shown beloes:
|
|
680
|
683
|
- python script: python script task will be enabled when select " GLUE模式(Python)"as task run mode.
|
|
681
|
684
|
|
|
682
|
685
|
|
|
683
|
|
-##### 5.5.4 executor
|
|
|
686
|
+#### 5.5.4 executor
|
|
684
|
687
|
Executor is actually an embedded Jetty server with default port 9999, as shown below（parameter:xxl.job.executor.port）.
|
|
685
|
688
|
|
|
686
|
689
|
|
|
|
@@ -689,7 +692,7 @@ Executor will identify Bean mode task in spring container through @JobHander Whe
|
|
689
|
692
|
|
|
690
|
693
|
When executor received schedule request from schedule center, if task type is “Bean模式” it will match bean mode task in Spring container and call it’s execute() method and execute task logic. if task type is “GLUE模式”, it will load Glue code, instantiate a Java object and inject other spring service（notice: the spring service injected in Glue code must exist in the same executor project）, then call execute() method and execute task logic.
|
|
691
|
694
|
|
|
692
|
|
-##### 5.5.5 task log
|
|
|
695
|
+#### 5.5.5 task log
|
|
693
|
696
|
XXL-JOB will generate a log file for every schedule request, the log info will be recorded by XxlJobLogger.log() method, the log file will be loaded when view log info through schedule center.
|
|
694
|
697
|
|
|
695
|
698
|
(history version is implemented by overriding LOG4J’s Appender so it exists dependency restrictions, The way has been discraded in the new version)
|
|
|
@@ -698,17 +701,17 @@ The location of log file can be specified in executor configuration file, defaul
|
|
698
|
701
|
|
|
699
|
702
|
When start child thread in JobHandler, child thread will print log in parent JobHandler thread’s execute log in order to trace execute log.
|
|
700
|
703
|
|
|
701
|
|
-#### 5.6 Communication module analysis
|
|
|
704
|
+### 5.6 Communication module analysis
|
|
702
|
705
|
|
|
703
|
|
-##### 5.6.1 A complete task schedule communication process
|
|
|
706
|
+#### 5.6.1 A complete task schedule communication process
|
|
704
|
707
|
- 1,schedule center send http request to executor, and the service in executor in fact is a jetty server with default port 9999.
|
|
705
|
708
|
- 2,executor execute task logic.
|
|
706
|
709
|
- 3,executor http callback with schedule center for schedule result, the service in schedule center used to receive callback request from executor is a set of api opended to executor.
|
|
707
|
710
|
|
|
708
|
|
-##### 5.6.2 Encrypt Communication data
|
|
|
711
|
+#### 5.6.2 Encrypt Communication data
|
|
709
|
712
|
When scheduler center send request to executor, it will use RequestModel and ResponseModel object to encapsulate schedule request parameters and response data, these two object will be serialized before communication, data protocol and time stamp will be checked so that achieve data encryption target.
|
|
710
|
713
|
|
|
711
|
|
-#### 5.7 task register and task auto discover
|
|
|
714
|
+### 5.7 task register and task auto discover
|
|
712
|
715
|
Task executor machine property has been canceled from v1.5, instead of task register and auto discovery, get remote machine address dynamic.
|
|
713
|
716
|
|
|
714
|
717
|
AppName: unique identify of executor cluster, executor is minimal unite of task register, every task recognize machine addresses under the executor on which it was binded.
|
|
|
@@ -717,10 +720,10 @@ Task executor machine property has been canceled from v1.5, instead of task regi
|
|
717
|
720
|
|
|
718
|
721
|
To ensure system lightweight and reduce learning costs, it did not use Zookeeper as register center, Use DB as register center to do task registration.
|
|
719
|
722
|
|
|
720
|
|
-#### 5.8 task execute result
|
|
|
723
|
+### 5.8 task execute result
|
|
721
|
724
|
Since v1.6.2, the task execute result is recognized through ReturnT of IJobHandler, it executes success when return value meets the condition "ReturnT.code == ReturnT.SUCCESS_CODE" , or it executes fail, and it can callback error message info to schedule center through ReturnT.msg, so it can control task execute results in the task logic.
|
|
722
|
725
|
|
|
723
|
|
-#### 5.9 slice broadcat & dynamic slice
|
|
|
726
|
+### 5.9 slice broadcat & dynamic slice
|
|
724
|
727
|
When “分片广播” is selected as route policy in executor cluster, one task schedule will broadcast all executor node in cluster to trigger task execute in every executor, pass slice parameter at the same time, so we can develop slice task by slice parameters.
|
|
725
|
728
|
|
|
726
|
729
|
"分片广播" break the task by the dimensions of executor, support dynamic extend executor cluster so that it can add slice number dynamically to do business process, In case of large amount of data process can significantly improve task processing capacity and speed.
|
|
|
@@ -738,7 +741,7 @@ This feature applies to scenes as shown below:
|
|
738
|
741
|
- 1、slice task scene:when 10 executor to handle 10w records, 1w records need to be handled per machine, time-consuming 10 times lower；
|
|
739
|
742
|
- 2、Broadcast task scene:broadcast all cluster nodes to execute shell script、broadcast all cluster nodes to update cache.
|
|
740
|
743
|
|
|
741
|
|
-#### 5.10 AccessToken
|
|
|
744
|
+### 5.10 AccessToken
|
|
742
|
745
|
To improve system security it is need to check security between schedule center and executor, just allow communication between them when AccessToken of each other matched.
|
|
743
|
746
|
|
|
744
|
747
|
The AccessToken of scheduler center and executor can be configured by xxl.job.accessToken.
|
|
|
@@ -750,7 +753,7 @@ There are only two settings when communication between scheduler center and exec
|
|
750
|
753
|
|
|
751
|
754
|
|
|
752
|
755
|
## 6 Version update log
|
|
753
|
|
-#### 6.1 version V1.1.x，New features [2015-12-05]
|
|
|
756
|
+### 6.1 version V1.1.x，New features [2015-12-05]
|
|
754
|
757
|
**【since V1.1.x，XXL-JOB was used by company hiring me，alias Ferrari inner company，the latest version is recommended for new project】**
|
|
755
|
758
|
- 1、simple:support CRUD operation through Web page, simple and one minute to get started;
|
|
756
|
759
|
- 2、dynamic:support dynamic update task status,pause/recover task and effective in real time;
|
|
|
@@ -761,7 +764,7 @@ There are only two settings when communication between scheduler center and exec
|
|
761
|
764
|
- 7、support for custom parameters;
|
|
762
|
765
|
- 8、Support pause task execution remotely .
|
|
763
|
766
|
|
|
764
|
|
-#### 6.2 version V1.2.x，New features [2016-01-17]
|
|
|
767
|
+### 6.2 version V1.2.x，New features [2016-01-17]
|
|
765
|
768
|
- 1、support task group;
|
|
766
|
769
|
- 2、suport local task, remote task;
|
|
767
|
770
|
- 3、support two types underlying communication ,Servlet or JETTY;
|
|
|
@@ -779,7 +782,7 @@ There are only two settings when communication between scheduler center and exec
|
|
779
|
782
|
- High scalability;
|
|
780
|
783
|
- stability;
|
|
781
|
784
|
|
|
782
|
|
-#### 6.3 version V1.3.0，New features [2016-05-19]
|
|
|
785
|
+### 6.3 version V1.3.0，New features [2016-05-19]
|
|
783
|
786
|
- 1、discard local task module, remote task was recommended, easy to decouple system, the JobHander of task was called executor.
|
|
784
|
787
|
- 2、dicard underlying communication type servlet, JETTY was recommended, schedule and callback bidirectional communication, rebuild the communication logic;
|
|
785
|
788
|
- 3、UI interactive optimization:optimize left menu expansion and menu item selected status , task list opens the table with compression optimization;
|
|
|
@@ -789,7 +792,7 @@ There are only two settings when communication between scheduler center and exec
|
|
789
|
792
|
- BEAN mode executor:every executor is a Spring Bean instance，it was recognized and scheduled by XXL-JOB through @JobHander annotation;
|
|
790
|
793
|
-GLUE mode executor:every executor corresponds to a piece of code，edited and maintained online by Web, Dynamic compile and takes effect in real time, executor is responsible for loading GLUE code and executing;
|
|
791
|
794
|
|
|
792
|
|
-#### 6.4 version V1.3.1，New features [2016-05-23]
|
|
|
795
|
+### 6.4 version V1.3.1，New features [2016-05-23]
|
|
793
|
796
|
- 1、Update project directory structure:
|
|
794
|
797
|
- /xxl-job-admin -------------------- 【schedule center】:Responsible for managing schedule information，send schedule request according to schedule configuration;
|
|
795
|
798
|
- /xxl-job-core ----------------------- Public core dependence
|
|
|
@@ -799,12 +802,12 @@ There are only two settings when communication between scheduler center and exec
|
|
799
|
802
|
- 2、Upgrade the user manual under the new directory structure;
|
|
800
|
803
|
- 3、Optimize some interactions and UI;
|
|
801
|
804
|
|
|
802
|
|
-#### 6.5 version V1.3.2，New features [2016-05-28]
|
|
|
805
|
+### 6.5 version V1.3.2，New features [2016-05-28]
|
|
803
|
806
|
- 1、Schedule logic for transactional handle;
|
|
804
|
807
|
- 2、executor asynchronous callback execution log;
|
|
805
|
808
|
- 3、【important】based on HA support of schedule center，extend executor’s Failover support，Support configure multiple execution addresses;
|
|
806
|
809
|
|
|
807
|
|
-#### 6.6 version V1.4.0 New features [2016-07-24]
|
|
|
810
|
+### 6.6 version V1.4.0 New features [2016-07-24]
|
|
808
|
811
|
- 1、Task dependency: it is implemented by trigger event, it will automatically trigger a child task schedule after Task execute success and callback, multiple child tasks are separated by commas;
|
|
809
|
812
|
- 2、executor source code has been reconstructed, optimize underlying db script;
|
|
810
|
813
|
- 3、optimize task thread group logic of executor, before it is group by executor’s JobHandler so when multiple task reuse Jobhanlder will cause block with each other. Now it is grouped by task of schedule center so tasks are isolated from task execution.
|
|
|
@@ -823,7 +826,7 @@ There are only two settings when communication between scheduler center and exec
|
|
823
|
826
|
|
|
824
|
827
|
Tips: V1.3.x release has been published , enter the maintenance phase, branch address is [V1.3](https://github.com/xuxueli/xxl-job/tree/v1.3) .New features will be updated continuously in the master branch.
|
|
825
|
828
|
|
|
826
|
|
-#### 6.7 version V1.4.1 New features [2016-09-06]
|
|
|
829
|
+### 6.7 version V1.4.1 New features [2016-09-06]
|
|
827
|
830
|
- 1、project successfully pushed to maven central warehouse, Central warehouse address and dependency as shown below:
|
|
828
|
831
|
```
|
|
829
|
832
|
<!-- http://repo1.maven.org/maven2/com/xuxueli/xxl-job-core/ -->
|
|
|
@@ -841,14 +844,14 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
841
|
844
|
- 7、optimize table structure and the table name now is upper case;
|
|
842
|
845
|
- 8、modify ContentType of JSON response from exception handler of schedule center to fix the bug that it is could not recognized by browser.
|
|
843
|
846
|
|
|
844
|
|
-#### 6.8 version V1.4.2 New features [2016-09-29]
|
|
|
847
|
+### 6.8 version V1.4.2 New features [2016-09-29]
|
|
845
|
848
|
- 1、push V1.4.2 to maven central warehouse, main version V1.4 enter maintenance phase;
|
|
846
|
849
|
- 2、fix problem task list offset when add task;
|
|
847
|
850
|
- 3、fix a style disorder problem that caused by bootstrap does not support the modal frame overlap , the problem occurs when the task is edited;
|
|
848
|
851
|
- 4、optimize schedule status when schedule timeout and Handler could not matched;
|
|
849
|
852
|
- 5、the task could not stop problem caused by catch exception has given solution;
|
|
850
|
853
|
|
|
851
|
|
-#### 6.9 version V1.5.0 New features [2016-11-13]
|
|
|
854
|
+### 6.9 version V1.5.0 New features [2016-11-13]
|
|
852
|
855
|
- 1、task register: executor registers the task automatically, schedule center will automatically discover the registered task and trigger execution.
|
|
853
|
856
|
- 2、add parameter AppName for executor: AppName is the unique identifier of each executor cluster, register periodically and automatically with AppName.
|
|
854
|
857
|
- 3、add column executor management in schedule center : manage online executors, automatically discover registered executors via the property AppName。Only managed executors are allowed to be used;
|
|
|
@@ -858,14 +861,14 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
858
|
861
|
- 7、adjust table structure, reconstruct the project;
|
|
859
|
862
|
- 8、schedule center automatically registered and found, failover: schedule center periodically registered automatically, task callback can recognize all online schedule center addresses, task callback support failover so that it can avoid single point of risk.
|
|
860
|
863
|
|
|
861
|
|
-#### 6.10 version V1.5.1 New features [2016-11-13]
|
|
|
864
|
+### 6.10 version V1.5.1 New features [2016-11-13]
|
|
862
|
865
|
- 1、Reconstruct the underlying code and optimize logic, clean POM and Clean Code;
|
|
863
|
866
|
- 2、Servlet/JSP Spec selected 3.0/2.2;
|
|
864
|
867
|
- 3、Spring updated to 3.2.17.RELEASE version;
|
|
865
|
868
|
- 4、Jetty updated to version 8.2.0.v20160908;
|
|
866
|
869
|
- 5、has push V1.5.0 and V1.5.1 to maven central warehouse;
|
|
867
|
870
|
|
|
868
|
|
-#### 6.10 version V1.5.2 New features [2017-02-28]
|
|
|
871
|
+### 6.10 version V1.5.2 New features [2017-02-28]
|
|
869
|
872
|
- 1、optimize IP tools class which used to gets IP address，IP static cache;
|
|
870
|
873
|
- 2、both executor and schedule center support customize registered IP address;Solve problem when machine has multiple network card and get the wrong card;
|
|
871
|
874
|
- 3、solve the problem that it will generate multiple log files when executed across days;
|
|
|
@@ -875,7 +878,7 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
875
|
878
|
- 7、reconstruct underlying code and optimize logic and Clean Code;
|
|
876
|
879
|
- 8、optimize Dependency Injection Logic of GLUE, support injected as alias;
|
|
877
|
880
|
|
|
878
|
|
-#### 6.11 version V1.6.0 New features [2017-03-13]
|
|
|
881
|
+### 6.11 version V1.6.0 New features [2017-03-13]
|
|
879
|
882
|
- 1、upgrade communication scheme，the HEX communication model is adjusted to the B-RPC model based on HTTP;
|
|
880
|
883
|
- 2、executor supports set execution address list manually，provide switch to use automatically registered address or manually set address;
|
|
881
|
884
|
- 3、executor route rules:第一个、最后一个、轮询、随机、一致性HASH、最不经常使用、最近最久未使用、故障转移;
|
|
|
@@ -886,20 +889,20 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
886
|
889
|
- 8、new created task defaults to a non-running state;
|
|
887
|
890
|
- 9、optimize update logic of GLUE mode task instance , The original update is based on the timeout value and now is updated according to the version number，version number plus one while source changed;
|
|
888
|
891
|
|
|
889
|
|
-#### 6.12 version V1.6.1 New features [2017-03-25]
|
|
|
892
|
+### 6.12 version V1.6.1 New features [2017-03-25]
|
|
890
|
893
|
- 1、Rolling log;
|
|
891
|
894
|
- 2、reconstruct WebIDE interactive;
|
|
892
|
895
|
- 3、enhanced communication check，filter unnormal requests effectively;
|
|
893
|
896
|
- 4、enhanced permission check，Using dynamic login TOKEN（recommend instead of internal SSO）;
|
|
894
|
897
|
- 5、optimize database configuration，solve garbled problem;
|
|
895
|
898
|
|
|
896
|
|
-#### 6.13 version V1.6.2 New features [2017-04-25]
|
|
|
899
|
+### 6.13 version V1.6.2 New features [2017-04-25]
|
|
897
|
900
|
- 1、execution report:support view run time data in real time, such as task number, total schedule number, executor number etc., include schedule report , such as scheduled distribution graph on date, scheduled success distribution graph etc;
|
|
898
|
901
|
- 2、JobHandler support set return value for tasks, it is easy to control task execute result in task logic;
|
|
899
|
902
|
- 3、the problem could not view exception info when resource path include space or chinese word casused resource file could not be loaded;
|
|
900
|
903
|
- 4、optimize route policy:fix problems that Loop and LFU routing policy counters are no limit and first route is focused on the first machine;
|
|
901
|
904
|
|
|
902
|
|
-#### 6.14 version V1.7.0 New features [2017-05-02]
|
|
|
905
|
+### 6.14 version V1.7.0 New features [2017-05-02]
|
|
903
|
906
|
- 1、script task:support develop and run script task by GLUE, include script type such as Shell、Python and Groovy;
|
|
904
|
907
|
- 2、add spring-boot type executor example project;
|
|
905
|
908
|
- 3、upgrade jetty to version 9.2;
|
|
|
@@ -907,7 +910,7 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
907
|
910
|
- 5、executor remove GlueLoader dependency，instead of push mode，thus GLUE source code load no longer rely on JDBC;
|
|
908
|
911
|
- 6、get the project name when login and redirect, solve 404 problem when it is not deployed by the directory;
|
|
909
|
912
|
|
|
910
|
|
-#### 6.15 version V1.7.1 New features [2017-05-08]
|
|
|
913
|
+### 6.15 version V1.7.1 New features [2017-05-08]
|
|
911
|
914
|
- 1、unified write and read code of execute log as UTF-8，solve log garbled problem under windows environment;
|
|
912
|
915
|
- 2、communication timeout period is limited to 10s，To avoid schedule thread is occupied under abnormal situation;
|
|
913
|
916
|
- 3、adjust executor , server stat, destroy and register logic;
|
|
|
@@ -917,7 +920,7 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
917
|
920
|
- 7、pop-up component is replaced by layer;
|
|
918
|
921
|
- 8、upgrade quartz to version 2.3.0;
|
|
919
|
922
|
|
|
920
|
|
-#### 6.16 version V1.7.2 New features [2017-05-17]
|
|
|
923
|
+### 6.16 version V1.7.2 New features [2017-05-17]
|
|
921
|
924
|
- 1、block handle policy:the policy when schedule is too frequently and the executor it too late to handle, include multiple strategies:single machine serially execute（default）、discard subsequent schedule、override before schedule;
|
|
922
|
925
|
- 2、fail handle policy:handle policy when scheduled fail, include :failure alarm（default）、failed to retry;
|
|
923
|
926
|
- 3、The communication timeout is adjusted to 180s;
|
|
|
@@ -925,7 +928,7 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
925
|
928
|
- 5、fix executor parameters lost bug when edit;
|
|
926
|
929
|
- 6、add task test Demo to make task logic test easier;
|
|
927
|
930
|
|
|
928
|
|
-#### 6.17 version V1.8.0 New features [2017-07-17]
|
|
|
931
|
+### 6.17 version V1.8.0 New features [2017-07-17]
|
|
929
|
932
|
- 1、optimize update logic of task Cron，instead of rescheduleJob，at the same time preventing set cron repeatedly;
|
|
930
|
933
|
- 2、optimize API callback service failed status code，facilitate troubleshooting;
|
|
931
|
934
|
- 3、XxlJobLogger support multi-parameter;
|
|
|
@@ -938,7 +941,7 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
938
|
941
|
- 10、supports filter execute log based on running status;
|
|
939
|
942
|
- 11、optimize scheduling Center Task Registration Detection Logic;
|
|
940
|
943
|
|
|
941
|
|
-#### 6.18 version V1.8.1 New features [2017-07-30]
|
|
|
944
|
+### 6.18 version V1.8.1 New features [2017-07-30]
|
|
942
|
945
|
- 1、slice broadcast task:When slice broadcast is selected as route policy in executor cluster, one task schedule will broadcast all executor node in cluster to trigger task execute in every executor, pass slice parameter at the same time, so we can develop slice task by slice parameters;
|
|
943
|
946
|
- 2、dynamic slice: break the task by the dimensions of executor, support dynamic extend executor cluster so that it can add slice number dynamically to do business process, In case of large amount of data process can significantly improve task processing capacity and speed;
|
|
944
|
947
|
- 3、executor JobHandler disables name conflicts;
|
|
|
@@ -951,12 +954,12 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
951
|
954
|
- 10、upgrade springboot version to 1.5.6.RELEASE of executor;
|
|
952
|
955
|
- 11、unify maven version dependency management;
|
|
953
|
956
|
|
|
954
|
|
-#### 6.19 version V1.8.2 New features[Coding]
|
|
|
957
|
+### 6.19 version V1.8.2 New features[Coding]
|
|
955
|
958
|
- 1,support configuring the HTTPS for executor callback URL;
|
|
956
|
959
|
- 2,Standardize project directory for extend multi executors;
|
|
957
|
960
|
- 3,add JFinal type executor sample project;
|
|
958
|
961
|
|
|
959
|
|
-#### TODO LIST
|
|
|
962
|
+### TODO LIST
|
|
960
|
963
|
- 1,Task privilege management:control privilege on executor, check privilege on core operations;
|
|
961
|
964
|
- 2,Task slice routing:using consistent Hash algorithm to calculate slice order as stable as possible, even if there is fluctuation in the registration machine will not cause large fluctuations in the order of slice. Currently using IP natural sorting can meet the demand，to be determined;
|
|
962
|
965
|
- 3,Failure retry optimization:The current failure to retry logic is execute the request logic once again after the scheduled request fails。The optimization point is retry for both scheduling and execution failures, retry a full schedule when retrying，This may lead schedule failure to an infinite loop，to be determined;
|
|
|
@@ -970,22 +973,22 @@ Tips: V1.3.x release has been published , enter the maintenance phase, branch a
|
|
970
|
973
|
|
|
971
|
974
|
## 7. Other
|
|
972
|
975
|
|
|
973
|
|
-#### 7.1 report problem
|
|
974
|
|
-XXL-JOB project is on Github，If you have any questions, you can ask questions at [ISSUES](https://github.com/xuxueli/xxl-job/issues/) You can also join the above technical exchange group;
|
|
|
976
|
+### 7.1 Contributing
|
|
|
977
|
+Contributions are welcome! Open a pull request to fix a bug, or open an [Issue](https://github.com/xuxueli/xxl-job/issues/) to discuss a new feature or change.
|
|
975
|
978
|
|
|
976
|
|
-#### 7.2 used records（record just for spread，Product is open source and free of charge）
|
|
|
979
|
+### 7.2 used records（record just for spread，Product is open source and free of charge）
|
|
977
|
980
|
Record for spread product and product is free and open source.
|
|
978
|
981
|
Welcome to [check in](https://github.com/xuxueli/xxl-job/issues/1 )on github.
|
|
979
|
982
|
|
|
980
|
|
-#### 7.3 Open source protocol and version
|
|
981
|
|
-Product is open source and free of charge，will provide free community technical support continuously,Personal or company internal free access and use.
|
|
|
983
|
+### 7.3 Copyright and License
|
|
|
984
|
+This product is open source and free, and will continue to provide free community technical support. Individual or enterprise users are free to access and use.
|
|
982
|
985
|
|
|
983
|
|
-XXL-JOB uses GPLv3 protocol to ensure the user's right of free use. The agreement avoids special risk of patent applications "the GPL assures that patents cannot be used to render the program non-free.（From GPLv3）".
|
|
984
|
|
-Copyright (c) 2015-present, xuxueli.
|
|
|
986
|
+- Licensed under the GNU General Public License (GPL) v3.
|
|
|
987
|
+- Copyright (c) 2015-present, xuxueli.
|
|
985
|
988
|
|
|
986
|
989
|
---
|
|
987
|
|
-#### donation
|
|
988
|
|
-scan it if want to support project，reward a cup of coffee to author:）
|
|
|
990
|
+### Donate
|
|
|
991
|
+To support the author, donate the author to buy a cup of coffee：）
|
|
989
|
992
|
|
|
990
|
993
|
webchat:
|
|
991
|
994
|
Alipay:
|
xuxueli