diff --git a/README.md b/README.md index 3224378..ad0c8ff 100644 --- a/README.md +++ b/README.md @@ -1,297 +1,100 @@ -UidGenerator +uid-generator-spring-boot-starter ========================== -[In Chinese 中文版](README.zh_cn.md) - -UidGenerator is a Java implemented, [Snowflake](https://github.com/twitter/snowflake) based unique ID generator. It -works as a component, and allows users to override workId bits and initialization strategy. As a result, it is much more -suitable for virtualization environment, such as [docker](https://www.docker.com/). Besides these, it overcomes -concurrency limitation of Snowflake algorithm by consuming future time; parallels UID produce and consume by caching -UID with RingBuffer; eliminates CacheLine pseudo sharing, which comes from RingBuffer, via padding. And finally, it -can offer over 6 million QPS per single instance. - -Requires：[Java8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)+, -[MySQL](https://dev.mysql.com/downloads/mysql/)(Default implement as WorkerID assigner; If there are other implements, MySQL is not required) - -Snowflake -------------- -![Snowflake](doc/snowflake.png) -** Snowflake algorithm：** An unique id consists of worker node, timestamp and sequence within that timestamp. Usually, -it is a 64 bits number(long), and the default bits of that three fields are as follows: - -* sign(1bit) - The highest bit is always 0. - -* delta seconds (28 bits) - The next 28 bits, represents delta seconds since a customer epoch(2016-05-20). The maximum time will be 8.7 years. - -* worker id (22 bits) - The next 22 bits, represents the worker node id, maximum value will be 4.2 million. UidGenerator uses a build-in - database based ```worker id assigner``` when startup by default, and it will dispose previous work node id after - reboot. Other strategy such like 'reuse' is coming soon. - -* sequence (13 bits) - the last 13 bits, represents sequence within the one second, maximum is 8192 per second by default. - -**The parameters above can be configured in spring bean** - - -CachedUidGenerator -------------------- -RingBuffer is an array，each item of that array is called 'slot', every slot keeps a uid or a flag(Double RingBuffer). -The size of RingBuffer is 2^n, where n is positive integer and equal or greater than bits of -```sequence```. Assign bigger value to ```boostPower``` if you want to enlarge RingBuffer to improve throughput. - -###### Tail & Cursor pointer -* Tail Pointer - - Represents the latest produced UID. If it catches up with cursor, the ring buffer will be full, at that moment, no put - operation should be allowed, you can specify a policy to handle it by assigning - property ```rejectedPutBufferHandler```. - -* Cursor Pointer - - Represents the latest already consumed UID. If cursor catches up with tail, the ring buffer will be empty, and - any take operation will be rejected. you can also specify a policy to handle it by assigning - property ```rejectedTakeBufferHandler```. - -![RingBuffer](doc/ringbuffer.png) - -CachedUidGenerator used double RingBuffer，one RingBuffer for UID, another for status(if valid for take or put) - -Array can improve performance of reading, due to the CUP cache mechanism. At the same time, it brought the side -effect of 「False Sharing」, in order to solve it, cache line padding is applied. - -![FalseSharing](doc/cacheline_padding.png) - -#### RingBuffer filling -* Initialization padding - During RingBuffer initializing，the entire RingBuffer will be filled. - -* In-time filling - Whenever the percent of available UIDs is less than threshold ```paddingFactor```, the fill task is triggered. You can - reassign that threshold in Spring bean configuration. - -* Periodic filling - Filling periodically in a scheduled thread. The```scheduleInterval``` can be reassigned in Spring bean configuration. - +基于 [百度UidGenerator](https://github.com/baidu/uid-generator), 做了以下改动： +- 改造为spring-boot-starter的形式，不用部署为分布式，直接建表、在项目中引入，即可使用 +- 针对时钟回拨，提供了修正选项（默认启用，可通过配置关闭），小于阈值直接休眠，大于阈值更改机器号 +- 对机器id用尽提供了复用策略：取余 +- 解除id位数限制，由“必须64位”改为“不大于64位”，可根据需要获取更短id + +参数均可通过Spring进行自定义，默认参数为： +- delta seconds (30 bits) +当前时间，相对于时间基点"2019-02-20"的增量值，单位：秒，最多可支持约34年，超出抛异常 +- worker id (16 bits) +机器id，最多可支持约6.5w次机器启动。内置实现为在启动时由数据库分配，默认分配策略为用后即弃，当前复用策略为取余。 +- sequence (7 bits) +每秒下的并发序列，7 bits可支持每秒128个并发，超出128则等待下一秒 + +默认参数下，初始id长度为12，最终随时间增加，最长到16位 Quick Start ------------ -Here we have a demo with 4 steps to introduce how to integrate UidGenerator into Spring based projects.
- -### Step 1: Install Java8, Maven, MySQL -If you have already installed maven, jdk8+ and Mysql or other DB which supported by Mybatis, just skip to next.
-Download [Java8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html), -[MySQL](https://dev.mysql.com/downloads/mysql/) and [Maven](https://maven.apache.org/download.cgi), -and install jdk, mysql. For maven, extracting and setting MAVEN_HOME is enough. -#### Set JAVA_HOME & MAVEN_HOME -Here is a sample script to set JAVA_HOME and MAVEN_HOME -```shell -export MAVEN_HOME=/xxx/xxx/software/maven/apache-maven-3.3.9 -export PATH=$MAVEN_HOME/bin:$PATH -JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home"; -export JAVA_HOME; -``` +这里介绍如何在SpringBoot项目中使用uid-generator-spring-boot-starter, 具体流程如下:
-### Step 2: Create table WORKER_NODE -Replace ```xxxxx``` with real database name, and run following script to create table, +### 步骤1: 创建表WORKER_NODE +在项目数据库里，运行sql脚本以导入表WORKER_NODE, 脚本如下: ```sql -DROP DATABASE IF EXISTS `xxxx`; -CREATE DATABASE `xxxx` ; -use `xxxx`; DROP TABLE IF EXISTS WORKER_NODE; CREATE TABLE WORKER_NODE ( ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name', PORT VARCHAR(64) NOT NULL COMMENT 'port', -TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER', +TYPE INT NOT NULL COMMENT 'node type: CONTAINER(1), ACTUAL(2), FAKE(3)', LAUNCH_DATE DATE NOT NULL COMMENT 'launch date', MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time', CREATED TIMESTAMP NOT NULL COMMENT 'created time', PRIMARY KEY(ID) -) - COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB; +) COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB; ``` +配置好数据库连接 -Reset property of 'jdbc.url', 'jdbc.username' and 'jdbc.password' in [mysql.properties](src/test/resources/uid/mysql.properties). - -### Step 3: Spring configuration -#### DefaultUidGenerator -There are two implements of UidGenerator: [DefaultUidGenerator](src/main/java/com/baidu/fsg/uid/impl/DefaultUidGenerator.java), [CachedUidGenerator](src/main/java/com/baidu/fsg/uid/impl/CachedUidGenerator.java).
-For performance sensitive application, CachedUidGenerator is recommended. - +### 步骤2: Maven引用 +当前项目打包，或从Maven仓库中引入uid-generator-spring-boot-starter包 ```xml - - - - - - - - - - - - - - + + com.github.wujun234 + uid-generator-spring-boot-starter + 1.0.3.RELEASE + ``` +### 步骤3: 开始使用 -#### CachedUidGenerator -Copy beans of CachedUidGenerator to 'test/resources/uid/cached-uid-spring.xml'. -```xml - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` - -#### Mybatis config -[mybatis-spring.xml](src/test/resources/uid/mybatis-spring.xml) shows as below: -```xml - - - - - - - - - - - - - - +UidGenerator接口提供了 UID 生成和解析的方法，提供了两种实现: +- [DefaultUidGenerator](src/main/java/com/github/wujun234/uid/impl/DefaultUidGenerator.java) +实时生成 +- [CachedUidGenerator](src/main/java/com/github/wujun234/uid/impl/CachedUidGenerator.java) +生成一次id之后，按序列号+1生成一批id，缓存，供之后请求 - - - - - - +如对UID生成性能有要求, 请使用CachedUidGenerator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` - -### Step 4: Run UnitTest -Run [CachedUidGeneratorTest](src/test/java/com/baidu/fsg/uid/CachedUidGeneratorTest.java), shows how to generate / parse UniqueID: ```java +//@Resource +//private UidGenerator defaultUidGenerator; + @Resource -private UidGenerator uidGenerator; +private UidGenerator cachedUidGenerator; @Test public void testSerialGenerate() { // Generate UID - long uid = uidGenerator.getUID(); + long uid = cachedUidGenerator.getUID(); // Parse UID into [Timestamp, WorkerId, Sequence] - // {"UID":"180363646902239241","parsed":{ "timestamp":"2017-01-19 12:15:46", "workerId":"4", "sequence":"9" }} - System.out.println(uidGenerator.parseUID(uid)); + // {"UID":"450795408770","timestamp":"2019-02-20 14:55:39","workerId":"27","sequence":"2"} + System.out.println(cachedUidGenerator.parseUID(uid)); } ``` - -### Tips -For low concurrency and long term application, less ```seqBits``` but more ```timeBits``` is recommended. For -example, if DisposableWorkerIdAssigner is adopted and the average reboot frequency is 12 per node per day, with the -configuration ```{"workerBits":23,"timeBits":31,"seqBits":9}```, one project can run for 68 years with 28 nodes -and entirely concurrency 14400 UID/s. - -For frequent reboot and long term application, less ```seqBits``` but more ```timeBits``` and ```workerBits``` is -recommended. For example, if DisposableWorkerIdAssigner is adopted and the average reboot frequency is 24 * 12 per node -per day, with the configuration ```{"workerBits":27,"timeBits":30,"seqBits":6}```, one project can run for 34 years -with 37 nodes and entirely concurrency 2400 UID/s. - -#### Experiment for Throughput -To figure out CachedUidGenerator's UID throughput, some experiments are carried out.
-Firstly, workerBits is arbitrarily fixed to 20, and change timeBits from 25(about 1 year) to 32(about 136 years),
- -|timeBits|25|26|27|28|29|30|31|32| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,831,465|7,007,279|6,679,625|6,499,205|6,534,971|7,617,440|6,186,930|6,364,997| - -![throughput1](doc/throughput1.png) - -Then, timeBits is arbitrarily fixed to 31, and workerBits is changed from 20(about 1 million total reboots) to 29(about - 500 million total reboots),
- -|workerBits|20|21|22|23|24|25|26|27|28|29| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,186,930|6,642,727|6,581,661|6,462,726|6,774,609|6,414,906|6,806,266|6,223,617|6,438,055|6,435,549| - -![throughput2](doc/throughput2.png) - -It is obvious that whatever the configuration is, CachedUidGenerator always has the ability to provide **6 million** -stable throughput, what sacrificed is just life expectancy, this is very cool. - -Finally, both timeBits and workerBits are fixed to 31 and 23 separately, and change the number of CachedUidGenerator -consumer. Since our CPU only has 4 cores, \[1, 8\] is chosen.
- -|consumers|1|2|3|4|5|6|7|8| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,462,726|6,542,259|6,077,717|6,377,958|7,002,410|6,599,113|7,360,934|6,490,969| - -![throughput3](doc/throughput3.png) +### 步骤4: 可选设置 +#### 自定义配置 +以下为可选配置, 如未指定将采用默认值 +```yml +uid: + timeBits: 30 # 时间位, 默认:30 + workerBits: 16 # 机器位, 默认:16 + seqBits: 7 # 序列号, 默认:7 + epochStr: "2019-02-20" # 初始时间, 默认:"2019-02-20" + enableBackward: true # 是否容忍时钟回拨, 默认:true + maxBackwardSeconds: 1 # 时钟回拨最长容忍时间（秒）, 默认:1 + CachedUidGenerator: # CachedUidGenerator相关参数 + boostPower: 3 # RingBuffer size扩容参数, 可提高UID生成的吞吐量, 默认:3 + paddingFactor: 50 # 指定何时向RingBuffer中填充UID, 取值为百分比(0, 100), 默认为50 + #scheduleInterval: 60 # 默认:不配置此项, 即不使用Schedule线程. 如需使用, 请指定Schedule线程时间间隔, 单位:秒 +``` +#### 可选实现 +选用CachedUidGenerator时，可以选择实现“拒绝策略”的拓展 +- 拒绝策略: 当环已满, 无法继续填充时 +默认无需指定, 将丢弃Put操作, 仅日志记录. 如有特殊需求, 请实现RejectedPutBufferHandler接口(支持Lambda表达式) +- 拒绝策略: 当环已空, 无法继续获取时 +默认无需指定, 将记录日志, 并抛出UidGenerateException异常. 如有特殊需求, 请实现RejectedTakeBufferHandler接口(支持Lambda表达式) diff --git a/README.zh_cn.md b/README.zh_cn.md deleted file mode 100644 index 8200c46..0000000 --- a/README.zh_cn.md +++ /dev/null @@ -1,278 +0,0 @@ -UidGenerator -========================== -[In English](README.md) - -UidGenerator是Java实现的, 基于[Snowflake](https://github.com/twitter/snowflake)算法的唯一ID生成器。UidGenerator以组件形式工作在应用项目中, -支持自定义workerId位数和初始化策略, 从而适用于[docker](https://www.docker.com/)等虚拟化环境下实例自动重启、漂移等场景。 -在实现上, UidGenerator通过借用未来时间来解决sequence天然存在的并发限制; 采用RingBuffer来缓存已生成的UID, 并行化UID的生产和消费, -同时对CacheLine补齐，避免了由RingBuffer带来的硬件级「伪共享」问题. 最终单机QPS可达600万。 - -依赖版本：[Java8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)及以上版本, -[MySQL](https://dev.mysql.com/downloads/mysql/)(内置WorkerID分配器, 启动阶段通过DB进行分配; 如自定义实现, 则DB非必选依赖） - -Snowflake算法 -------------- -![Snowflake](doc/snowflake.png) -Snowflake算法描述：指定机器 & 同一时刻 & 某一并发序列，是唯一的。据此可生成一个64 bits的唯一ID（long）。默认采用上图字节分配方式： - -* sign(1bit) - 固定1bit符号标识，即生成的UID为正数。 - -* delta seconds (28 bits) - 当前时间，相对于时间基点"2016-05-20"的增量值，单位：秒，最多可支持约8.7年 - -* worker id (22 bits) - 机器id，最多可支持约420w次机器启动。内置实现为在启动时由数据库分配，默认分配策略为用后即弃，后续可提供复用策略。 - -* sequence (13 bits) - 每秒下的并发序列，13 bits可支持每秒8192个并发。 - -**以上参数均可通过Spring进行自定义** - - -CachedUidGenerator -------------------- -RingBuffer环形数组，数组每个元素成为一个slot。RingBuffer容量，默认为Snowflake算法中sequence最大值，且为2^N。可通过```boostPower```配置进行扩容，以提高RingBuffer -读写吞吐量。 - -Tail指针、Cursor指针用于环形数组上读写slot： - -* Tail指针 - 表示Producer生产的最大序号(此序号从0开始，持续递增)。Tail不能超过Cursor，即生产者不能覆盖未消费的slot。当Tail已赶上curosr，此时可通过```rejectedPutBufferHandler```指定PutRejectPolicy - -* Cursor指针 - 表示Consumer消费到的最小序号(序号序列与Producer序列相同)。Cursor不能超过Tail，即不能消费未生产的slot。当Cursor已赶上tail，此时可通过```rejectedTakeBufferHandler```指定TakeRejectPolicy - -![RingBuffer](doc/ringbuffer.png) - -CachedUidGenerator采用了双RingBuffer，Uid-RingBuffer用于存储Uid、Flag-RingBuffer用于存储Uid状态(是否可填充、是否可消费) - -由于数组元素在内存中是连续分配的，可最大程度利用CPU cache以提升性能。但同时会带来「伪共享」FalseSharing问题，为此在Tail、Cursor指针、Flag-RingBuffer中采用了CacheLine -补齐方式。 - -![FalseSharing](doc/cacheline_padding.png) - -#### RingBuffer填充时机 #### -* 初始化预填充 - RingBuffer初始化时，预先填充满整个RingBuffer. - -* 即时填充 - Take消费时，即时检查剩余可用slot量(```tail``` - ```cursor```)，如小于设定阈值，则补全空闲slots。阈值可通过```paddingFactor```来进行配置，请参考Quick Start中CachedUidGenerator配置 - -* 周期填充 - 通过Schedule线程，定时补全空闲slots。可通过```scheduleInterval```配置，以应用定时填充功能，并指定Schedule时间间隔 - - -Quick Start ------------- - -这里介绍如何在基于Spring的项目中使用UidGenerator, 具体流程如下:
- -### 步骤1: 安装依赖 -先下载[Java8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html), [MySQL](https://dev.mysql.com/downloads/mysql/)和[Maven](https://maven.apache.org/download.cgi) - -#### 设置环境变量 -maven无须安装, 设置好MAVEN_HOME即可. 可像下述脚本这样设置JAVA_HOME和MAVEN_HOME, 如已设置请忽略. -```shell -export MAVEN_HOME=/xxx/xxx/software/maven/apache-maven-3.3.9 -export PATH=$MAVEN_HOME/bin:$PATH -JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home"; -export JAVA_HOME; -``` - -### 步骤2: 创建表WORKER_NODE -运行sql脚本以导入表WORKER_NODE, 脚本如下: -```sql -DROP DATABASE IF EXISTS `xxxx`; -CREATE DATABASE `xxxx` ; -use `xxxx`; -DROP TABLE IF EXISTS WORKER_NODE; -CREATE TABLE WORKER_NODE -( -ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', -HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name', -PORT VARCHAR(64) NOT NULL COMMENT 'port', -TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER', -LAUNCH_DATE DATE NOT NULL COMMENT 'launch date', -MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time', -CREATED TIMESTAMP NOT NULL COMMENT 'created time', -PRIMARY KEY(ID) -) - COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB; -``` - -修改[mysql.properties](src/test/resources/uid/mysql.properties)配置中, jdbc.url, jdbc.username和jdbc.password, 确保库地址, 名称, 端口号, 用户名和密码正确. - -### 步骤3: 修改Spring配置 -提供了两种生成器: [DefaultUidGenerator](src/main/java/com/baidu/fsg/uid/impl/DefaultUidGenerator.java)、[CachedUidGenerator](src/main/java/com/baidu/fsg/uid/impl/CachedUidGenerator.java)。如对UID生成性能有要求, 请使用CachedUidGenerator
-对应Spring配置分别为: [default-uid-spring.xml](src/test/resources/uid/default-uid-spring.xml)、[cached-uid-spring.xml](src/test/resources/uid/cached-uid-spring.xml) - -#### DefaultUidGenerator配置 -```xml - - - - - - - - - - - - - - -``` - -#### CachedUidGenerator配置 -```xml - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` - -#### Mybatis配置 -[mybatis-spring.xml](src/test/resources/uid/mybatis-spring.xml)配置说明如下: - -```xml - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -``` - -### 步骤4: 运行示例单测 -运行单测[CachedUidGeneratorTest](src/test/java/com/baidu/fsg/uid/CachedUidGeneratorTest.java), 展示UID生成、解析等功能 -```java -@Resource -private UidGenerator uidGenerator; - -@Test -public void testSerialGenerate() { - // Generate UID - long uid = uidGenerator.getUID(); - - // Parse UID into [Timestamp, WorkerId, Sequence] - // {"UID":"180363646902239241","parsed":{ "timestamp":"2017-01-19 12:15:46", "workerId":"4", "sequence":"9" }} - System.out.println(uidGenerator.parseUID(uid)); - -} -``` - -### 关于UID比特分配的建议 -对于并发数要求不高、期望长期使用的应用, 可增加```timeBits```位数, 减少```seqBits```位数. 例如节点采取用完即弃的WorkerIdAssigner策略, 重启频率为12次/天, -那么配置成```{"workerBits":23,"timeBits":31,"seqBits":9}```时, 可支持28个节点以整体并发量14400 UID/s的速度持续运行68年. - -对于节点重启频率频繁、期望长期使用的应用, 可增加```workerBits```和```timeBits```位数, 减少```seqBits```位数. 例如节点采取用完即弃的WorkerIdAssigner策略, 重启频率为24*12次/天, -那么配置成```{"workerBits":27,"timeBits":30,"seqBits":6}```时, 可支持37个节点以整体并发量2400 UID/s的速度持续运行34年. - -#### 吞吐量测试 -在MacBook Pro（2.7GHz Intel Core i5, 8G DDR3）上进行了CachedUidGenerator（单实例）的UID吞吐量测试.
-首先固定住workerBits为任选一个值(如20), 分别统计timeBits变化时(如从25至32, 总时长分别对应1年和136年)的吞吐量, 如下表所示:
- -|timeBits|25|26|27|28|29|30|31|32| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,831,465|7,007,279|6,679,625|6,499,205|6,534,971|7,617,440|6,186,930|6,364,997| - -![throughput1](doc/throughput1.png) - -再固定住timeBits为任选一个值(如31), 分别统计workerBits变化时(如从20至29, 总重启次数分别对应1百万和500百万)的吞吐量, 如下表所示:
- -|workerBits|20|21|22|23|24|25|26|27|28|29| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,186,930|6,642,727|6,581,661|6,462,726|6,774,609|6,414,906|6,806,266|6,223,617|6,438,055|6,435,549| - -![throughput1](doc/throughput2.png) - -由此可见, 不管如何配置, CachedUidGenerator总能提供**600万/s**的稳定吞吐量, 只是使用年限会有所减少. 这真的是太棒了. - -最后, 固定住workerBits和timeBits位数(如23和31), 分别统计不同数目(如1至8,本机CPU核数为4)的UID使用者情况下的吞吐量,
- -|workerBits|1|2|3|4|5|6|7|8| -|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| -|throughput|6,462,726|6,542,259|6,077,717|6,377,958|7,002,410|6,599,113|7,360,934|6,490,969| - -![throughput1](doc/throughput3.png) \ No newline at end of file diff --git a/doc/cacheline_padding.png b/doc/cacheline_padding.png deleted file mode 100644 index ed921c9..0000000 Binary files a/doc/cacheline_padding.png and /dev/null differ diff --git a/doc/ringbuffer.png b/doc/ringbuffer.png deleted file mode 100644 index d1a88a2..0000000 Binary files a/doc/ringbuffer.png and /dev/null differ diff --git a/doc/snowflake.png b/doc/snowflake.png deleted file mode 100644 index d9d0890..0000000 Binary files a/doc/snowflake.png and /dev/null differ diff --git a/doc/throughput1.png b/doc/throughput1.png deleted file mode 100644 index 41c7cfd..0000000 Binary files a/doc/throughput1.png and /dev/null differ diff --git a/doc/throughput2.png b/doc/throughput2.png deleted file mode 100644 index 2e89c13..0000000 Binary files a/doc/throughput2.png and /dev/null differ diff --git a/doc/throughput3.png b/doc/throughput3.png deleted file mode 100644 index 1527135..0000000 Binary files a/doc/throughput3.png and /dev/null differ diff --git a/pom.xml b/pom.xml index ceffec5..0407c48 100644 --- a/pom.xml +++ b/pom.xml @@ -3,144 +3,169 @@ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 - UID-Generator - An unique id generator - com.baidu.fsg - uid-generator - 1.0.0-SNAPSHOT + com.github.wujun234 + uid-generator-spring-boot-starter + 1.0.3.RELEASE jar + uid-generator-spring-boot-starter + An unique id generator based on baidu uid-generator + https://github.com/wujun234/uid-generator-spring-boot-starter + UTF-8 1.8 - 4.2.5.RELEASE - 1.7.7 + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + https://github.com/wujun234/uid-generator-spring-boot-starter + https://github.com/wujun234/uid-generator-spring-boot-starter.git + https://github.com/wujun234 + + + + + wujun + wujun234@gmail.com + + + - - - org.springframework - spring-core - ${spring.version} - - - org.springframework - spring-beans - ${spring.version} - - - org.springframework - spring-context - ${spring.version} - + - org.springframework - spring-jdbc - ${spring.version} + org.springframework.boot + spring-boot-autoconfigure - - - org.mybatis - mybatis - 3.2.3 - + - org.mybatis - mybatis-spring - 1.2.4 + org.mybatis.spring.boot + mybatis-spring-boot-starter + 1.3.2 - - commons-collections - commons-collections - 3.2.2 - commons-lang commons-lang 2.6 - - - ch.qos.logback - logback-classic - 1.1.3 - - - org.slf4j - slf4j-api - ${slf4j-version} - - - org.slf4j - log4j-over-slf4j - ${slf4j-version} - - - junit - junit - 4.10 + org.springframework.boot + spring-boot-starter-test test - - org.springframework - spring-test - ${spring.version} - test - - mysql mysql-connector-java - 5.1.18 test - - - com.alibaba - druid - 1.0.19 - test - - + + + + org.springframework.boot + spring-boot-dependencies + 2.1.3.RELEASE + pom + import + + + + + org.apache.maven.plugins maven-compiler-plugin + 3.8.0 ${jdk.version} ${jdk.version} ${project.build.sourceEncoding} - 3.5.1 - - - org.apache.maven.plugins - maven-source-plugin - 2.3 - - - package - - jar - - - - + + + release + + + + + org.apache.maven.plugins + maven-source-plugin + 3.0.1 + + + attach-sources + + jar-no-fork + + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.0.1 + + + none + + + + package + + jar + + + + + + + org.apache.maven.plugins + maven-gpg-plugin + 1.6 + + + verify + + sign + + + + + + + + + oss + https://oss.sonatype.org/content/repositories/snapshots/ + + + oss + https://oss.sonatype.org/service/local/staging/deploy/maven2/ + + + + \ No newline at end of file diff --git a/src/main/java/com/baidu/fsg/uid/BitsAllocator.java b/src/main/java/com/github/wujun234/uid/BitsAllocator.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/BitsAllocator.java rename to src/main/java/com/github/wujun234/uid/BitsAllocator.java index 065e7ca..a7933b3 100644 --- a/src/main/java/com/baidu/fsg/uid/BitsAllocator.java +++ b/src/main/java/com/github/wujun234/uid/BitsAllocator.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid; +package com.github.wujun234.uid; import org.apache.commons.lang.builder.ToStringBuilder; import org.apache.commons.lang.builder.ToStringStyle; @@ -59,7 +59,7 @@ public class BitsAllocator { public BitsAllocator(int timestampBits, int workerIdBits, int sequenceBits) { // make sure allocated 64 bits int allocateTotalBits = signBits + timestampBits + workerIdBits + sequenceBits; - Assert.isTrue(allocateTotalBits == TOTAL_BITS, "allocate not enough 64 bits"); + Assert.isTrue(allocateTotalBits <= TOTAL_BITS, "allocate larger than 64 bits"); // initialize bits this.timestampBits = timestampBits; diff --git a/src/main/java/com/github/wujun234/uid/UidAutoConfigure.java b/src/main/java/com/github/wujun234/uid/UidAutoConfigure.java new file mode 100644 index 0000000..2fc8a09 --- /dev/null +++ b/src/main/java/com/github/wujun234/uid/UidAutoConfigure.java @@ -0,0 +1,51 @@ +package com.github.wujun234.uid; + +import com.github.wujun234.uid.impl.CachedUidGenerator; +import com.github.wujun234.uid.impl.UidProperties; +import com.github.wujun234.uid.worker.DisposableWorkerIdAssigner; +import com.github.wujun234.uid.worker.WorkerIdAssigner; +import com.github.wujun234.uid.impl.DefaultUidGenerator; +import org.mybatis.spring.annotation.MapperScan; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.context.annotation.Lazy; + +/** + * UID 的自动配置 + * + * @author wujun + * @date 2019.02.20 10:57 + */ +@Configuration +@ConditionalOnClass({ DefaultUidGenerator.class, CachedUidGenerator.class }) +@MapperScan({ "com.github.wujun234.uid.worker.dao" }) +@EnableConfigurationProperties(UidProperties.class) +public class UidAutoConfigure { + + @Autowired + private UidProperties uidProperties; + + @Bean + @ConditionalOnMissingBean + @Lazy + DefaultUidGenerator defaultUidGenerator() { + return new DefaultUidGenerator(uidProperties); + } + + @Bean + @ConditionalOnMissingBean + @Lazy + CachedUidGenerator cachedUidGenerator() { + return new CachedUidGenerator(uidProperties); + } + + @Bean + @ConditionalOnMissingBean + WorkerIdAssigner workerIdAssigner() { + return new DisposableWorkerIdAssigner(); + } +} diff --git a/src/main/java/com/baidu/fsg/uid/UidGenerator.java b/src/main/java/com/github/wujun234/uid/UidGenerator.java similarity index 91% rename from src/main/java/com/baidu/fsg/uid/UidGenerator.java rename to src/main/java/com/github/wujun234/uid/UidGenerator.java index 90cd7c7..382e93d 100644 --- a/src/main/java/com/baidu/fsg/uid/UidGenerator.java +++ b/src/main/java/com/github/wujun234/uid/UidGenerator.java @@ -13,9 +13,9 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid; +package com.github.wujun234.uid; -import com.baidu.fsg.uid.exception.UidGenerateException; +import com.github.wujun234.uid.exception.UidGenerateException; /** * Represents a unique id generator. diff --git a/src/main/java/com/baidu/fsg/uid/buffer/BufferPaddingExecutor.java b/src/main/java/com/github/wujun234/uid/buffer/BufferPaddingExecutor.java similarity index 97% rename from src/main/java/com/baidu/fsg/uid/buffer/BufferPaddingExecutor.java rename to src/main/java/com/github/wujun234/uid/buffer/BufferPaddingExecutor.java index 80c37ad..dee6623 100644 --- a/src/main/java/com/baidu/fsg/uid/buffer/BufferPaddingExecutor.java +++ b/src/main/java/com/github/wujun234/uid/buffer/BufferPaddingExecutor.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.buffer; +package com.github.wujun234.uid.buffer; import java.util.List; import java.util.concurrent.ExecutorService; @@ -22,13 +22,12 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import com.github.wujun234.uid.utils.NamingThreadFactory; +import com.github.wujun234.uid.utils.PaddedAtomicLong; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.util.Assert; -import com.baidu.fsg.uid.utils.NamingThreadFactory; -import com.baidu.fsg.uid.utils.PaddedAtomicLong; - /** * Represents an executor for padding {@link RingBuffer}
* There are two kinds of executors: one for scheduled padding, the other for padding immediately. @@ -41,7 +40,8 @@ public class BufferPaddingExecutor { /** Constants */ private static final String WORKER_NAME = "RingBuffer-Padding-Worker"; private static final String SCHEDULE_NAME = "RingBuffer-Padding-Schedule"; - private static final long DEFAULT_SCHEDULE_INTERVAL = 5 * 60L; // 5 minutes + /** 5 minutes */ + private static final long DEFAULT_SCHEDULE_INTERVAL = 5 * 60L; /** Whether buffer padding is running */ private final AtomicBoolean running; diff --git a/src/main/java/com/baidu/fsg/uid/buffer/BufferedUidProvider.java b/src/main/java/com/github/wujun234/uid/buffer/BufferedUidProvider.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/buffer/BufferedUidProvider.java rename to src/main/java/com/github/wujun234/uid/buffer/BufferedUidProvider.java index 6b332cf..1e68f50 100644 --- a/src/main/java/com/baidu/fsg/uid/buffer/BufferedUidProvider.java +++ b/src/main/java/com/github/wujun234/uid/buffer/BufferedUidProvider.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.buffer; +package com.github.wujun234.uid.buffer; import java.util.List; diff --git a/src/main/java/com/baidu/fsg/uid/buffer/RejectedPutBufferHandler.java b/src/main/java/com/github/wujun234/uid/buffer/RejectedPutBufferHandler.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/buffer/RejectedPutBufferHandler.java rename to src/main/java/com/github/wujun234/uid/buffer/RejectedPutBufferHandler.java index c9f9d34..80aa260 100644 --- a/src/main/java/com/baidu/fsg/uid/buffer/RejectedPutBufferHandler.java +++ b/src/main/java/com/github/wujun234/uid/buffer/RejectedPutBufferHandler.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.buffer; +package com.github.wujun234.uid.buffer; /** * If tail catches the cursor it means that the ring buffer is full, any more buffer put request will be rejected. diff --git a/src/main/java/com/baidu/fsg/uid/buffer/RejectedTakeBufferHandler.java b/src/main/java/com/github/wujun234/uid/buffer/RejectedTakeBufferHandler.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/buffer/RejectedTakeBufferHandler.java rename to src/main/java/com/github/wujun234/uid/buffer/RejectedTakeBufferHandler.java index 0afe679..7bbb630 100644 --- a/src/main/java/com/baidu/fsg/uid/buffer/RejectedTakeBufferHandler.java +++ b/src/main/java/com/github/wujun234/uid/buffer/RejectedTakeBufferHandler.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.buffer; +package com.github.wujun234.uid.buffer; /** * If cursor catches the tail it means that the ring buffer is empty, any more buffer take request will be rejected. diff --git a/src/main/java/com/baidu/fsg/uid/buffer/RingBuffer.java b/src/main/java/com/github/wujun234/uid/buffer/RingBuffer.java similarity index 99% rename from src/main/java/com/baidu/fsg/uid/buffer/RingBuffer.java rename to src/main/java/com/github/wujun234/uid/buffer/RingBuffer.java index 2b5ab62..b04141e 100644 --- a/src/main/java/com/baidu/fsg/uid/buffer/RingBuffer.java +++ b/src/main/java/com/github/wujun234/uid/buffer/RingBuffer.java @@ -13,16 +13,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.buffer; +package com.github.wujun234.uid.buffer; import java.util.concurrent.atomic.AtomicLong; +import com.github.wujun234.uid.utils.PaddedAtomicLong; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.util.Assert; -import com.baidu.fsg.uid.utils.PaddedAtomicLong; - /** * Represents a ring buffer based on array.
* Using array could improve read element performance due to the CUP cache line. To prevent diff --git a/src/main/java/com/baidu/fsg/uid/exception/UidGenerateException.java b/src/main/java/com/github/wujun234/uid/exception/UidGenerateException.java similarity index 97% rename from src/main/java/com/baidu/fsg/uid/exception/UidGenerateException.java rename to src/main/java/com/github/wujun234/uid/exception/UidGenerateException.java index ae7aaff..8fe67f1 100644 --- a/src/main/java/com/baidu/fsg/uid/exception/UidGenerateException.java +++ b/src/main/java/com/github/wujun234/uid/exception/UidGenerateException.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.exception; +package com.github.wujun234.uid.exception; /** * UidGenerateException diff --git a/src/main/java/com/baidu/fsg/uid/impl/CachedUidGenerator.java b/src/main/java/com/github/wujun234/uid/impl/CachedUidGenerator.java similarity index 67% rename from src/main/java/com/baidu/fsg/uid/impl/CachedUidGenerator.java rename to src/main/java/com/github/wujun234/uid/impl/CachedUidGenerator.java index 4f9689c..891b411 100644 --- a/src/main/java/com/baidu/fsg/uid/impl/CachedUidGenerator.java +++ b/src/main/java/com/github/wujun234/uid/impl/CachedUidGenerator.java @@ -13,67 +13,98 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.impl; +package com.github.wujun234.uid.impl; import java.util.ArrayList; import java.util.List; +import com.github.wujun234.uid.BitsAllocator; +import com.github.wujun234.uid.UidGenerator; +import com.github.wujun234.uid.exception.UidGenerateException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.DisposableBean; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.util.Assert; -import com.baidu.fsg.uid.BitsAllocator; -import com.baidu.fsg.uid.UidGenerator; -import com.baidu.fsg.uid.buffer.BufferPaddingExecutor; -import com.baidu.fsg.uid.buffer.RejectedPutBufferHandler; -import com.baidu.fsg.uid.buffer.RejectedTakeBufferHandler; -import com.baidu.fsg.uid.buffer.RingBuffer; -import com.baidu.fsg.uid.exception.UidGenerateException; +import com.github.wujun234.uid.buffer.BufferPaddingExecutor; +import com.github.wujun234.uid.buffer.RejectedPutBufferHandler; +import com.github.wujun234.uid.buffer.RejectedTakeBufferHandler; +import com.github.wujun234.uid.buffer.RingBuffer; /** * Represents a cached implementation of {@link UidGenerator} extends * from {@link DefaultUidGenerator}, based on a lock free {@link RingBuffer}

- * + *

* The spring properties you can specified as below:
- *

  • boostPower: RingBuffer size boost for a power of 2, Sample: boostPower is 3, it means the buffer size - * will be ({@link BitsAllocator#getMaxSequence()} + 1) << - * {@link #boostPower}, Default as {@value #DEFAULT_BOOST_POWER} - *
  • paddingFactor: Represents a percent value of (0 - 100). When the count of rest available UIDs reach the - * threshold, it will trigger padding buffer. Default as{@link RingBuffer#DEFAULT_PADDING_PERCENT} - * Sample: paddingFactor=20, bufferSize=1000 -> threshold=1000 * 20 /100, padding buffer will be triggered when tail-cursorboostPower: RingBuffer size boost for a power of 2, Sample: boostPower is 3, it means the buffer size + * will be ({@link BitsAllocator#getMaxSequence()} + 1) << + * {@link #boostPower}, Default as {@value #DEFAULT_BOOST_POWER} + *
  • paddingFactor: Represents a percent value of (0 - 100). When the count of rest available UIDs reach the + * threshold, it will trigger padding buffer. Default as{@link RingBuffer#DEFAULT_PADDING_PERCENT} + * Sample: paddingFactor=20, bufferSize=1000 -> threshold=1000 * 20 /100, padding buffer will be triggered when tail-cursorscheduleInterval: Padding buffer in a schedule, specify padding buffer interval, Unit as second *
  • rejectedPutBufferHandler: Policy for rejected put buffer. Default as discard put request, just do logging *
  • rejectedTakeBufferHandler: Policy for rejected take buffer. Default as throwing up an exception - * + * * @author yutianbao + * @author wujun */ +@ConfigurationProperties(prefix = "uid.cached-uid-generator") public class CachedUidGenerator extends DefaultUidGenerator implements DisposableBean { private static final Logger LOGGER = LoggerFactory.getLogger(CachedUidGenerator.class); private static final int DEFAULT_BOOST_POWER = 3; - /** Spring properties */ + // --------------------- 配置属性 begin --------------------- + /** + * RingBuffer size扩容参数, 可提高UID生成的吞吐量. + * 默认:3， 原bufferSize=8192, 扩容后bufferSize= 8192 << 3 = 65536 + */ private int boostPower = DEFAULT_BOOST_POWER; + /** + * 指定何时向RingBuffer中填充UID, 取值为百分比(0, 100), 默认为50 + * 举例: bufferSize=1024, paddingFactor=50 -> threshold=1024 * 50 / 100 = 512. + * 当环上可用UID数量 < 512时, 将自动对RingBuffer进行填充补全 + */ private int paddingFactor = RingBuffer.DEFAULT_PADDING_PERCENT; + /** + * 另外一种RingBuffer填充时机, 在Schedule线程中, 周期性检查填充 + * 默认:不配置此项, 即不使用Schedule线程. 如需使用, 请指定Schedule线程时间间隔, 单位:秒 + */ private Long scheduleInterval; - + // --------------------- 配置属性 end ----------------------- + + /** 拒绝策略: 当环已满, 无法继续填充时 + 默认无需指定, 将丢弃Put操作, 仅日志记录. 如有特殊需求, 请实现RejectedPutBufferHandler接口(支持Lambda表达式)并以@Autowired方式注入 */ + @Autowired(required = false) private RejectedPutBufferHandler rejectedPutBufferHandler; + + /** 拒绝策略: 当环已空, 无法继续获取时 + 默认无需指定, 将记录日志, 并抛出UidGenerateException异常. 如有特殊需求, 请实现RejectedTakeBufferHandler接口(支持Lambda表达式)并以@Autowired方式注入 */ + @Autowired(required = false) private RejectedTakeBufferHandler rejectedTakeBufferHandler; - /** RingBuffer */ + /** + * RingBuffer + */ private RingBuffer ringBuffer; private BufferPaddingExecutor bufferPaddingExecutor; + public CachedUidGenerator(UidProperties uidProperties) { + super(uidProperties); + } + @Override public void afterPropertiesSet() throws Exception { // initialize workerId & bitsAllocator super.afterPropertiesSet(); - + // initialize RingBuffer & RingBufferPaddingExecutor this.initRingBuffer(); LOGGER.info("Initialized RingBuffer successfully."); } - + @Override public long getUID() { try { @@ -88,7 +119,7 @@ public long getUID() { public String parseUID(long uid) { return super.parseUID(uid); } - + @Override public void destroy() throws Exception { bufferPaddingExecutor.shutdown(); @@ -96,7 +127,7 @@ public void destroy() throws Exception { /** * Get the UIDs in the same specified second under the max sequence - * + * * @param currentSecond * @return UID list, size of {@link BitsAllocator#getMaxSequence()} + 1 */ @@ -106,14 +137,14 @@ protected List nextIdsForOneSecond(long currentSecond) { List uidList = new ArrayList<>(listSize); // Allocate the first sequence of the second, the others can be calculated with the offset - long firstSeqUid = bitsAllocator.allocate(currentSecond - epochSeconds, workerId, 0L); + long firstSeqUid = bitsAllocator.allocate(currentSecond - uidProperties.getEpochSeconds(), workerId, 0L); for (int offset = 0; offset < listSize; offset++) { uidList.add(firstSeqUid + offset); } return uidList; } - + /** * Initialize RingBuffer & RingBufferPaddingExecutor */ @@ -129,9 +160,9 @@ private void initRingBuffer() { if (usingSchedule) { bufferPaddingExecutor.setScheduleInterval(scheduleInterval); } - + LOGGER.info("Initialized BufferPaddingExecutor. Using schdule:{}, interval:{}", usingSchedule, scheduleInterval); - + // set rejected put/take handle policy this.ringBuffer.setBufferPaddingExecutor(bufferPaddingExecutor); if (rejectedPutBufferHandler != null) { @@ -140,10 +171,10 @@ private void initRingBuffer() { if (rejectedTakeBufferHandler != null) { this.ringBuffer.setRejectedTakeHandler(rejectedTakeBufferHandler); } - + // fill in all slots of the RingBuffer bufferPaddingExecutor.paddingBuffer(); - + // start buffer padding threads bufferPaddingExecutor.start(); } @@ -155,7 +186,12 @@ public void setBoostPower(int boostPower) { Assert.isTrue(boostPower > 0, "Boost power must be positive!"); this.boostPower = boostPower; } - + + public void setPaddingFactor(int paddingFactor) { + Assert.isTrue(paddingFactor > 0 && paddingFactor < 100, "padding factor must be in (0, 100)!"); + this.paddingFactor = paddingFactor; + } + public void setRejectedPutBufferHandler(RejectedPutBufferHandler rejectedPutBufferHandler) { Assert.notNull(rejectedPutBufferHandler, "RejectedPutBufferHandler can't be null!"); this.rejectedPutBufferHandler = rejectedPutBufferHandler; diff --git a/src/main/java/com/baidu/fsg/uid/impl/DefaultUidGenerator.java b/src/main/java/com/github/wujun234/uid/impl/DefaultUidGenerator.java similarity index 64% rename from src/main/java/com/baidu/fsg/uid/impl/DefaultUidGenerator.java rename to src/main/java/com/github/wujun234/uid/impl/DefaultUidGenerator.java index 20f9a2a..664f162 100644 --- a/src/main/java/com/baidu/fsg/uid/impl/DefaultUidGenerator.java +++ b/src/main/java/com/github/wujun234/uid/impl/DefaultUidGenerator.java @@ -13,86 +13,94 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.impl; +package com.github.wujun234.uid.impl; import java.util.Date; import java.util.concurrent.TimeUnit; -import org.apache.commons.lang.StringUtils; +import com.github.wujun234.uid.BitsAllocator; +import com.github.wujun234.uid.UidGenerator; +import com.github.wujun234.uid.exception.UidGenerateException; +import com.github.wujun234.uid.impl.UidProperties; +import com.github.wujun234.uid.utils.DateUtils; +import com.github.wujun234.uid.worker.WorkerIdAssigner; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.InitializingBean; -import com.baidu.fsg.uid.BitsAllocator; -import com.baidu.fsg.uid.UidGenerator; -import com.baidu.fsg.uid.exception.UidGenerateException; -import com.baidu.fsg.uid.utils.DateUtils; -import com.baidu.fsg.uid.worker.WorkerIdAssigner; +import org.springframework.beans.factory.annotation.Autowired; /** * Represents an implementation of {@link UidGenerator} - * + *

    * The unique id has 64bits (long), default allocated as blow:
    *

  • sign: The highest bit is 0 *
  • delta seconds: The next 28 bits, represents delta seconds since a customer epoch(2016-05-20 00:00:00.000). - * Supports about 8.7 years until to 2024-11-20 21:24:16 + * Supports about 8.7 years until to 2024-11-20 21:24:16 *
  • worker id: The next 22 bits, represents the worker's id which assigns based on database, max id is about 420W *
  • sequence: The next 13 bits, represents a sequence within the same second, max for 8192/s

    - * + *

    * The {@link DefaultUidGenerator#parseUID(long)} is a tool method to parse the bits - * + *

    * 

    {@code
  * +------+----------------------+----------------+-----------+
  * | sign |     delta seconds    | worker node id | sequence  |
  * +------+----------------------+----------------+-----------+
  *   1bit          28bits              22bits         13bits
  * }
    - * + *

    * You can also specified the bits by Spring property setting. *

  • timeBits: default as 28 *
  • workerBits: default as 22 *
  • seqBits: default as 13 *
  • epochStr: Epoch date string format 'yyyy-MM-dd'. Default as '2016-05-20'

    - * + *

    * Note that: The total bits must be 64 -1 * * @author yutianbao + * @author wujun */ public class DefaultUidGenerator implements UidGenerator, InitializingBean { private static final Logger LOGGER = LoggerFactory.getLogger(DefaultUidGenerator.class); - /** Bits allocate */ - protected int timeBits = 28; - protected int workerBits = 22; - protected int seqBits = 13; - - /** Customer epoch, unit as second. For example 2016-05-20 (ms: 1463673600000)*/ - protected String epochStr = "2016-05-20"; - protected long epochSeconds = TimeUnit.MILLISECONDS.toSeconds(1463673600000L); + protected UidProperties uidProperties; - /** Stable fields after spring bean initializing */ + /** + * Bit分配器,Stable fields after spring bean initializing + */ protected BitsAllocator bitsAllocator; protected long workerId; - /** Volatile fields caused by nextId() */ + /** + * Volatile fields caused by nextId() + */ protected long sequence = 0L; protected long lastSecond = -1L; - /** Spring property */ + /** + * Spring property + */ + @Autowired protected WorkerIdAssigner workerIdAssigner; + public DefaultUidGenerator(UidProperties uidProperties) { + this.uidProperties = uidProperties; + } + @Override public void afterPropertiesSet() throws Exception { // initialize bits allocator - bitsAllocator = new BitsAllocator(timeBits, workerBits, seqBits); + bitsAllocator = new BitsAllocator(uidProperties.getTimeBits(), uidProperties.getWorkerBits(), uidProperties.getSeqBits()); // initialize worker id workerId = workerIdAssigner.assignWorkerId(); if (workerId > bitsAllocator.getMaxWorkerId()) { - throw new RuntimeException("Worker id " + workerId + " exceeds the max " + bitsAllocator.getMaxWorkerId()); + LOGGER.error("Worker id " + workerId + " exceeds the max " + bitsAllocator.getMaxWorkerId()); + workerId = workerId % bitsAllocator.getMaxWorkerId(); + LOGGER.info("new Worker id = " + workerId); } - LOGGER.info("Initialized bits(1, {}, {}, {}) for workerID:{}", timeBits, workerBits, seqBits, workerId); + LOGGER.info("Initialized bits(1, {}, {}, {}) for workerID:{}", uidProperties.getTimeBits(), uidProperties.getWorkerBits(), uidProperties.getSeqBits(), workerId); } @Override @@ -108,17 +116,15 @@ public long getUID() throws UidGenerateException { @Override public String parseUID(long uid) { long totalBits = BitsAllocator.TOTAL_BITS; - long signBits = bitsAllocator.getSignBits(); - long timestampBits = bitsAllocator.getTimestampBits(); long workerIdBits = bitsAllocator.getWorkerIdBits(); long sequenceBits = bitsAllocator.getSequenceBits(); // parse UID long sequence = (uid << (totalBits - sequenceBits)) >>> (totalBits - sequenceBits); - long workerId = (uid << (timestampBits + signBits)) >>> (totalBits - workerIdBits); + long workerId = (uid << (totalBits - workerIdBits - sequenceBits)) >>> (totalBits - workerIdBits); long deltaSeconds = uid >>> (workerIdBits + sequenceBits); - Date thatTime = new Date(TimeUnit.SECONDS.toMillis(epochSeconds + deltaSeconds)); + Date thatTime = new Date(TimeUnit.SECONDS.toMillis(uidProperties.getEpochSeconds() + deltaSeconds)); String thatTimeStr = DateUtils.formatByDateTimePattern(thatTime); // format as string @@ -138,7 +144,24 @@ protected synchronized long nextId() { // Clock moved backwards, refuse to generate uid if (currentSecond < lastSecond) { long refusedSeconds = lastSecond - currentSecond; - throw new UidGenerateException("Clock moved backwards. Refusing for %d seconds", refusedSeconds); + if (uidProperties.isEnableBackward()) { + if (refusedSeconds <= uidProperties.getMaxBackwardSeconds()) { + LOGGER.error("Clock moved backwards. wait for %d seconds", refusedSeconds); + while (currentSecond < lastSecond) { + currentSecond = getCurrentSecond(); + } + } else { + workerId = workerIdAssigner.assignFakeWorkerId(); + LOGGER.error("Clock moved backwards. Assigned New WorkerId %d", workerId); + if (workerId > bitsAllocator.getMaxWorkerId()) { + LOGGER.error("Worker id " + workerId + " exceeds the max " + bitsAllocator.getMaxWorkerId()); + workerId = workerId % bitsAllocator.getMaxWorkerId(); + LOGGER.info("new Worker id = " + workerId); + } + } + } else { + throw new UidGenerateException("Clock moved backwards. Refusing for %d seconds", refusedSeconds); + } } // At the same second, increase sequence @@ -149,7 +172,7 @@ protected synchronized long nextId() { currentSecond = getNextSecond(lastSecond); } - // At the different second, sequence restart from zero + // At the different second, sequence restart from zero } else { sequence = 0L; } @@ -157,11 +180,11 @@ protected synchronized long nextId() { lastSecond = currentSecond; // Allocate bits for UID - return bitsAllocator.allocate(currentSecond - epochSeconds, workerId, sequence); + return bitsAllocator.allocate(currentSecond - uidProperties.getEpochSeconds(), workerId, sequence); } /** - * Get next millisecond + * Get next second */ private long getNextSecond(long lastTimestamp) { long timestamp = getCurrentSecond(); @@ -177,7 +200,7 @@ private long getNextSecond(long lastTimestamp) { */ private long getCurrentSecond() { long currentSecond = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()); - if (currentSecond - epochSeconds > bitsAllocator.getMaxDeltaSeconds()) { + if (currentSecond - uidProperties.getEpochSeconds() > bitsAllocator.getMaxDeltaSeconds()) { throw new UidGenerateException("Timestamp bits is exhausted. Refusing UID generate. Now: " + currentSecond); } @@ -190,29 +213,4 @@ private long getCurrentSecond() { public void setWorkerIdAssigner(WorkerIdAssigner workerIdAssigner) { this.workerIdAssigner = workerIdAssigner; } - - public void setTimeBits(int timeBits) { - if (timeBits > 0) { - this.timeBits = timeBits; - } - } - - public void setWorkerBits(int workerBits) { - if (workerBits > 0) { - this.workerBits = workerBits; - } - } - - public void setSeqBits(int seqBits) { - if (seqBits > 0) { - this.seqBits = seqBits; - } - } - - public void setEpochStr(String epochStr) { - if (StringUtils.isNotBlank(epochStr)) { - this.epochStr = epochStr; - this.epochSeconds = TimeUnit.MILLISECONDS.toSeconds(DateUtils.parseByDayPattern(epochStr).getTime()); - } - } } diff --git a/src/main/java/com/github/wujun234/uid/impl/UidProperties.java b/src/main/java/com/github/wujun234/uid/impl/UidProperties.java new file mode 100644 index 0000000..f002369 --- /dev/null +++ b/src/main/java/com/github/wujun234/uid/impl/UidProperties.java @@ -0,0 +1,113 @@ +package com.github.wujun234.uid.impl; + +import com.github.wujun234.uid.utils.DateUtils; +import org.apache.commons.lang.StringUtils; +import org.springframework.boot.context.properties.ConfigurationProperties; + +import java.util.concurrent.TimeUnit; + +/** + * UID 的配置 + * + * @author wujun + * @date 2019.02.20 10:31 + */ +@ConfigurationProperties(prefix = "uid") +public class UidProperties { + + /** + * 时间增量值占用位数。当前时间相对于时间基点的增量值，单位为秒 + */ + private int timeBits = 30; + + /** + * 工作机器ID占用的位数 + */ + private int workerBits = 16; + + /** + * 序列号占用的位数 + */ + private int seqBits = 7; + + /** + * 时间基点. 例如 2019-02-20 (毫秒: 1550592000000) + */ + private String epochStr = "2019-02-20"; + + /** + * 时间基点对应的毫秒数 + */ + private long epochSeconds = TimeUnit.MILLISECONDS.toSeconds(1550592000000L); + + /** + * 是否容忍时钟回拨, 默认:true + */ + private boolean enableBackward = true; + + /** + * 时钟回拨最长容忍时间（秒） + */ + private long maxBackwardSeconds = 1L; + + public int getTimeBits() { + return timeBits; + } + + public void setTimeBits(int timeBits) { + if (timeBits > 0) { + this.timeBits = timeBits; + } + } + + public int getWorkerBits() { + return workerBits; + } + + public void setWorkerBits(int workerBits) { + if (workerBits > 0) { + this.workerBits = workerBits; + } + } + + public int getSeqBits() { + return seqBits; + } + + public void setSeqBits(int seqBits) { + if (seqBits > 0) { + this.seqBits = seqBits; + } + } + + public String getEpochStr() { + return epochStr; + } + + public void setEpochStr(String epochStr) { + if (StringUtils.isNotBlank(epochStr)) { + this.epochStr = epochStr; + this.epochSeconds = TimeUnit.MILLISECONDS.toSeconds(DateUtils.parseByDayPattern(epochStr).getTime()); + } + } + + public long getEpochSeconds() { + return epochSeconds; + } + + public boolean isEnableBackward() { + return enableBackward; + } + + public void setEnableBackward(boolean enableBackward) { + this.enableBackward = enableBackward; + } + + public long getMaxBackwardSeconds() { + return maxBackwardSeconds; + } + + public void setMaxBackwardSeconds(long maxBackwardSeconds) { + this.maxBackwardSeconds = maxBackwardSeconds; + } +} diff --git a/src/main/java/com/baidu/fsg/uid/utils/DateUtils.java b/src/main/java/com/github/wujun234/uid/utils/DateUtils.java similarity index 98% rename from src/main/java/com/baidu/fsg/uid/utils/DateUtils.java rename to src/main/java/com/github/wujun234/uid/utils/DateUtils.java index e60b8cb..a5a7677 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/DateUtils.java +++ b/src/main/java/com/github/wujun234/uid/utils/DateUtils.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.utils; +package com.github.wujun234.uid.utils; import java.text.ParseException; import java.util.Calendar; diff --git a/src/main/java/com/baidu/fsg/uid/utils/DockerUtils.java b/src/main/java/com/github/wujun234/uid/utils/DockerUtils.java similarity index 95% rename from src/main/java/com/baidu/fsg/uid/utils/DockerUtils.java rename to src/main/java/com/github/wujun234/uid/utils/DockerUtils.java index ac0fd41..bcb155c 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/DockerUtils.java +++ b/src/main/java/com/github/wujun234/uid/utils/DockerUtils.java @@ -1,104 +1,104 @@ -/* - * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.baidu.fsg.uid.utils; - -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * DockerUtils - * - * @author yutianbao - */ -public abstract class DockerUtils { - private static final Logger LOGGER = LoggerFactory.getLogger(DockerUtils.class); - - /** Environment param keys */ - private static final String ENV_KEY_HOST = "JPAAS_HOST"; - private static final String ENV_KEY_PORT = "JPAAS_HTTP_PORT"; - private static final String ENV_KEY_PORT_ORIGINAL = "JPAAS_HOST_PORT_8080"; - - /** Docker host & port */ - private static String DOCKER_HOST = ""; - private static String DOCKER_PORT = ""; - - /** Whether is docker */ - private static boolean IS_DOCKER; - - static { - retrieveFromEnv(); - } - - /** - * Retrieve docker host - * - * @return empty string if not a docker - */ - public static String getDockerHost() { - return DOCKER_HOST; - } - - /** - * Retrieve docker port - * - * @return empty string if not a docker - */ - public static String getDockerPort() { - return DOCKER_PORT; - } - - /** - * Whether a docker - * - * @return - */ - public static boolean isDocker() { - return IS_DOCKER; - } - - /** - * Retrieve host & port from environment - */ - private static void retrieveFromEnv() { - // retrieve host & port from environment - DOCKER_HOST = System.getenv(ENV_KEY_HOST); - DOCKER_PORT = System.getenv(ENV_KEY_PORT); - - // not found from 'JPAAS_HTTP_PORT', then try to find from 'JPAAS_HOST_PORT_8080' - if (StringUtils.isBlank(DOCKER_PORT)) { - DOCKER_PORT = System.getenv(ENV_KEY_PORT_ORIGINAL); - } - - boolean hasEnvHost = StringUtils.isNotBlank(DOCKER_HOST); - boolean hasEnvPort = StringUtils.isNotBlank(DOCKER_PORT); - - // docker can find both host & port from environment - if (hasEnvHost && hasEnvPort) { - IS_DOCKER = true; - - // found nothing means not a docker, maybe an actual machine - } else if (!hasEnvHost && !hasEnvPort) { - IS_DOCKER = false; - - } else { - LOGGER.error("Missing host or port from env for Docker. host:{}, port:{}", DOCKER_HOST, DOCKER_PORT); - throw new RuntimeException( - "Missing host or port from env for Docker. host:" + DOCKER_HOST + ", port:" + DOCKER_PORT); - } - } - -} +/* + * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.wujun234.uid.utils; + +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * DockerUtils + * + * @author yutianbao + */ +public abstract class DockerUtils { + private static final Logger LOGGER = LoggerFactory.getLogger(DockerUtils.class); + + /** Environment param keys */ + private static final String ENV_KEY_HOST = "JPAAS_HOST"; + private static final String ENV_KEY_PORT = "JPAAS_HTTP_PORT"; + private static final String ENV_KEY_PORT_ORIGINAL = "JPAAS_HOST_PORT_8080"; + + /** Docker host & port */ + private static String DOCKER_HOST = ""; + private static String DOCKER_PORT = ""; + + /** Whether is docker */ + private static boolean IS_DOCKER; + + static { + retrieveFromEnv(); + } + + /** + * Retrieve docker host + * + * @return empty string if not a docker + */ + public static String getDockerHost() { + return DOCKER_HOST; + } + + /** + * Retrieve docker port + * + * @return empty string if not a docker + */ + public static String getDockerPort() { + return DOCKER_PORT; + } + + /** + * Whether a docker + * + * @return + */ + public static boolean isDocker() { + return IS_DOCKER; + } + + /** + * Retrieve host & port from environment + */ + private static void retrieveFromEnv() { + // retrieve host & port from environment + DOCKER_HOST = System.getenv(ENV_KEY_HOST); + DOCKER_PORT = System.getenv(ENV_KEY_PORT); + + // not found from 'JPAAS_HTTP_PORT', then try to find from 'JPAAS_HOST_PORT_8080' + if (StringUtils.isBlank(DOCKER_PORT)) { + DOCKER_PORT = System.getenv(ENV_KEY_PORT_ORIGINAL); + } + + boolean hasEnvHost = StringUtils.isNotBlank(DOCKER_HOST); + boolean hasEnvPort = StringUtils.isNotBlank(DOCKER_PORT); + + // docker can find both host & port from environment + if (hasEnvHost && hasEnvPort) { + IS_DOCKER = true; + + // found nothing means not a docker, maybe an actual machine + } else if (!hasEnvHost && !hasEnvPort) { + IS_DOCKER = false; + + } else { + LOGGER.error("Missing host or port from env for Docker. host:{}, port:{}", DOCKER_HOST, DOCKER_PORT); + throw new RuntimeException( + "Missing host or port from env for Docker. host:" + DOCKER_HOST + ", port:" + DOCKER_PORT); + } + } + +} diff --git a/src/main/java/com/baidu/fsg/uid/utils/EnumUtils.java b/src/main/java/com/github/wujun234/uid/utils/EnumUtils.java similarity index 94% rename from src/main/java/com/baidu/fsg/uid/utils/EnumUtils.java rename to src/main/java/com/github/wujun234/uid/utils/EnumUtils.java index 0264911..c95cf07 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/EnumUtils.java +++ b/src/main/java/com/github/wujun234/uid/utils/EnumUtils.java @@ -1,64 +1,64 @@ -/* - * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.baidu.fsg.uid.utils; - -import org.springframework.util.Assert; - -/** - * EnumUtils provides the operations for {@link ValuedEnum} such as Parse, value of... - * - * @author yutianbao - */ -public abstract class EnumUtils { - - /** - * Parse the bounded value into ValuedEnum - * - * @param clz - * @param value - * @return - */ - public static , V> T parse(Class clz, V value) { - Assert.notNull(clz, "clz can not be null"); - if (value == null) { - return null; - } - - for (T t : clz.getEnumConstants()) { - if (value.equals(t.value())) { - return t; - } - } - return null; - } - - /** - * Null-safe valueOf function - * - * @param - * @param enumType - * @param name - * @return - */ - public static > T valueOf(Class enumType, String name) { - if (name == null) { - return null; - } - - return Enum.valueOf(enumType, name); - } - -} +/* + * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.wujun234.uid.utils; + +import org.springframework.util.Assert; + +/** + * EnumUtils provides the operations for {@link ValuedEnum} such as Parse, value of... + * + * @author yutianbao + */ +public abstract class EnumUtils { + + /** + * Parse the bounded value into ValuedEnum + * + * @param clz + * @param value + * @return + */ + public static , V> T parse(Class clz, V value) { + Assert.notNull(clz, "clz can not be null"); + if (value == null) { + return null; + } + + for (T t : clz.getEnumConstants()) { + if (value.equals(t.value())) { + return t; + } + } + return null; + } + + /** + * Null-safe valueOf function + * + * @param + * @param enumType + * @param name + * @return + */ + public static > T valueOf(Class enumType, String name) { + if (name == null) { + return null; + } + + return Enum.valueOf(enumType, name); + } + +} diff --git a/src/main/java/com/baidu/fsg/uid/utils/NamingThreadFactory.java b/src/main/java/com/github/wujun234/uid/utils/NamingThreadFactory.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/utils/NamingThreadFactory.java rename to src/main/java/com/github/wujun234/uid/utils/NamingThreadFactory.java index 8b11a2c..9121245 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/NamingThreadFactory.java +++ b/src/main/java/com/github/wujun234/uid/utils/NamingThreadFactory.java @@ -1,164 +1,165 @@ -/* - * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.baidu.fsg.uid.utils; - -import java.lang.Thread.UncaughtExceptionHandler; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ThreadFactory; -import java.util.concurrent.atomic.AtomicLong; - -import org.apache.commons.lang.ClassUtils; -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Named thread in ThreadFactory. If there is no specified name for thread, it - * will auto detect using the invoker classname instead. - * - * @author yutianbao - */ -public class NamingThreadFactory implements ThreadFactory { - private static final Logger LOGGER = LoggerFactory.getLogger(NamingThreadFactory.class); - - /** - * Thread name pre - */ - private String name; - /** - * Is daemon thread - */ - private boolean daemon; - /** - * UncaughtExceptionHandler - */ - private UncaughtExceptionHandler uncaughtExceptionHandler; - /** - * Sequences for multi thread name prefix - */ - private final ConcurrentHashMap sequences; - - /** - * Constructors - */ - public NamingThreadFactory() { - this(null, false, null); - } - - public NamingThreadFactory(String name) { - this(name, false, null); - } - - public NamingThreadFactory(String name, boolean daemon) { - this(name, daemon, null); - } - - public NamingThreadFactory(String name, boolean daemon, UncaughtExceptionHandler handler) { - this.name = name; - this.daemon = daemon; - this.uncaughtExceptionHandler = handler; - this.sequences = new ConcurrentHashMap(); - } - - @Override - public Thread newThread(Runnable r) { - Thread thread = new Thread(r); - thread.setDaemon(this.daemon); - - // If there is no specified name for thread, it will auto detect using the invoker classname instead. - // Notice that auto detect may cause some performance overhead - String prefix = this.name; - if (StringUtils.isBlank(prefix)) { - prefix = getInvoker(2); - } - thread.setName(prefix + "-" + getSequence(prefix)); - - // no specified uncaughtExceptionHandler, just do logging. - if (this.uncaughtExceptionHandler != null) { - thread.setUncaughtExceptionHandler(this.uncaughtExceptionHandler); - } else { - thread.setUncaughtExceptionHandler(new UncaughtExceptionHandler() { - public void uncaughtException(Thread t, Throwable e) { - LOGGER.error("unhandled exception in thread: " + t.getId() + ":" + t.getName(), e); - } - }); - } - - return thread; - } - - /** - * Get the method invoker's class name - * - * @param depth - * @return - */ - private String getInvoker(int depth) { - Exception e = new Exception(); - StackTraceElement[] stes = e.getStackTrace(); - if (stes.length > depth) { - return ClassUtils.getShortClassName(stes[depth].getClassName()); - } - return getClass().getSimpleName(); - } - - /** - * Get sequence for different naming prefix - * - * @param invoker - * @return - */ - private long getSequence(String invoker) { - AtomicLong r = this.sequences.get(invoker); - if (r == null) { - r = new AtomicLong(0); - AtomicLong previous = this.sequences.putIfAbsent(invoker, r); - if (previous != null) { - r = previous; - } - } - - return r.incrementAndGet(); - } - - /** - * Getters & Setters - */ - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public boolean isDaemon() { - return daemon; - } - - public void setDaemon(boolean daemon) { - this.daemon = daemon; - } - - public UncaughtExceptionHandler getUncaughtExceptionHandler() { - return uncaughtExceptionHandler; - } - - public void setUncaughtExceptionHandler(UncaughtExceptionHandler handler) { - this.uncaughtExceptionHandler = handler; - } - -} +/* + * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.wujun234.uid.utils; + +import java.lang.Thread.UncaughtExceptionHandler; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.commons.lang.ClassUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Named thread in ThreadFactory. If there is no specified name for thread, it + * will auto detect using the invoker classname instead. + * + * @author yutianbao + */ +public class NamingThreadFactory implements ThreadFactory { + private static final Logger LOGGER = LoggerFactory.getLogger(NamingThreadFactory.class); + + /** + * Thread name pre + */ + private String name; + /** + * Is daemon thread + */ + private boolean daemon; + /** + * UncaughtExceptionHandler + */ + private UncaughtExceptionHandler uncaughtExceptionHandler; + /** + * Sequences for multi thread name prefix + */ + private final ConcurrentHashMap sequences; + + /** + * Constructors + */ + public NamingThreadFactory() { + this(null, false, null); + } + + public NamingThreadFactory(String name) { + this(name, false, null); + } + + public NamingThreadFactory(String name, boolean daemon) { + this(name, daemon, null); + } + + public NamingThreadFactory(String name, boolean daemon, UncaughtExceptionHandler handler) { + this.name = name; + this.daemon = daemon; + this.uncaughtExceptionHandler = handler; + this.sequences = new ConcurrentHashMap(); + } + + @Override + public Thread newThread(Runnable r) { + Thread thread = new Thread(r); + thread.setDaemon(this.daemon); + + // If there is no specified name for thread, it will auto detect using the invoker classname instead. + // Notice that auto detect may cause some performance overhead + String prefix = this.name; + if (StringUtils.isBlank(prefix)) { + prefix = getInvoker(2); + } + thread.setName(prefix + "-" + getSequence(prefix)); + + // no specified uncaughtExceptionHandler, just do logging. + if (this.uncaughtExceptionHandler != null) { + thread.setUncaughtExceptionHandler(this.uncaughtExceptionHandler); + } else { + thread.setUncaughtExceptionHandler(new UncaughtExceptionHandler() { + @Override + public void uncaughtException(Thread t, Throwable e) { + LOGGER.error("unhandled exception in thread: " + t.getId() + ":" + t.getName(), e); + } + }); + } + + return thread; + } + + /** + * Get the method invoker's class name + * + * @param depth + * @return + */ + private String getInvoker(int depth) { + Exception e = new Exception(); + StackTraceElement[] stes = e.getStackTrace(); + if (stes.length > depth) { + return ClassUtils.getShortClassName(stes[depth].getClassName()); + } + return getClass().getSimpleName(); + } + + /** + * Get sequence for different naming prefix + * + * @param invoker + * @return + */ + private long getSequence(String invoker) { + AtomicLong r = this.sequences.get(invoker); + if (r == null) { + r = new AtomicLong(0); + AtomicLong previous = this.sequences.putIfAbsent(invoker, r); + if (previous != null) { + r = previous; + } + } + + return r.incrementAndGet(); + } + + /** + * Getters & Setters + */ + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public boolean isDaemon() { + return daemon; + } + + public void setDaemon(boolean daemon) { + this.daemon = daemon; + } + + public UncaughtExceptionHandler getUncaughtExceptionHandler() { + return uncaughtExceptionHandler; + } + + public void setUncaughtExceptionHandler(UncaughtExceptionHandler handler) { + this.uncaughtExceptionHandler = handler; + } + +} diff --git a/src/main/java/com/baidu/fsg/uid/utils/NetUtils.java b/src/main/java/com/github/wujun234/uid/utils/NetUtils.java similarity index 95% rename from src/main/java/com/baidu/fsg/uid/utils/NetUtils.java rename to src/main/java/com/github/wujun234/uid/utils/NetUtils.java index d74aadd..311d36f 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/NetUtils.java +++ b/src/main/java/com/github/wujun234/uid/utils/NetUtils.java @@ -1,84 +1,84 @@ -/* - * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.baidu.fsg.uid.utils; - -import java.net.InetAddress; -import java.net.NetworkInterface; -import java.net.SocketException; -import java.util.Enumeration; - -/** - * NetUtils - * - * @author yutianbao - */ -public abstract class NetUtils { - - /** - * Pre-loaded local address - */ - public static InetAddress localAddress; - - static { - try { - localAddress = getLocalInetAddress(); - } catch (SocketException e) { - throw new RuntimeException("fail to get local ip."); - } - } - - /** - * Retrieve the first validated local ip address(the Public and LAN ip addresses are validated). - * - * @return the local address - * @throws SocketException the socket exception - */ - public static InetAddress getLocalInetAddress() throws SocketException { - // enumerates all network interfaces - Enumeration enu = NetworkInterface.getNetworkInterfaces(); - - while (enu.hasMoreElements()) { - NetworkInterface ni = enu.nextElement(); - if (ni.isLoopback()) { - continue; - } - - Enumeration addressEnumeration = ni.getInetAddresses(); - while (addressEnumeration.hasMoreElements()) { - InetAddress address = addressEnumeration.nextElement(); - - // ignores all invalidated addresses - if (address.isLinkLocalAddress() || address.isLoopbackAddress() || address.isAnyLocalAddress()) { - continue; - } - - return address; - } - } - - throw new RuntimeException("No validated local address!"); - } - - /** - * Retrieve local address - * - * @return the string local address - */ - public static String getLocalAddress() { - return localAddress.getHostAddress(); - } - -} +/* + * Copyright (c) 2017 Baidu, Inc. All Rights Reserve. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.wujun234.uid.utils; + +import java.net.InetAddress; +import java.net.NetworkInterface; +import java.net.SocketException; +import java.util.Enumeration; + +/** + * NetUtils + * + * @author yutianbao + */ +public abstract class NetUtils { + + /** + * Pre-loaded local address + */ + public static InetAddress localAddress; + + static { + try { + localAddress = getLocalInetAddress(); + } catch (SocketException e) { + throw new RuntimeException("fail to get local ip."); + } + } + + /** + * Retrieve the first validated local ip address(the Public and LAN ip addresses are validated). + * + * @return the local address + * @throws SocketException the socket exception + */ + public static InetAddress getLocalInetAddress() throws SocketException { + // enumerates all network interfaces + Enumeration enu = NetworkInterface.getNetworkInterfaces(); + + while (enu.hasMoreElements()) { + NetworkInterface ni = enu.nextElement(); + if (ni.isLoopback()) { + continue; + } + + Enumeration addressEnumeration = ni.getInetAddresses(); + while (addressEnumeration.hasMoreElements()) { + InetAddress address = addressEnumeration.nextElement(); + + // ignores all invalidated addresses + if (address.isLinkLocalAddress() || address.isLoopbackAddress() || address.isAnyLocalAddress()) { + continue; + } + + return address; + } + } + + throw new RuntimeException("No validated local address!"); + } + + /** + * Retrieve local address + * + * @return the string local address + */ + public static String getLocalAddress() { + return localAddress.getHostAddress(); + } + +} diff --git a/src/main/java/com/baidu/fsg/uid/utils/PaddedAtomicLong.java b/src/main/java/com/github/wujun234/uid/utils/PaddedAtomicLong.java similarity index 97% rename from src/main/java/com/baidu/fsg/uid/utils/PaddedAtomicLong.java rename to src/main/java/com/github/wujun234/uid/utils/PaddedAtomicLong.java index c338312..5c9a34c 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/PaddedAtomicLong.java +++ b/src/main/java/com/github/wujun234/uid/utils/PaddedAtomicLong.java @@ -13,7 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.utils; +package com.github.wujun234.uid.utils; import java.util.concurrent.atomic.AtomicLong; diff --git a/src/main/java/com/baidu/fsg/uid/utils/ValuedEnum.java b/src/main/java/com/github/wujun234/uid/utils/ValuedEnum.java similarity index 90% rename from src/main/java/com/baidu/fsg/uid/utils/ValuedEnum.java rename to src/main/java/com/github/wujun234/uid/utils/ValuedEnum.java index 22a4964..a2d1b92 100644 --- a/src/main/java/com/baidu/fsg/uid/utils/ValuedEnum.java +++ b/src/main/java/com/github/wujun234/uid/utils/ValuedEnum.java @@ -13,15 +13,20 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.utils; +package com.github.wujun234.uid.utils; /** * {@code ValuedEnum} defines an enumeration which is bounded to a value, you * may implements this interface when you defines such kind of enumeration, that * you can use {@link EnumUtils} to simplify parse and valueOf operation. - * + * * @author yutianbao */ public interface ValuedEnum { + /** + * get value + * + * @return T + */ T value(); } diff --git a/src/main/java/com/baidu/fsg/uid/worker/DisposableWorkerIdAssigner.java b/src/main/java/com/github/wujun234/uid/worker/DisposableWorkerIdAssigner.java similarity index 63% rename from src/main/java/com/baidu/fsg/uid/worker/DisposableWorkerIdAssigner.java rename to src/main/java/com/github/wujun234/uid/worker/DisposableWorkerIdAssigner.java index 26570c9..6f51029 100644 --- a/src/main/java/com/baidu/fsg/uid/worker/DisposableWorkerIdAssigner.java +++ b/src/main/java/com/github/wujun234/uid/worker/DisposableWorkerIdAssigner.java @@ -13,12 +13,13 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.worker; -import com.baidu.fsg.uid.utils.DockerUtils; -import com.baidu.fsg.uid.utils.NetUtils; -import com.baidu.fsg.uid.worker.dao.WorkerNodeDAO; -import com.baidu.fsg.uid.worker.entity.WorkerNodeEntity; +package com.github.wujun234.uid.worker; + +import com.github.wujun234.uid.worker.dao.WorkerNodeDAO; +import com.github.wujun234.uid.worker.entity.WorkerNodeEntity; +import com.github.wujun234.uid.utils.DockerUtils; +import com.github.wujun234.uid.utils.NetUtils; import org.apache.commons.lang.math.RandomUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -27,12 +28,13 @@ import javax.annotation.Resource; /** - * Represents an implementation of {@link WorkerIdAssigner}, + * Represents an implementation of {@link WorkerIdAssigner}, * the worker id will be discarded after assigned to the UidGenerator - * + * * @author yutianbao */ public class DisposableWorkerIdAssigner implements WorkerIdAssigner { + private static final Logger LOGGER = LoggerFactory.getLogger(DisposableWorkerIdAssigner.class); @Resource @@ -42,14 +44,21 @@ public class DisposableWorkerIdAssigner implements WorkerIdAssigner { * Assign worker id base on database.

    * If there is host name & port in the environment, we considered that the node runs in Docker container
    * Otherwise, the node runs on an actual machine. - * + * * @return assigned worker id */ - @Transactional + @Transactional(rollbackFor = Exception.class) + @Override public long assignWorkerId() { // build worker node entity WorkerNodeEntity workerNodeEntity = buildWorkerNode(); + WorkerNodeEntity oldWorkerNode = workerNodeDAO + .getWorkerNodeByHostPort(workerNodeEntity.getHostName(), workerNodeEntity.getPort()); + if (null != oldWorkerNode) { + return oldWorkerNode.getId(); + } + // add worker node for new (ignore the same IP + PORT) workerNodeDAO.addWorkerNode(workerNodeEntity); LOGGER.info("Add worker node:" + workerNodeEntity); @@ -57,6 +66,12 @@ public long assignWorkerId() { return workerNodeEntity.getId(); } + @Transactional(rollbackFor = Exception.class) + @Override + public long assignFakeWorkerId() { + return buildFakeWorkerNode().getId(); + } + /** * Build worker node entity by IP and PORT */ @@ -66,7 +81,6 @@ private WorkerNodeEntity buildWorkerNode() { workerNodeEntity.setType(WorkerNodeType.CONTAINER.value()); workerNodeEntity.setHostName(DockerUtils.getDockerHost()); workerNodeEntity.setPort(DockerUtils.getDockerPort()); - } else { workerNodeEntity.setType(WorkerNodeType.ACTUAL.value()); workerNodeEntity.setHostName(NetUtils.getLocalAddress()); @@ -76,4 +90,16 @@ private WorkerNodeEntity buildWorkerNode() { return workerNodeEntity; } + private WorkerNodeEntity buildFakeWorkerNode() { + WorkerNodeEntity workerNodeEntity = new WorkerNodeEntity(); + workerNodeEntity.setType(WorkerNodeType.FAKE.value()); + if (DockerUtils.isDocker()) { + workerNodeEntity.setHostName(DockerUtils.getDockerHost()); + workerNodeEntity.setPort(DockerUtils.getDockerPort() + "-" + RandomUtils.nextInt(100000)); + } else { + workerNodeEntity.setHostName(NetUtils.getLocalAddress()); + workerNodeEntity.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000)); + } + return workerNodeEntity; + } } diff --git a/src/main/java/com/baidu/fsg/uid/worker/WorkerIdAssigner.java b/src/main/java/com/github/wujun234/uid/worker/WorkerIdAssigner.java similarity index 68% rename from src/main/java/com/baidu/fsg/uid/worker/WorkerIdAssigner.java rename to src/main/java/com/github/wujun234/uid/worker/WorkerIdAssigner.java index 25c14c3..c7088e4 100644 --- a/src/main/java/com/baidu/fsg/uid/worker/WorkerIdAssigner.java +++ b/src/main/java/com/github/wujun234/uid/worker/WorkerIdAssigner.java @@ -13,20 +13,29 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.worker; +package com.github.wujun234.uid.worker; + +import com.github.wujun234.uid.impl.DefaultUidGenerator; /** - * Represents a worker id assigner for {@link com.baidu.fsg.uid.impl.DefaultUidGenerator} - * + * Represents a worker id assigner for {@link DefaultUidGenerator} + * * @author yutianbao */ public interface WorkerIdAssigner { /** - * Assign worker id for {@link com.baidu.fsg.uid.impl.DefaultUidGenerator} - * + * Assign worker id for {@link DefaultUidGenerator} + * * @return assigned worker id */ long assignWorkerId(); + /** + * Assign fake worker id + * + * @return assigned fake worker id + */ + long assignFakeWorkerId(); + } diff --git a/src/main/java/com/baidu/fsg/uid/worker/WorkerNodeType.java b/src/main/java/com/github/wujun234/uid/worker/WorkerNodeType.java similarity index 82% rename from src/main/java/com/baidu/fsg/uid/worker/WorkerNodeType.java rename to src/main/java/com/github/wujun234/uid/worker/WorkerNodeType.java index b69b240..41fefcf 100644 --- a/src/main/java/com/baidu/fsg/uid/worker/WorkerNodeType.java +++ b/src/main/java/com/github/wujun234/uid/worker/WorkerNodeType.java @@ -13,20 +13,31 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.worker; +package com.github.wujun234.uid.worker; -import com.baidu.fsg.uid.utils.ValuedEnum; +import com.github.wujun234.uid.utils.ValuedEnum; /** * WorkerNodeType *

  • CONTAINER: Such as Docker *
  • ACTUAL: Actual machine - * + * * @author yutianbao */ public enum WorkerNodeType implements ValuedEnum { - CONTAINER(1), ACTUAL(2); + /** + * 容器 + */ + CONTAINER(1), + /** + * 物理机 + */ + ACTUAL(2), + /** + * 虚拟 + */ + FAKE(3); /** * Lock type diff --git a/src/main/java/com/baidu/fsg/uid/worker/dao/WorkerNodeDAO.java b/src/main/java/com/github/wujun234/uid/worker/dao/WorkerNodeDAO.java similarity index 52% rename from src/main/java/com/baidu/fsg/uid/worker/dao/WorkerNodeDAO.java rename to src/main/java/com/github/wujun234/uid/worker/dao/WorkerNodeDAO.java index 86d7f27..b1ecc4f 100644 --- a/src/main/java/com/baidu/fsg/uid/worker/dao/WorkerNodeDAO.java +++ b/src/main/java/com/github/wujun234/uid/worker/dao/WorkerNodeDAO.java @@ -13,16 +13,20 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.worker.dao; +package com.github.wujun234.uid.worker.dao; -import com.baidu.fsg.uid.worker.entity.WorkerNodeEntity; +import com.github.wujun234.uid.worker.entity.WorkerNodeEntity; +import org.apache.ibatis.annotations.Insert; +import org.apache.ibatis.annotations.Options; import org.apache.ibatis.annotations.Param; +import org.apache.ibatis.annotations.Select; import org.springframework.stereotype.Repository; /** * DAO for M_WORKER_NODE * * @author yutianbao + * @author wujun */ @Repository public interface WorkerNodeDAO { @@ -34,6 +38,18 @@ public interface WorkerNodeDAO { * @param port * @return */ + @Select("SELECT " + + " ID," + + " HOST_NAME," + + " PORT," + + " TYPE," + + " LAUNCH_DATE," + + " MODIFIED," + + " CREATED" + + " FROM" + + " WORKER_NODE" + + " WHERE" + + " HOST_NAME = #{host,jdbcType=VARCHAR} AND PORT = #{port,jdbcType=VARCHAR} limit 1") WorkerNodeEntity getWorkerNodeByHostPort(@Param("host") String host, @Param("port") String port); /** @@ -41,6 +57,21 @@ public interface WorkerNodeDAO { * * @param workerNodeEntity */ + @Insert("INSERT INTO WORKER_NODE" + + "(HOST_NAME," + + "PORT," + + "TYPE," + + "LAUNCH_DATE," + + "MODIFIED," + + "CREATED)" + + "VALUES (" + + "#{hostName}," + + "#{port}," + + "#{type}," + + "#{launchDate}," + + "NOW()," + + "NOW())") + @Options(useGeneratedKeys = true, keyProperty = "id", keyColumn = "id") void addWorkerNode(WorkerNodeEntity workerNodeEntity); } diff --git a/src/main/java/com/baidu/fsg/uid/worker/entity/WorkerNodeEntity.java b/src/main/java/com/github/wujun234/uid/worker/entity/WorkerNodeEntity.java similarity index 96% rename from src/main/java/com/baidu/fsg/uid/worker/entity/WorkerNodeEntity.java rename to src/main/java/com/github/wujun234/uid/worker/entity/WorkerNodeEntity.java index 78964a1..1dc5112 100644 --- a/src/main/java/com/baidu/fsg/uid/worker/entity/WorkerNodeEntity.java +++ b/src/main/java/com/github/wujun234/uid/worker/entity/WorkerNodeEntity.java @@ -13,15 +13,14 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package com.baidu.fsg.uid.worker.entity; +package com.github.wujun234.uid.worker.entity; import java.util.Date; +import com.github.wujun234.uid.worker.WorkerNodeType; import org.apache.commons.lang.builder.ReflectionToStringBuilder; import org.apache.commons.lang.builder.ToStringStyle; -import com.baidu.fsg.uid.worker.WorkerNodeType; - /** * Entity for M_WORKER_NODE * diff --git a/src/main/resources/META-INF/mybatis/mapper/WORKER_NODE.xml b/src/main/resources/META-INF/mybatis/mapper/WORKER_NODE.xml deleted file mode 100644 index e8cefcb..0000000 --- a/src/main/resources/META-INF/mybatis/mapper/WORKER_NODE.xml +++ /dev/null @@ -1,47 +0,0 @@ - - - - - - - - - - - - - - - INSERT INTO WORKER_NODE - (HOST_NAME, - PORT, - TYPE, - LAUNCH_DATE, - MODIFIED, - CREATED) - VALUES ( - #{hostName}, - #{port}, - #{type}, - #{launchDate}, - NOW(), - NOW()) - - - - \ No newline at end of file diff --git a/src/main/resources/META-INF/spring.factories b/src/main/resources/META-INF/spring.factories new file mode 100644 index 0000000..5de415c --- /dev/null +++ b/src/main/resources/META-INF/spring.factories @@ -0,0 +1 @@ +org.springframework.boot.autoconfigure.EnableAutoConfiguration=com.github.wujun234.uid.UidAutoConfigure \ No newline at end of file diff --git a/src/main/scripts/WORKER_NODE.sql b/src/main/scripts/WORKER_NODE.sql index d9f4212..4190d6a 100644 --- a/src/main/scripts/WORKER_NODE.sql +++ b/src/main/scripts/WORKER_NODE.sql @@ -4,7 +4,7 @@ CREATE TABLE WORKER_NODE ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id', HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name', PORT VARCHAR(64) NOT NULL COMMENT 'port', -TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER', +TYPE INT NOT NULL COMMENT 'node type: CONTAINER(1), ACTUAL(2), FAKE(3)', LAUNCH_DATE DATE NOT NULL COMMENT 'launch date', MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time', CREATED TIMESTAMP NOT NULL COMMENT 'created time', diff --git a/src/test/java/com/baidu/fsg/uid/CachedUidGeneratorTest.java b/src/test/java/com/github/wujun234/uid/CachedUidGeneratorTest.java similarity index 87% rename from src/test/java/com/baidu/fsg/uid/CachedUidGeneratorTest.java rename to src/test/java/com/github/wujun234/uid/CachedUidGeneratorTest.java index f6801a3..890a59b 100644 --- a/src/test/java/com/baidu/fsg/uid/CachedUidGeneratorTest.java +++ b/src/test/java/com/github/wujun234/uid/CachedUidGeneratorTest.java @@ -1,127 +1,131 @@ -package com.baidu.fsg.uid; - -import com.baidu.fsg.uid.impl.CachedUidGenerator; -import org.apache.commons.lang.StringUtils; -import org.junit.Assert; -import org.junit.Test; -import org.junit.runner.RunWith; -import org.springframework.test.context.ContextConfiguration; -import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; - -import javax.annotation.Resource; -import java.io.IOException; -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Set; -import java.util.concurrent.ConcurrentSkipListSet; -import java.util.concurrent.atomic.AtomicInteger; - -/** - * Test for {@link CachedUidGenerator} - * - * @author yutianbao - */ -@RunWith(SpringJUnit4ClassRunner.class) -@ContextConfiguration(locations = { "classpath:uid/cached-uid-spring.xml" }) -public class CachedUidGeneratorTest { - private static final int SIZE = 7000000; // 700w - private static final boolean VERBOSE = false; - private static final int THREADS = Runtime.getRuntime().availableProcessors() << 1; - - @Resource - private UidGenerator uidGenerator; - - /** - * Test for serially generate - * - * @throws IOException - */ - @Test - public void testSerialGenerate() throws IOException { - // Generate UID serially - Set uidSet = new HashSet<>(SIZE); - for (int i = 0; i < SIZE; i++) { - doGenerate(uidSet, i); - } - - // Check UIDs are all unique - checkUniqueID(uidSet); - } - - /** - * Test for parallel generate - * - * @throws InterruptedException - * @throws IOException - */ - @Test - public void testParallelGenerate() throws InterruptedException, IOException { - AtomicInteger control = new AtomicInteger(-1); - Set uidSet = new ConcurrentSkipListSet<>(); - - // Initialize threads - List threadList = new ArrayList<>(THREADS); - for (int i = 0; i < THREADS; i++) { - Thread thread = new Thread(() -> workerRun(uidSet, control)); - thread.setName("UID-generator-" + i); - - threadList.add(thread); - thread.start(); - } - - // Wait for worker done - for (Thread thread : threadList) { - thread.join(); - } - - // Check generate 700w times - Assert.assertEquals(SIZE, control.get()); - - // Check UIDs are all unique - checkUniqueID(uidSet); - } - - /** - * Woker run - */ - private void workerRun(Set uidSet, AtomicInteger control) { - for (;;) { - int myPosition = control.updateAndGet(old -> (old == SIZE ? SIZE : old + 1)); - if (myPosition == SIZE) { - return; - } - - doGenerate(uidSet, myPosition); - } - } - - /** - * Do generating - */ - private void doGenerate(Set uidSet, int index) { - long uid = uidGenerator.getUID(); - String parsedInfo = uidGenerator.parseUID(uid); - boolean existed = !uidSet.add(uid); - if (existed) { - System.out.println("Found duplicate UID " + uid); - } - - // Check UID is positive, and can be parsed - Assert.assertTrue(uid > 0L); - Assert.assertTrue(StringUtils.isNotBlank(parsedInfo)); - - if (VERBOSE) { - System.out.println(Thread.currentThread().getName() + " No." + index + " >>> " + parsedInfo); - } - } - - /** - * Check UIDs are all unique - */ - private void checkUniqueID(Set uidSet) throws IOException { - System.out.println(uidSet.size()); - Assert.assertEquals(SIZE, uidSet.size()); - } - -} +package com.github.wujun234.uid; + +import com.github.wujun234.uid.impl.CachedUidGenerator; +import org.apache.commons.lang.StringUtils; +import org.junit.Assert; +import org.junit.Ignore; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; + +import javax.annotation.Resource; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.ConcurrentSkipListSet; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Test for {@link CachedUidGenerator} + * + * @author yutianbao + * @author wujun + */ +@Ignore +@RunWith(SpringJUnit4ClassRunner.class) +@SpringBootTest(webEnvironment = WebEnvironment.NONE) +public class CachedUidGeneratorTest { + private static final int SIZE = 7000000; // 700w + private static final boolean VERBOSE = false; + private static final int THREADS = Runtime.getRuntime().availableProcessors() << 1; + + @Resource + private UidGenerator cachedUidGenerator; + + /** + * Test for serially generate + * + * @throws IOException + */ + @Test + public void testSerialGenerate() throws IOException { + // Generate UID serially + Set uidSet = new HashSet<>(SIZE); + for (int i = 0; i < SIZE; i++) { + doGenerate(uidSet, i); + } + + // Check UIDs are all unique + checkUniqueID(uidSet); + } + + /** + * Test for parallel generate + * + * @throws InterruptedException + * @throws IOException + */ + @Test + public void testParallelGenerate() throws InterruptedException, IOException { + AtomicInteger control = new AtomicInteger(-1); + Set uidSet = new ConcurrentSkipListSet<>(); + + // Initialize threads + List threadList = new ArrayList<>(THREADS); + for (int i = 0; i < THREADS; i++) { + Thread thread = new Thread(() -> workerRun(uidSet, control)); + thread.setName("UID-generator-" + i); + + threadList.add(thread); + thread.start(); + } + + // Wait for worker done + for (Thread thread : threadList) { + thread.join(); + } + + // Check generate 700w times + Assert.assertEquals(SIZE, control.get()); + + // Check UIDs are all unique + checkUniqueID(uidSet); + } + + /** + * Woker run + */ + private void workerRun(Set uidSet, AtomicInteger control) { + for (;;) { + int myPosition = control.updateAndGet(old -> (old == SIZE ? SIZE : old + 1)); + if (myPosition == SIZE) { + return; + } + + doGenerate(uidSet, myPosition); + } + } + + /** + * Do generating + */ + private void doGenerate(Set uidSet, int index) { + long uid = cachedUidGenerator.getUID(); + String parsedInfo = cachedUidGenerator.parseUID(uid); + boolean existed = !uidSet.add(uid); + if (existed) { + System.out.println("Found duplicate UID " + uid); + } + + // Check UID is positive, and can be parsed + Assert.assertTrue(uid > 0L); + Assert.assertTrue(StringUtils.isNotBlank(parsedInfo)); + + if (VERBOSE) { + System.out.println(Thread.currentThread().getName() + " No." + index + " >>> " + parsedInfo); + } + } + + /** + * Check UIDs are all unique + */ + private void checkUniqueID(Set uidSet) throws IOException { + System.out.println(uidSet.size()); + Assert.assertEquals(SIZE, uidSet.size()); + } + +} diff --git a/src/test/java/com/baidu/fsg/uid/DefaultUidGeneratorTest.java b/src/test/java/com/github/wujun234/uid/DefaultUidGeneratorTest.java similarity index 83% rename from src/test/java/com/baidu/fsg/uid/DefaultUidGeneratorTest.java rename to src/test/java/com/github/wujun234/uid/DefaultUidGeneratorTest.java index 62364c5..91b438c 100644 --- a/src/test/java/com/baidu/fsg/uid/DefaultUidGeneratorTest.java +++ b/src/test/java/com/github/wujun234/uid/DefaultUidGeneratorTest.java @@ -1,122 +1,124 @@ -package com.baidu.fsg.uid; - -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Set; -import java.util.concurrent.ConcurrentSkipListSet; -import java.util.concurrent.atomic.AtomicInteger; - -import javax.annotation.Resource; - -import org.apache.commons.lang.StringUtils; -import org.junit.Assert; -import org.junit.Test; -import org.junit.runner.RunWith; -import org.springframework.test.context.ContextConfiguration; -import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; - -import com.baidu.fsg.uid.impl.DefaultUidGenerator; - -/** - * Test for {@link DefaultUidGenerator} - * - * @author yutianbao - */ -@RunWith(SpringJUnit4ClassRunner.class) -@ContextConfiguration(locations = { "classpath:uid/default-uid-spring.xml" }) -public class DefaultUidGeneratorTest { - private static final int SIZE = 100000; // 10w - private static final boolean VERBOSE = true; - private static final int THREADS = Runtime.getRuntime().availableProcessors() << 1; - - @Resource - private UidGenerator uidGenerator; - - /** - * Test for serially generate - */ - @Test - public void testSerialGenerate() { - // Generate UID serially - Set uidSet = new HashSet<>(SIZE); - for (int i = 0; i < SIZE; i++) { - doGenerate(uidSet, i); - } - - // Check UIDs are all unique - checkUniqueID(uidSet); - } - - /** - * Test for parallel generate - * - * @throws InterruptedException - */ - @Test - public void testParallelGenerate() throws InterruptedException { - AtomicInteger control = new AtomicInteger(-1); - Set uidSet = new ConcurrentSkipListSet<>(); - - // Initialize threads - List threadList = new ArrayList<>(THREADS); - for (int i = 0; i < THREADS; i++) { - Thread thread = new Thread(() -> workerRun(uidSet, control)); - thread.setName("UID-generator-" + i); - - threadList.add(thread); - thread.start(); - } - - // Wait for worker done - for (Thread thread : threadList) { - thread.join(); - } - - // Check generate 10w times - Assert.assertEquals(SIZE, control.get()); - - // Check UIDs are all unique - checkUniqueID(uidSet); - } - - /** - * Worker run - */ - private void workerRun(Set uidSet, AtomicInteger control) { - for (;;) { - int myPosition = control.updateAndGet(old -> (old == SIZE ? SIZE : old + 1)); - if (myPosition == SIZE) { - return; - } - - doGenerate(uidSet, myPosition); - } - } - - /** - * Do generating - */ - private void doGenerate(Set uidSet, int index) { - long uid = uidGenerator.getUID(); - String parsedInfo = uidGenerator.parseUID(uid); - uidSet.add(uid); - - // Check UID is positive, and can be parsed - Assert.assertTrue(uid > 0L); - Assert.assertTrue(StringUtils.isNotBlank(parsedInfo)); - - if (VERBOSE) { - System.out.println(Thread.currentThread().getName() + " No." + index + " >>> " + parsedInfo); - } - } - - /** - * Check UIDs are all unique - */ - private void checkUniqueID(Set uidSet) { - System.out.println(uidSet.size()); - Assert.assertEquals(SIZE, uidSet.size()); - } - -} +package com.github.wujun234.uid; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.ConcurrentSkipListSet; +import java.util.concurrent.atomic.AtomicInteger; + +import com.github.wujun234.uid.impl.DefaultUidGenerator; +import org.apache.commons.lang.StringUtils; +import org.junit.Assert; +import org.junit.Ignore; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; + +/** + * Test for {@link DefaultUidGenerator} + * + * @author yutianbao + * @author wujun + */ +@Ignore +@RunWith(SpringJUnit4ClassRunner.class) +@SpringBootTest(webEnvironment = WebEnvironment.NONE) +public class DefaultUidGeneratorTest { + private static final int SIZE = 100000; // 10w + private static final boolean VERBOSE = true; + private static final int THREADS = Runtime.getRuntime().availableProcessors() << 1; + + @Autowired + private UidGenerator defaultUidGenerator; + + /** + * Test for serially generate + */ + @Test + public void testSerialGenerate() { + // Generate UID serially + Set uidSet = new HashSet<>(SIZE); + for (int i = 0; i < SIZE; i++) { + doGenerate(uidSet, i); + } + + // Check UIDs are all unique + checkUniqueID(uidSet); + } + + /** + * Test for parallel generate + * + * @throws InterruptedException + */ + @Test + public void testParallelGenerate() throws InterruptedException { + AtomicInteger control = new AtomicInteger(-1); + Set uidSet = new ConcurrentSkipListSet<>(); + + // Initialize threads + List threadList = new ArrayList<>(THREADS); + for (int i = 0; i < THREADS; i++) { + Thread thread = new Thread(() -> workerRun(uidSet, control)); + thread.setName("UID-generator-" + i); + + threadList.add(thread); + thread.start(); + } + + // Wait for worker done + for (Thread thread : threadList) { + thread.join(); + } + + // Check generate 10w times + Assert.assertEquals(SIZE, control.get()); + + // Check UIDs are all unique + checkUniqueID(uidSet); + } + + /** + * Worker run + */ + private void workerRun(Set uidSet, AtomicInteger control) { + for (;;) { + int myPosition = control.updateAndGet(old -> (old == SIZE ? SIZE : old + 1)); + if (myPosition == SIZE) { + return; + } + + doGenerate(uidSet, myPosition); + } + } + + /** + * Do generating + */ + private void doGenerate(Set uidSet, int index) { + long uid = defaultUidGenerator.getUID(); + String parsedInfo = defaultUidGenerator.parseUID(uid); + uidSet.add(uid); + + // Check UID is positive, and can be parsed + Assert.assertTrue(uid > 0L); + Assert.assertTrue(StringUtils.isNotBlank(parsedInfo)); + + if (VERBOSE) { + System.out.println(Thread.currentThread().getName() + " No." + index + " >>> " + parsedInfo); + } + } + + /** + * Check UIDs are all unique + */ + private void checkUniqueID(Set uidSet) { + System.out.println(uidSet.size()); + Assert.assertEquals(SIZE, uidSet.size()); + } + +} diff --git a/src/test/java/com/github/wujun234/uid/UidTestApplication.java b/src/test/java/com/github/wujun234/uid/UidTestApplication.java new file mode 100644 index 0000000..61e2d11 --- /dev/null +++ b/src/test/java/com/github/wujun234/uid/UidTestApplication.java @@ -0,0 +1,19 @@ +package com.github.wujun234.uid; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + + +/** + * UID 测试主类 + * + * @author wujun + * @date 2019.02.20 11:01 + */ +@SpringBootApplication +public class UidTestApplication { + + public static void main(String[] args) { + SpringApplication.run(UidTestApplication.class, args); + } +} diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml new file mode 100644 index 0000000..ed4e115 --- /dev/null +++ b/src/test/resources/application.yml @@ -0,0 +1,24 @@ +mybatis: + configuration: + default-fetch-size: 100 + default-statement-timeout: 30 + map-underscore-to-camel-case: true +spring: + datasource: + driver-class-name: com.mysql.jdbc.Driver + url: jdbc:mysql://xxx + username: xxx + password: xxx + +# 以下为可选配置, 如未指定将采用默认值 +uid: + timeBits: 30 # 时间位, 默认:30 + workerBits: 16 # 机器位, 默认:16 + seqBits: 7 # 序列号, 默认:7 + epochStr: "2019-02-20" # 初始时间, 默认:"2019-02-20" + enableBackward: true # 是否容忍时钟回拨, 默认:true + maxBackwardSeconds: 1 # 时钟回拨最长容忍时间（秒）, 默认:1 + CachedUidGenerator: # CachedUidGenerator相关参数 + boostPower: 3 # RingBuffer size扩容参数, 可提高UID生成的吞吐量, 默认:3 + paddingFactor: 50 # 指定何时向RingBuffer中填充UID, 取值为百分比(0, 100), 默认为50 + #scheduleInterval: 60 # 默认:不配置此项, 即不实用Schedule线程. 如需使用, 请指定Schedule线程时间间隔, 单位:秒 diff --git a/src/test/resources/uid/cached-uid-spring.xml b/src/test/resources/uid/cached-uid-spring.xml deleted file mode 100644 index 7f81032..0000000 --- a/src/test/resources/uid/cached-uid-spring.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/test/resources/uid/default-uid-spring.xml b/src/test/resources/uid/default-uid-spring.xml deleted file mode 100644 index 188be7f..0000000 --- a/src/test/resources/uid/default-uid-spring.xml +++ /dev/null @@ -1,22 +0,0 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/src/test/resources/uid/mybatis-spring.xml b/src/test/resources/uid/mybatis-spring.xml deleted file mode 100644 index 6339e9e..0000000 --- a/src/test/resources/uid/mybatis-spring.xml +++ /dev/null @@ -1,80 +0,0 @@ - - - - - - - - - - - - classpath:/uid/*.properties - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/test/resources/uid/mysql.properties b/src/test/resources/uid/mysql.properties deleted file mode 100644 index 01f1d6b..0000000 --- a/src/test/resources/uid/mysql.properties +++ /dev/null @@ -1,23 +0,0 @@ -#datasource db info -mysql.driver=com.mysql.jdbc.Driver -jdbc.url=jdbc:mysql://localhost:xxxx/xxxx -jdbc.username=xxxx -jdbc.password=xxxx -jdbc.maxActive=2 - -#datasource base -datasource.defaultAutoCommit=true -datasource.initialSize=2 -datasource.minIdle=0 -datasource.maxWait=5000 -datasource.testWhileIdle=true -datasource.testOnBorrow=true -datasource.testOnReturn=false -datasource.validationQuery=SELECT 1 FROM DUAL -datasource.timeBetweenEvictionRunsMillis=30000 -datasource.minEvictableIdleTimeMillis=60000 -datasource.logAbandoned=true -datasource.removeAbandoned=true -datasource.removeAbandonedTimeout=120 -datasource.filters=stat -