使用 Spring Batch 構建企業級批處理應用-1

摘要: 引言總述本系列文章旨在通過示例搭建以及特性介紹，詳細講述如何利用 Spring Batch 開發企業批處理應用。本系列文章共分為三部分，第一部分初步介紹了批處理以及 Spring Batch 的相關概念，同時搭建了一個簡單的基于 Spring Batch 的批處理應用。第二部分介紹了 Step Flow 以及并發支持。第三部分則主要介紹了 Spring Batch 對任務監控的支持。下面讓我們進入... 閱讀全文

posted @ 2020-03-06 13:45 paulwong 閱讀(414) | 評論 (0) | 編輯收藏

MobaXterm注冊版

LINUX 的shell神器
https://www.upload.ee/files/11132815/MobaXterm_Pro_Portable_SetuP_v20_fu11.rar.html

posted @ 2020-02-27 17:44 paulwong 閱讀(514) | 評論 (0) | 編輯收藏

各種獲取JVM DUMP的方法

JVM 的線程堆棧 dump 也稱 core dump，內容為文本，主要包含當時 JVM 的線程堆棧，堆 dump 也稱 heap dump，內容為二進制格式，主要包含當時 JVM 堆內存中的內容。由于各個操作系統、各個 JVM 實現不同，即使同一 JVM 實現，各個版本也有差異，本文描述的方法都基于 64 位 Linux 操作系統環境，Java 8 Oracle HotSpot JVM 實現。

堆棧和堆的內容在定位問題的時候，都是非常重要的信息。線程堆棧 dump 可以了解當時 JVM 中所有線程的運行情況，比如線程的狀態和當前正在運行的代碼行。堆 dump 可以了解當時堆的使用情況，各個類實例的數量及各個實例所占用的空間大小。

線程堆棧

使用 jstack

jstack 是 JDK 自帶的工具，用于 dump 指定進程 ID(PID)的 JVM 的線程堆棧信息。

# 打印堆棧信息到標準輸出 jstack PID  
# 打印堆棧信息到標準輸出，會打印關于鎖的信息 jstack -l PID  
強制打印堆棧信息到標準輸出，如果使用 jstack PID 沒有響應的情況下(此時 JVM 進程可能掛起)，
加 -F 參數 jstack -F PID

使用 jcmd

jcmd 是 JDK 自帶的工具，用于向 JVM 進程發送命令，根據命令的不同，可以代替或部分代替 jstack、jmap 等。可以發送命令 Thread.print 來打印出 JVM 的線程堆棧信息。

# 下面的命令同等于 jstack PID 
jcmd PID Thread.print  

# 同等于 jstack -l PID 
jcmd PID Thread.print -l

使用 kill -3

kill 可以向特定的進程發送信號(SIGNAL)，缺省情況是發送終止(TERM) 的信號，即 kill PID 與 kill -15 PID 或 kill -TERM PID 是等價的。JVM 進程會監聽 QUIT 信號(其值為 3)，當收到這個信號時，會打印出當時的線程堆棧和堆內存使用概要，相比 jstack，此時多了堆內存的使用概要情況。但 jstack 可以指定 -l 參數，打印鎖的信息。

kill -3 PID 
# 或 kill -QUIT PID

堆

-XX:+HeapDumpOnOutOfMemoryError

添加 JVM 參數 -XX:+HeapDumpOnOutOfMemoryError 后，當發生 OOM(OutOfMemory)時，自動堆 dump。缺省情況下，JVM 會創建一個名稱為 java_pidPID.hprof 的堆 dump 文件在 JVM 的工作目錄下。但可以使用參數 -XX:HeapDumpPath=PATH 來指定 dump 文件的保存位置。

# JVM 發生 OOM 時，會自動在 /var/log/abc 目錄下產生堆 dump 文件 java_pidPID.hprof 
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/abc/

jmap

jmap 也是 JDK 自帶的工具，主要用于獲取堆相關的信息。

堆 dump

# 將 JVM 的堆 dump 到指定文件，如果堆中對象較多，需要的時間會較長，子參數 format 只支持 b，
即二進制格式
jmap -dump:format=b,file=FILE_WITH_PATH

# 如果 JVM 進程未響應命令，可以加上參數 -F 嘗試
jmap -F -dump:format=b,file=FILE_WITH_PATH

# 可以只 dump 堆中的存活對象，加上 live 子參數，但使用 -F 時不支持 live
jmap -dump:live,format=b,file=FILE_WITH_PATH

獲取堆概要信息

# -heap 參數用于查看指定 JVM 進程的堆的信息，包括堆的各個參數的值，堆中新生代、年老代的內存大小、使用率等 
jmap -heap PID  

# 同樣，如果 JVM 進程未響應命令，可以加上參數 -F 嘗試 
jmap -F -heap PID

一個實例輸出如下：

Attaching to process ID 68322, please wait
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b16

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 268435456 (256.0MB)
   NewSize                  = 8388608 (8.0MB)
   MaxNewSize               = 89128960 (85.0MB)
   OldSize                  = 16777216 (16.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 41943040 (40.0MB)
   used     = 1701504 (1.6226806640625MB)
   free     = 40241536 (38.3773193359375MB)
   4.05670166015625% used
From Space:
   capacity = 4194304 (4.0MB)
   used     = 0 (0.0MB)
   free     = 4194304 (4.0MB)
   0.0% used
To Space:
   capacity = 5242880 (5.0MB)
   used     = 0 (0.0MB)
   free     = 5242880 (5.0MB)
   0.0% used
PS Old Generation
   capacity = 30408704 (29.0MB)
   used     = 12129856 (11.56793212890625MB)
   free     = 18278848 (17.43206787109375MB)
   39.889421134159484% used

16658 interned Strings occupying 1428472 bytes.

獲取堆中的類實例統計

# 打印 JVM 堆中的類實例統計信息，以占用內存的大小排序，同樣，如果 JVM 未響應命令，也可以使用 -F 參數 
jmap -histo PID  

# 也可以只統計堆中的存活對象，加上 live 子參數，但使用 -F 時不支持 live 
jmap -histo:live PID

使用 jcmd

# 等同 jmap -dump:live,format=b,file=FILE_WITH_PATH
jcmd PID GC.heap_dump FILE_WITH_PATH

# 等同 jmap -dump:format=b,file=FILE_WITH_PATH
jcmd PID GC.heap_dump -all FILE_WITH_PATH

# 等同 jmap -histo:live PID
jcmd PID GC.class_histogram

# 等同 jmap -histo PID
jcmd PID GC.class_histogram -all

posted @ 2020-02-24 22:03 paulwong 閱讀(1251) | 評論 (0) | 編輯收藏

Mongodb shell中select in 的實現

var bookIds = db.likes.find({userId:100}).map(function(like) {
return like.bookId;
});
var books = db.books.find({_id:{$in:bookIds}});

db.REPORT_ITEM.count({REQUEST_ID : db.BATCH_CONTROL.find({ FILE_NAME : "20200218_100000.file" }).map(function(like) {
return like._id;
})[0].str, JOB_TYPE_ENUM:"CHECK"})

posted @ 2020-02-21 23:10 paulwong 閱讀(528) | 評論 (0) | 編輯收藏

采用 jstatd 監控服務器

https://www.jianshu.com/p/97f09e1648a6

posted @ 2020-02-21 17:08 paulwong 閱讀(343) | 評論 (0) | 編輯收藏

mongodb錯誤記錄

https://blog.csdn.net/wangxiaotongfan/article/details/81560463

posted @ 2020-02-21 08:50 paulwong 閱讀(282) | 評論 (0) | 編輯收藏

JAVA進程無故退出而沒有留下LOG？

LINUX通常有個OOM KILLER進程，對于他認為吃內存的進程，會根據一定的算分，執行kill -9殺掉，查看日志如下：

#less /tmp/messages
Feb 20 03:55:09 ip kernel: Out of memory: Kill process 5974 (java) score 494 or sacrifice child
Feb 20 03:55:09 ip kernel: Killed process 5974 (java), UID 1001, total-vm:23674020kB, anon-rss:17503912kB, file-rss:0kB, shmem-rss:0kB

那如何屏蔽呢？

#/etc/cron.d/oom_disable

*/1**** root pgrep -f "java" | while read PID;do echo -17 > /proc/$PID/oom_adj;done

參考文檔：
Linux內核OOM機制的詳細分析
http://blog.chinaunix.net/uid-29242873-id-3942763.html

posted @ 2020-02-20 15:57 paulwong 閱讀(902) | 評論 (0) | 編輯收藏

!!21 MOST IMPORTANT JAVA 8 VM OPTIONS FOR SERVERS

27/03/2016 MAURICE KNOPP 7 KOMMENTARE

In this video I explain some 21 JVM parameters which are suited for most server applications. If you have any questions, you can read those links below for more information or just ask in the comments section.

I run several Java enterprise server applications. I often wondered – what are the best „default“ JVM settings for a server application to start with in production? I read a lot on the web and tried several things myself and wanted to share what I found out, so far. Links containing more information about JVM optimization can be found here:

http://blog.sokolenko.me/2014/11/javavm-options-production.html

http://www.petefreitag.com/articles/gctuning/

http://stas-blogspot.blogspot.de/2011/07/most-complete-list-of-xx-options-for.html

So let’s start:

-server

Use „-server“: All 64-bit JVMs use the server VM as default anyway. This setting generally optimizes the JVM for long running server applications instead of startup time. The JVM will collect more data about the Java byte code during program execution and generate the most efficient machine code via JIT.

-Xms=<heap size>[g|m|k] -Xmx=<heap size>[g|m|k]

The „-Xmx/-Xms“ settings specify the maximum and minimum values for the JVM heap memory. For servers, both params should have the same value to avoid heap resizing during runtime. I’ve applications running with 16GB heap sizes without an issue.

Depending on your application, you will have to try out how much memory will be best suited for your use case.

-XX:MaxMetaspaceSize=<metaspace size>[g|m|k]

Java 8 has no „Permanent Generation“ (PermGen) anymore but requires additional „Metaspace“ memory instead. This memory is used, in addition to the heap memory we specified before, for storing class meta data information.

The default size will be unlimited – I tend to limit MaxMetaspaceSize with a somewhat high value. Just in case something goes wrong with the application, the JVM will not hog all the memory of the server.

I suggest: Let your application run for a couple of days to get a feeling for how much Metaspace Size it uses normally. Upon next restart of the application set the limit to e.g. double the value.

-XX:+CMSClassUnloadingEnabled

Additionally, you might want to allow the JVM to unload classes which are held in memory but no code is pointing to them any more. If your application generates lots of dynamic classes, this is what you want.

-XX:+UseConcMarkSweepGC

This option makes the JVM use the ConcurrentMarkSweepGC – It can do much work in parallel to program execution but in some circumstances a „full GC“ with a „STW pause“ might still occur. I’ve read many articles and came to the conclusion that this GC is still the best one for server workloads.

-XX:+CMSParallelRemarkEnabled

The option CMSParallelRemarkEnabled means the remarking is done in parallel to program execution – which is what you want if your server has many cores (and most servers do).

 -XX:+UseCMSInitiatingOccupancyOnly  -XX:CMSInitiatingOccupancyFraction=<percent>

Normally the GC will use heuristics to know when it’s time to clear memory. GC might kick in too late with default settings (causing full-Gcs).
Some sources say it might be a good idea to disable heuristics altogether and just use generation occupancy to start a CMS collection cycle. Setting values around 70% worked fine for all of my applications and use cases.

-XX:+ScavengeBeforeFullGC

The first option tells the GC to first free memory by clearing out the „young generation“ or fairly new objects before doing a full GC.

-XX:+CMSScavengeBeforeRemark

CMSScavengeBeforeRemark does attempt a minor collection before the CMS remark phase – thus keeping the remark pause afterwards short.

-XX:+CMSClassUnloadingEnabled

The option „-XX:+CMSClassUnloadingEnabled“ here tells the JVM to unload classes, which are not needed any more by the running application. If you deploy war files to an application server like wildfly, tomcat or glassfish without restarting the server after the deployment, this flag is for you.

-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

The option „-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses“ is especially important if your application uses RMI (remote method invocation). The usage of RMI will cause the JVM to do a FULL-GC EVERY HOUR! This might be a very bad idea for large heap sizes because the FULL-GC pause might take up to several seconds. It would be better to do a concurrent GC and try to unload unused classes to free up more memory – which is exactly what the second option does.

-XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:"<path to log>"

These options shown here will write out all GC related information to a specified log file. You can see how well your GC configuration works by looking into it.

I personally prefer to use the „Visual GC“ plug in for the „Visual VM“ tool to monitor the general JVM and GC behavior.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path to dump>`date`.hprof

When your JVM runs out of memory, you will want to know why. Since the OOM error might be hard to reproduce and you want to get your production server up and running again – you should specify a path for a heap dump. When things have settled down, you can analyze the dump afterwards.

-Djava.rmi.server.hostname=<external IP> -Dcom.sun.management.jmxremote.port=<port>

These options will help you to specify an IP and port for JMX – you will need those ports open to connect remotely to a JVM running on a server for tools like VisualVM. You can gain deep insights over cpu and memory usage, gc behaviour, class loading, thread count and usage of your application this way.

Visual VM
Lastly, I would like to recommend to you the VisualVM tool which is bundled with the Java 8 JDK. You can use it to gain more insights about your specific application behaviour on the JVM – like cpu and memory usage, thread utilisation and much more.

VisualVM can be extended with a plug in called „Visual GC“. It will briefly show you VERY detailed information about the usage of the young and old generation object spaces. You can easily spot problems with garbage collection simply by analyzing these graphs during application runtime.

Thank you very much for watching! If you liked the video you might consider giving it a „thumbs up“. If you have any questions – just put them in the comments section. I will reply as quickly as possible.

-------------------------------------------------------

-XX:+UseCompressedOops [If Max Heap allocation is less than 32GB]
This can save a significant amount of memory and this option should already be enabled by default on recent java 8 versions. This option allowes object references to be stored as 32-bit values instead of 64-bit on 64-bit JVMs. This leads to before mentioned memory savings.

-XX:+AggressiveOpts
This option will enable performance options which are hoped to become enabled by default in upcoming released of the JVM. This option sets some performance settings but is marked as experimental! So you should only enable it, when you have to possibility to test your application thoroughly before enabling this flag on an production server.

-XX:+UseStringDeduplication
Since Java 8 update 20 you can use this option to reduce the memory usage of your application. The JVM will spot identical strings in memory, remove the duplicated and point all references to the remaining, single instance of the string.

-XX:+UseG1GC
Will tell the JVM to use the most recent G1 garbage collector. You are trading better application response times (due to shorter gc times with G1) against lower throughput (compared against good old ConcMarkSweepGC / CMS). If your application can deliver more value through short gc times, then G1 is definately better suited. Otherwise on Java 8, I’d recommend sticking with CMS.

Concerning your Tomcat 8 question, I’d suggest you have a look into it with the „VisualVM“ tool. Look at memory usage, GC times (visual GC plugin), pull and analyse stack traces or thread dumps to find the weak spot. You might also consider attaching a debugger to tomcat to find the bug.

https://www.maknesium.de/21-most-important-java-8-vm-options-for-servers

posted @ 2020-02-16 22:30 paulwong 閱讀(359) | 評論 (0) | 編輯收藏

windows下jenkins提示文件名太長

由于jenkins是調用windows的git取代碼，因此是git的問題，進行如下配置即可：

git config --global core.longpaths true

posted @ 2020-02-14 14:37 paulwong 閱讀(540) | 評論 (0) | 編輯收藏

SPRING BOOT 環境下減少中間件依賴的UNIT測試

SPRING BOOT 環境下，測試有時會依賴于外部的中間件，如Mysql，Activemq，Mongodb等，那如何能減少這種依賴呢？
SPRING BOOT其實已經實現了自動化配置。

Mongodb

SPRING BOOT的自動化配置文件：org.springframework.boot.autoconfigure.mongo.embeddedEmbedded.MongoAutoConfiguration.java

在pom.xml中新增一test profile，并添加相應jar包，這樣可防止對其他profile的影響，如果是在Eclipse跑測試，需在Project的屬性中指定Active Profile為test，以覆蓋pom.xml的定義。
這種方式即使是使用SPRING DATA MONGODB的REPOSITORY也是適用的。

  <profile>
        <id>test</id>
        <dependencies>
            <dependency>
                <groupId>de.flapdoodle.embed</groupId>
                <artifactId>de.flapdoodle.embed.mongo</artifactId>
                <scope>test</scope>
            </dependency>
        </dependencies>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
    </profile>

在application-test.yaml中添加端口，其他如IP那些信息都不需要

spring:
   data:
      mongodb:
         port: 27017

unit test config

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.util.List;

import javax.annotation.PostConstruct;
import javax.sql.DataSource;

import org.bson.Document;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
import org.springframework.jdbc.datasource.init.DatabasePopulatorUtils;
import org.springframework.jdbc.datasource.init.ResourceDatabasePopulator;
import org.springframework.util.FileCopyUtils;

@Configuration
@Profile({"test", "integrationTest"})
@EnableMongoRepositories(
        basePackages = {"paul.com.repository"
        }
)
public class EmbeddedDataSourceConfiguration {

    @Value("classpath:/initdata/USER.json")
    private Resource userResource;

    @Value("classpath:/initdata/MEMBERS.json")
    private Resource membersResource;

    @Autowired
    private ResourceLoader resourceLoader;

    @Autowired
    private DataSource dataSource;

    @Autowired
    private MongoTemplate  mongoTemplate;

    @PostConstruct
    protected void initialize() throws FileNotFoundException, IOException {
        this.initializeHsqldb();
        this.initializeMongodb();
    }

    private void initializeHsqldb() {
        ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
        populator.addScript(resourceLoader.getResource("classpath:/org/springframework/batch/core/schema-hsqldb.sql"));
        populator.setContinueOnError(true);
        DatabasePopulatorUtils.execute(populator , dataSource);
    }

    private void initializeMongodb() throws FileNotFoundException, IOException {
        this.saveResource(userResource, "USER");

        this.saveDocumentList(membersResource, "MEMBER");
    }

    private void saveResource(Resource resource, String collectionName) {
        String resourceJson = this.asString(resource);
        Document resourceDocument = Document.parse(resourceJson);
        this.mongoTemplate.save(resourceDocument, collectionName);
    }

    private void saveDocumentList(Resource resource, String collectionName) {
        String resourceJson = this.asString(resource);
        Document resourceDocument = Document.parse("{ \"list\":" + resourceJson + "}");
        List<Document> documentList = resourceDocument.get("list", List.class);
        documentList.forEach(document -> this.mongoTemplate.save(document, collectionName));
    }

    private String asString(Resource resource) {
        try (Reader reader = new InputStreamReader(resource.getInputStream(), StandardCharsets.UTF_8)) {
            return FileCopyUtils.copyToString(reader);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

//    @Bean(destroyMethod="close")
//    public DataSource dataSource() {
//        BasicDataSource dataSource = new BasicDataSource();
//        dataSource.setDriverClassName(environment.getProperty("batch.jdbc.driver"));
//        dataSource.setUrl(environment.getProperty("batch.jdbc.url"));
//        dataSource.setUsername(environment.getProperty("batch.jdbc.user"));
//        dataSource.setPassword(environment.getProperty("batch.jdbc.password"));
//        return dataSource;
//    }
}

ActiveMQ

只需更改application-test.yml中的brokerUrl為vm://embedded即可

spring:
   activemq:
      broker-url: vm://embedded?broker.persistent=false,useShutdownHook=false
      in-memory: true
      non-blocking-redelivery: true
      #packages:
        #trust-all: false
        #trusted: com.memorynotfound
      pool:
        block-if-full: true
        block-if-full-timeout: -1
        create-connection-on-startup: true
        enabled: false
        expiry-timeout: 0
        idle-timeout: 30000
        max-connections: 1
        maximum-active-session-per-connection: 500
        reconnect-on-exception: true
        time-between-expiration-check: -1
        use-anonymous-producers: true
        user: admin
        #password: ENC(hWJHuMyhydTqyF32neasTw==)
        password: admin

關系型數據庫

將在application-test.yml中的數據庫信息刪除，同時在pom.xml中添加jar包依賴，這邊是采用HSQL數據庫

  <profile>
        <id>test</id>
        <dependencies>
            <dependency>
                <groupId>org.hsqldb</groupId>
                <artifactId>hsqldb</artifactId>
                <scope>test</scope>
            </dependency>
        </dependencies>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
    </profile>

非SPRING BOOT/SPRING的純JDK環境可參考
https://github.com/yandex-qatools/embedded-services

https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo

https://github.com/jonyfs/spring-boot-data-embedded-mongodb/blob/master/src/main/java/br/com/jonyfs/spring/boot/data/embedded/mongodb/config/MongoConfig.java

ActiveMQ:
https://memorynotfound.com/spring-boot-embedded-activemq-configuration-example/

posted @ 2020-02-07 10:28 paulwong 閱讀(676) | 評論 (0) | 編輯收藏

僅列出標題

My Links

Blog Stats

常用鏈接

留言簿(66)

隨筆分類(1386)

隨筆檔案(1144)

文章分類(7)

文章檔案(10)

相冊

收藏夾(2)

AI

Develop

E-BOOK

Other

養生

微服務

搜索

最新評論

閱讀排行榜

評論排行榜

60天內閱讀排行

線程堆棧

使用 jstack

使用 jcmd

使用 kill -3

堆

-XX:+HeapDumpOnOutOfMemoryError

jmap

堆 dump

獲取堆概要信息

使用 jcmd

Mongodb

ActiveMQ

關系型數據庫