序

本文主要研究一下jdbc的batch的使用以及jpa的batch設(shè)置

batch

statement的batch操作，可以批量進行insert或update操作包雀，提升操作性能亲铡，特別是在大數(shù)據(jù)量的insert或update的時候。

使用方式

    @Test
    public void testSqlInjectSafeBatch(){
        String sql = "insert into employee (name, city, phone) values (?, ?, ?)";

        Connection conn = null;
        PreparedStatement pstmt = null;

        try{
            conn = dataSource.getConnection();
            conn.setAutoCommit(false);
            pstmt = conn.prepareStatement(sql);

            for (int i=0;i<3;i++) {
                pstmt.setString(1,"name"+i);
                pstmt.setString(2,"city"+i);
                pstmt.setString(3,"iphone"+i);
                pstmt.addBatch();
            }
            pstmt.executeBatch();

            conn.commit();

        }catch (SQLException e){
            e.printStackTrace();
            try {
                conn.rollback();
            } catch (SQLException e1) {
                e1.printStackTrace();
            }
        }finally {
            DbUtils.closeQuietly(pstmt);
            DbUtils.closeQuietly(conn);
        }
    }

主要就是每條操作參數(shù)設(shè)置完之后赞草，調(diào)用addBatch方法吆鹤，然后再所有操作都pstmt.addBatch()完之后，調(diào)用pstmt.executeBatch()
這種方式有個缺陷就是數(shù)據(jù)量大容易消耗內(nèi)存轰异，因此建議再分批次處理

@Test
    public void testSqlInjectSafeAndOOMSafeBatch(){
        String sql = "insert into employee (name, city, phone) values (?, ?, ?)";

        Connection conn = null;
        PreparedStatement pstmt = null;

        final int batchSize = 1000;
        int count = 0;

        try{
            conn = dataSource.getConnection();
            pstmt = conn.prepareStatement(sql);

            for (int i=0;i<10000;i++) {
                pstmt.setString(1,"name"+i);
                pstmt.setString(2,"city"+i);
                pstmt.setString(3,"iphone"+i);
                pstmt.addBatch();

                //小批量提交,避免OOM
                if(++count % batchSize == 0) {
                    pstmt.executeBatch();
                }
            }

            pstmt.executeBatch(); //提交剩余的數(shù)據(jù)

        }catch (SQLException e){
            e.printStackTrace();
        }finally {
            DbUtils.closeQuietly(pstmt);
            DbUtils.closeQuietly(conn);
        }
    }

jpa的batch設(shè)置

spring:
  jpa:
    database-platform: org.hibernate.dialect.PostgreSQLDialect
    hibernate:
      ddl-auto: update
      naming:
        implicit-strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
        physical-strategy: org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
    show-sql: true
    properties:
      hibernate:
        format_sql: true
        jdbc:
          batch_size: 5000
          batch_versioned_data: true
        order_inserts: true
        order_updates: true

通過設(shè)置spring.jpa.properties.hibernate.jdbc.batch_size來設(shè)置批量

實例測試

    @Test
    public void testJpaBatch() {
        List<DemoUser> demoUsers = new ArrayList<>();
        for(int i=0;i<10;i++){
            DemoUser demoUser = new DemoUser();
            demoUser.setPrincipal("demo");
            demoUser.setAccessToken(UUID.randomUUID().toString());
            demoUser.setAuthType(UUID.randomUUID().toString());
            demoUser.setDeptName(UUID.randomUUID().toString());
            demoUser.setOrgName(UUID.randomUUID().toString());
            demoUsers.add(demoUser);
        }
        StopWatch stopWatch = new StopWatch("jpa batch");
        stopWatch.start();
        demoUserDao.save(demoUsers);
        stopWatch.stop();
        System.out.println(stopWatch.prettyPrint());
    }

調(diào)整batch_size參數(shù)的測試結(jié)果

     沒有設(shè)置批量
     * StopWatch 'jpa batch': running time (millis) = 21383
     -----------------------------------------
     ms     %     Task name
     -----------------------------------------
     21383  100%

     設(shè)置批量500
     StopWatch 'jpa batch': running time (millis) = 16790
     -----------------------------------------
     ms     %     Task name
     -----------------------------------------
     16790  100%

     批量1000
     StopWatch 'jpa batch': running time (millis) = 12317
     -----------------------------------------
     ms     %     Task name
     -----------------------------------------
     12317  100%

     批量5000
     StopWatch 'jpa batch': running time (millis) = 13190
     -----------------------------------------
     ms     %     Task name
     -----------------------------------------
     13190  100%

小結(jié)

jdbc的batch參數(shù)對于大數(shù)據(jù)量的新增/更新操作來說搭独，非常有用廊镜，可以提升批量操作的效率。

doc

• Batch Processing in JDBC
• JDBC: Batch Updates
• 三種JDBC批量插入編程方法的比較
• Batch Insert In Java – JDBC
• JDBC PreparedStatement example – Batch Update
• JDBC - Batch Processing
• Java Jdbc減少交互提升批量處理性能配椭，到底該如何優(yōu)化才好雹姊？
• Spring Data JPA: Batch insert for nested entities
• Spring JPA Hibernate - JpaRepository Insert (Batch)