1老虫,什么是垂直/水平拆分模式
2叶骨,基于客戶端與服務(wù)器端實(shí)現(xiàn)分表分庫(kù)的區(qū)別
3,單表達(dá)到多大的量開(kāi)始分表分庫(kù)
4祈匙,數(shù)據(jù)庫(kù)分表分庫(kù)策略有哪些
5忽刽, shardingSphere 實(shí)戰(zhàn)分庫(kù)分表
6,為什么不推薦使用mycat 實(shí)現(xiàn)分庫(kù)分表
7夺欲,分表分庫(kù)后查詢存在哪些優(yōu)缺點(diǎn)
8跪帝,分表分庫(kù)后如何實(shí)現(xiàn)分頁(yè)查詢
9,分表分庫(kù)后如何實(shí)現(xiàn)排序查詢些阅;
垂直/水平拆分模式
1伞剑,數(shù)據(jù)庫(kù)拆分主要指分庫(kù)分表,其目的主要是分散數(shù)據(jù)庫(kù)壓力市埋,達(dá)到橫向發(fā)展黎泣,滿足均衡訪問(wèn)等。
2缤谎, 數(shù)據(jù)庫(kù)拆分主要有2種形式: 垂直拆分和水平拆分抒倚。
A 垂直拆分:
將不同的業(yè)務(wù)功能相關(guān)的表放到不同的數(shù)據(jù)庫(kù)中,也就是類似于 微服務(wù)架構(gòu)中 會(huì)員數(shù)據(jù)庫(kù)/訂單數(shù)據(jù)庫(kù)/支付數(shù)據(jù)庫(kù)
B 水平拆分:
當(dāng)一張表的業(yè)務(wù)量行數(shù)如果超過(guò) 500 行坷澡,分頁(yè)/排序效率還是非常低托呕,可以對(duì)同一個(gè)表數(shù)據(jù)實(shí)現(xiàn)拆分放到多個(gè)不同的數(shù)據(jù)庫(kù)表中存放。
Mayikt_memberdb01
Mayikt_user
Mayikt_memberdb02
Mayikt_user
基于客戶端與服務(wù)器端實(shí)現(xiàn)分表分庫(kù)的區(qū)別:
1,基于服務(wù)器端mycat 實(shí)現(xiàn)數(shù)據(jù)庫(kù)代理
優(yōu)點(diǎn): 能夠保證數(shù)據(jù)庫(kù) 的安全性
缺點(diǎn): 效率比較低
2项郊,基于客戶端Shardingjdbc 實(shí)現(xiàn)數(shù)據(jù)庫(kù)代理:
優(yōu)點(diǎn): 效率比較高
缺點(diǎn): 不能夠保證數(shù)據(jù)庫(kù)的安全性馅扣,內(nèi)存溢出;
單表達(dá)到多大量開(kāi)始分表分庫(kù):
【推薦】單表行數(shù)超過(guò) 500 萬(wàn)行或者單表容量超過(guò) 2GB着降,才推薦進(jìn)行分庫(kù)分表差油。
說(shuō)明:如果預(yù)計(jì)三年后的數(shù)據(jù)量根本達(dá)不到這個(gè)級(jí)別,請(qǐng)不要在創(chuàng)建表時(shí)就分庫(kù)分表任洞。
摘自:阿里巴巴java開(kāi)發(fā)手冊(cè)
數(shù)據(jù)庫(kù)分表分庫(kù)策略有哪些:
取余/取模 均勻存放數(shù)據(jù) 缺點(diǎn): 不夠擴(kuò)容厌殉。
按照范圍分片: 1-500萬(wàn) 501萬(wàn)-10000萬(wàn);
按照日期進(jìn)行分片 日志侈咕,訂單信息 同居
按照枚舉值分片;
二進(jìn)制取模范圍分片:
一致性hash 分片,類似于HashMap 缺點(diǎn) 數(shù)據(jù)存放不均勻器紧。
按照目前字段前綴指定的進(jìn)行分區(qū) mayikt wuhan
按照前綴ASCII 碼和值進(jìn)行取模范圍分片耀销。
主流分片算法: 取余/取模 日期 一致性hash 分片
根據(jù)user_id
Userid=1%2=1
Userid=2%2=0
Userid=3%2=1
Userid=4%2=0
不能實(shí)現(xiàn)擴(kuò)容。
ShardingSphere 實(shí)戰(zhàn)分表分庫(kù)
ShardingSphere 是一套開(kāi)源的分布式數(shù)據(jù)庫(kù)中間件解決方案組成的生態(tài)圈铲汪,它由Sharding-JDBC熊尉, Sharding-PROXY 和 Sahriding 這三款 相互獨(dú)立的產(chǎn)品組成。
他們均提供標(biāo)準(zhǔn)化的數(shù)據(jù)分片掌腰,分布式事務(wù) 和數(shù)據(jù)庫(kù)治理功能狰住,可適用于java同構(gòu),已購(gòu)語(yǔ)言齿梁,云原生等各種多樣化的應(yīng)用場(chǎng)景催植。
Sharding_JDBC 可以實(shí)現(xiàn)
分庫(kù) & 分表
- 讀寫分離
- 分片策略定制化
- 無(wú)中心化分布式主鍵
SharingdSphere 環(huán)境搭建:
數(shù)據(jù)庫(kù)準(zhǔn)備;
DROP TABLE IF EXISTS `mayikt_user_0`;
CREATE TABLE `mayikt_user_0` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`age` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;
-- ----------------------------
-- Records of mayikt_user_0
-- ----------------------------
INSERT INTO `mayikt_user_0` VALUES ('2', 'mayikt2', '2');
INSERT INTO `mayikt_user_0` VALUES ('4', 'mayikt4', '4');
INSERT INTO `mayikt_user_0` VALUES ('6', 'mayikt6', '6');
INSERT INTO `mayikt_user_0` VALUES ('8', 'mayikt8', '8');
-- ----------------------------
-- Table structure for mayikt_user_1
-- ----------------------------
DROP TABLE IF EXISTS `mayikt_user_1`;
CREATE TABLE `mayikt_user_1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`age` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;
-- ----------------------------
-- Records of mayikt_user_1
-- ----------------------------
INSERT INTO `mayikt_user_1` VALUES ('1', 'mayikt1', '1');
INSERT INTO `mayikt_user_1` VALUES ('3', 'mayikt3', '3');
INSERT INTO `mayikt_user_1` VALUES ('5', 'mayikt5', '5');
INSERT INTO `mayikt_user_1` VALUES ('7', 'mayikt7', '7');
INSERT INTO `mayikt_user_1` VALUES ('9', 'mayikt9', '9');
Maven 依賴:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.5.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.taotao</groupId>
<artifactId>sharding-jdbc</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>sharding-jdbc</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.3.2</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.20</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.62</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>io.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-boot-starter</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>io.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-namespace</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
application.yml 配置文件
# 數(shù)據(jù)源 mayiktdb
sharding:
jdbc:
datasource:
names: shard1,shard2
# 第一個(gè)數(shù)據(jù)庫(kù)
shard1:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
jdbc-url: jdbc:mysql://127.0.0.1:3306/mayikt-member?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
username: root
password: root
shard2:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
jdbc-url: jdbc:mysql://127.0.0.1:3306/mayikt?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
username: root
password: root
# 水平拆分的數(shù)據(jù)庫(kù)(表) 配置分庫(kù) + 分表策略 行表達(dá)式分片策略
config:
sharding:
#默認(rèn)數(shù)據(jù)源(未設(shè)置分表策略的表將默認(rèn)用此數(shù)據(jù)源)
default-database-name: shard1
#默認(rèn)分庫(kù)策略
default-database-strategy:
standard:
sharding-column: id
precise-algorithm-class-name: com.taotao.rfspringboot.config.DataBasePreciseRule
tables:
mayikt_user: ##虛擬表名稱 mayikt_user_0 mayikt_user_1
actual-data-nodes: shard$->{1..2}.mayikt_user_$->{0..2} # 沒(méi)有帶上 分片字段
table-strategy:
standard: ##SELECT * FROM mayikt_user_1 where id =1
precise-algorithm-class-name: com.taotao.rfspringboot.config.MayiktRangeShardingAlgorithm
sharding-column: id
# 打印執(zhí)行的數(shù)據(jù)庫(kù)
props:
sql:
show: true
# 打印執(zhí)行的sql語(yǔ)句
spring:
main:
allow-bean-definition-overriding: true
server:
port: 8081
mybatis-plus:
# 如果是放在src/main/java目錄下 classpath:/com/yourpackage/*/com.exchange.mapper/*Mapper.com.exchange.mapper
# 如果是放在resource目錄 classpath:/com.exchange.mapper/*Mapper.com.exchange.mapper
mapper-locations: classpath:/mapper/*.xml
#實(shí)體掃描,多個(gè)package用逗號(hào)或者分號(hào)分隔
typeAliasesPackage: com.taotao.rfspringboot.entity
global-config:
#主鍵類型 0:"數(shù)據(jù)庫(kù)ID自增", 1:"用戶輸入ID",2:"全局唯一ID (數(shù)字類型唯一ID)", 3:"全局唯一ID UUID";
#若采用雪花算法生成id勺择,需要在生成的實(shí)體類中將id的type = IdType.AUTO去掉
id-type: 0
#字段策略 0:"忽略判斷",1:"非 NULL 判斷"),2:"非空判斷"
field-strategy: 2
#駝峰下劃線轉(zhuǎn)換
db-column-underline: true
#刷新mapper 調(diào)試神器
#refresh-mapper: true
#數(shù)據(jù)庫(kù)大寫下劃線轉(zhuǎn)換
#capital-mode: true
# Sequence序列接口實(shí)現(xiàn)類配置
#邏輯刪除配置(下面3個(gè)配置)
logic-delete-value: Y
logic-not-delete-value: N
#sql-injector: com.nky.pork.quality.standard.conf.MybatisPlusConfig
configuration:
#配置返回?cái)?shù)據(jù)庫(kù)(column下劃線命名&&返回java實(shí)體是駝峰命名)创南,自動(dòng)匹配無(wú)需as(沒(méi)開(kāi)啟這個(gè),SQL需要寫as: select user_id as userId)
map-underscore-to-camel-case: true
cache-enabled: false
#配置JdbcTypeForNull, oracle數(shù)據(jù)庫(kù)必須配置
jdbc-type-for-null: 'null'
entity:
package com.taotao.shardingjdbc.entity;
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;
/**
*@author tom
*Date 2020/11/6 0006 8:27
*
*/
@Data
@TableName("mayikt_user")
public class Mayikt_user {
private Integer id;
private String name;
private Integer age;
public Mayikt_user(Integer id, String name, Integer age) {
this.id = id;
this.name = name;
this.age = age;
}
}
數(shù)據(jù)庫(kù)訪問(wèn):
package com.taotao.shardingjdbc.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.taotao.shardingjdbc.entity.MayiktUser;
import org.apache.ibatis.annotations.Select;
import java.util.List;
/**
*@author tom
*Date 2020/11/6 0006 8:29
*
*/
public interface MayiktUserMapper extends BaseMapper<MayiktUser> {
/**
* 查詢所有
*
* @return
*/
@Select("SELECT * FROM mayikt_user")
List<MayiktUser> userList();
/**
* 分頁(yè)查詢
*
* @return
*/
@Select("SELECT * FROM mayikt_user limit 0,2")
List<MayiktUser> userListPage();
/**
* 排序
*
* @return
*/
@Select("SELECT * FROM mayikt_user order by id desc ")
List<MayiktUser> userOrderBy();
/**
* get by id
*
* @return
*/
@Select("SELECT * FROM mayikt_user where id =#{id} ")
List<MayiktUser> getByUserId(Long id);
}
Controller:
package com.taotao.shardingjdbc.controller;
import com.taotao.shardingjdbc.entity.MayiktUser;
import com.taotao.shardingjdbc.mapper.MayiktUserMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.List;
/**
*@author tom
*Date 2020/11/6 0006 8:41
*
*/
@RestController
public class UserController {
@Autowired
private MayiktUserMapper mayiktUserMapper;
@RequestMapping("/insertUser")
public String insertUser(){
for (int i = 0; i <10 ; i++) {
MayiktUser mayiktUser=new MayiktUser(i,"mykt"+i,i);
try {
mayiktUserMapper.insert(mayiktUser);
}catch (Exception e){
e.printStackTrace();
}
}
return "success";
}
/**
* 查詢所有
*/
@RequestMapping("/userList")
public List<MayiktUser> userList(){
return mayiktUserMapper.userList();
}
/**
* 分頁(yè)查詢
*/
@RequestMapping("/userListPage")
public List<MayiktUser> userListpage(){
return mayiktUserMapper.userListPage();
}
/**
* 排序
*/
@RequestMapping("/userOrderBy")
public List<MayiktUser> userOrderBy(){
return mayiktUserMapper.userOrderBy();
}
@RequestMapping("/getByUserId")
public List<MayiktUser> getByUserId(Long id){
return mayiktUserMapper.getByUserId(id);
}
}
package com.taotao.rfspringboot.config;
import io.shardingsphere.api.algorithm.sharding.PreciseShardingValue;
import io.shardingsphere.api.algorithm.sharding.standard.PreciseShardingAlgorithm;
import java.util.Collection;
/**
* 數(shù)據(jù)庫(kù)分片規(guī)則
*
* @author Mos
*/
public class DataBasePreciseRule implements PreciseShardingAlgorithm<Long> {
@Override
public String doSharding(Collection<String> databaseNames, PreciseShardingValue<Long> shardingValue) {
/**
* databaseNames 所有分片庫(kù)的集合
* shardingValue 為分片屬性省核,其中 logicTableName 為邏輯表稿辙,columnName 分片健(字段)气忠,value 為從 SQL 中解析出的分片健的值
*/
for (String databaseName : databaseNames) {
String value = shardingValue.getValue() % databaseNames.size()+1 + "";
if (databaseName.endsWith(value)) {
System.out.println(databaseName+"&&&"+shardingValue.getValue() +"*********"+databaseNames.size());
return databaseName;
}
}
throw new IllegalArgumentException();
}
}
package com.taotao.rfspringboot.config;
import io.shardingsphere.api.algorithm.sharding.PreciseShardingValue;
import io.shardingsphere.api.algorithm.sharding.standard.PreciseShardingAlgorithm;
import lombok.extern.slf4j.Slf4j;
import java.util.Collection;
@Slf4j
public class MayiktRangeShardingAlgorithm implements PreciseShardingAlgorithm<Long> {
@Override
public String doSharding(Collection<String> databaseNames, PreciseShardingValue<Long> shardingValue) {
/**
* databaseNames 所有分片庫(kù)的集合
* shardingValue 為分片屬性邻储,其中 logicTableName 為邏輯表,columnName 分片骄稍搿(字段)吨娜,value 為從 SQL 中解析出的分片健的值
*/
for (String databaseName : databaseNames) {
String value = shardingValue.getValue() % databaseNames.size() + "";
if (databaseName.endsWith(value)) {
return databaseName;
}
}
throw new IllegalArgumentException();
}
}
歸并數(shù)據(jù)原理
1.SELECT * FROM mayikt_user
底層發(fā)出多條語(yǔ)句查詢每張表數(shù)據(jù),在本地實(shí)現(xiàn)數(shù)據(jù)合并