第1章 分片的概念
1.有了副本集思恐,為什么還需要分片悯许?
副本集資源利用率不高
分片可以提高資源利用率
2.分片的缺點(diǎn)
理想情況下需要機(jī)器比較多
配置和運(yùn)維變得復(fù)雜且困難
提前規(guī)劃好特別重要杨箭,一旦建立后在想改變架構(gòu)變得很困難
第2章 分片涉及到的名詞
1.路由服務(wù)mongos
路由服務(wù)凫碌,提供代理鸳址,替用戶去向后請求shard分片的數(shù)據(jù)
2.數(shù)據(jù)節(jié)點(diǎn)shard
負(fù)責(zé)處理數(shù)據(jù)的節(jié)點(diǎn)狱意,每個shard都是分片集群的一部分
3.分片配置信息服務(wù)器config
保存數(shù)據(jù)分配在哪個shard上
保存所有shard的配置信
提供給mongos查詢服務(wù)
4.片鍵
數(shù)據(jù)存放到哪個shard的區(qū)分規(guī)則
片鍵就是索引
選擇片鍵的依據(jù):
能夠被經(jīng)常訪問到的字段
索引字段基數(shù)夠大
第3章 分片的分類
1.區(qū)間片鍵
id name host sex
1 zhang SH boy
2 ya BJ boy
3 yaya SZ girl
如果以id作為片鍵:
id
1-100 shard1
100-200 shard2
200-300 shard3
300-+無窮 shard4
如果以host作為片鍵:
SH shard1
BJ shard2
SZ shard3
2.hash片鍵:
特點(diǎn):足夠平均式散,足夠隨機(jī)
id name host sex
1 zhang SH boy
2 ya BJ boy
3 yaya SZ girl
如果以id作為片鍵:
索引:id
1 hash 之后 shard2
2 hash 之后 shard3
3 hash 之后 shard1
第4章 IP端口目錄規(guī)劃
0.分片架構(gòu)圖
1.IP端口規(guī)劃
db01 10.0.0.51
Shard1_Master 28100
Shard3_Slave 28200
Shard2_Arbiter 28300
Config Server 40000
mongos Server 60000
db02 10.0.0.52
Shard2_Master 28100
Shard1_Slave 28200
Shard3_Arbiter 28300
Config Server 40000
mongos Server 60000
db03 10.0.0.53
Shard3_Master 28100
Shard2_Slave 28200
Shard1_Arbiter 28300
Config Server 40000
mongos Server 60000
2.目錄規(guī)劃
服務(wù)目錄:
/opt/master/{conf,log,pid}
/opt/slave//{conf,log,pid}
/opt/arbiter/{conf,log,pid}
/data/config/{conf,log,pid}
/data/mongos/{conf,log,pid}
數(shù)據(jù)目錄:
/data/master/
/data/slave/
/data/arbiter/
/data/config/
第5章 分片集群搭建副本集步驟
0.整體步驟
1.先搭shard建副本集
2.搭建config副本集
3.搭建mongos副本集
4.數(shù)據(jù)庫啟用分片功能
5.集合設(shè)置片鍵
6.插入測試數(shù)據(jù)
7.檢查是否分片
8.安裝圖形化工具
1.安裝軟件
注意:db01操作即可
pkill mongo
rm -rf /opt/*
rm -rf /data
tar zxf mongodb-linux-x86_64-rhel70-4.0.14.tgz -C /opt/
cd /opt/
ln -s mongodb-linux-x86_64-rhel70-4.0.14 mongodb
rsync -avz /opt/mongodb* db02:/opt/
rsync -avz /opt/mongodb* db03:/opt/
echo 'export PATH=$PATH:/opt/mongodb/bin' >> /etc/profile
source /etc/profile
2.創(chuàng)建目錄
注意:三臺服務(wù)器都操作V才邸>迕摺!
mkdir -p /opt/master/{conf,log,pid}
mkdir -p /opt/slave/{conf,log,pid}
mkdir -p /opt/arbiter/{conf,log,pid}
mkdir -p /data/master/
mkdir -p /data/slave/
mkdir -p /data/arbiter/
3.db01創(chuàng)建配置文件
master節(jié)點(diǎn)配置文件
cat >/opt/master/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/master/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/master/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/master/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28100
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
slave節(jié)點(diǎn)配置文件
cat >/opt/slave/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/slave/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/slave/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/slave/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28200
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard3
sharding:
clusterRole: shardsvr
EOF
aribiter節(jié)點(diǎn)配置文件
cat >/opt/arbiter/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/arbiter/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/arbiter/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/arbiter/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28300
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
4.db02創(chuàng)建配置文件
master節(jié)點(diǎn)配置文件
cat >/opt/master/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/master/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/master/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/master/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28100
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
slave節(jié)點(diǎn)配置文件
cat >/opt/slave/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/slave/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/slave/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/slave/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28200
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
aribiter節(jié)點(diǎn)配置文件
cat >/opt/arbiter/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/arbiter/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/arbiter/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/arbiter/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28300
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard3
sharding:
clusterRole: shardsvr
EOF
5.db03創(chuàng)建配置文件
master節(jié)點(diǎn)配置文件
cat >/opt/master/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/master/log/mongod.log
storage:
journal:
enabled: true
dbPath: /data/master/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/master/pid/mongodb.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28100
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard3
sharding:
clusterRole: shardsvr
EOF
slave節(jié)點(diǎn)配置文件
cat >/opt/slave/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/slave/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/slave/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/slave/pid/mongod.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28200
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard2
sharding:
clusterRole: shardsvr
EOF
aribiter節(jié)點(diǎn)配置文件
cat >/opt/arbiter/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/arbiter/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/arbiter/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/arbiter/pid/mongod.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 28300
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
oplogSizeMB: 1024
replSetName: shard1
sharding:
clusterRole: shardsvr
EOF
6.優(yōu)化警告
注意于个!三臺服務(wù)器都操作7湛!厅篓!
echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
7.啟動服務(wù)
mongod -f /opt/master/conf/mongod.conf
mongod -f /opt/slave/conf/mongod.conf
mongod -f /opt/arbiter/conf/mongod.conf
netstat -lntup|grep mongod
8.初始化副本集
db01 master節(jié)點(diǎn)初始化shard1的副本集
mongo --port 28100
rs.initiate()
rs.add("10.0.0.52:28200")
rs.addArb("10.0.0.53:28300")
db02 master節(jié)點(diǎn)初始化shard2的副本集
mongo --port 28100
rs.initiate()
rs.add("10.0.0.53:28200")
rs.addArb("10.0.0.51:28300")
db03 master節(jié)點(diǎn)初始化shard3的副本集
mongo --port 28100
rs.initiate()
rs.add("10.0.0.51:28200")
rs.addArb("10.0.0.52:28300")
9.檢查命令
mongo --port 28100
rs.status()
第6章 分片集群搭建config步驟
1.創(chuàng)建目錄
mkdir -p /opt/config/{conf,log,pid}
mkdir -p /data/config/
2.創(chuàng)建配置文件
cat >/opt/config/conf/mongod.conf<<EOF
systemLog:
destination: file
logAppend: true
path: /opt/config/log/mongodb.log
storage:
journal:
enabled: true
dbPath: /data/config/
directoryPerDB: true
wiredTiger:
engineConfig:
cacheSizeGB: 0.5
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
processManagement:
fork: true
pidFilePath: /opt/config/pid/mongod.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 40000
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
replication:
replSetName: configset
sharding:
clusterRole: configsvr
EOF
3.啟動
mongod -f /opt/config/conf/mongod.conf
4.初始化副本集
mongo --port 40000
rs.initiate()
rs.add("10.0.0.52:40000")
rs.add("10.0.0.53:40000")
rs.status()
第7章 分片集群搭建mongos配置
1.創(chuàng)建目錄
mkdir -p /opt/mongos/{conf,log,pid}
2.創(chuàng)建配置文件
cat >/opt/mongos/conf/mongos.conf <<EOF
systemLog:
destination: file
logAppend: true
path: /opt/mongos/log/mongos.log
processManagement:
fork: true
pidFilePath: /opt/mongos/pid/mongos.pid
timeZoneInfo: /usr/share/zoneinfo
net:
port: 60000
bindIp: 127.0.0.1,$(ifconfig eth0|awk 'NR==2{print $2}')
sharding:
configDB:
configset/10.0.0.51:40000,10.0.0.52:40000,10.0.0.53:40000
EOF
3.啟動
mongos -f /opt/mongos/conf/mongos.conf
4.登錄mongos
mongo --port 60000
5.添加分片成員
use admin
db.runCommand({addShard:'shard1/10.0.0.51:28100,10.0.0.52:28200,10.0.0.53:28300'})
db.runCommand({addShard:'shard2/10.0.0.52:28100,10.0.0.53:28200,10.0.0.51:28300'})
db.runCommand({addShard:'shard3/10.0.0.53:28100,10.0.0.51:28200,10.0.0.52:28300'})
6.查看分片信息
db.runCommand( { listshards : 1 } )
第8章 hash分片配置
1.數(shù)據(jù)庫開啟分片
mongo --port 60000
use admin
db.runCommand( { enablesharding : "oldboy" } )
2.創(chuàng)建索引
use oldboy
db.hash.ensureIndex( { id: "hashed" } )
3.集合開啟哈希分片
use admin
sh.shardCollection( "oldboy.hash", { id: "hashed" } )
4.生成測試數(shù)據(jù)
use oldboy
for(i=1;i<10000;i++){db.hash.insert({"id":i,"name":"shenzheng","age":18});}
5.分片驗(yàn)證
shard1:
mongo db01:28100
use oldboy
db.hash.count()
3349
shard2:
mongo db02:28100
use oldboy
db.hash.count()
3366
shard3:
mongo db03:28100
use oldboy
db.hash.count()
3284
第9章 分片集群常用管理命令
1.列出分片所有詳細(xì)信息
db.printShardingStatus()
sh.status()
2.列出所有分片成員信息
use admin
db.runCommand({ listshards : 1})
3.列出開啟分片的數(shù)據(jù)庫
use config
db.databases.find({"partitioned": true })
4.查看分片的片鍵
use config
db.collections.find().pretty()
5.查看集合的分片信息
db.getCollection('range').getShardDistribution()
db.getCollection('hash').getShardDistribution()
第11章 正確的啟動順序
mongos
所有的config
所有的master
所有的arbiter
所有的slave