本人博客同步發(fā)表礁蔗,排版更佳
教訓(xùn):MongoDB在分片后的集合上進(jìn)行db.collection.count()操作時(shí),出現(xiàn)結(jié)果不準(zhǔn)確的現(xiàn)象,需要采用聚合的方法獲取集合的count結(jié)果
在使用MongoDB-Java客戶端做簡單的插入操作(10W條)以后,使用Studio 3T查看插入結(jié)果時(shí)炫七,發(fā)現(xiàn)顯示的count結(jié)果與插入的數(shù)據(jù)不一致,偶然會(huì)多出幾條或十幾條钾唬,插入操作很簡單万哪,其中table3被分片
//創(chuàng)建客戶端連接
MongoClient mongoClient = new MongoClient( "xxx.xxx.xxx.xxx" , 20000 );
MongoDatabase database = mongoClient.getDatabase("testdb");
MongoCollection<Document> collection = database.getCollection("table3");
//插入文檔
long start = System.currentTimeMillis();
Random r = new Random();
for (int i=0;i<100000;i++){
Document doc = new Document("id", i)
.append("type", "database"+i)
.append("test","testval"+i)
.append("name", "name"+i);
System.out.println("insertOne = " + i);
collection.insertOne(doc);
}
System.out.println("use time = " + (System.currentTimeMillis()- start));
System.out.println("count = " + collection.count());
發(fā)現(xiàn)問題后,通過在shell里面查詢count抡秆,命令如下
db.table3.count()
使用該命令依然會(huì)出現(xiàn)統(tǒng)計(jì)信息不準(zhǔn)確的現(xiàn)象奕巍,通過谷歌發(fā)現(xiàn),官方文檔——(https://docs.mongodb.com/manual/reference/method/db.collection.count/)中有解釋這種現(xiàn)象:
On a sharded cluster,
db.collection.count()
can result in an inaccurate count if orphaned documentsexist or if a chunk migration is in progress.To avoid these situations, on a sharded cluster, use the
$group
stage of thedb.collection.aggregate()
method to$sum
the documents. For example, the following operation counts the documents in a collection:
db.collection.aggregate(
[
{ $group: { _id: null, count: { $sum: 1 } } }
]
)
官方文檔解釋了這種現(xiàn)象的原因以及解決方法:
不準(zhǔn)確的原因:
- 操作的是分片的集合(前提)儒士;
- shard分片正在做塊遷移的止,導(dǎo)致有重復(fù)數(shù)據(jù)出現(xiàn)
- 存在孤立文檔(因?yàn)椴徽jP(guān)機(jī)、塊遷移失敗等原因?qū)е拢?/li>
解決方法
使用聚合aggregate的方式查詢count數(shù)量着撩,shell命令如下:
db.collection.aggregate(
[
{ $group: { _id: null, count: { $sum: 1 } } }
]
)
java代碼
所以在Java中也可以采用聚合的方式獲取count結(jié)果诅福,使用聚合aggregate的方式可以準(zhǔn)確的獲取sharding后的集合的count結(jié)果匾委。
DBObject groupFields = new BasicDBObject("_id", null);
groupFields.put("count", new BasicDBObject("$sum", 1));
BasicDBObject group = new BasicDBObject("$group", groupFields);
List<BasicDBObject> aggreList = new ArrayList<BasicDBObject>();
aggreList.add(group);
AggregateIterable<Document> output = collection.aggregate(aggreList);
for (Document dbObject : output)
{
System.out.println("Aggregates count = "+ dbObject);
}
完整代碼
//創(chuàng)建客戶端連接
MongoClient mongoClient = new MongoClient( "xxx.xxx.xxx.xxx" , 20000 );
MongoDatabase database = mongoClient.getDatabase("testdb");
MongoCollection<Document> collection = database.getCollection("table3");
//插入文檔
long start = System.currentTimeMillis();
Random r = new Random();
for (int i=0;i<100000;i++){
Document doc = new Document("id", i)
.append("type", "database"+i)
.append("test","testval"+i)
.append("name", "name"+i);
System.out.println("insertOne = " + i);
collection.insertOne(doc);
}
System.out.println("use time = " + (System.currentTimeMillis()- start));
System.out.println("count = " + collection.count());
DBObject groupFields = new BasicDBObject("_id", null);
groupFields.put("count", new BasicDBObject("$sum", 1));
BasicDBObject group = new BasicDBObject("$group", groupFields);
List<BasicDBObject> aggreList = new ArrayList<BasicDBObject>();
aggreList.add(group);
AggregateIterable<Document> output = collection.aggregate(aggreList);
for (Document dbObject : output)
{
System.out.println("Aggregates count = "+ dbObject);
}