2015/12/7

如何在 MongoDB 處理最佳化的問題

MongoDB 有許多查詢的機制,但需要搭配使用一些工具,了解怎麼讓查詢的效能最佳化。

索引

MongoDB 可針對所有文件類型建立索引,但查詢時是否有用到索引的條件是根據索引建立的順序來決定的。

針對 age 欄位建立索引

> db.mate.ensureIndex({age:1});
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}

取得 collection 所有 indexes

> db.mate.getIndexes();
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test.mate"
    },
    {
        "v" : 1,
        "key" : {
            "age" : 1
        },
        "name" : "age_1",
        "ns" : "test.mate"
    }
]

建立複合索引

> db.mate.ensureIndex({age:1, room:1});
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 2,
    "numIndexesAfter" : 3,
    "ok" : 1
}

> db.mate.find().sort( {age:1, room:1});
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
> db.mate.find().sort( {age:-1, room:1});
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

刪除所有索引,除了 _id 無法刪除之外

> db.mate.dropIndexes();
{
    "nIndexesWas" : 3,
    "msg" : "non-_id indexes dropped for collection",
    "ok" : 1
}

以 unique 指定該索引值必須要是 唯一的,不能有重複的紀錄

> db.mate.ensureIndex({userid:1}, {unique:true});
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}
> db.mate.ensureIndex({age:1}, {unique:true});
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 2,
    "errmsg" : "exception: E11000 duplicate key error index: test.mate.$age_1 dup key: { : null }",
    "code" : 11000,
    "ok" : 0
}

以下為所有測試指令的集合

db.mate.ensureIndex({age:1});
db.mate.getIndexes();

db.mate.dropIndexes();
db.mate.ensureIndex({age:1, room:1});
db.mate.find().sort( {age:1, room:1});
db.mate.find().sort( {age:-1, room:1});
db.mate.getIndexes();

db.mate.dropIndexes();
db.mate.ensureIndex({userid:1}, {unique:true});
db.mate.ensureIndex({age:1}, {unique:true});

查詢最佳化

首先確認 db 有為 age, room 建立 index

> db.mate.dropIndexes();
{
    "nIndexesWas" : 2,
    "msg" : "non-_id indexes dropped for collection",
    "ok" : 1
}
> db.mate.ensureIndex({age:1, room:1});
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}
> db.mate.getIndexes();
[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test.mate"
    },
    {
        "v" : 1,
        "key" : {
            "age" : 1,
            "room" : 1
        },
        "name" : "age_1_room_1",
        "ns" : "test.mate"
    }
]

執行 find 條件查詢,可以用 explain() 確認是否有使用到 index。Explain Results 頁面可查看裡面每個欄位的意義。

結果中主要應該是看 winningPlan,這是 query optimizer 最後所選擇的 query plan。

Stage 表示該階段的 operation,有下列幾種

  1. COLLSCAN: collection scan,掃描整個 collection,如果有產生這個 stage,就表示這個查詢的速度會很慢
  2. IXSCAN: scanning index keys,掃描 index 的 keys
  3. FETCH: retrieveing documents,取得文件
  4. SHARD_MERGE: 由 shards 整合結果
  5. PROJECTION: Projections 表示針對collection的某些欄位進行篩檢
  6. KEEP_MUTATIONS: 還沒查到說明的頁面
    > db.mate.find( {age:{ $gt: 22}, username:{ $regex:/^M.*/ }} );
    { "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
    > db.mate.find( {age:{ $gt: 22}, username:{ $regex:/^M.*/ }} ).explain();
    {
     "queryPlanner" : {
         "plannerVersion" : 1,
         "namespace" : "test.mate",
         "indexFilterSet" : false,
         "parsedQuery" : {
             "$and" : [
                 {
                     "age" : {
                         "$gt" : 22
                     }
                 },
                 {
                     "username" : /^M.*/
                 }
             ]
         },
         "winningPlan" : {
             "stage" : "KEEP_MUTATIONS",
             "inputStage" : {
                 "stage" : "FETCH",
                 "filter" : {
                     "username" : /^M.*/
                 },
                 "inputStage" : {
                     "stage" : "IXSCAN",
                     "keyPattern" : {
                         "age" : 1,
                         "room" : 1
                     },
                     "indexName" : "age_1_room_1",
                     "isMultiKey" : false,
                     "direction" : "forward",
                     "indexBounds" : {
                         "age" : [
                             "(22.0, inf.0]"
                         ],
                         "room" : [
                             "[MinKey, MaxKey]"
                         ]
                     }
                 }
             }
         },
         "rejectedPlans" : [ ]
     },
     "serverInfo" : {
         "host" : "server",
         "port" : 27017,
         "version" : "3.0.7",
         "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd"
     },
     "ok" : 1
    }
    

搜尋的另一個例子

> db.mate.find( {age:{ $gt: 22}}, {username:1} );
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "username" : "Mary Wang" }
> db.mate.find( {age:{ $gt: 22}}, {username:1} ).explain();
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "test.mate",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "age" : {
                "$gt" : 22
            }
        },
        "winningPlan" : {
            "stage" : "PROJECTION",
            "transformBy" : {
                "username" : 1
            },
            "inputStage" : {
                "stage" : "FETCH",
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "keyPattern" : {
                        "age" : 1,
                        "room" : 1
                    },
                    "indexName" : "age_1_room_1",
                    "isMultiKey" : false,
                    "direction" : "forward",
                    "indexBounds" : {
                        "age" : [
                            "(22.0, inf.0]"
                        ],
                        "room" : [
                            "[MinKey, MaxKey]"
                        ]
                    }
                }
            }
        },
        "rejectedPlans" : [ ]
    },
    "serverInfo" : {
        "host" : "kokola",
        "port" : 27017,
        "version" : "3.0.7",
        "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd"
    },
    "ok" : 1
}

由於沒有 city 欄位的 index,因此產生了 COLLSCAN,當資料量大,這個查詢的速度將會變得很慢。

> db.mate.find( {city:{ $regex:/^T.*/ } } );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }
> db.mate.find( {city:{ $regex:/^T.*/ } } ).explain();
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "test.mate",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "city" : /^T.*/
        },
        "winningPlan" : {
            "stage" : "COLLSCAN",
            "filter" : {
                "city" : /^T.*/
            },
            "direction" : "forward"
        },
        "rejectedPlans" : [ ]
    },
    "serverInfo" : {
        "host" : "kokola",
        "port" : 27017,
        "version" : "3.0.7",
        "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd"
    },
    "ok" : 1
}

以下為所有測試指令的集合

db.mate.dropIndexes();
db.mate.ensureIndex({age:1});
db.mate.ensureIndex({room:1});
db.mate.getIndexes();


db.mate.find( {age:{ $gt: 22}, username:{ $regex:/^M.*/ }} );
db.mate.find( {age:{ $gt: 22}, username:{ $regex:/^M.*/ }} ).explain();

db.mate.find( {age:{ $gt: 22}}, {username:1} );
db.mate.find( {age:{ $gt: 22}}, {username:1} ).explain();

db.mate.find( {city:{ $regex:/^T.*/ } } );
db.mate.find( {city:{ $regex:/^T.*/ } } ).explain();

Profiler

MongoDB 中,可以啟動 profiler,用來記錄一些速度較慢的查詢指令,這可以用來作為最佳化資料庫的資訊來源。

有兩種方式可以啟動 profile 的功能,第一種是直接在 mongod 的參數中,加上 -profile ,第二種是透過 db.setProfilingLevel()來啟動 profiler。

profiler level 有三種

  • 0: 不啟動
  • 1: 記錄慢指令(預設為 > 100ms)
  • 2: 記錄所有指令

可用以下指令設定 ProfilingLevel

db.setProfilingLevel(1);

也可以同時指定慢指令的時間,例如設定為 10ms

db.setProfilingLevel(1,10);

查看執行時間超過 5ms 的 profiler 記錄

db.system.profile.find({millis:{$gt:5}});

查看最新的 profiler 記錄

db.system.profile.find().sort({$natural:-1}).limit(5);

> db.system.profile.find().sort({$natural:-1}).limit(5);
{ "op" : "query", "ns" : "test.mate", "query" : { "age" : { "$gt" : 22 }, "username" : { "$regex" : /^M.*/ } }, "ntoreturn" : 0, "ntoskip" : 0, "nscanned" : 1, "nscannedObjects" : 1, "keyUpdates" : 0, "writeConflicts" : 0, "numYield" : 0, "locks" : { "Global" : { "acquireCount" : { "r" : NumberLong(2) } }, "MMAPV1Journal" : { "acquireCount" : { "r" : NumberLong(1) } }, "Database" : { "acquireCount" : { "r" : NumberLong(1) } }, "Collection" : { "acquireCount" : { "R" : NumberLong(1) } } }, "nreturned" : 1, "responseLength" : 127, "millis" : 0, "execStats" : { "stage" : "KEEP_MUTATIONS", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "works" : 2, "advanced" : 1, "needTime" : 0, "needFetch" : 0, "saveState" : 0, "restoreState" : 0, "isEOF" : 1, "invalidates" : 0, "inputStage" : { "stage" : "FETCH", "filter" : { "username" : /^M.*/ }, "nReturned" : 1, "executionTimeMillisEstimate" : 0, "works" : 2, "advanced" : 1, "needTime" : 0, "needFetch" : 0, "saveState" : 0, "restoreState" : 0, "isEOF" : 1, "invalidates" : 0, "docsExamined" : 1, "alreadyHasObj" : 0, "inputStage" : { "stage" : "IXSCAN", "nReturned" : 1, "executionTimeMillisEstimate" : 0, "works" : 1, "advanced" : 1, "needTime" : 0, "needFetch" : 0, "saveState" : 0, "restoreState" : 0, "isEOF" : 1, "invalidates" : 0, "keyPattern" : { "age" : 1 }, "indexName" : "age_1", "isMultiKey" : false, "direction" : "forward", "indexBounds" : { "age" : [ "(22.0, inf.0]" ] }, "keysExamined" : 1, "dupsTested" : 0, "dupsDropped" : 0, "seenInvalidated" : 0, "matchTested" : 0 } } }, "ts" : ISODate("2015-11-11T03:22:04.503Z"), "client" : "127.0.0.1", "allUsers" : [ ], "user" : "" }

最佳化的策略方案

  1. 建立索引
    針對查詢條件的欄位建立索引
  2. 限制回傳結果的筆數
    使用 limit() 限制筆數,可減少 db 的資源消耗並減少網路傳輸量
  3. 只查詢使用到的欄位,不查詢所有欄位
    只查詢使用到的欄位
  4. 使用 Capped Collection
    Capped Collection 比普通的 Collection 的讀寫效率高,它有以下的特性
    • 固定大小:必須手動建立並設定固定的大小,db.createcollection("testc", {capped:true, size:1000000});
    • 可以 insert 與 update 但不能 delete
    • 只能用 drop 刪除整個 collection
    • 預設以 insert 的順序排序
    • FIFO,如果超過了限制的大小,依照 FIFO 原則,自動刪除最舊的資料
  5. 使用 Server Side Code Execution
    類似關聯式資料庫的 Stored Procedure
  6. hint
    一般情況下使用 query optimizer 就好了,有時候可以用 hint() 強制要求查詢要使用某個 index。例如在多個條件的查詢中,如果其中一個欄位有索引,可以用 hint

     > db.mate.find( {age:{ $gt: 22}}, {username:1} ).hint({age:1});
     { "_id" : ObjectId("5641980c997bc79c93b12e77"), "username" : "Mary Wang" }
     > db.mate.find( {age:{ $gt: 22}}, {username:1} ).hint({age:1}).explain)(;
     2015-11-11T11:35:22.023+0800 E QUERY    SyntaxError: Unexpected token )
     > db.mate.find( {age:{ $gt: 22}}, {username:1} ).hint({age:1}).explain();
     {
         "queryPlanner" : {
             "plannerVersion" : 1,
             "namespace" : "test.mate",
             "indexFilterSet" : false,
             "parsedQuery" : {
                 "age" : {
                     "$gt" : 22
                 }
             },
             "winningPlan" : {
                 "stage" : "PROJECTION",
                 "transformBy" : {
                     "username" : 1
                 },
                 "inputStage" : {
                     "stage" : "FETCH",
                     "inputStage" : {
                         "stage" : "IXSCAN",
                         "keyPattern" : {
                             "age" : 1
                         },
                         "indexName" : "age_1",
                         "isMultiKey" : false,
                         "direction" : "forward",
                         "indexBounds" : {
                             "age" : [
                                 "(22.0, inf.0]"
                             ]
                         }
                     }
                 }
             },
             "rejectedPlans" : [ ]
         },
         "serverInfo" : {
             "host" : "server",
             "port" : 27017,
             "version" : "3.0.7",
             "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd"
         },
         "ok" : 1
     }
    
  7. 使用 profiling
    利用 profiling 記錄一些比較慢的查詢,用來作為後續最佳化的參考資料

資料庫監控

  • mongosniff
    binary 安裝包裡面並沒有包含這個工具,必須要自己從 source code 編譯,這個工具可以監聽 27017 port,記錄所有封包。

  • mongostat
    可查詢某個 mongod 的統計資訊

    [root@kokola bin]# ./mongostat
    insert query update delete getmore command flushes mapped vsize   res faults qr|qw ar|aw netIn netOut conn     time
      *0    *0     *0     *0       0     1|0       0 160.0M  1.7G 73.0M      0   0|0   0|0   79b    10k    2 11:42:14
      *0    *0     *0     *0       0     1|0       0 160.0M  1.7G 73.0M      0   0|0   0|0   79b    10k    2 11:42:15
      *0    *0     *0     *0       0     1|0       0 160.0M  1.7G 73.0M      0   0|0   0|0   79b    10k    2 11:42:16
      *0    *0     *0     *0       0     1|0       0 160.0M  1.7G 73.0M      0   0|0   0|0   79b    10k    2 11:42:17
      *0    *0     *0     *0       0     2|0       0 160.0M  1.7G 73.0M      0   0|0   0|0  133b    10k    2 11:42:18
      *0    *0     *0     *0       0     1|0       0 160.0M  1.7G 73.0M      0   0|0   0|0   79b    10k    2 11:42:19
    
      insert
          每秒 insert 的operation數量
      query
      update
      delete
      getmore
          每秒進行 get more (ex: cursor batch) operations 的數量
      command
          每秒執行的指令數量
      flushes
          如果是 WiredTiger Storage Engine就是 WiredTiger checkpoints triggered 的數量,如果是 MMAPv1 Storage Engine,就是 fsync operations 的數量
      dirty
          只有 WiredTiger Storage Engine 會有,這是 WiredTiger cache with dirty bytes 的比例
      used
          WiredTiger Storage Engine 才有,這是 WiredTiger cache 的使用比例
      mapped
          MMAPv1 Storage Engine 才有,總資料量,單位是 MegaBytes
      vsize
          mongod proecess 使用的 virtual memory 數量,單位是 MegaBytes
      non-mapped
          (optional) MMAPv1 Storage Engine 才有,去掉所有 mapped memory 的 virtual memory 總量
      res
          process 使用的 resident memory 總量,單位是 MegaBytes
      faults
          MMAPv1 Storage Engine 才有,每秒產生的 page faults 數量
      idx miss
          MMAPv1 Storage Engine才有,載入一個 btree node 所產生的 page fault,所需要使用的 index 比例
      qr
          所有 clients 等待讀取資料的 queue 的長度
      qw
          所有 clients 等待寫入資料的 queue 的長度
      ar
          執行 read operations 的 active clients 數量
      aw
          執行 write operations 的 active clients 數量
      netIn
          mongod 由網路接收到的資料量,單位為 bytes,包含 mongostat 本身的資料量
      netOut
          mongod 由網路發送的資料量,單位為 bytes
      conn
          總連線數
      set
          replica set 的名稱
      repl
          replication status of the members
          (1) M: master
          (2) SEC: secondary
          (3) REC: recovering
          (4) UNK: unknown
          (5) SLV: slave
          (6) RTR: mongos process(router)
          (7) ARB: arbiter
    

db.serverStatus

透過 db.serverStatus() 指令,我們可以即時檢視資料庫的狀態。

> db.serverStatus();
{
    "host" : "server",
    "version" : "3.0.7",    // MongoDB 版本
    "process" : "mongod",
    "pid" : NumberLong(12533), // processid
    "uptime" : 160841,    // 啟動的時間(seconds)
    "uptimeMillis" : NumberLong(160841652),
    "uptimeEstimate" : 159741,
    "localTime" : ISODate("2015-11-11T06:27:31.655Z"),
    "asserts" : {    // server 產生 error or assertions 的數量
        "regular" : 0,
        "warning" : 0,
        "msg" : 0,
        "user" : 9,
        "rollovers" : 0
    },
    "backgroundFlushing" : {    // 將資料寫入 disk 的 report
        "flushes" : 2680,
        "total_ms" : 4357,
        "average_ms" : 1.6257462686567163,
        "last_ms" : 1,
        "last_finished" : ISODate("2015-11-11T06:26:50.271Z")
    },
    "connections" : {    // 目前clients的連線狀況
        "current" : 1,    // active client 連線數量
        "available" : 52427,    // 空閒的連線數量
        "totalCreated" : NumberLong(4)
    },
    "cursors" : {    // 目前的 cursor 使用狀況
        "note" : "deprecated, use server status metrics",
        "clientCursors_size" : 0,
        "totalOpen" : 0,
        "pinned" : 0,
        "totalNoTimeout" : 0,
        "timedOut" : 3
    },
    "dur" : {    // journaling 的 report
        "commits" : 22,
        "journaledMB" : 0,
        "writeToDataFilesMB" : 0,
        "compression" : 0,
        "commitsInWriteLock" : 0,
        "earlyCommits" : 0,
        "timeMs" : {
            "dt" : 0,
            "prepLogBuffer" : 0,
            "writeToJournal" : 0,
            "writeToDataFiles" : 0,
            "remapPrivateView" : 0,
            "commits" : 0,
            "commitsInWriteLock" : 0
        }
    },
    "extra_info" : {
        "note" : "fields vary by platform",
        "heap_usage_bytes" : 63700688,
        "page_faults" : 65
    },
    "globalLock" : {    // global system lock 的 report
        "totalTime" : NumberLong("160841653000"),
        "currentQueue" : {
            "total" : 0,    // 全部 queue 的數量
            "readers" : 0,
            "writers" : 0
        },
        "activeClients" : {
            "total" : 9,    // client連線數量
            "readers" : 0,
            "writers" : 0
        }
    },
    "locks" : {        // 每個 lock type 與 mode 的統計報告
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(539117),
                "w" : NumberLong(559),
                "W" : NumberLong(50)
            }
        },
        "MMAPV1Journal" : {
            "acquireCount" : {
                "r" : NumberLong(269221),
                "w" : NumberLong(1214),
                "R" : NumberLong(1573490),
                "W" : NumberLong(6)
            },
            "acquireWaitCount" : {
                "w" : NumberLong(1),
                "R" : NumberLong(2)
            },
            "timeAcquiringMicros" : {
                "w" : NumberLong(18),
                "R" : NumberLong(173851)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(269199),
                "w" : NumberLong(391),
                "R" : NumberLong(55),
                "W" : NumberLong(201)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "R" : NumberLong(281400),
                "W" : NumberLong(442)
            }
        },
        "Metadata" : {
            "acquireCount" : {
                "W" : NumberLong(180)
            }
        }
    },
    "network" : {    // 網路的使用報告
        "bytesIn" : NumberLong(111181),
        "bytesOut" : NumberLong(354420),
        "numRequests" : NumberLong(1227)
    },
    "opcounters" : {    // 這個 instance 處理了幾個 operations
        "insert" : 200,
        "query" : 158,
        "update" : 3,
        "delete" : 2,
        "getmore" : 3,
        "command" : 973
    },
    "opcountersRepl" : {    // 處理了幾個 replicated operations
        "insert" : 0,
        "query" : 0,
        "update" : 0,
        "delete" : 0,
        "getmore" : 0,
        "command" : 0
    },
    "storageEngine" : {        // 目前使用那一種 storage engine
        "name" : "mmapv1"
    },
    "writeBacksQueued" : false,
    "mem" : {    // 目前記憶體的使用報告
        "bits" : 64,    // 64 bits
        "resident" : 73,    // 佔用的物理記憶體總量
        "virtual" : 1750,    // 虛擬記憶體總量
        "supported" : true,    // 是否支援擴充記憶體
        "mapped" : 160,
        "mappedWithJournal" : 320
    },
    "metrics" : {    // 監控 mongod 的 state 與 workload 的一些 operational metrics
        "commands" : {
            "<UNKNOWN>" : NumberLong(1),
            "collStats" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(2)
            },
            "count" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(2)
            },
            "createIndexes" : {
                "failed" : NumberLong(2),
                "total" : NumberLong(13)
            },
            "delete" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(2)
            },
            "drop" : {
                "failed" : NumberLong(63),
                "total" : NumberLong(85)
            },
            "dropIndexes" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(7)
            },
            "explain" : {
                "failed" : NumberLong(1),
                "total" : NumberLong(18)
            },
            "getLog" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(3)
            },
            "getnonce" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(1)
            },
            "insert" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(200)
            },
            "isMaster" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(740)
            },
            "listCollections" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(38)
            },
            "listIndexes" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(11)
            },
            "mapReduce" : {
                "failed" : NumberLong(1),
                "total" : NumberLong(16)
            },
            "ping" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(3)
            },
            "profile" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(2)
            },
            "renameCollection" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(15)
            },
            "replSetGetStatus" : {
                "failed" : NumberLong(3),
                "total" : NumberLong(3)
            },
            "serverStatus" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(11)
            },
            "update" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(3)
            },
            "whatsmyuri" : {
                "failed" : NumberLong(0),
                "total" : NumberLong(3)
            }
        },
        "cursor" : {
            "timedOut" : NumberLong(3),
            "open" : {
                "noTimeout" : NumberLong(0),
                "pinned" : NumberLong(0),
                "total" : NumberLong(0)
            }
        },
        "document" : {
            "deleted" : NumberLong(1),
            "inserted" : NumberLong(199),
            "returned" : NumberLong(1279),
            "updated" : NumberLong(3)
        },
        "getLastError" : {
            "wtime" : {
                "num" : 0,
                "totalMillis" : 0
            },
            "wtimeouts" : NumberLong(0)
        },
        "operation" : {
            "fastmod" : NumberLong(0),
            "idhack" : NumberLong(0),
            "scanAndOrder" : NumberLong(7),
            "writeConflicts" : NumberLong(0)
        },
        "queryExecutor" : {
            "scanned" : NumberLong(34),
            "scannedObjects" : NumberLong(1824)
        },
        "record" : {
            "moves" : NumberLong(0)
        },
        "repl" : {
            "apply" : {
                "batches" : {
                    "num" : 0,
                    "totalMillis" : 0
                },
                "ops" : NumberLong(0)
            },
            "buffer" : {
                "count" : NumberLong(0),
                "maxSizeBytes" : 268435456,
                "sizeBytes" : NumberLong(0)
            },
            "network" : {
                "bytes" : NumberLong(0),
                "getmores" : {
                    "num" : 0,
                    "totalMillis" : 0
                },
                "ops" : NumberLong(0),
                "readersCreated" : NumberLong(0)
            },
            "preload" : {
                "docs" : {
                    "num" : 0,
                    "totalMillis" : 0
                },
                "indexes" : {
                    "num" : 0,
                    "totalMillis" : 0
                }
            }
        },
        "storage" : {
            "freelist" : {
                "search" : {
                    "bucketExhausted" : NumberLong(0),
                    "requests" : NumberLong(611),
                    "scanned" : NumberLong(0)
                }
            }
        },
        "ttl" : {
            "deletedDocuments" : NumberLong(0),
            "passes" : NumberLong(2680)
        }
    },
    "ok" : 1
}

db.stats

db.stats 指令是用來查看 dbStats 資料庫的狀態

> db.stats();
{
    "db" : "test",
    "collections" : 5,    // connection 數量
    "objects" : 28,        // 物件數量
    "avgObjSize" : 307.42857142857144,    // 平均物件大小
    "dataSize" : 8608,    // 資料大小
    "storageSize" : 1110016,    // storage大小
    "numExtents" : 6,    // 事件數量
    "indexes" : 4,        // 索引數量
    "indexSize" : 32704,    // 索引大小
    "fileSize" : 67108864,    // 檔案大小
    "nsSizeMB" : 16,
    "extentFreeList" : {
        "num" : 3,
        "totalSize" : 147456
    },
    "dataFileVersion" : {
        "major" : 4,
        "minor" : 22
    },
    "ok" : 1
}

2015/11/30

如何使用 MongoDB

以下記錄如何以 cli 方式啟動 MongoDB,以及基本的 CRUD operations。

安裝 MongoDB

要在 centos linux 環境安裝 MongoDB,除了用 yum 安裝之外,還可以直接在 網頁 下載 tgz 檔案,以 CentOS 6 為例,選擇作業系統版本 RHEL 6 Linux 64-bit,點擊 DOWNLOAD 後,就可以下載檔案 mongodb-linux-x86_64-rhel62-3.0.7.tgz。

該壓縮檔裡面,只有一個 bin 目錄,裡面是 mongodb 的所有執行檔,我們將裡面的資料解壓縮到 /usr/share/mongodb 裡面。

tar zxvf mongodb-linux-x86_64-rhel62-3.0.7.tgz /usr/share/
cd /usr/share
mv mongodb-linux-x86_64-rhel62-3.0.7 mongodb
cd mongodb

mkdir -p data/db
mkdir -p logs

執行 mongod ,同時指定 db 以及 logfile 路徑,就可以啟動 mongodb。

/usr/share/mongodb/bin/mongod --dbpath=/usr/share/mongodb/data/db -logpath=/usr/share/mongodb/logs/mongodb.log

這是直接將啟動的參數放在 mongod 命令列中,也可以製作一個設定檔 mongodb.conf,並以 -f 指定設定檔就可以了。

/usr/share/mongodb/bin/mongod -f /etc/mongod.conf

如果希望在背景啟動 mongodb,必須加上 --fork 的參數。

[root@server mongodb]# /usr/share/mongodb/bin/mongod -dbpath=/usr/share/mongodb/data/db -logpath=/usr/share/mongodb/logs/mongodb.log --fork
about to fork child process, waiting until server is ready for connections.
forked process: 9544
child process started successfully, parent exiting
[root@server mongodb]# ps aux|grep mongo
root      9544  1.0  1.3 499800 50308 ?        Sl   15:36   0:00 /usr/share/mongodb/bin/mongod -dbpath=/usr/share/mongodb/data/db -logpath=/usr/share/mongodb/logs/mongodb.log --fork
root      9556  0.0  0.0 105388   912 pts/0    S+   15:36   0:00 grep --color mongo

mongod 的參數分為 一般參數、 windows參數、replication 參數、replica set 參數以及隱藏參數。常用的參數如下:

  1. dbpath
  2. logpath
  3. logappend: logfile 以 append 方式附加到檔案後面
  4. bind_ip
  5. port
  6. fork: 以 daemon 方式啟動 mongod
  7. journal: 開啟記錄檔功能
  8. syncdelay: 系統同步更新 disk 的時間,預設為 60s
  9. directoryperdb: 每個 db 儲存在單獨的目錄中,這樣比較容易區分不同的資料庫
  10. maxConns: 最大的 client 連線數
  11. repairpath: 執行 repair 的臨時目錄,如果沒有啟用 journal 功能,異常當機重新啟動後,必須執行 repair

如果要停止 mongod,可使用 db.shutdownServer() 指令

# mongo
> use admin;
switched to db admin
> db.shutdownServer();
2015-11-09T17:23:30.415+0800 I NETWORK  DBClientCursor::init call() failed
server should be down...
2015-11-09T17:23:30.417+0800 I NETWORK  trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2015-11-09T17:23:30.417+0800 W NETWORK  Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-11-09T17:23:30.417+0800 I NETWORK  reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed

不能用 kill -9 PID 直接將 mongodb process 刪除,這樣可能會導致 mongodb 的資料損壞。只能透過 kill -2 PID 或 kill -15 PID 停止 mongod process。

CRUD

mongo: MongoDB Shell 是 MongoDB 操作及管理的界面,同時也是一個 JavaScript shell 可以執行 js code。

以下幾點是 mongodb 的一些基本使用概念

  1. mongo 不需要預先建立 collection,在第一次儲存資料時,就會自動建立
  2. 文件資料可以儲存任何結構的資料,也不需要修改 schema
  3. 每一次儲存資料,都會產生一個欄位 _id,該欄位可儲存任何資料類型,未指定時會自動產生,資料類型為 ObjectId,ObjectId 裡面就隱藏了產生這筆資料的時間。

    ObjectId: 是 12 bytes BSON 格式,包含以下資訊:

    1. 4-byte: 表示 Unix epoch 到現在的秒數,也就是 timestamp
    2. 3-byte: machine identifier
    3. 2-byte: process id
    4. 3-byte: counter,啟始值為一個亂數值
  • 儲存

    > user1={name:"sun", age: 12};
    { "name" : "sun", "age" : 12 }
    > user2={name:"moon", age: 14};
    { "name" : "moon", "age" : 14 }
    > 
    > db.solar.save(user1);
    WriteResult({ "nInserted" : 1 })
    > db.solar.save(user2);
    WriteResult({ "nInserted" : 1 })
    > 
    > db.solar.find();
    { "_id" : ObjectId("56416af4997bc79c93b12e23"), "name" : "sun", "age" : 12 }
    { "_id" : ObjectId("56416af4997bc79c93b12e24"), "name" : "moon", "age" : 14 }
    
  • 修改

    > db.solar.update({name:"moon"},{$set:{age:"16"}});
    WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
    > db.solar.find();
    { "_id" : ObjectId("56416af4997bc79c93b12e23"), "name" : "sun", "age" : 12 }
    { "_id" : ObjectId("56416af4997bc79c93b12e24"), "name" : "moon", "age" : "16" }
    
  • 刪除

    > db.solar.remove({name:"moon"});
    WriteResult({ "nRemoved" : 1 })
    > db.solar.find();
    { "_id" : ObjectId("56416af4997bc79c93b12e23"), "name" : "sun", "age" : 12 }
    > db.solar.remove({name:"moon"});
    WriteResult({ "nRemoved" : 0 })
    > db.solar.find();
    { "_id" : ObjectId("56416af4997bc79c93b12e23"), "name" : "sun", "age" : 12 }
    
  • _id 欄位可以自訂,但不能重複

    > user3={_id:"id_star", name:"star", age: 18};
    { "_id" : "id_star", "name" : "star", "age" : 18 }
    > db.solar.save(user3);
    WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "id_star" })
    >
    > user4={_id:"id_star", name:"star2", age: 20};
    { "_id" : "id_star", "name" : "star2", "age" : 20 }
    > db.solar.save(user4);
    WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
    > 
    > db.solar.find();
    { "_id" : ObjectId("56416af4997bc79c93b12e23"), "name" : "sun", "age" : 12 }
    { "_id" : "id_star", "name" : "star2", "age" : 20 }
    > db.solar.insert(user4);
    WriteResult({
      "nInserted" : 0,
      "writeError" : {
          "code" : 11000,
          "errmsg" : "E11000 duplicate key error index: test.solar.$_id_ dup key: { : \"id_star\" }"
      }
    })
    
  • 查詢後回傳 cursor,it可取得下一頁資料

    > for( var i=1; i<=40; i++ ) db.users.save( {userid:"user_"+i} );
    WriteResult({ "nInserted" : 1 })
    > db.users.find( );
    { "_id" : ObjectId("5641882a997bc79c93b12e39"), "userid" : "user_1" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3a"), "userid" : "user_2" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3b"), "userid" : "user_3" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3c"), "userid" : "user_4" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3d"), "userid" : "user_5" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3e"), "userid" : "user_6" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3f"), "userid" : "user_7" }
    { "_id" : ObjectId("5641882a997bc79c93b12e40"), "userid" : "user_8" }
    { "_id" : ObjectId("5641882a997bc79c93b12e41"), "userid" : "user_9" }
    { "_id" : ObjectId("5641882a997bc79c93b12e42"), "userid" : "user_10" }
    { "_id" : ObjectId("5641882a997bc79c93b12e43"), "userid" : "user_11" }
    { "_id" : ObjectId("5641882a997bc79c93b12e44"), "userid" : "user_12" }
    { "_id" : ObjectId("5641882a997bc79c93b12e45"), "userid" : "user_13" }
    { "_id" : ObjectId("5641882a997bc79c93b12e46"), "userid" : "user_14" }
    { "_id" : ObjectId("5641882a997bc79c93b12e47"), "userid" : "user_15" }
    { "_id" : ObjectId("5641882a997bc79c93b12e48"), "userid" : "user_16" }
    { "_id" : ObjectId("5641882a997bc79c93b12e49"), "userid" : "user_17" }
    { "_id" : ObjectId("5641882a997bc79c93b12e4a"), "userid" : "user_18" }
    { "_id" : ObjectId("5641882a997bc79c93b12e4b"), "userid" : "user_19" }
    { "_id" : ObjectId("5641882a997bc79c93b12e4c"), "userid" : "user_20" }
    Type "it" for more
    > it
    { "_id" : ObjectId("5641882a997bc79c93b12e4d"), "userid" : "user_21" }
    { "_id" : ObjectId("5641882a997bc79c93b12e4e"), "userid" : "user_22" }
    { "_id" : ObjectId("5641882a997bc79c93b12e4f"), "userid" : "user_23" }
    { "_id" : ObjectId("5641882a997bc79c93b12e50"), "userid" : "user_24" }
    { "_id" : ObjectId("5641882a997bc79c93b12e51"), "userid" : "user_25" }
    { "_id" : ObjectId("5641882a997bc79c93b12e52"), "userid" : "user_26" }
    { "_id" : ObjectId("5641882a997bc79c93b12e53"), "userid" : "user_27" }
    { "_id" : ObjectId("5641882a997bc79c93b12e54"), "userid" : "user_28" }
    { "_id" : ObjectId("5641882a997bc79c93b12e55"), "userid" : "user_29" }
    { "_id" : ObjectId("5641882a997bc79c93b12e56"), "userid" : "user_30" }
    { "_id" : ObjectId("5641882a997bc79c93b12e57"), "userid" : "user_31" }
    { "_id" : ObjectId("5641882a997bc79c93b12e58"), "userid" : "user_32" }
    { "_id" : ObjectId("5641882a997bc79c93b12e59"), "userid" : "user_33" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5a"), "userid" : "user_34" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5b"), "userid" : "user_35" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5c"), "userid" : "user_36" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5d"), "userid" : "user_37" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5e"), "userid" : "user_38" }
    { "_id" : ObjectId("5641882a997bc79c93b12e5f"), "userid" : "user_39" }
    { "_id" : ObjectId("5641882a997bc79c93b12e60"), "userid" : "user_40" }
    

    可用 while 搭配 cursor.hasNext 進行迴圈處理

    > var cursor = db.users.find();
    > while ( cursor.hasNext() ) printjson(cursor.next());
    { "_id" : ObjectId("5641882a997bc79c93b12e39"), "userid" : "user_1" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3a"), "userid" : "user_2" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3b"), "userid" : "user_3" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3c"), "userid" : "user_4" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3d"), "userid" : "user_5" }
    //..... 省略
    

    也可以用 forEach

    > db.users.find().forEach(printjson);
    { "_id" : ObjectId("5641882a997bc79c93b12e39"), "userid" : "user_1" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3a"), "userid" : "user_2" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3b"), "userid" : "user_3" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3c"), "userid" : "user_4" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3d"), "userid" : "user_5" }
    

    也可以把 cursor 當成陣列

    > var cursor = db.users.find();
    > printjson(cursor[5]);
    { "_id" : ObjectId("5641882a997bc79c93b12e3e"), "userid" : "user_6" }
    > while ( cursor.hasNext() ) printjson(cursor.next());
    > printjson(cursor[6]);
    { "_id" : ObjectId("5641882a997bc79c93b12e3f"), "userid" : "user_7" }
    

    _id 裡面就隱藏了資料建立時間

    > db.users.find().forEach(function (doc){ d = doc._id.getTimestamp(); print(d.getFullYear()+"-"+(d.getMonth()+1)+"-"+d.getDate() + " " + d.getHours() + ":" + d.getMinutes() + ":" + d.getSeconds()) })
    2015-11-10 14:1:14
    2015-11-10 14:1:14
    2015-11-10 14:1:14
    2015-11-10 14:1:14
    //.... 省略
    

    條件查詢

    > db.users.find({userid:"user_10"}).forEach(printjson);
    { "_id" : ObjectId("5641882a997bc79c93b12e42"), "userid" : "user_10" }
    

    自訂回傳欄位的條件查詢

    > db.users.find({userid:"user_10"},{_id:true}).forEach(printjson);
    { "_id" : ObjectId("5641882a997bc79c93b12e42") }
    

    findOne: 只取得第一筆資料

    > printjson(db.users.findOne({userid:"user_10"}));
    { "_id" : ObjectId("5641882a997bc79c93b12e42"), "userid" : "user_10" }
    

    limit: 限制回傳資料的筆數

    > db.users.find().limit(3);
    { "_id" : ObjectId("5641882a997bc79c93b12e39"), "userid" : "user_1" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3a"), "userid" : "user_2" }
    { "_id" : ObjectId("5641882a997bc79c93b12e3b"), "userid" : "user_3" }
    

進階查詢

首先準備測試資料

> db.mate.save({userid:"john", username:"John Lin", city:"Taipei", age: 20, room:"301"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"muder", username:"Muder Yen", city:"Taichung", age: 19, room:"301"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"mary", username:"Mary Wang", city:"Tainan", age: 23, room:"601"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"celina", username:"Celina Lin", city:"Taichung", age: 20, room:"601"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"lunar", username:"Lunar Wang", city:"I Lan", age: 22, room:"302"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"danny", username:"Danny Huang", city:"Taipei"});
WriteResult({ "nInserted" : 1 })
> db.mate.save({userid:"johnny", username:"Johnny Lo", city:"Taipei", age: null});
WriteResult({ "nInserted" : 1 })
> db.mate.find();
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

$gt 超過 22 歲的人

> db.mate.find( {age:{ $gt: 22}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }

$gte 大於等於 22 歲

> db.mate.find( {age:{ $gte: 22}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }

$ne: 不等於 22

> db.mate.find( {age:{ $ne: 22}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

$mod: 除以10餘0

> db.mate.find( {age:{ $mod: [10,0]}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }

$all: 完全等於 20

> db.mate.find( {age:{ $all:[20]}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }

$in: 22 或 23 歲

> db.mate.find( {age:{ $in:[22, 23]}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }

$nin: 不等於 22 或 23

> db.mate.find( {age:{ $nin:[22, 23]}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

$exists: 不存在 age 欄位

> db.mate.find( {age:{$exists:false}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }

null: age 欄位為 null,不存在 age 也包含在內

> db.mate.find( {age:null} );
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

有 age 欄位且 age 為 null

> db.mate.find( {age:{$in:[null], $exists:true}} );
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

$regex: username 以 M 為開頭

> db.mate.find( {username:{ $regex:/^M.*/ }} );
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }

$not: username 不以 M 為開頭

> db.mate.find( {username:{ $not:/^M.*/ }} );
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

以 javascript function 進行搜尋

> f=function() {return this.age>22;}
function () {return this.age>22;}
> db.mate.find(f);
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }

count(): 計算資料筆數

> db.mate.find().count();
7

skip(): 跳過幾筆資料
limit(): 限制回傳筆數

> db.mate.find().skip(2).limit(2);
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
> db.mate.find().skip(3).limit(1);
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }

sort: 針對 age 排序

> db.mate.find().sort({age:1});
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }

> db.mate.find().sort({age:-1});
{ "_id" : ObjectId("5641980c997bc79c93b12e77"), "userid" : "mary", "username" : "Mary Wang", "city" : "Tainan", "age" : 23, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e79"), "userid" : "lunar", "username" : "Lunar Wang", "city" : "I Lan", "age" : 22, "room" : "302" }
{ "_id" : ObjectId("5641980c997bc79c93b12e75"), "userid" : "john", "username" : "John Lin", "city" : "Taipei", "age" : 20, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e78"), "userid" : "celina", "username" : "Celina Lin", "city" : "Taichung", "age" : 20, "room" : "601" }
{ "_id" : ObjectId("5641980c997bc79c93b12e76"), "userid" : "muder", "username" : "Muder Yen", "city" : "Taichung", "age" : 19, "room" : "301" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7a"), "userid" : "danny", "username" : "Danny Huang", "city" : "Taipei" }
{ "_id" : ObjectId("5641980c997bc79c93b12e7b"), "userid" : "johnny", "username" : "Johnny Lo", "city" : "Taipei", "age" : null }

以下為所有測試的指令

db.mate.save({userid:"john", username:"John Lin", city:"Taipei", age: 20, room:"301"});
db.mate.save({userid:"muder", username:"Muder Yen", city:"Taichung", age: 19, room:"301"});
db.mate.save({userid:"mary", username:"Mary Wang", city:"Tainan", age: 23, room:"601"});
db.mate.save({userid:"celina", username:"Celina Lin", city:"Taichung", age: 20, room:"601"});
db.mate.save({userid:"lunar", username:"Lunar Wang", city:"I Lan", age: 22, room:"302"});
db.mate.save({userid:"danny", username:"Danny Huang", city:"Taipei"});
db.mate.save({userid:"johnny", username:"Johnny Lo", city:"Taipei", age: null});

db.mate.find( {age:{ $gt: 22}} );

db.mate.find( {age:{ $gte: 22}} );

db.mate.find( {age:{ $ne: 22}} );

db.mate.find( {age:{ $mod: [10,0]}} );

db.mate.find( {age:{ $all:[20]}} );

db.mate.find( {age:{ $in:[22, 23]}} );

db.mate.find( {age:{ $nin:[22, 23]}} );

db.mate.find( {age:{$exists:false}} );

db.mate.find( {age:null} );

db.mate.find( {age:{$in:[null], $exists:true}} );

db.mate.find( {username:{ $regex:/^M.*/ }} );

db.mate.find( {username:{ $not:/^M.*/ }} );


f=function() {return this.age>22;}
db.mate.find(f);


db.mate.find().count();

db.mate.find().skip(2).limit(2);

db.mate.find().skip(3).limit(1);


db.mate.find().sort({age:1});

db.mate.find().sort({age:-1});

MapReduce

MongoDB 的 Map, Reduce 函數都可以用 javascript 實作,Map 函數中必須呼叫 emit(key, value),對 collection 中所有紀錄運算過一次之後,再將 {key, [value1, value2, ....]} 傳遞給 reduce 進行處理,最後可再利用 finalize(),針對 reduce 的結果做更進一步的處理。

在 mongo 中,可以用 mapReduce 或是 runCommand 進行 MapReduce,help mr 看到線上的簡略說明

> help mr

See also http://dochub.mongodb.org/core/mapreduce

function mapf() {
  // 'this' holds current document to inspect
  emit(key, value);
}

function reducef(key,value_array) {
  return reduced_value;
}

db.mycollection.mapReduce(mapf, reducef[, options])

options
{[query : <query filter object>]
 [, sort : <sort the query.  useful for optimization>]
 [, limit : <number of objects to return from collection>]
 [, out : <output-collection name>]
 [, keeptemp: <true|false>]
 [, finalize : <finalizefunction>]
 [, scope : <object where fields go into javascript global scope >]
 [, verbose : true]}

query: 在進行 Map 運算前,先針對 collection 進行一次資料篩檢
sort: 針對 collection 資料排序,可減少 reduce 的處理時間
limit: 限制由 collection 回傳的資料筆數
out: 儲存統計結果的 collection,如果不指定,會放在暫存的 collection 中,連線中斷後,會自動刪除
keeptemp: 是否要保留暫存的 collection
finalize: 對 reduce 的結果再進行一次處理
scope: 指定可在 map, reduce, finalize 中使用的 global 變數
verbose: 顯示詳細的統計時間資訊

首先準備 map 與 reduce function

> mfunction = function() { emit(this.room, 1 )};
function () { emit(this.room, 1 )}
> 
> rfunction = function(key, values) {
... var x = 0;
... values.forEach( function(v) { x += v; } );
... return x;
... }

function (key, values) {
var x = 0;
values.forEach( function(v) { x += v; } );
return x;
}

runCommand,並指定輸出到 mate_res

> 
> res = db.runCommand( {
... mapreduce:"mate",
... map: mfunction,
... reduce: rfunction,
... out: "mate_res"
... });
{
    "result" : "mate_res",
    "timeMillis" : 1,
    "counts" : {
        "input" : 7,
        "emit" : 7,
        "reduce" : 3,
        "output" : 4
    },
    "ok" : 1
}
> 
> db.mate_res.find();
{ "_id" : null, "value" : 2 }
{ "_id" : "301", "value" : 2 }
{ "_id" : "302", "value" : 1 }
{ "_id" : "601", "value" : 2 }

加上一個 finalize 函數,將結果變成 room 與 count

> ffunction = function(key,value) {return {room:key, count:value};}
function (key,value) {return {room:key, count:value};}
> res = db.runCommand( {
... mapreduce:"mate",
... map: mfunction,
... reduce: rfunction,
... out: "mate_res",
... finalize: ffunction
... });
{
    "result" : "mate_res",
    "timeMillis" : 1,
    "counts" : {
        "input" : 7,
        "emit" : 7,
        "reduce" : 3,
        "output" : 4
    },
    "ok" : 1
}
> 
> db.mate_res.find();
{ "_id" : null, "value" : { "room" : null, "count" : 2 } }
{ "_id" : "301", "value" : { "room" : "301", "count" : 2 } }
{ "_id" : "302", "value" : { "room" : "302", "count" : 1 } }
{ "_id" : "601", "value" : { "room" : "601", "count" : 2 } }

加上 query,過濾 mate collection,只需要大於等於 22 歲的人

> res = db.runCommand( {
... mapreduce:"mate",
... map: mfunction,
... reduce: rfunction,
... out: "mate_res",
... finalize: ffunction,
... query: {age:{$gte: 22}}
... });
{
    "result" : "mate_res",
    "timeMillis" : 1,
    "counts" : {
        "input" : 2,
        "emit" : 2,
        "reduce" : 0,
        "output" : 2
    },
    "ok" : 1
}
> 
> db.mate_res.find();
{ "_id" : "302", "value" : { "room" : "302", "count" : 1 } }
{ "_id" : "601", "value" : { "room" : "601", "count" : 1 } }

以下為測試指令的集合

mfunction = function() { emit(this.room, 1 )};

rfunction = function(key, values) {
    var x = 0;
    values.forEach( function(v) { x += v; } );
    return x;
}

res = db.runCommand( {
    mapreduce:"mate",
    map: mfunction,
    reduce: rfunction,
    out: "mate_res"
});

db.mate_res.find();


ffunction = function(key,value) {return {room:key, count:value};}
res = db.runCommand( {
    mapreduce:"mate",
    map: mfunction,
    reduce: rfunction,
    out: "mate_res",
    finalize: ffunction
});

db.mate_res.find();


res = db.runCommand( {
    mapreduce:"mate",
    map: mfunction,
    reduce: rfunction,
    out: "mate_res",
    finalize: ffunction,
    query: {age:{$gte: 22}}
});

db.mate_res.find();

2015/11/26

13.67 - 陳浩基

13.67從書名開始,本身就是一個謎題,整本書六段故事,不僅時間部份是以倒敘的方式組成,每一個故事的發生時間,恰好跟香港當地發生的大事相關。作者陳浩基在後記中提到,香港的警民衝突,就像是一個螺旋一樣,回到了1967年當時的時代狀況,他也期待未來的香港,能像過去的歷史演進一般,恢復成一個和諧又美麗的香港。

13.67其實就是第一以及最後一個故事發生的年份,這六個故事都跟主角香港警探關振鐸有關,時間分別發生在2013年、2002年、1997年、1989年、1974年和1967年。故事的起點反而是關振鐸生命的終點,而故事的終點 1967年,卻是關振鐸從一個制服警員升遷成為警探的起點。

2013 年的故事「黑與白的真實」是一件謀殺案,香港警隊中的傳奇人物,有「天眼」之稱的關振鐸因為重病躺在病床上,只能透過腦波偵測儀器進行案件的分析,推理過程中,原本以為嫌疑犯已經慢慢地沉不住氣,案件即將水落石出了,但卻峰迴路轉,讓一個更縝密,計畫了二十年的謀殺案件現形。

故事的最後一句話:「或許手法是黑色的,但目的是白色的。」很明確地告訴我們,身為警察難免有時候也必須使用一些手段達到目的,但必須得拿捏掌握分寸,才能在灰黑的世界中,看到白色的光亮。

2002年「囚徒道義」表面上是一件謀殺案,實際上卻是幫派之間的衝突,而小明的師父關振鐸雖然已經退休成為顧問,但為了替警方解決兩大幫派的問題,刻意以一個人為製造的「囚徒兩難」的情境,在兩個幫派老大之間,製造了一個矛盾。不過到了故事發展的中途,才讓讀者知道,被謀殺的明星唐穎,其實是為了報仇,自願當作棋子假造了一起謀殺案。

1997年的故事「最長的一日」,因為香港九七大限,關振鐸在警隊服務了三十二年之後,決定要提早一個月退休,鏹水案以及石本添越獄的案子,讓關振鐸在退休前也不得安寧。

駱小明跟著關振鐸處理鏹水案,讓其他人手負責麻煩的石本添越獄案,在尋找線索的過程中,天眼關振鐸不但找出鏹水案的兇手,同時還找到了失蹤的石本添,這才讓人知道,發生的多起鏹水案,都是為了製造石本添越獄的最佳條件。

1989年的故事「泰美斯的天秤」,就是在說明石本添石本勝兄弟落網的過程,原本佈署在大樓周遭的警隊,不知道什麼原因,讓嫌犯突然警覺被警隊包圍,而突圍發生槍戰衝突,過程中造成了數個民眾死亡。眾多線索,讓大家懷疑有內奸,但都不曉得是誰。

關振鐸感嘆,立法局(應該是檢察機關)前面拿著天秤與劍,雙眼蒙布的女神,代表著公平與權力,他認為警察手中掌握的是劍,但沒有天秤,他必須用一切手段達到目的,抓到犯人,但是判斷的天秤,只能交給法院去審理。

1974年的故事「Borrowed Place」是廉政公署剛起步的主管夏嘉瀚的兒子被綁架的案子,在繳交贖款的過程中,關振鐸才發現綁架只是幌子,重點是夏嘉瀚放在家裡的重要文件,也就是所有涉入貪污的警員名單。

1967年的故事「Borrowed Time」是在關振鐸二十出頭的時候,還在當制服警員的時候,是一位少年跟一個警察追查炸彈兇手的過程,也因為這起事件,關振鐸升官成為一名警探。厲害的是,在最後一頁,故事點名了王冠棠以及阮文彬他們計畫加入俞家一同奮鬥的決定。

有網友將這幾個年代發生的特殊事件列出來:

  • 2013年是政改爭議和警民關係惡劣的一年;
  • 2002年是回歸五年香港經濟衰退,工廠大幅北上,第二年更爆發非典型肺炎(SARS)
  • 1997年香港回歸
  • 1989年天安門事件令香港成為政治避難所
  • 1974年廉政公署成立,大舉打擊警察內部貪污
  • 1967年為六七暴動

13.67以香港最擅長的警匪對峙為主題,以一位警探的一生作為引線,帶領我們看著香港這四十幾年的變化,雖然天眼關振鐸風風光光地不斷偵破許多案件,但作者也不斷提醒我們,警察辦案的手段,只要不是絕對的違法,為了破案是什麼都能做的,在過程中,即使有些許違反警例,重點完全只在結果。

看了很多翻譯的小說,但都遠遠不及一開始就以中文創作的故事,不管是「三體」或是這本「13.67」都可說是近幾年看到,令人印象深刻的故事。

陳浩基《13.67》讀後感 兼談正義香港警察

香港警察故事——讀陳浩基《13.67》後感

《13·67》讀後感:What's Right?

1367 -陳浩基

六七暴動,亦稱1967年香港左派暴動

2015/11/23

如何在 CentOS 安裝 MongoDB

MongoDB 屬於 NoSQL Database 中的文件資料庫這種類型,可以儲存 XML, JSON 或是其他格式的文件資料,這些資料都是 self-describing 的,也就是說文件之間並不需要完全一樣,換句話說,在更新 schema 時,並不需要修改既有的資料。

依照這個網頁的說明 Install MongoDB on Red Hat Enterprise or CentOS Linux,就可以把 mongodb 裝好。

Packages

  1. mongodb-org
    這是自動安裝其他四個 packages 的 metapackage
  2. mongodb-org-server
    包含 mongod daemon、configurations以及 init scripts
  3. mongodb-org-mongos
    包含 mongos daemon
  4. mongodb-org-shell
    包含 mongo shell
  5. mongodb-org-tools
    包含以下的 MongoDB 工具: mongoimport bsondump, mongodump, mongoexport, mongofiles, mongooplog, mongoperf, mongorestore, mongostat, and mongotop

mongod 的 service script 在 /etc/init.d/mongod

設定檔在 /etc/mongod.conf

資料存放在 /var/lib/mongo

log 存放在 /var/log/mongodb

vi /etc/yum.repos.d/mongodb-org-3.0.repo

[mongodb-org-3.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1

用以下指令安裝並啟動 mongodb

yum install -y mongodb-org
service mongod start

在 /var/log/mongodb/mongod.log 裡面看到這一行,基本上就是把 mongodb 環境安裝好了。

[initandlisten] waiting for connections on port 27017

簡單的 DB 測試

執行 mongo 就會直接連接到 test 這個 database

show dbs 是顯示所有資料庫, show collections 是顯示資料庫中的 collections,相當於關聯式資料庫的 table

> show dbs
local  0.078GB
> show dbs;
local  0.078GB
> show collections;

切換到 mydatabase 資料庫

> use mydatabase;
switched to db mydatabase
> db
mydatabase
  1. insert 新增
    建立一個 post 文件,將 post 放進

    > post = { "title": "今日天氣", "content": "晴朗", "date": new Date()}
    {
     "title" : "今日天氣",
     "content" : "晴朗",
     "date" : ISODate("2015-11-05T08:33:54.761Z")
    }
    > db.blog.insert(post)
    WriteResult({ "nInserted" : 1 })
    
  2. find 查詢
    以 find 取得所有資料

    > db.blog.find()
    { "_id" : ObjectId("563b147db835798ae60f4bae"), "title" : "今日天氣", "content" : "晴朗", "date" : ISODate("2015-11-05T08:33:54.761Z") }
    
  3. update 修改
    先修改剛剛的 post 資料,然後用 update 更新到資料庫中。

    > post.comment = ["Great!"]
    [ "Great!" ]
    > post
    {
     "title" : "今日天氣",
     "content" : "晴朗",
     "date" : ISODate("2015-11-05T08:33:54.761Z"),
     "comment" : [
         "Great!"
     ]
    }
    > db.blog.update({title: "今日天氣"}, post)
    WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
    > db.blog.find()
    { "_id" : ObjectId("563b147db835798ae60f4bae"), "title" : "今日天氣", "content" : "晴朗", "date" : ISODate("2015-11-05T08:33:54.761Z"), "comment" : [ "Great!" ] }
    
  4. remove 刪除
    以 remove 刪除資料

    > db.blog.remove({title: "今日天氣"});
    WriteResult({ "nRemoved" : 1 })
    > db.blog.find();
    

如何用 R 語言連接 MongoDB

先在 mongodb 重新新增兩筆資料

# mongo
> post = { "title": "今日天氣", "content": "晴朗", "date": new Date()}
> db.blog.insert(post)

> post2 = { "title": "明日天氣", "content": "陰天", "date": new Date()}
> db.blog.insert(post2)

在 R 語言的界面中,依照以下的過程進行測試

  1. 安裝 rmongodb 套件

    > install.packages("rmongodb")
    
  2. 使用 rmongodb library

    > library(rmongodb)
    
  3. 建立 mongodb 連線

    > mongo <- mongo.create(host="192.168.1.11" , db="test", username="", password="")
    > dbns <- mongo.get.database.collections(mongo, db="test")
    
  4. 建立查詢條件,並以 find 查詢資料

    > query <- '{ "title": { "$exists": true }, "content": { "$exists": true } }'
    > 
    > cur <- mongo.find(mongo, dbns, query=query)
    
  5. 將 cursor 結果轉換成 data.frame

     > title <- content <- date <- NULL
     > while (mongo.cursor.next(cur)) {
     +     value <- mongo.cursor.value(cur)
     +     title <- rbind(title, mongo.bson.value(value, 'title'))
     +     content <- rbind(content, mongo.bson.value(value, 'content'))
     +  
     +     date <- rbind(date, mongo.bson.value(value, 'date'))
     + }
     > 
     > posts <- data.frame(title=title, content=content, date=date)
    
     > posts
            title           content       date
     1 隞憭拇除 \xe6\xe6\x9c\x97 1446714342
     2 \xe6\x98憭拇除       \xe9憭\xa9 1446715746
    
  6. 中文顯示有問題
    在產生 data.frame 之前,必須先設定 UTF-8 Encoding

    > Encoding(title)="UTF-8"
    > 
    > Encoding(content)="UTF-8"
    > 
    > posts <- data.frame(title=title, content=content, date=date)
    > 
    > posts
      title content       date
    1 今日天氣    晴朗 1446714342
    2 明日天氣    陰天 1446715746
    

濃縮所有測試過程的指令如下

install.packages("rmongodb")
library(rmongodb)
mongo <- mongo.create(host="192.168.1.11" , db="test", username="", password="")
dbns <- mongo.get.database.collections(mongo, db="test")
query <- '{ "title": { "$exists": true }, "content": { "$exists": true } }'

cur <- mongo.find(mongo, dbns, query=query)

title <- content <- date <- NULL

while (mongo.cursor.next(cur)) {
value <- mongo.cursor.value(cur)
title <- rbind(title, mongo.bson.value(value, 'title'))
content <- rbind(content, mongo.bson.value(value, 'content'))
date <- rbind(date, mongo.bson.value(value, 'date'))
}
Encoding(title)="UTF-8"
Encoding(content)="UTF-8"
posts <- data.frame(title=title, content=content, date=date)
posts

在 rmongodb 寫入中文也會遇到編碼的問題,參考 R實踐之中文編碼轉換,必須先用 iconv 先將中文字串轉碼。

2015/11/16

如何在 CentOS 安裝 Cassandra

Cassandra 屬於 column-family 類型的 NoSQL database,特性是 ring cluster network topology,支援類似 SQL 語法的 CQL,適合儲存依照時間排序且scheme固定的大量資料。以下嘗試在 centos 安裝 cassandra。

單一節點

tar zxvf apache-cassandra-2.2.3-bin.tar.gz -C /usr/share

#啟動
/usr/share/apache-cassandra-2.2.3/bin/cassandra

資料會放在
/usr/share/apache-cassandra-2.2.3/data 這個目錄中

系統紀錄放在
/usr/share/apache-cassandra-2.2.3/logs 這個目錄中

然後參考 cassandra Getting Started 的範例,使用 CQL 測試這個 DB。

// 建立資料庫 mykeyspace
CREATE KEYSPACE mykeyspace
WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

// 使用資料庫
USE mykeyspace;

// 建立 table: users
CREATE TABLE users (
  user_id int PRIMARY KEY,
  fname text,
  lname text
);


// insert 三筆測試資料
INSERT INTO users (user_id,  fname, lname)
  VALUES (1745, 'john', 'smith');
INSERT INTO users (user_id,  fname, lname)
  VALUES (1744, 'john', 'doe');
INSERT INTO users (user_id,  fname, lname)
  VALUES (1746, 'john', 'smith');

// 取得 users 所有資料
SELECT * FROM users;

 user_id | fname | lname
---------+-------+-------
    1745 |  john | smith
    1744 |  john |   doe
    1746 |  john | smith

(3 rows)

// 針對 lname 建立 index
CREATE INDEX ON users (lname);

// 條件搜尋
SELECT * FROM users WHERE lname = 'smith';

 user_id | fname | lname
---------+-------+-------
    1745 |  john | smith
    1746 |  john | smith

直接用 kill -9 停止此單一節點的 cassandra 的 process。

ps aux | grep cassandra
kill -9 20109

兩個節點

在兩台機器上,都分別先將單一節點的設定方式做好。記得一定要先以單機方式先啟動一次,否則會出現 Token 錯誤的問題。

使用 /usr/share/apache-cassandra-2.2.3/tools/bin/token-generator,計算這兩個節點在 Murmur3Partitioner 時,token 分別要設定為多少?

[root@server bin]# /usr/share/apache-cassandra-2.2.3/tools/bin/token-generator
Token Generator Interactive Mode
--------------------------------

 How many datacenters will participate in this Cassandra cluster? 2
 How many nodes are in datacenter #1? ^C[root@kokola bin]# ./token-generator
Token Generator Interactive Mode
--------------------------------

 How many datacenters will participate in this Cassandra cluster? 1
 How many nodes are in datacenter #1? 2

DC #1:
  Node #1:  -9223372036854775808
  Node #2:                     0

然後分別編輯兩台機器上的 /usr/share/apache-cassandra-2.2.3/conf/cassandra.yaml

把

- seeds: "127.0.0.1"

改為

- seeds: "192.168.1.11,192.168.1.12"

把

listen_address: localhost

分別改為各自的 IP: 192.168.1.11,192.168.1.12

listen_address: 192.168.1.11

在 192.168.1.11 增加一行

initial_token: -9223372036854775808

在 192.168.1.12 則改為

initial_token: 0

然後分別啟動兩台機器的 cassandra。

就可以在第二台機器,用 CQL 取得剛剛在單一節點時建立的 mykeyspace

[root@server bin]# ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.2.3 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> use mykeyspace ;
/usr/share/apache-cassandra-2.2.3/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py:3331: DeprecationWarning: ResponseFuture.result timeout argument is deprecated. Specify the request timeout via Session.execute[_async].
cqlsh:mykeyspace> 
cqlsh:mykeyspace> select * from users;

 user_id | fname | lname
---------+-------+-------
    1745 |  john | smith
    1744 |  john |   doe
    1746 |  john | smith

(3 rows)

可以用 nodetool 查看 cluster 的狀態

[root@server bin]# ./nodetool -h localhost status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  192.168.1.11  68.09 KB   256          ?       53b41601-c145-4c1a-81d1-9b5f8b09ac73  rack1
UN  192.168.1.12  207.75 KB  256          ?       66642998-9179-4b06-b341-e654dc8e8ca6  rack1

service script

以下為 service script,儲存這個檔案到 /etc/init.d/cassandra,chmod 755 後就可以用 service cassandra start 啟動 cassandra。

#!/bin/bash
# chkconfig: 2345 99 01
# description: Cassandra

. /etc/rc.d/init.d/functions

CASSANDRA_HOME=/usr/share/apache-cassandra-2.2.3/
CASSANDRA_BIN=$CASSANDRA_HOME/bin/cassandra
CASSANDRA_NODETOOL=$CASSANDRA_HOME/bin/nodetool
CASSANDRA_LOG=$CASSANDRA_HOME/logs/cassandra.log
CASSANDRA_PID=/var/run/cassandra.pid
CASSANDRA_LOCK=/var/lock/subsys/cassandra
PROGRAM="cassandra"

if [ ! -f $CASSANDRA_BIN ]; then
  echo "File not found: $CASSANDRA_BIN"
  exit 1
fi

RETVAL=0

start() {
  if [ -f $CASSANDRA_PID ] && checkpid `cat $CASSANDRA_PID`; then
    echo "Cassandra is already running."
    exit 0
  fi
  echo -n $"Starting $PROGRAM: "
  daemon $CASSANDRA_BIN -p $CASSANDRA_PID >> $CASSANDRA_LOG 2>&1
  usleep 500000
  RETVAL=$?
  if [ $RETVAL -eq 0 ]; then
    touch $CASSANDRA_LOCK
    echo_success
  else
    echo_failure
  fi
  echo
  return $RETVAL
}

stop() {
  if [ ! -f $CASSANDRA_PID ]; then
    echo "Cassandra is already stopped."
    exit 0
  fi
  echo -n $"Stopping $PROGRAM: "
  $CASSANDRA_NODETOOL -h 127.0.0.1 decommission
  if kill `cat $CASSANDRA_PID`; then
    RETVAL=0
    rm -f $CASSANDRA_LOCK
    echo_success
  else
    RETVAL=1
    echo_failure
  fi
  echo
  [ $RETVAL = 0 ]
}

status_fn() {
  if [ -f $CASSANDRA_PID ] && checkpid `cat $CASSANDRA_PID`; then
    echo "Cassandra is running."
    exit 0
  else
    echo "Cassandra is stopped."
    exit 1
  fi
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status_fn
    ;;
  restart)
    stop
    start
    ;;
  *)
    echo $"Usage: $PROGRAM {start|stop|restart|status}"
    RETVAL=3
esac

exit $RETVAL

如何用 R 語言連接 cassandra 資料庫

以下簡述如何使用 R 語言連接 cassandra 資料庫。

  1. 調整 cassandra 設定,將 rpc server 打開,並重新啟動 cassandra

    修改 /usr/share/apache-cassandra-2.2.3/conf/cassandra.yaml,將 rpc server 打開

     start_rpc: true
    
  2. 下載 cassandra-jdbc library
    到 cassandra-jdbc 網頁,下載 cassandra-jdbc-2.1.1.jar 以及 cassandra-jdbc dependencies,將所有 jar 檔都放在要執行 R 語言的電腦的一個資料夾中。

  3. 在 R 語言,安裝 RJDBC 套件

     install.packages("RJDBC")
    
  4. R 語言的範例程式

    • 載入 RJDBC library

      > library(RJDBC)
      Loading required package: DBI
      Loading required package: rJava
      
    • 載入 Cassandra JDBC driver,以及相關 jars,我是放在 D:/git/system/pkg/cassandra/cassandra-jdbc/libs/ 這個目錄中,要注意目錄最後面的 libs 後面一定要加上 /

      > cassdrv <- JDBC("org.apache.cassandra.cql.jdbc.CassandraDriver", list.files("D:/git/system/pkg/cassandra/cassandra-jdbc/libs/",pattern="jar$",full.names=T))
      
    • 連接到 cassandra

      > casscon <- dbConnect(cassdrv, "jdbc:cassandra://192.168.1.11:9160/mykeyspace")
      
    • 對此 cassandra connection 執行 CQL

      > res <- dbGetQuery(casscon, "select * from users limit 10")
      
    • 取得 CQL 結果

      > res
      user_id fname lname
      1    1745  john smith
      2    1744  john   doe
      3    1746  john smith
      
  5. 錯誤訊息
    如果在連接 cassandra 時,發生 SQLNonTransientConnectionException,就表示 cassandra server 並沒有打開 rpc server 連接的設定。

     > casscon <- dbConnect(cassdrv, "jdbc:cassandra://192.168.1.11:9160/mykeyspace")
     Error in .jcall(drv@jdrv, "Ljava/sql/Connection;", "connect", as.character(url)[1],  : 
       java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
    

References

[研究] Apache Cassandra 2.0.3 安裝 (CentOS 6.5 x64)

設定 Cassandra cluster 的 token

cassandra token calculator