歡迎關(guān)注個(gè)人公眾號(hào) DailyJobOps

原文地址：Gitlab整理匯總（安裝、非22端口克隆番宁、升級(jí)及遇到問(wèn)題元莫、備份恢復(fù)）

1、安裝

如果不采用自建數(shù)據(jù)庫(kù)蝶押，而是gitlab自集成的數(shù)據(jù)庫(kù)踱蠢，那么安裝很簡(jiǎn)單

1.1、采用外置自建數(shù)據(jù)庫(kù)

• 需要先成功安裝數(shù)據(jù)庫(kù)棋电，確保數(shù)據(jù)庫(kù)可以正常訪問(wèn)
• 在數(shù)據(jù)庫(kù)中創(chuàng)建對(duì)應(yīng)的數(shù)據(jù)庫(kù)茎截、賬號(hào)、密碼赶盔，然后授權(quán)
• 下載rpm包企锌，執(zhí)行yum install gitlab-ce.xxx.rpm
• 修改配置文件 /etc/gitlab/gitlab.rb
• 重載配置 gitlab-ctl reconfigure
• 重啟服務(wù) gitlab-ctl restart

1.2、采用Gitlab自集成數(shù)據(jù)庫(kù)

• 下載rpm包招刨，執(zhí)行yum install gitlab-ce.xxx.rpm
• 重載配置 gitlab-ctl reconfigure
• 重啟服務(wù) gitlab-ctl restart

2霎俩、非22端口克隆

常規(guī)git倉(cāng)庫(kù)clone的方式為：

• http方式

  git clone http://gitlab.xxx.com/devops-group/devops-dbp-platform.git
  

• ssh方式（默認(rèn)22端口）

  git clone git@gitlab.xxx.com:devops-group/devops-dbp-platform.git
  

如果gitlab安裝的ssh端口（其實(shí)就是安裝主機(jī)的ssh服務(wù)的啟動(dòng)端口）是非22端口，比如是2022端口沉眶，則ssh方式克隆地址變更為

git clone ssh://git@gitlab.xxx.com:2022/devops-group/devops-dbp-platform.git

注意和上面默認(rèn)22端口的地址做對(duì)比打却，看看不一樣的地方在哪里

3、升級(jí)

跨大版本升級(jí)的原則是：

先升級(jí)到下一個(gè)大版本中的最高版本谎倔，沒(méi)有問(wèn)題之后再如此繼續(xù)直到升級(jí)到最后一個(gè)大版本中需要的版本

比如柳击，實(shí)際環(huán)境中當(dāng)前版本是 8.1.4 那么升級(jí)路線是 8.1.4 -> 8.17.8 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12

?? 升級(jí)過(guò)程中需要注意一個(gè)數(shù)據(jù)庫(kù)的問(wèn)題，在 12.1 及之后版本片习，官方移除了對(duì)MySQL的支持捌肴，數(shù)據(jù)庫(kù)使用PostgreSQL

image.png

升級(jí)步驟
1、先停止數(shù)據(jù)寫入（當(dāng)然如果在一個(gè)確定不會(huì)有寫入的時(shí)候藕咏，比如下班之后状知，獲取停止域名解析等可以不進(jìn)行如下操作）

sudo gitlab-ctl stop unicorn 
sudo gitlab-ctl stop puma 
sudo gitlab-ctl stop sidekiq 

2、按照整理出來(lái)的升級(jí)路徑下載對(duì)應(yīng)的rpm包
這里推薦 https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7
3孽查、更新升級(jí)

# 安裝更新rpm包
rpm -Uvh gitlab-ce-10.8.7-ce.0.el7.x86_64.rpm
# 重新生效配置
gitlab-ctl reconfigure
# 重啟服務(wù)
gitlab-ctl restart

4饥悴、驗(yàn)證當(dāng)前版本是否正常
5、重復(fù)如上過(guò)程盲再，升級(jí)到下一個(gè)版本

4西设、備份及恢復(fù)

備份配置

# 配置相關(guān)配置
# vim /etc/gitlab/gitlab.rb
# gitlab 訪問(wèn)域名
external_url 'http://192.168.8.127'  
# 備份配置
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/data/gitlab/backups" 
gitlab_rails['backup_archive_permissions'] = 0644  
# 備份有效期，單位為秒答朋，比如如下7天 贷揽，也可以根據(jù)自己實(shí)際情況來(lái)做異地轉(zhuǎn)存
# 這里的有效期是針對(duì)本地存儲(chǔ)，而非remote storage梦碗，比如阿里云的OSS
gitlab_rails['backup_keep_time'] = 604800  

備份

gitlab-rake gitlab:backup:create

image.png

注意上述告警提示禽绪，需要自己備份如下兩個(gè)文件蓖救，恢復(fù)的時(shí)候也確保這兩個(gè)位置有對(duì)應(yīng)的問(wèn)題

• /etc/gitlab/gitlab.rb
• /etc/gitlab/gitlab-secrets.json

遠(yuǎn)端備份

另外從上述備份記錄中注意到 Uploading backup archive to remote storage ... skipped， 從7.4版本支持直接轉(zhuǎn)存本地備份到遠(yuǎn)程（Starting with GitLab 7.4 you can let the backup script upload the ‘.tar’ file it creates）

舉例

gitlab_rails['backup_upload_connection'] = {
  'provider' => 'Aliyun',
  'aliyun_accesskey_id' => 'Access Key',
  'aliyun_secretekey_id' => 'Secret Key',
  'aliyun_oss_endpoint' => 'http://oss-cn-beijing-internal.aliyuncs.com',
  'aliyun_oss_bucket' => 'devops-backup',
  'aliyun_oss_location' => 'beijing'
}
# 對(duì)象存儲(chǔ)bucket中的子目錄
gitlab_rails['backup_upload_remote_directory'] = 'gitlab'

注意：一般備份不用長(zhǎng)久保存丐一，備份在OSS藻糖，為了節(jié)約成本，可以配置阿里云OSS bucket的生命周期库车，讓其保留最近30天的備份即可

具體參考
https://docs.gitlab.com/12.10/ee/administration/object_storage.html

恢復(fù)

注意事項(xiàng)：
1巨柒、恢復(fù)的文件必須是在配置的備份路徑下，比如這里的 /data/gitlab/backups
2柠衍、恢復(fù)的時(shí)候指定備份的序號(hào)洋满，比如 637134260_2021_11_17_12.0.12_gitlab_backup.tar 這里取637134260_2021_11_17_12.0.12
3、備份文件的權(quán)限必須是 644
4珍坊、檢查恢復(fù)狀態(tài) gitlab-rake gitlab:check SANITIZE=true

sudo gitlab-ctl stop unicorn 
sudo gitlab-ctl stop puma 
sudo gitlab-ctl stop sidekiq 
# Verify 
sudo gitlab-ctl status
# restore
gitlab-rake gitlab:backup:restore BACKUP=1629369869

5牺勾、內(nèi)存消耗問(wèn)題

#進(jìn)程超時(shí)時(shí)間
unicorn['worker_timeout'] = 60
#進(jìn)程數(shù)
unicorn['worker_processes'] = 10
#進(jìn)程最小內(nèi)存 200KB
unicorn['worker_memory_limit_min'] = "200 * 1 << 20"
#進(jìn)程最大內(nèi)存 300MB 
unicorn['worker_memory_limit_max'] = "300 * 1 << 20"
#并發(fā)數(shù)
sidekiq['concurrency'] = 16
#數(shù)據(jù)庫(kù)緩存
postgresql['shared_buffers'] = "256MB"
#數(shù)據(jù)庫(kù)并發(fā)數(shù)
postgresql['max_worker_processes'] = 8

6、升級(jí)過(guò)程問(wèn)題整理：

1阵漏、mysql2 adapter 問(wèn)題

報(bào)錯(cuò)信息：

NameError: uninitialized constant Mysql2::Client::SECURE_CONNECTION

解決方案：

升級(jí)到0.3.20 然后修改Gemfile.lock

-> https://github.com/brianmario/mysql2/issues/711
-> Use mysql2 0.3.17 or higher for MySQL 5.7 compatibility.

/opt/gitlab/embedded/bin/gem install -i/opt/gitlab/embedded/service/gem/ruby/2.1.0 mysql2 -v 0.3.20
vim /opt/gitlab/embedded/service/gitlab-rails/Gemfile.lock
mysql2 (0.3.20)

2驻民、安裝gitlab-ce之后，且在mysql2 adapter正常的情況履怯，需要執(zhí)行 gitlab-rake setup 進(jìn)行數(shù)據(jù)庫(kù)表安裝

3回还、gitlab基于備份進(jìn)行恢復(fù)報(bào)錯(cuò)

報(bào)錯(cuò)信息：

• ERROR 1227 (42000) at line 27: Access denied; you need (at least one of) the SUPER privilege(s) for this operation

解決方案：

管理員登錄數(shù)據(jù)庫(kù)然后給gitlab用戶賦權(quán) `**grant super on *.* to 'gitlab'@'localhost';**`

報(bào)錯(cuò)信息

• ERROR 1840 (HY000) at line 33: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty.

解決方案：

管理員登錄數(shù)據(jù)庫(kù) 執(zhí)行 **reset master;**  因?yàn)間itlab使用的MySQL是本地自建，單節(jié)點(diǎn)不需要主從叹洲，故reset不影響柠硕；
還有一種方式是注銷掉 備份文件中涉及到 `@@GLOBAL.GTID_PURGED`

4、權(quán)限報(bào)錯(cuò)

報(bào)錯(cuò)信息

Failed asserting that mode permissions on "/data/git-data/repositories" is 2770

解決方案

chmod 2770 /data/git-data/repositories

5运提、Bundler::GemNotFound

報(bào)錯(cuò)信息

Bundler::GemNotFound: Your bundle is locked to mysql2 (0.3.20), but that version could not be found in any of the sources listed in your Gemfile. If you haven't changed sources, that means the author of mysql2 (0.3.20) has removed it. You'll need to update your bundle to a different version of mysql2 (0.3.20) that hasn't been removed in order to install

昨天按照

/opt/gitlab/embedded/bin/gem install -i/opt/gitlab/embedded/service/gem/ruby/2.1.0 mysql2 -v 0.3.20

已經(jīng)安裝0.3.20版本蝗柔，但是升級(jí)之后提示不存在，在 /opt/gitlab/embedded/service/gem/ruby/2.1.0/extensions/x86_64-linux/2.1.0/也看到對(duì)應(yīng)的版本存在民泵。
猜測(cè)是因?yàn)樯?jí)之后安裝方式是否不一樣

• /opt/gitlab/embedded/bin/gem list |grep mysql → 提示沒(méi)有MySQL
• 重新安裝癣丧，只不過(guò)不指定安裝目錄，/opt/gitlab/embedded/bin/gem install mysql2 -v 0.3.20
• 再次 /opt/gitlab/embedded/bin/gem list |grep mysql 提示安裝成功
• 通過(guò) find /opt/gitlab/ -name 'mysql2-0.3.20' 查找發(fā)現(xiàn)栈妆，新版本應(yīng)該是從 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/ 下找擴(kuò)展

image.png

6胁编、升級(jí)到 9.5.10 報(bào)錯(cuò)

報(bào)錯(cuò)信息：

LoadError: cannot load such file -- peek-mysql2

解決方案：

[root@devops-gitlab-vpc ~]# /opt/gitlab/embedded/bin/gem install peek-mysql2
Fetching: peek-mysql2-1.2.0.gem (100%)
Successfully installed peek-mysql2-1.2.0
Parsing documentation for peek-mysql2-1.2.0
Installing ri documentation for peek-mysql2-1.2.0
Done installing documentation for peek-mysql2 after 0 seconds
1 gem installed

但是在安裝了之后還是提示 cannot load such file -- peek-mysql2
最終經(jīng)過(guò)排查和試驗(yàn)，在 Gemfile.lock 中 修改

gem 'peek-mysql2', '~> 1.1.0', group: :mysql 為 gem 'peek-mysql2' 和 之前 mysql2 的保持一致签钩，重新 gitlab:check 該錯(cuò)誤消失

7、升級(jí)到 9.5.10 報(bào)錯(cuò)

報(bào)錯(cuò)信息：

Mysql2::Error: SELECT command denied to user 'gitlab'@'localhost' for table 'user': SHOW FULL FIELDS FROM `mysql`.`user`

解決方案

mysql> grant select on mysql.* to gitlab@'localhost'; 
Query OK, 0 rows affected, 1 warning (0.00 sec)

報(bào)錯(cuò)信息：

Mysql2::Error: Thread stack overrun:  14000 bytes used of a 131072 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.: UPDATE `web_hooks` SET `job_events` = `web_hooks`.`build_events` WHERE `web_hooks`.`id` >= 1 AND `web_hooks`.`id` < 2

thread_stack=256K

Mysql2::Error: Duplicate column name 'job_events': ALTER TABLE web_hooks ADD job_events tinyint(1)

解決方案：

mysql> select job_events from web_hooks ;
+------------+
| job_events |
+------------+
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
+------------+
18 rows in set (0.00 sec)

mysql> alter table web_hooks drop column job_events ;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

報(bào)錯(cuò)信息：

Mysql2::Error: Trigger already exists: CREATE TRIGGER trigger_688beaaec90d_insert / Mysql2::Error: Trigger already exists: CREATE TRIGGER trigger_688beaaec90d_update

解決方案：

mysql> show triggers ;
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
| Trigger                     | Event  | Table     | Statement                                 | Timing | Created                | sql_mode          | Definer          | character_set_client | collation_connection | Database Collation |
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
| trigger_688beaaec90d_insert | INSERT | web_hooks | SET NEW.`job_events` = NEW.`build_events` | BEFORE | 2021-08-12 11:40:22.88 | STRICT_ALL_TABLES | gitlab@localhost | utf8                 | utf8_general_ci      | utf8_general_ci    |
| trigger_688beaaec90d_update | UPDATE | web_hooks | SET NEW.`job_events` = NEW.`build_events` | BEFORE | 2021-08-12 11:40:22.88 | STRICT_ALL_TABLES | gitlab@localhost | utf8                 | utf8_general_ci      | utf8_general_ci    |
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
2 rows in set (0.00 sec)

mysql> drop trigger trigger_688beaaec90d_insert ;
Query OK, 0 rows affected (0.01 sec)

mysql> drop trigger trigger_688beaaec90d_update ;
Query OK, 0 rows affected (0.01 sec)

報(bào)錯(cuò)信息：

Mysql2::Error: Statement violates GTID consistency: CREATE TABLE ... SELECT.: CREATE TABLE issue_assignees AS
SELECT assignee_id AS user_id, id AS issue_id FROM issues WHERE assignee_id IS NOT NULL

問(wèn)題分析：

MySQL5.6及以上的版本坏快，開啟了 enforce_gtid_consistency=true 功能導(dǎo)致的铅檩，MySQL官方解釋說(shuō)當(dāng)啟用 enforce_gtid_consistency 功能的時(shí)候，MySQL只允許能夠保障事務(wù)安全莽鸿，并且能夠被日志記錄的SQL語(yǔ)句被執(zhí)行昧旨，像create table … select 和 create temporarytable語(yǔ)句拾给，以及同時(shí)更新事務(wù)表和非事務(wù)表的SQL語(yǔ)句或事務(wù)都不允許執(zhí)行。

解決方案：

# GTID_MODE = ON requires ENFORCE_GTID_CONSISTENCY = ON.
gtid_mode = 0
enforce_gtid_consistency = 0

8兔沃、升級(jí) 10.8.7 報(bào)錯(cuò)

報(bào)錯(cuò)信息

• mkmf.rb can't find header files for ruby at /opt/gitlab/embedded/lib/ruby/include/ruby.h

參考 https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3635

解決方案

[root@devops-gitlab-vpc embedded]# cd /opt/gitlab/embedded/lib/ruby/include
-bash: cd: /opt/gitlab/embedded/lib/ruby/include: No such file or directory
[root@devops-gitlab-vpc embedded]# cd /opt/gitlab/embedded/lib/ruby/
[root@devops-gitlab-vpc ruby]# ll
total 12
drwxr-xr-x 31 root root 4096 Aug 12 12:18 2.3.0
drwxr-xr-x  3 root root 4096 Aug 12 12:19 gems
drwxr-xr-x  3 root root 4096 Jul 26  2018 site_ruby
 
 
# complie ruby 2.3.0
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.0.tar.gz
tar -zxvf ruby-2.3.0.tar.gz
cd ruby-2.3.0
./configure
make
 
cp -ra include /opt/gitlab/embedded/lib/ruby
 
cp -ra .ext/include/x86_64-linux /opt/gitlab/embedded/lib/ruby/include/ruby/
 
 
# Error: /opt/gitlab/embedded/lib/ruby/include/ruby/ruby.h:24:25: fatal error: ruby/config.h: No such file or directory
cd /opt/gitlab/embedded/lib/ruby/include/
cp ruby/x86_64-linux/ruby/config.h ruby/
 
 
# Error: make: *** No rule to make target `/include/x86_64-linux/ruby/config.h', needed by `client.o'.  Stop.
mkdir -p /include/x86_64-linux/ruby/
cp /opt/gitlab/embedded/lib/ruby/include/ruby/config.h /include/x86_64-linux/ruby/
 
 
最后在安裝0.4.10
/opt/gitlab/embedded/bin/gem install mysql2 -v 0.4.10
 
 
# Error: LoadError: cannot load such file -- peek-mysql2
/opt/gitlab/embedded/bin/gem install peek-mysql2 -v 1.1.0
vim /opt/gitlab/embedded/service/gitlab-rails/Gemfile 
gem 'peek-mysql2', '~> 1.1.0', group: :mysql -> gem 'peek-mysql2', '~> 1.1.0'

再次 gitlab-ctl reconfigure

報(bào)錯(cuò)信息：

ERROR: Encountered unsupported config key 'gitlab_git_http_server' in /etc/gitlab/gitlab.rb.

gitlab_git_http_server['repo_root'] = "xxx" → gitlab_workhorse['repo_root'] = "xxx"

Error: Mysql2::Error: Illegal mix of collations (utf8_general_ci,IMPLICIT) and  (utf8_unicode_ci,IMPLICIT) for operation '=':         INSERT INTO user_synced_attributes_metadata   (user_id, provider, email_synced)

報(bào)錯(cuò)分析：

user_synced_attributes_metadata 采用默認(rèn)字符集和字符集排序 utf8 和  utf8_general_ci （show variables like '%collation%';）

users 表和很多字段都是  utf8_unicode_ci 字符集排序蒋得，

解決方案：

根據(jù)上面的報(bào)錯(cuò)修復(fù) email_provider 字符集排序。

mysql> alter table users modify email_provider varchar(255) COLLATE utf8_general_ci  DEFAULT NULL; 
Query OK, 115 rows affected (0.06 sec) Records: 115  Duplicates: 0  Warnings: 0

然后有問(wèn)題乒疏，就把 users 表整個(gè)字符集都修改了

mysql> alter table users convert to character set utf8 collate utf8_general_ci ;

最后還是存在字符集排序的問(wèn)題额衙，修改整個(gè)數(shù)據(jù)庫(kù)中不是 utf8_general_ci 的所有表 字符集排序

9、升級(jí)11.11.8 報(bào)錯(cuò)

報(bào)錯(cuò)信息

compiling statement.c
linking shared-object mysql2/mysql2.so
/bin/ld: unrecognized option '--compress-debug-sections=zlib'
/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
make: *** [mysql2.so] Error 1

排查ld依賴

[root@devops-gitlab-vpc ruby-2.5.0]# ld --help |grep compress
[root@devops-gitlab-vpc ruby-2.5.0]# ld -v
GNU ld version 2.25.1-32.base.el7_4.1
[root@devops-gitlab-vpc ~]# ld -v
GNU ld version 2.27-44.base.el7

注意：
1怕吴、提示無(wú) xxx 包的時(shí)候：

1窍侧、建議采用 /opt/gitlab/embedded/bin/gem install xxxx -v x.y.z 安裝
2、至于 x.y.z 應(yīng)該是多少转绷，建議先從 Gemfile.lock 中獲取 
grep xxx /opt/gitlab/embedded/service/gitlab-rails/Gemfile.lock

2伟件、過(guò)程中需要不同版本的ruby相關(guān)頭文件
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.0.tar.gz
wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.0.tar.gz
這里在不同的版本安裝的時(shí)候，清理之前版本的议经，重新copy最新版本

3斧账、批量更新表字符集排序

4、Gemfile 文件中mysql2 和 peek-mysql2 的配置都不帶后續(xù)的group
gem 'peek-mysql2', '~> 1.1.0', group: :mysql -> gem 'peek-mysql2', '~> 1.1.0'

10煞肾、Mysql2::Error: Incorrect string value 亂碼

報(bào)錯(cuò)信息：

Mysql2::Error: Incorrect string value: '\xF0\x9F\x94\xA8\xE6\x89...' for column 'commit_title' at row 1: INSERT INTO `push_event_payloads` (`event_id`, `commit_count`, `ref_type`, `action`, `commit_from`, `commit_to`, `ref`, `commit_title`) VALUES (259421, 1, 0, 2, x'a2ad3a918aa4d9aac226c430b365bb5cb05f5619', x'1041d9cdfaed711af45455b65497284194aed02c', 'dev_test1', '1咧织，修改??手機(jī)空布局不居中問(wèn)題。')

Fix:

解決方案

alter table push_event_payloads change commit_title commit_title varchar(70) character set utf8mb4 collate utf8mb4_general_ci ;

11扯旷、gitlab-ctl reconfigure 卡住不動(dòng)

報(bào)錯(cuò)信息：

Error: gitlab-ctl reconfigure 過(guò)程中如果卡主不動(dòng)

解決方案：

systemctl restart gitlab-runsvdir