Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible

Posted on Oct 11, 2023

Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible

GitLab, a popular web-based Git repository manager, uses PostgreSQL as its default database. In a self-managed GitLab, the storage we allocated may not be able to fulfill our requirement in the long run, and we might need to move our database to a new location.

In this blog we  will see how we can migrate our existing database to a custom location using Ansible. We will cover the necessary steps and configurations required to achieve this automation and provide a step-by-step guide to help you get started. You can easily migrate GitLab PostgreSQL to a custom location with just three steps using Ansible; create a custom location for migration, migrate the existing database to the new site, changing and apply the GitLab configuration file.

We already have some custom configurations on my self-managed GitLab server. I will show how we can change the configuration file to use the custom location for the GitLab PostgreSQL database.

Read more about: AWS vs Google Cloud Vs Microsoft Azure



Prerequisites An up-and-running self-managed GitLab server. User with appropriate permission to make changes to GitLab configuration file. The private key to SSH to the remote server using Ansible. So, let’s dive in and explore how to automate GitLab PostgreSQL database migration using Ansible.

Prerequisites

  • An up-and-running self-managed GitLab server.
  • User with appropriate permission to make changes to GitLab configuration file.
  • The private key to SSH to the remote server using Ansible.



So, let’s dive in and explore how to automate GitLab PostgreSQL database migration using Ansible.


  • An up-and-running self-managed GitLab server.
  • User with appropriate permission to make changes to GitLab configuration file.
  • The private key to SSH to the remote server using Ansible.


So, let’s dive in and explore how to automate GitLab PostgreSQL database migration using Ansible.

File structure

|__ roles
|_______migrate
|__________defaults
|_____________main.yml
|__________tasks
|_____________main.yml
|_______reconfigure
|__________tasks
|_____________main.yml
|__________handler
|_____________main.yml
|__________template
|_____________gitlab.rb.j2
|__________vars
|_____________main.yml
|__ Configure.yml
|__ inventory.txt

Inventory file format

[hosts]
 gitlab ansible_host= 

[all:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_user= 

The inventory file needs to be in this format. The “inventory_hostname “value is required in the later step.

Step 1: Create a custom location for migrating

We will create a custom location in the initial step if we don’t use the ansible task yet. Run the following code on the Ansible playbook role:

- name: Create a directory if it does not exist
ansible.builtin.file:
path:
state: directory

make sure to change the <custom path> with your own path

Step 2: Migrating the existing database data to a new custom location using Ansible

To ensure no data changes occur during the migration, we must stop the GitLab and PostgreSQL services.

Read More about: How To Insert Data Into a DynamoDB Table with Boto3

After halting the GitLab service, we now have to migrate our PostgreSQL server to the new custom location by running this code:

- name: stop Gitlab service
service:
name: gitlab-runsvdir
state: stopped

- name: Ensure database_path dir exists ansible.builtin.file: path: "{{ database_path }}" state: directory recurse: yes - name: Synchronize postresql to new path. ansible.posix.synchronize: src: /var/opt/gitlab/ #This is the default path for GitLab postgreSQL Database dest: "{{ database_path }}" delegate_to: "{{ inventory_hostname }}" - name: Start Gitlab service service: name: gitlab-runsvdir
state: started

Step 3: Changing GitLab configuration

Now is the time we make changes to our GitLab configuration file located at '/etc/gitlab/gitlab.rb' . For this we are going to  use  ansible template module to change the configuration file and reconfigure our GitLab service.

We already have some existing configurations in our GitLab, and now we will add path variables for the GitLab PostgreSQL database in our Configuration file.

Explore more about “How to Install and Upgrade the AWS CDK CLI

Step 4: add some other Configuration

Now we are going to add  some other configurations for setting up the database custom location. The jinja2 file should look like this and  have should contain other existing configuration.

Now use the following code to perform this task on the Ansible playbook task:

external_url "{{ gitlab_external_url }}"

######################################
##### other configurations ##########
######################################

postgresql['home'] = "{{ database_path }}/postgresql" postgresql['data_dir'] = "{{ database_path }}/postgresql/data"
postgresql['unix_socket_directories'] = "{{ database_path }}/postgresql"

You must include another existing configuration line because the template module replaces your configuration file with the jinja2 (.j2) file content, which may misconfigure your current setup.

The ansible task for changing the configuration file and setting. Use the below code to complete these steps:

- name: backup existing configuration file to the /tmp folder
copy:
src: "/etc/gitlab/gitlab.rb"
dest: "/tmp"
remote_src: yes

- name: Copy GitLab configuration file. template: src: "{{ gitlab_config_template }}" #location of the .j2 jinja2 file dest: /etc/gitlab/gitlab.rb owner: root group: root mode: 0600 backup: yes
notify: restart gitlab

The handler task. To handle the job, you need to run the code which is mentioned below:

---
- name: restart gitlab
command: gitlab-ctl reconfigure
register: gitlab_restart
failed_when: gitlab_restart_handler_failed_when | bool

After all the task is completed,  you should have your GitLab database should be migrated to a new destination, as shown in the image below.

Database connection info before migration


Database connection info after migration



Roll Back 

In case something goes wrong, you can roll back to your previous configuration by copying the last configuration file we copied to /tmp/gitlab.rb . Or the backup file created while Ansible ran the template module. Then reconfiguring the GitLab.
This can also be done using an ansible task that looks like this.

In case of any unexpected error, you may need to SSH into your GitLab server and make the configuration changes manually.

- name: backup existing configuration file to the /tmp folder
copy:
src: "/tmp/gitlab.rb"
dest: "/etc/gitlab/"
remote_src: yes
notify: restart gitlab

The handler task:

---
- name: restart gitlab
command: gitlab-ctl reconfigure
register: gitlab_restart
failed_when: gitlab_restart_handler_failed_when | bool

Frequently Asked Questions

How can we check the remaining storage? Can it be seen through GitLab’s dashboard? How to check from cli?
To check the remaining space in your GitLab Linux server, you can use the df -h command to display the disk space usage in a more human-readable format, with sizes in gigabytes, megabytes, etc.
Unfortunately, for a self-managed GitLab server, we can’t use the dashboard to view the remaining storage.
Will the process be the same if I need to change any other configuration?

Answer:

The simple answer is Yes.

To make any configuration changes to your self-managed GitLab server, you should change the configuration file located in the '/etc/gitlab/gitlab.rb' and then run the command 'sudo gitlab-ctl reconfigure' to apply the changes.
For example, you can change the repository storage location, add LDAP authentication, etc.
Can and How I view the other configurations from the GitLab’s Dashboard?

Answer:

To see the overview of your GitLab server, you must first log in using a user with the administrator right, then click on the top left sidebar and choose admin. Here you can see the overview of all the configurations of your GitLab, like, Statistics, Features enabled, component versions, etc.

Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible
Saugat Tiwari

Latest Blogs

New AWS Announcement for October 2023

New AWS Announcement for October 2023


New AWS Announcement for October 2023

Adex International

Nov 08, 2023

Sustainability in the AWS Well-Architected Framework: A Comprehensive Guide

Sustainability in the AWS Well-Architected Framework: A Comprehensive Guide


Sustainability in the AWS Well-Architected Framework: A Comprehensive Guide

Adex International

Oct 19, 2023

AWS New Announcement Sept 2023

AWS New Announcement Sept 2023


AWS New Announcement Sept 2023

Adex International

Oct 17, 2023

Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible

Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible


Migrate Gitlab PostgreSQL Database to Custom Location Using Ansible

Saugat Tiwari

Oct 11, 2023

Mastering DevOps: Your Ultimate Guide to DevOps Managed Services

Mastering DevOps: Your Ultimate Guide to DevOps Managed Services


Mastering DevOps: Your Ultimate Guide to DevOps Managed Services

Biswash Giri

Oct 11, 2023

Discover the Benefits of Security as a Service (SECaaS) for your Business

Discover the Benefits of Security as a Service (SECaaS) for your Business


Discover the Benefits of Security as a Service (SECaaS) for your Business

Saugat Tiwari

Oct 11, 2023

Port Forwarding Using AWS System Manager Session Manager

Port Forwarding Using AWS System Manager Session Manager


Port Forwarding Using AWS System Manager Session Manager

Saugat Tiwari

Oct 11, 2023

Maximizing Directory Services with LDAP: Creating OUs, Groups, and Users for Improved Authentication and Access Control

Maximizing Directory Services with LDAP: Creating OUs, Groups, and Users for Improved Authentication and Access Control


Maximizing Directory Services with LDAP: Creating OUs, Groups, and Users for Improved Authentication and Access Control

Biswash Giri

Oct 11, 2023

AWS Migration Tools: A Comprehensive Guide

AWS Migration Tools: A Comprehensive Guide

IntroductionAWS migration tools are a comprehensive set of services and utilities provided by Amazon...


AWS Migration Tools: A Comprehensive Guide

Binaya Puri

Oct 11, 2023

Difference Between AWS Cloudwatch and Cloudtrail

Difference Between AWS Cloudwatch and Cloudtrail

AWS CloudWatch and AWS CloudTrails are sometimes difficult to distinguish. This article seeks to d...


Difference Between AWS Cloudwatch and Cloudtrail

Sabin Joshi

Oct 11, 2023

New AWS Announcements for June 2023 - Adex

New AWS Announcements for June 2023 - Adex


New AWS Announcements for June 2023 - Adex

Ravi Gupta

Oct 11, 2023

Top 7 Applications Of Cloud Computing In Various Field

Top 7 Applications Of Cloud Computing In Various Field


Top 7 Applications Of Cloud Computing In Various Field

Susmita Karki Chhetri

Oct 11, 2023

Ingesting and Monitoring Custom Metrics in CloudWatch With AWS Lambda

Ingesting and Monitoring Custom Metrics in CloudWatch With AWS Lambda


Ingesting and Monitoring Custom Metrics in CloudWatch With AWS Lambda

Tej pandey

Oct 11, 2023

7 Types of Security in Cloud Computing?

7 Types of Security in Cloud Computing?


7 Types of Security in Cloud Computing?

Mukesh Awasthi

Oct 11, 2023

Cost-effective Use cases & Benefits of Amazon S3

Cost-effective Use cases & Benefits of Amazon S3


Cost-effective Use cases & Benefits of Amazon S3

Nischal Gautam

Oct 11, 2023

IT Outsourcing: Everything You Need To Know

IT Outsourcing: Everything You Need To Know

The world has changed, and as technology advances, so does the world of work. Gone are the day...


IT Outsourcing: Everything You Need To Know

Roshan Raman Giri

Oct 11, 2023

Getting Started with Amazon Redshift in 6 Simple Steps

Getting Started with Amazon Redshift in 6 Simple Steps


Getting Started with Amazon Redshift in 6 Simple Steps

Tej pandey

Oct 11, 2023

How to Host Static Websites on AWS S3?

How to Host Static Websites on AWS S3?

How to Host Static Websites on AWS S3? Hosting a Static Website on AWS S3 has a lot of benefits....


How to Host Static Websites on AWS S3?

Ravi Gupta

Oct 11, 2023

The Importance of Managed Cloud Security for Businesses

The Importance of Managed Cloud Security for Businesses


The Importance of Managed Cloud Security for Businesses

Roshan Raman Giri

Oct 11, 2023

How To Use Amazon S3 For Personal Backup?

How To Use Amazon S3 For Personal Backup?


How To Use Amazon S3 For Personal Backup?

Tej pandey

Oct 11, 2023

Major AWS Updates &Announcements of 2023 - March

Major AWS Updates &Announcements of 2023 - March


Major AWS Updates &Announcements of 2023 - March

Roshan Raman Giri

Oct 11, 2023

How To Insert Data Into a DynamoDB Table with Boto3

How To Insert Data Into a DynamoDB Table with Boto3

DynamoDB is used for many use cases, including web and mobile applications, gaming, ad tech,...


How To Insert Data Into a DynamoDB Table with Boto3

Binaya Puri

Oct 11, 2023

How to Install and Upgrade the AWS CDK CLI

How to Install and Upgrade the AWS CDK CLI


How to Install and Upgrade the AWS CDK CLI

Nischal Gautam

Oct 11, 2023

Ultimate Guide on Creating Terraform Modules

Ultimate Guide on Creating Terraform Modules


Ultimate Guide on Creating Terraform Modules

Tej pandey

Oct 11, 2023

What is serverless computing?

What is serverless computing?


What is serverless computing?

Tej pandey

Oct 11, 2023

AWS Well-Architected Framework Security Pillar

AWS Well-Architected Framework Security Pillar

The Amazon Well-Architected Framework is a set of recommendations and practice guidelines for develo...


AWS Well-Architected Framework Security Pillar

Binaya Puri

Oct 11, 2023

Amazon FSx for Lustre, Windows, and NetApp ONTAP

Amazon FSx for Lustre, Windows, and NetApp ONTAP

Amazon FSx for Lustre, Windows, and NetApp ONTAPAmazon FSx is known for its fully managed, hig...


Amazon FSx for Lustre, Windows, and NetApp ONTAP

Ravi Gupta

Oct 11, 2023

How to Choose the Right Cloud Service Provider?

How to Choose the Right Cloud Service Provider?


How to Choose the Right Cloud Service Provider?

Tej pandey

Oct 11, 2023

25 New AWS Services Updates from AWS Re:Invent 2022

25 New AWS Services Updates from AWS Re:Invent 2022


25 New AWS Services Updates from AWS Re:Invent 2022

Susmita Karki Chhetri

Oct 11, 2023

AWS Managed Hosting Services And Dedicated Hosting Benefits

AWS Managed Hosting Services And Dedicated Hosting Benefits


AWS Managed Hosting Services And Dedicated Hosting Benefits

Tej pandey

Oct 11, 2023

What is Serverless Security? Risk & Best Practices

What is Serverless Security? Risk & Best Practices

Serverless computing  is a rising topic right now in the cloud tech industry. As per a Datad...


What is Serverless Security? Risk & Best Practices

Anup Giri

Oct 11, 2023

Difference Between Cloud Computing and Cybersecurity

Difference Between Cloud Computing and Cybersecurity


Difference Between Cloud Computing and Cybersecurity

Mukesh Awasthi

Oct 11, 2023

DevOps for Developers: How It Helps Streamline the Development Process

DevOps for Developers: How It Helps Streamline the Development Process

As per a survey done by Puppet, firms with DevOps practice have increased recovery speeds by 24 ti...


DevOps for Developers: How It Helps Streamline the Development Process

Roshan Raman Giri

Oct 11, 2023

New AWS Announcements for August 2023

New AWS Announcements for August 2023


New AWS Announcements for August 2023

Rohan Jha

Oct 11, 2023

The FinOps Chronicles

The FinOps Chronicles


The FinOps Chronicles

Anup Giri

Oct 11, 2023

AWS Auto scale Instance-Based on RabbitMQ Custom Metrics

AWS Auto scale Instance-Based on RabbitMQ Custom Metrics


AWS Auto scale Instance-Based on RabbitMQ Custom Metrics

Anup Giri

Oct 11, 2023

Overcome Merge Hell with Trunk based development and Continuous Integration

Overcome Merge Hell with Trunk based development and Continuous Integration


Overcome Merge Hell with Trunk based development and Continuous Integration

Rohan Jha

Oct 11, 2023

What's the difference between CapEX Vs OpEX in Cloud Computing?

What's the difference between CapEX Vs OpEX in Cloud Computing?


What's the difference between CapEX Vs OpEX in Cloud Computing?

Tej pandey

Oct 11, 2023

How Does Your Organization Keep Cloud Costs Under Control?

How Does Your Organization Keep Cloud Costs Under Control?


How Does Your Organization Keep Cloud Costs Under Control?

Susmita Karki Chhetri

Oct 11, 2023

Microsoft Azure vs AWS vs Google Cloud Comparison

Microsoft Azure vs AWS vs Google Cloud Comparison


Microsoft Azure vs AWS vs Google Cloud Comparison

Mukesh Awasthi

Oct 11, 2023

What are the Benefits of Amazon S3 Glacier?

What are the Benefits of Amazon S3 Glacier?


What are the Benefits of Amazon S3 Glacier?

Anup Giri

Oct 11, 2023

Leverage Azure Migrate to Discover and Assess Your AWS Instances for Smooth Migration to Azure

Leverage Azure Migrate to Discover and Assess Your AWS Instances for Smooth Migration to Azure


Leverage Azure Migrate to Discover and Assess Your AWS Instances for Smooth Migration to Azure

Rohan Jha

Oct 11, 2023