How to reindex using Reindex API on Elastic Search
This provides instructions on how to complete a reindex within Elastic Search using the reindexing API.
This will be made as generic as possible with places to enter the values needed.
A change request should be raised for production, which will be scheduled for out of hours. As part of the change get confirmation of case_type_id and the name of the index. Example implementation plan
Example index name: nfd_cases-000001
Example case_type_id: NFD
Example index clone1 name:cloned-nfd_cases-000001
Example index clone2 name: cloned-nfd_cases-000002
Some changes will include both non-production and production environments.
1) Request JIT access for the relevant environments bastion. (https://myaccess.microsoft.com/)
2) Make sure you are connected to the F5 VPN
3) Connect to the bastion server
az ssh config --ip \*.platform.hmcts.net --file ~/.ssh/config
Non-prod:
ssh bastion-nonprod.platform.hmcts.net
Prod:
ssh bastion-prod.platform.hmcts.net
4) Find the IP address for the service you wish to connect to and connect to Elastic Search node: For a list of IP addresses for each env, go to:
Connect to Elastic Search node from relevant bastion:
ssh -p22 elkadmin@<ip address> -i <logstash ssh key>
5) Stop the logstash instance that the team uses, this is to avoid any issues from scheduled jobs
Most teams do not have dedicated instances but some do.
You can scale the instance down to 0 to stop it running in this repo
Running this command on the relevant AKS cluster will let you see all logstash instances which can be scaled up and down in the cnp-flux-config repo.
kubectl get pods -A |grep logstash
6) Check if app team need to shutter
Also ensure that app team have removed themselves from case disposer flux config.
7) List the index you are re indexing prior to moving forward
Please make sure to remove the 000001 from the end of the index name when running the below command, this ensure all clones will also be listed.
curl --silent localhost:9200/_cat/indices | awk '{print $3, $7}' | grep <Your index name>
8) Check overall DB count prior to deletion
Connect to ccd-data-store DB from Bastion server and run this query:
SELECT count(*) FROM case_data WHERE case_type_id = '<your case type id>';
9) Get a count of the Dead Letter Queue on elastic search for the case_type_id
curl --header "Content-Type: application/json" 'http://localhost:9200/.logstash_dead_letter/_count' -d '{"query": {"match_phrase": {"failed_case": "\"case_type_id\":\"<your case type id>\""}}}'
10) Block writing to main index ahead of cloning
curl -X PUT "localhost:9200/<Your index name>/_settings?pretty" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write": true}}'
11) Clone the index for backup incase of rollback
Make sure this clones name ends in a 01 like the main index.
curl -X POST "localhost:9200/<Your index name>/_clone/cloned-<Your index name>?pretty"
After this command, please check both indexes exist by running the command from step 7 again.
12) Clone index a second time, this will be used as a source for the reindex.
For clearer referencing end this clone name with a 02 instead of a 01.
curl -X POST "localhost:9200/<Your index name>/_clone/cloned-<Your index name>?pretty"
After this command, please check both indexes exist by running the command from step 7 again. The original index and the both clones should have the same counts.
Note: For this cloned index please ensure the name has a 2 at the end instead of a 1.
13) Set the second cloned index to read only mode
curl -X PUT "localhost:9200/<Your second clones name>/_settings?pretty" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.read_only": true}}'
14) Delete the original index
curl -X DELETE localhost:9200/<Your index name>;
15) Wait for the new definition to be imported by the service team.
Confirm its been imported by running the command from step 7 again, it will show as having no cases in it but it should exist.
16) Reindex using the second clone as the source for the reindex
This will take a while depending on the size of the index, so please be patient. The curl we run can time out but rest assured it will still run in the background.
curl -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'{"source": {"index": "<Your second clones name>","size":100},"dest": {"index": "<your index name>"}}'
17) Check the reindex was successful
Even after Step 17 is complete this can take some time to complete, so please be patient.
curl --silent localhost:9200/_cat/indices | awk '{print $3, $7}' | grep <Your index name>
If the index count isnt up to what is expected but no error was shown in step 17, then please run the below command and check again.
curl -X GET "localhost:9200/_refresh?pretty"
18) Ensure you scale the logstash instance back to 1.
Post Implementation steps
A day or two after the reindex has been completed and verified by the service team, please follow these steps to clean up.
1) Check for all indexes
curl --silent localhost:9200/_cat/indices | awk '{print $3, $7}' | grep <Your index name>
2) Remove the read only setting on the second clone
curl -X PUT "localhost:9200/<Your second clones name>/_settings?pretty" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.read_only": false}}'
3) Delete the cloned indexes
Run the below command for each of the clones.
Please run these 1 at a time to ensure no mistakes are made.
curl -X DELETE localhost:9200/<clone index name>;
Rollback steps
If at any point you need to roll back the changes the following steps should be followed:
1) Delete the new index that was created as part of the re index
curl -X DELETE localhost:9200/<Your index name>;
2) Clone the backup index back to the original name
curl -X POST "localhost:9200/cloned-<Your index name>/_clone/<Your index name>?pretty"
3) Verify that the index has been cloned back correctly
curl --silent localhost:9200/_cat/indices | awk '{print $3, $7}' | grep <your index name>
You should see the clone alongside the original index with the original count.
4) Set the index to write mode
curl -X PUT "localhost:9200/<Your index name>/_settings?pretty" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write": false}}'
If you run this command afterwards it should show the blocks write is set to false.
curl -X GET http://localhost:9200/<Your index name>/_settings?pretty
5) Set the index alias of our restore index
Check with service team what the alias should be if unsure.
curl -X POST "localhost:9200/_aliases?pretty" -H 'Content-Type: application/json' -d' { "actions": [ { "add": { "index": "<your index name>", "alias": "<your index alias>" } } ] } '
Run this command to verify the alias has been set correctly
curl -X GET "localhost:9200/_alias/<your index alias>?pretty"
6) Service teams to verify if the rollback has been successful
7) If rollback is successful clean up the cloned index