Cayenne/SOLR instances, and the sequence for updating the pools

Public pool

Disable haproxy overflow that api-plus also uses echo “disable server rest-api-plus/svc11f” | nc cr5c 8298

Deploy to svc11b/mds7 and svc11g/mds3 Disable svc11b and svc11g

echo "disable server rest-api-public/svc11b" | nc cr5c 8298
echo "disable server rest-api-public/svc11g" | nc cr5c 8298

Run the reconfiguration and replication steps on mds7 and mds3 thru step6

When above complete, deploy to svc11f/mds9, svc11h/mds11 (aka svcplus2/mdsplus2) Verify step 7

Enable svc11b and svc11g

echo "enable server rest-api-public/svc11b" | nc cr5c 8298
echo "enable server rest-api-public/svc11g" | nc cr5c 8298

Verify step 8

Disable svc11f and svc11h

echo "disable server rest-api-public/svc11f" | nc cr5c 8298
echo "disable server rest-api-public/svc11h" | nc cr5c 8298

Run the reconfiguration and replication steps on mds9 and mds11 thru step6

Verify step 7

Enable svc11f and svc11h

echo "enable server rest-api-public/svc11f" | nc cr5c 8298
echo "enable server rest-api-public/svc11h" | nc cr5c 8298

Verify step 8 Enable haproxy overflow that api-plus also uses

echo "enable server rest-api-plus/svc11f" | nc cr5c 8298

Polite pool

Disable haproxy overflow that api-plus also uses

echo "disable server rest-api-plus/svc11c" | nc cr5c 8298

Deploy to svc11d/mds4 and svc11e/mds8 at same time Disable svc11d and svc11e

echo "disable server rest-api-polite/svc11d" | nc cr5c 8298
echo "disable server rest-api-polite/svc11e" | nc cr5c 8298

Run the reconfiguration and replication steps on mds9 and mds5 thru step6

When above complete, deploy to svc11c/mds5 Verify step 7

Enable svc11d and svc11e

echo "enable server rest-api-polite/svc11d" | nc cr5c 8298
echo "enable server rest-api-polite/svc11e" | nc cr5c 8298

Verify step 8

Disable svc11c

echo "disable server rest-api-public/svc11c" | nc cr5c 8298

Run the reconfiguration and replication steps on mds5 thru step7

When above completes, verify step 7

Enable svc11c

echo "enable server rest-api-polite/svc11c" | nc cr5c 8298

Verify step 8

Enable haproxy overflow that api-plus also uses

echo "enable server rest-api-plus/svc11c" | nc cr5c 8298

Plus pool

Deploy to svcplus1/mdsplus1 Disable svcplus1

echo "disable server rest-api-plus/svcplus1" | nc cr5c 8298

Run the reconfiguration and replication steps on mdsplus1 thru step6

When above complete, deploy to svc11a/mds10 Verify step 7

Enable svcplus1

echo "enable server rest-api-plus/svcplus1" | nc cr5c 8298

Verify step 8

Disable svc11a

echo "disable server rest-api-plus/svc11a" | nc cr5c 8298

Run the reconfiguration and replication steps on mds10 thru step 6

When above completes, verify step 7

Enable svc11a

echo "enable server rest-api-plus/svc11a" | nc cr5c 8298

Verify step 8

CRMDS

Deploy to mds1 Disable mds1

echo "disable server rest-api-crmds/mds1" | nc cr5d 3315

Run the reconfiguration and replication steps on mds1

Enable mds1

echo "enable server rest-api-crmds/mds1" | nc cr5d 3315

Deploy to mds2 Disable mds2

echo "disable server rest-api-crmds/mds2" | nc cr5d 3315

Run the reconfiguration and replication steps on mds2

Enable mds2

echo "enable server rest-api-crmds/mds2" | nc cr5d 3315

Deploy to mds6 Disable mds6

echo "disable server rest-api-crmds/mds6" | nc cr5d 3315

Run the reconfiguration and replication steps on mds6

Enable mds6

echo "enable server rest-api-crmds/mds6" | nc cr5d 3315

Snapshots

Deploy to mds12 There is no HAProxy in front of mds12, simply run the reconfiguration and replication steps on mds12

General Steps to reconfigure/replicate any Cayenne/SOLR service instance

1. Stop the back-end SOLR service

2. Delete the monitor enable indicator file. From service root folder:

rm /home/crossref/ssddata/cayenne-solr/enable

3. Stop the service instance

/home/crossref/ssddata/cayenne-solr/bin/solr stop -p 8984

4. Delete the contents of the data folder

From app root folder:

rm -rf /home/crossref/ssddata/cayenne-solr/server/solr/crmds1/data/*

5. Reconfigure the Solr slave to point at master ssmds0a

From service root folder, edit: /home/crossref/solrmon.sh and change ssmds0b to ssmds0a in this line:

APP="$1bin/solr start -p 8983 -Denable.slave=true -Dmaster.url=http://ssmds0b:8984/solr/crmds1"

6. Start the back-end SOLR service

Restore the monitor script flag. From service root folder, execute:

touch /home/crossref/ssddata/cayenne-solr/enable

Ensure there’s a PID file there

The monitor will trigger (within a minute) and restart the solr instance. Confirm it has started: go to Solr admin, confirm it is replicating from ssmds0a.

(http://mds4:8983/solr/#/crmds1/replication) (Older UI: http://mds4:8983/solr/old.html#/crmds1/replication)

7. Wait for replication to complete:

8. Verify api/solr pair is functioning, e.g. from a browser, enter

http://<server>.crossref.org:3000/works/10.1002/14651858.cd000179

Verify that service is taking public traffic by checking HAProxy stats page