====== Optum-specific Conventions ======
<code>timbrado-${__time(yyyyMMddHHmmss)}</code>

==== Storage Conventions ====
  * Add disk to VM via Enterprise Service Catalog
  * Rename new volume group to esosvg, or add new disk to existing esosvg volume group
  * LV ''esosvg/home'' mounted at /home
  * LV ''esosvg/mariadb'' mounted at /var/lib/mysql
  * LV ''esosvg/opt'' mounted at /opt //(only if ''rootvg'' /opt isn't sufficient)//
  * LV ''esosvg/arc'' mounted at /opt/arc
  * **Alternative:**  LV ''esosvg/home'' mounted at /home with all other locations sym-linked under /home

===== Bluejay Controller =====
==== Platform Installation Overview ====
  * Update system with ''yum update''
  * Lay out storage per conventions
  * Install/enable EPEL: ''dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm''
  * Install ''newterm''
  * Create users
  * Change ''/etc/sudoers'' to switch to ''NOPASSWD:'' wheel group entry
  * Install basic packages
    * yum install wget htop vim gvim git
    * yum groupinstall “X Window System”
    * yum install dejavu-sans-mono-fonts.noarch 
    * yum install xorg-x11-fonts-misc
    * //Optional: ''yum groupinstall “Fonts”'' for all fonts //
    * yum install tightvnc-server
    * yum install tightvnc
    * yum install xorg-x11-server-devel
  * Install ...
    * yum install java-1.8.0-openjdk-devel
    * yum install mariadb-server
    * yum install mariadb-devel
    * yum install python-devel
    * yum install MySQL-python
    * yum install httpd
    * yum install php
    * yum install php-mysql
    * yum install php-ldap
    * yum install php-gd
    * yum install gd-devel
  * Install Maven
  * Install InfluxDB (requires EPEL)
  * Install Grafana
  * Install Jenkins
==== Configure Jenkins for Bluejay ====
  * Ensure SSH key and known_hosts are populated for self-connection
  * Add plugins
  * Add credentials, use "Git" provider (not "GitHub")
  * Configure global tools section (maven, etc...)
  * Add environment variables in Global properties
    * ''M3_HOME'' = ''/usr/maven''
    * ''PATH'' = ''/var/lib/jenkins/bin:/usr/maven/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin''
    * **Optional:** ''JAVA_HOME'' = ''/usr/java''
  * Addition packages:
    * yum install python-pip
    * pip install --upgrade pip
    * pip install jenkinsapi
    * Correct permissions, if necessary, at ''/usr/lib/python2.7/site-packages'' for:
      * ''jenkinsapi''
      * ''six''
      * ''pytz''

===== Component Installation =====

==== MariaDB Installation ====
Previously-mentioned package install via ''yum'' installs the software \\
Configuration is in ''/etc/my.cnf'' and ''/etc/my.cnf.d'':
  * Setting the ''datadir'' attribute to where database storage is to live is advisable, to separate it from ''/var/lib/mysql'', which is where the socket is created
  * Trying to symbolic-link ''/var/lib/mysql'' to some other location causes permissions quirks
  * For some reason, ''/var/run/mariadb'' is sometimes not created - to remediate:<code>mkdir /var/run/mariadb
chown mysql:mysql /var/run/mariadb
chmod 775 /var/run/mariadb
chmod g+s /var/run/mariadb
cd /var/run/mariadb ; chcon -t var_run_t .
</code>
  * Make sure ''/var/lib/mysql'' has ''mysqld_db_t'' type, in the directory:  ''chcon -t mysqld_db_t .''

------
==== Apache Installation ====
Previously-mentioned package install via ''yum'' installs the software
Edit ''/etc/httpd/conf/httpd.conf'':
  * Set ''DocumentRoot'' to suitable location, ensure location exists and resides in suitable storage
Open firewall for Apache:<code>
firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --reload
</code>
Perform the typical ''systemctl enable httpd'' and ''service httpd start'' for startup

------
==== InfluxDB Installation ====
Resource [[https://influxdbcom.readthedocs.io/en/latest/content/docs/v0.9/guides/clustering|Clustering InfluxDB ...]] \\

Create the file ''/etc/yum.repos.d/influxdb.repo'' with this content:
<code>
[influxdb]
name = InfluxDB Repository - RHEL $releasever
baseurl = https://repos.influxdata.com/rhel/$releasever/$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
</code>
... then, do a ''yum update'' and ''yum install influxdb'' \\
Open firewall for InfluxDB:<code>
firewall-cmd --permanent --zone=public --add-port=8086/tcp
firewall-cmd --permanent --zone=public --add-port=8083/tcp
firewall-cmd --reload
</code>
Edit ''/etc/influxdb/influxdb.conf'' and:
  * In the ''[http]'' section:
    * Set ''enabled = true''
    * Change the ''bind = :8086'' entry, if needed
    * Set ''unix-socket-enabled = true''
    * Change ''bind-socket = '' value to ''/var/run/influxdb/influxdb.sock''
  * In the ''[udp]'' section:
    * Set ''enabled = true''
    * Change the ''bind = :8089'' entry, if needed
Ensure ''/var/run/influxdb'' and ''/var/lib/influxdb'' exist and are owned correctly \\
  * For some reason, ''/var/run/influxdb'' is sometimes not created - to remediate:<code>mkdir /var/run/influxdb
chown influxdb:influxdb /var/run/influxdb
chmod 775 /var/run/influxdb
chmod g+s /var/run/influxdb
cd /var/run/influxdb ; chcon -t var_run_t .
</code>
Remember to ''systemctl enable influxdb'' and ''service influxdb start'' \\
=== InfluxDB Auth ===
See: [[https://docs.influxdata.com/influxdb/v1.5/query_language/authentication_and_authorization|InfluxDB auth]] \\
<code>create user admin with password 'somepassword' with all privileges;
create user jmeter with password 'somepassword';
create database jmeter;
grant [READ,WRITE,ALL] on "jmeter" TO "jmeter";
revoke [READ,WRITE,ALL] on "jmeter" from "jmeter";</code>

------
==== Grafana Installation ====
Create the file ''/etc/yum.repos.d/grafana.repo'' with this content:
<code>
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
</code>
... then, do a ''yum update'' and ''yum install grafana'' \\
Open firewall for Grafana:<code>
firewall-cmd --permanent --zone=public --add-port=3000/tcp
firewall-cmd --reload
</code>
Remember to ''systemctl enable grafana-server'' and ''service grafana-server start'' \\
Log in (default login is ''admin'' with password ''admin'') and change password, add user(s) \\
Install the InfluxDB datasource plugin (find in plugins section, follow instructions)
==== LDAP for Grafana ====
Edit ''/etc/grafana/grafana.ini'':<code>[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true
</code>
Edit ''/etc/grafana/ldap.toml'':
  * Under ''servers.attributes'' section, make sure username is set to ''sAMAccountName'' for Active Directory
  * Under ''servers.group_mappings'' section, group_dn should be ''cn=bluejay_adm,cn=Users,dc=ms,dc=ds,dc=uhc,dc=com''
  * Editor role: ''cn=github_users,cn=Users,dc=ms,dc=ds,dc=uhc,dc=com''
  * Host: ''"ADC21101.ms.ds.uhc.com"''
  * ssl_skip_verify = true
  * bind_dn: ''"oaptbot@ms.ds.uhc.com"''
  * bind_password: '''BlueJay1!'''
  * search_filter: ''"(sAMAccountName=%s)"''
  * search_base_dns: ''["cn=Users,dc=ms,dc=ds,dc=uhc,dc=com"]''
  * 

------
==== Jenkins & Maven Installation ====
See [[tech:devops:jenkins:start|Jenkins Topics]] \\

------
===== Bluejay LG Node =====
  * Ensure central repository at ''/var/www/html/pub/eptrepo/jmeter'' is populated, and create SSL cert
    * Edit the ''create-rmi-keystore.sh'' script, changing the validity to ''99999'' and run to gen cert
    * Ensure cert file ''rmi_keystore.jks'' has same ownership/permissions/seattr as other files
  * On Jenkins slave, perform basic installation, update, and pkgs for: wget, htop, vim, git, and OpenJDK
  * Populate /etc/hosts
  * Create ''bluejay'' login<code>groupadd -g 39092 bluejay
useradd -u 441849 -g 39092 -d /home/bluejay -s /bin/bash -c "Bluejay" -m bluejay
passwd bluejay
chage -I -1 -m 0 -M 99999 -E -1 bluejay
</code>
  * Extended storage setup:<code>lvcreate -l 100%FREE -n bluejay rootvg
mkfs -t ext4 /dev/rootvg/bluejay
blkid /dev/rootvg/bluejay >> /etc/fstab
vi /etc/fstab
ls -lZd /home/bluejay/.
mkdir -p ~/tmp
mv /home/bluejay/.??* ~/tmp/
chmod 755 /home/bluejay
mount /home/bluejay
chown bluejay:bluejay /home/bluejay/.
chmod 775 /home/bluejay/.
chcon -u unconfined_u -t user_home_dir_t /home/bluejay/.
mv ~/tmp/.??* /home/bluejay/</code>
  * Populate ''bluejay'' login's ''~/.ssh'' directory:<code>cd ~bluejay ; mkdir .ssh ; cd .ssh
ssh-keygen -t rsa -f id_rsa -C bluejay@hostname
cp -a id_rsa.pub authorized_keys
chown -R bluejay:bluejay .
chcon -R -u unconfined_u -t ssh_home_t .
chmod 700 . ; chmod 600 *</code>
  * Copy public key to central staging location:<code>scp ./id_rsa.pub someone@somewhere:/home/someone/dot-ssh/bluejay_host.pub</code>
  * Create ''jenkins'' login<code>groupadd -g 822 jenkins
useradd -u 990 -g 822 -d /var/lib/jenkins -s /bin/bash -c "Jenkins" jenkins
usermod -a -G bluejay jenkins
usermod -a -G jenkins bluejay
chcon -u system_u -t var_lib_t /var/lib/jenkins/.
chmod 750 /var/lib/jenkins/.</code>
  * Populate ''jenkins'' login's ''~/.ssh'' directory:<code>cd ~jenkins ; mkdir .ssh ; cd .ssh
ssh-keygen -t rsa -f id_rsa -C jenkins@hostname
cp -a id_rsa.pub authorized_keys
chown -R jenkins:jenkins .
chcon -R -u unconfined_u -t ssh_home_t .
chmod 700 . ; chmod 600 *</code>
  * Copy public key to central staging location, and make sure it gets populated into Bluejay web server's ''bluejay'' login ''~/.ssh/authorized_keys'' file:<code>scp ./id_rsa.pub someone@somewhere:/home/someone/dot-ssh/jenkins_host.pub</code>
  * Create Jenkins agent workspace:<code>cd ~bluejay ; mkdir -p jenkins
chown bluejay:jenkins ./jenkins/.
chmod 770 ./jenkins/.
chmod g+s ./jenkins/.
chcon -u system_u -t var_lib_t ./jenkins/.
</code>
  * Populate Jenkins controller's ''jenkins'' login SSH key into ''bluejay'' login's ''authorized_keys''
  * Populate jenkins ''known_hosts'' file on Jenkins host

 \\

------
====== Trivia ======
===== Optum-specific Admin Concerns =====
==== UID/GID ====
  * msamud1: uid=''7301''
  * jenkins: uid=''990'', gid=''822''
  * bluejay: uid=''441849'', gid=''39092''
  * mosquito: uid=''263407'', gid=''28606''
  * canary: uid=''783478'', gid=''31708''
  * apache: uid=48, gid=1739
  * mantis: uid=''17236'', gid=''?''
  * wildfly: uid=''185'', gid=''185'' <del>wildfly: uid=''282'', gid=''28202''</del>
  * thorium: gid=''34625''
  * perfmgmt: gid=''4035''

==== EPEL ====
<code>
## RHEL/CentOS 7 64-Bit ##
# wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# rpm -ivh epel-release-latest-7.noarch.rpm
</code>

''yum --enablerepo=epel info pkgname'' \\
''yum --enablerepo=epel install pkgname''

==== Re-sizing /opt ====
If ''/opt'' happens to be an XFS filesystem, resizing can be done "live".  Otherwise, in order to unmount ''/opt'' to manipulate it, all processes holding files open on the filesystem have to be shut down, including:
<code>
/etc/init.d/opsware-agent stop
/etc/init.d/vasd stop
service postfix stop
service dbus stop
</code>
Edit ''/etc/nsswitch.conf'' to comment out lines with ''vas4'' in them, then ''kill -HUP //<pid-of-nss-process>//'' \\

Perform the whole thing in reverse when ''/opt'' is resized \\

===== Some Problems/Resolutions =====
==== FIPS-compliant Venafi-generated SSL certs ====
This problem: [[http://openssl.6102.n7.nabble.com/DEK-info-FIPS-openssl-td23289.html]] \\
... is resolved by downloading cert from Venafi in ''PKCS#8'' format \\