timbrado-${__time(yyyyMMddHHmmss)}

• Add disk to VM via Enterprise Service Catalog
• Rename new volume group to esosvg, or add new disk to existing esosvg volume group
• LV esosvg/home mounted at /home
• LV esosvg/mariadb mounted at /var/lib/mysql
• LV esosvg/opt mounted at /opt (only if rootvg /opt isn't sufficient)
• LV esosvg/arc mounted at /opt/arc
• Alternative: LV esosvg/home mounted at /home with all other locations sym-linked under /home

• Update system with yum update
• Lay out storage per conventions
• Install/enable EPEL
• Install newterm
• Create users
• Change /etc/sudoers to switch to NOPASSWD: wheel group entry
• Install basic packages
  • yum install wget htop vim gvim git
  • yum groupinstall “X Window System”
  • yum install dejavu-sans-mono-fonts.noarch
  • yum install xorg-x11-fonts-misc
  • Optional: yum groupinstall “Fonts” for all fonts
  • yum install tightvnc-server
  • yum install tightvnc
  • yum install xorg-x11-server-devel
• Install …
  • yum install java-1.8.0-openjdk-devel
  • yum install mariadb-server
  • yum install mariadb-devel
  • yum install python-devel
  • yum install MySQL-python
  • yum install httpd
  • yum install php
  • yum install php-mysql
  • yum install php-ldap
  • yum install php-gd
  • yum install gd-devel
• Install Maven
• Install InfluxDB (requires EPEL)
• Install Grafana
• Install Jenkins

• Ensure SSH key and known_hosts are populated for self-connection
• Add plugins
• Add credentials, use “Git” provider (not “GitHub”)
• Configure global tools section (maven, etc…)
• Add environment variables in Global properties
  • M3_HOME = /usr/maven
  • PATH = /var/lib/jenkins/bin:/usr/maven/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin
  • Optional: JAVA_HOME = /usr/java
• Addition packages:
  • yum install python-pip
  • pip install –upgrade pip
  • pip install jenkinsapi
  • Correct permissions, if necessary, at /usr/lib/python2.7/site-packages for:
    • jenkinsapi
    • six
    • pytz

Previously-mentioned package install via yum installs the software
Configuration is in /etc/my.cnf and /etc/my.cnf.d:

• Setting the datadir attribute to where database storage is to live is advisable, to separate it from /var/lib/mysql, which is where the socket is created
• Trying to symbolic-link /var/lib/mysql to some other location causes permissions quirks
• For some reason, /var/run/mariadb is sometimes not created - to remediate:

  mkdir /var/run/mariadb
  chown mysql:mysql /var/run/mariadb
  chmod 775 /var/run/mariadb
  chmod g+s /var/run/mariadb
  cd /var/run/mariadb ; chcon -t var_run_t .

• Make sure /var/lib/mysql has mysqld_db_t type, in the directory: chcon -t mysqld_db_t .

Previously-mentioned package install via yum installs the software Edit /etc/httpd/conf/httpd.conf:

• Set DocumentRoot to suitable location, ensure location exists and resides in suitable storage

Open firewall for Apache:

firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --reload

Perform the typical systemctl enable httpd and service httpd start for startup

Resource Clustering InfluxDB ...

Create the file /etc/yum.repos.d/influxdb.repo with this content:

[influxdb]
name = InfluxDB Repository - RHEL $releasever
baseurl = https://repos.influxdata.com/rhel/$releasever/$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key

… then, do a yum update and yum install influxdb
Open firewall for InfluxDB:

firewall-cmd --permanent --zone=public --add-port=8086/tcp
firewall-cmd --permanent --zone=public --add-port=8083/tcp
firewall-cmd --reload

Edit /etc/influxdb/influxdb.conf and:

• In the [http] section:
  • Set enabled = true
  • Change the bind = :8086 entry, if needed
  • Set unix-socket-enabled = true
  • Change bind-socket = value to /var/run/influxdb/influxdb.sock
• In the [udp] section:
  • Set enabled = true
  • Change the bind = :8089 entry, if needed

Ensure /var/run/influxdb and /var/lib/influxdb exist and are owned correctly

• For some reason, /var/run/influxdb is sometimes not created - to remediate:

  mkdir /var/run/influxdb
  chown influxdb:influxdb /var/run/influxdb
  chmod 775 /var/run/influxdb
  chmod g+s /var/run/influxdb
  cd /var/run/influxdb ; chcon -t var_run_t .

Remember to systemctl enable influxdb and service influxdb start
See: InfluxDB auth

Create the file /etc/yum.repos.d/grafana.repo with this content:

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

… then, do a yum update and yum install grafana
Open firewall for Grafana:

firewall-cmd --permanent --zone=public --add-port=3000/tcp
firewall-cmd --reload

Remember to systemctl enable grafana-server and service grafana-server start
Log in (default login is admin with password admin) and change password, add user(s)
Install the InfluxDB datasource plugin (find in plugins section, follow instructions)

Edit /etc/grafana/grafana.ini:

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

Edit /etc/grafana/ldap.toml:

• Under servers.attributes section, make sure username is set to sAMAccountName for Active Directory
• Under servers.group_mappings section, group_dn should be cn=bluejay_adm,cn=Users,dc=ms,dc=ds,dc=uhc,dc=com
• Editor role: cn=github_users,cn=Users,dc=ms,dc=ds,dc=uhc,dc=com
• Host: “ADC21101.ms.ds.uhc.com”
• ssl_skip_verify = true
• bind_dn: “oaptbot@ms.ds.uhc.com”
• bind_password: 'BlueJay1!'
• search_filter: “(sAMAccountName=%s)”
• search_base_dns: [“cn=Users,dc=ms,dc=ds,dc=uhc,dc=com”] * ==== Jenkins Installation ==== Pull the Jenkins repository file and import its GPG key: <code> wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins.io/redhat-stable/jenkins.repo rpm –import http://pkg.jenkins.io/redhat-stable/jenkins.io.key </code> … then, do a yum update and yum install jenkins
If the jenkins login isn't locked down off-server: * Enable login for Jenkins with usermod -s /bin/bash jenkins * Log in as jenkins and create SSH key with: ssh-keygen -t rsa Otherwise: * Create SSH key for jenkins from root: ssh-keygen -f id_rsa -C “jenkins@bluejay” Open firewall for Jenkins:<code> firewall-cmd –permanent –zone=public –add-port=8080/tcp firewall-cmd –reload </code> Remember to systemctl enable jenkins and service jenkins start
Log in (default login is admin with password admin) and change password, add user(s)
==== Maven Installation ==== Get latest maven archive, example:<code>wget http://www-eu.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz</code> Extract to suitable location and sym-link to it (/usr/maven or /opt/maven)
Set environment variables in /etc/profile.d/maven.sh:<code> export M3_HOME=/usr/maven export M2_HOME=${M3_HOME} export PATH=${M3_HOME}/bin:${PATH} </code> … perform a source /etc/profile.d/maven.sh to get these in current shell. ===== Bluejay LG Node ===== * Ensure central repository at /var/www/html/pub/eptrepo/jmeter is populated, and create SSL cert * Edit the create-rmi-keystore.sh script, changing the validity to 99999 and run to gen cert * Ensure cert file rmi_keystore.jks has same ownership/permissions/seattr as other files * On Jenkins slave, perform basic installation, update, and pkgs for: wget, htop, vim, git, and OpenJDK * Populate /etc/hosts * Create bluejay login<code>groupadd -g 39092 bluejay useradd -u 441849 -g 39092 -d /home/bluejay -s /bin/bash -c “Bluejay” -m bluejay passwd bluejay chage -I -1 -m 0 -M 99999 -E -1 bluejay </code> * Extended storage setup:<code>lvcreate -l 100%FREE -n bluejay rootvg mkfs -t ext4 /dev/rootvg/bluejay blkid /dev/rootvg/bluejay » /etc/fstab vi /etc/fstab ls -lZd /home/bluejay/. mkdir -p ~/tmp mv /home/bluejay/.??* ~/tmp/ chmod 755 /home/bluejay mount /home/bluejay chown bluejay:bluejay /home/bluejay/. chmod 775 /home/bluejay/. chcon -u unconfined_u -t user_home_dir_t /home/bluejay/. mv ~/tmp/.??* /home/bluejay/</code> * Populate bluejay login's ~/.ssh directory:<code>cd ~bluejay ; mkdir .ssh ; cd .ssh ssh-keygen -t rsa -f id_rsa -C bluejay@hostname cp -a id_rsa.pub authorized_keys chown -R bluejay:bluejay . chcon -R -u unconfined_u -t ssh_home_t . chmod 700 . ; chmod 600 *</code> * Copy public key to central staging location:<code>scp ./id_rsa.pub someone@somewhere:/home/someone/dot-ssh/bluejay_host.pub</code> * Create jenkins login<code>groupadd -g 822 jenkins useradd -u 990 -g 822 -d /var/lib/jenkins -s /bin/bash -c “Jenkins” jenkins usermod -a -G bluejay jenkins usermod -a -G jenkins bluejay chcon -u system_u -t var_lib_t /var/lib/jenkins/. chmod 750 /var/lib/jenkins/.</code> * Populate jenkins login's ~/.ssh directory:<code>cd ~jenkins ; mkdir .ssh ; cd .ssh ssh-keygen -t rsa -f id_rsa -C jenkins@hostname cp -a id_rsa.pub authorized_keys chown -R jenkins:jenkins . chcon -R -u unconfined_u -t ssh_home_t . chmod 700 . ; chmod 600 *</code> * Copy public key to central staging location, and make sure it gets populated into Bluejay web server's bluejay login ~/.ssh/authorized_keys file:<code>scp ./id_rsa.pub someone@somewhere:/home/someone/dot-ssh/jenkins_host.pub</code> * Create Jenkins agent workspace:<code>cd ~bluejay ; mkdir -p jenkins chown bluejay:jenkins ./jenkins/. chmod 770 ./jenkins/. chmod g+s ./jenkins/. chcon -u system_u -t var_lib_t ./jenkins/. </code> * Populate Jenkins controller's jenkins login SSH key into bluejay login's authorized_keys * Populate jenkins known_hosts file on Jenkins host
—— ====== Trivia ====== ===== Optum-specific Admin Concerns ===== ==== UID/GID ==== * msamud1: uid=7301 * jenkins: uid=990, gid=822 * bluejay: uid=441849, gid=39092 * mosquito: uid=263407, gid=28606 * canary: uid=783478, gid=31708 * apache: uid=48, gid=1739 * mantis: uid=17236, gid=? * wildfly: uid=282, gid=28202 * perfmgmt: gid=4035 ==== EPEL ==== <code> ## RHEL/CentOS 7 64-Bit ## # wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # rpm -ivh epel-release-latest-7.noarch.rpm </code> yum –enablerepo=epel info pkgname
yum –enablerepo=epel install pkgname ==== Re-sizing /opt ==== If /opt happens to be an XFS filesystem, resizing can be done “live”. Otherwise, in order to unmount /opt to manipulate it, all processes holding files open on the filesystem have to be shut down, including: <code> /etc/init.d/opsware-agent stop /etc/init.d/vasd stop service postfix stop service dbus stop </code> Edit /etc/nsswitch.conf to comment out lines with vas4 in them, then kill -HUP <pid-of-nss-process>
Perform the whole thing in reverse when /opt'' is resized