AnsweredAssumed Answered

Errors in cluster-audit lab 2.3 of Adm 200 course

Question asked by bbouman on Jul 12, 2018
Latest reply on Jul 18, 2018 by bbouman

Below are details pertaining to lab 2.3 Audit the Cluster.     I have highlighted a few of the errors.    Specifically there are missing services and failed connections.  How can I move forward with the lab?

Required RPMs:

package syslinux is not installed

package hdparm is not installed

package sdparm is not installed

           JDK not installed

DNS lookup

clush: 192.168.168.128: exited with exit code 1

....

 

192.168.168.129: Host MapR-VCLEhost not found: 2(SERVFAIL)

 

            .....

            ################################################################################

Reverse DNS lookup

clush: 192.168.168.129: exited with exit code 1

 

......

192.168.168.129: ;; connection timed out; no servers could be reached

 

            ......

 

 

FULL DETAILS and actions below;

 

IP Address 1 = 192.168.168.130

IP Address 2 = 192.168.168.128

IP Address 3 = 192.168.168.129

 

 

Your environment is ready to use.

 

For command line access, open the Terminal on the desktop and ssh to node1, node2, or node3.

 

Log in using the credentials below:

 

Username: root

Password: mapr

 

Press enter to close this window.

 

As root.

  1. Verify that clustershell is installed:

[root@node1 ~]# rpm -qa | grep clustershell

clustershell-1.7.3-1.fc25.noarch

 

  1. Check the connections (ssh) to the other nodes and issue date command to ensure password not needed.
  2. Create /etc/clustershell/groups and add nodes

all:  192.168.168.130 192.168.168.128 192.168.168.129

 

  1. [root@node1 clustershell]# clush -a date

192.168.168.129: Warning: Permanently added '192.168.168.129' (ECDSA) to the list of known hosts.

192.168.168.128: Warning: Permanently added '192.168.168.128' (ECDSA) to the list of known hosts.

192.168.168.130: Warning: Permanently added '192.168.168.130' (ECDSA) to the list of known hosts.

192.168.168.129: Thu Jul 12 10:07:50 PDT 2018

192.168.168.128: Thu Jul 12 10:07:50 PDT 2018

192.168.168.130: Thu Jul 12 10:07:50 PDT 2018

 

  1. clush -a --copy /root/labfiles
  2. [root@node1 ~]# clush -a ls -la /root/ | grep labfiles

192.168.168.130: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

192.168.168.128: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

192.168.168.129: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

?? if I ssh to node 2 and cd to /root, I cannot see the labfiles directory ??

 

[root@node2 ~]# ./root/labfiles/pre-install/cluster-audit.sh | tee audit.log

-bash: ./root/labfiles/pre-install/cluster-audit.sh: No such file or directory

[root@node2 ~]# cd /root/labfiles/pre-install

-bash: cd: /root/labfiles/pre-install: No such file or directory

[root@node2 ~]# ls

anaconda-ks.cfg  audit.log

[root@node2 ~]# cd /root

[root@node2 ~]# cd labfiles

-bash: cd: labfiles: No such file or directory

[root@node2 ~]# exit

logout

Connection to node2 closed.

You have new mail in /var/spool/mail/root

[root@node1 ~]# cd /root

[root@node1 ~]# ls

anaconda-ks.cfg  ClusterTests.zip  labfiles

[root@node1 ~]# cd labfiles

[root@node1 labfiles]# ls

post-install  pre-install

[root@node1 labfiles]# clush -a --copy /root/labfiles

[root@node1 labfiles]# clush -a ls -la /root/ | grep labfiles

192.168.168.128: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

192.168.168.130: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

192.168.168.129: drwxr-xr-x   4 root root  4096 Jul 12 10:09 labfiles

[root@node1 labfiles]# ssh node2

Last login: Thu Jul 12 17:10:36 2018 from 172.17.0.2

[root@node2 ~]# cd /root

[root@node2 ~]# ls

anaconda-ks.cfg  audit.log

[root@node2 ~]# cat audit.log

[root@node2 ~]# exit

logout

Connection to node2 closed.

You have new mail in /var/spool/mail/root

[root@node1 labfiles]# cd ..

[root@node1 ~]# ls

anaconda-ks.cfg  ClusterTests.zip  labfiles

[root@node1 ~]# /root/labfiles/pre-install/cluster-audit.sh | tee audit.log

sed: can't read /root/.ssh/config: No such file or directory

To suppres ssh noise, /root/.ssh/config has been modified

 

#################### Hardware audits ###############################

Thu Jul 12 17:15:21 UTC 2018

################################################################################

NodeSet: 192.168.168.128 192.168.168.129 192.168.168.130

################################################################################

All the groups currently defined for clush:

@all

groups zk, cldb, rm, and hist needed for clush based install

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

DMI Sys Info:

System Information

              Manufacturer: VMware, Inc.

              Product Name: VMware Virtual Platform

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

DMI BIOS:

BIOS Information

              Vendor: Phoenix Technologies LTD

              Version: 6.00

              Release Date: 07/02/2015

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

model name      : Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Architecture:          x86_64

CPU(s):                2

On-line CPU(s) list:   0,1

Thread(s) per core:    1

Core(s) per socket:    1

Socket(s):             2

Model name:            Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz

CPU MHz:               2496

CPU max MHz:           0.0000

CPU min MHz:           0.0000

Hypervisor vendor:     VMware

L1d cache:             32K

L1i cache:             32K

L2 cache:              256K

L3 cache:              3072K

NUMA node0 CPU(s):     0,1

Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch epb invpcid_single fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Thread(s) per core:    1

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

MemTotal:        5946372 kB

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

DIMM slots: 128

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

DIMM count is: 2

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

DIMM Details

              Configured Clock Speed: Unknown

Memory Device

              Size: 2048 MB

              Size: 4096 MB

              Speed: Unknown

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

02:01.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

02:02.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

02:03.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

---------------

192.168.168.[128-130] (3)

---------------

Settings for ens33:

              Speed: 1000Mb/s

              Link detected: yes

Settings for ens34:

              Speed: 1000Mb/s

              Link detected: yes

Settings for ens35:

              Speed: 1000Mb/s

              Link detected: yes

Settings for virbr0:

              Link detected: no

Settings for docker0:

              Link detected: yes

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Storage Controller:

00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)

00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

SCSI RAID devices in dmesg:

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Block Devices:

      1 NAME SIZE TYPE MOUNTPOINT

      1 sda   50G disk

      6 sdb   10G disk

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Udev rules:

60-vboxadd.rules

70-persistent-ipoib.rules

80-docker.rules

################################################################################

 

#################### Linux audits ################################

---------------

192.168.168.[128-130] (3)

---------------

CentOS Linux release 7.3.1611 (Core)

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Linux 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018

x86_64 GNU/Linux

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Time Sync Check:

Thu Jul 12 10:15:27 PDT 2018

################################################################################

192.168.168.129: Existing lock /var/run/yum.pid: another copy is running as pid 19071.

192.168.168.129: Another app is currently holding the yum lock; waiting for it to exit...

192.168.168.129:   The other application is: yum

192.168.168.129:     Memory :  25 M RSS (344 MB VSZ)

192.168.168.129:     Started: Thu Jul 12 10:15:27 2018 - 00:01 ago

192.168.168.129:     State  : Running, pid: 19071

192.168.168.128: Existing lock /var/run/yum.pid: another copy is running as pid 19071.

192.168.168.128: Another app is currently holding the yum lock; waiting for it to exit...

192.168.168.128:   The other application is: yum

192.168.168.128:     Memory :  31 M RSS (349 MB VSZ)

192.168.168.128:     Started: Thu Jul 12 10:15:27 2018 - 00:01 ago

192.168.168.128:     State  : Running, pid: 19071

192.168.168.128: Existing lock /var/run/yum.pid: another copy is running as pid 19069.

192.168.168.128: Another app is currently holding the yum lock; waiting for it to exit...

192.168.168.128:   The other application is: yum

192.168.168.128:     Memory :  39 M RSS (357 MB VSZ)

192.168.168.128:     Started: Thu Jul 12 10:15:27 2018 - 00:03 ago

192.168.168.128:     State  : Running, pid: 19069

---------------

192.168.168.[128-130] (3)

---------------

MapR Repos Check

!maprecosystem                     MapR Technologies                      45

!maprtech                          MapR Technologies                      23

Available Packages

Name        : mapr-core

Arch        : x86_64

Version     : 6.0.0.20171109191718.GA

Release     : 1

Size        : 3.5 k

Repo        : maprtech

Summary     : MapR

License     : MapR Technologies, Inc., CopyRight

Description : MapR: core package

            : Commit: e892229b271c98c75ccbb19fca93a30931496ac2

            : Branch: HEAD detached at e892229

 

Name        : mapr-spark

Arch        : noarch

Version     : 2.1.0.201711121518

Release     : 1

Size        : 193 M

Repo        : maprecosystem

Summary     : MapR

License     : MapR Technologies, Inc., CopyRight

Description : MapR: Spark Package

            : Tag: 2.1.0-mapr-1710

            : Commit:

 

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

SElinux status: SELINUX=disabled

Disabled

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

NFS packages installed

libnfsidmap-0.25-17.el7.x86_64

nfs4-acl-tools-0.3.3-15.el7.x86_64

nfs-utils-1.3.0-0.48.el7_4.1.x86_64

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Required RPMs:

package syslinux is not installed

package hdparm is not installed

package sdparm is not installed

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Optional RPMs:

package atop is not installed

package dstat is not installed

package ftop is not installed

package git is not installed

package htop is not installed

package iftop is not installed

package iotop is not installed

package jq is not installed

package mysql is not installed

package mysql-server is not installed

package nc is not installed

package nmap is not installed

package ntop is not installed

package tmux is not installed

package tree is not installed

package vim is not installed

package xml2 is not installed

package zsh is not installed

################################################################################

clush: 192.168.168.[128-130] (3): exited with exit code 3

---------------

192.168.168.[128-130] (3)

---------------

NTPD Active:

inactive

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

iptables.service

################################################################################

192.168.168.129: Unit iptables.service could not be found.

192.168.168.128: Unit iptables.service could not be found.

192.168.168.130: Unit iptables.service could not be found.

clush: 192.168.168.[128-130] (3): exited with exit code 4

################################################################################

clush: 192.168.168.[128-130] (3): exited with exit code 3

---------------

192.168.168.[128-130] (3)

---------------

  • firewalld.service - firewalld - dynamic firewall daemon

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

   Active: inactive (dead)

     Docs: man:firewalld(1)

################################################################################

192.168.168.130: Unit cpuspeed.service could not be found.

192.168.168.129: Unit cpuspeed.service could not be found.

192.168.168.128: Unit cpuspeed.service could not be found.

clush: 192.168.168.[128-130] (3): exited with exit code 4

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Sysctl Values:

vm.swappiness = 60

net.ipv4.tcp_retries2 = 15

vm.overcommit_memory = 0

################################################################################

/etc/sysctl.conf values should be:

vm.swappiness = 1

net.ipv4.tcp_retries2 = 5

vm.overcommit_memory = 0

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Transparent Huge Pages: [always] madvise never

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Checking for LUKS

---------------

192.168.168.[128-130] (3)

---------------

Disk Controller Max Transfer Size:

      7 /sys/block/sda/queue/max_hw_sectors_kb: 4096

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Disk Controller Configured Transfer Size:

      7 /sys/block/sda/queue/max_sectors_kb: 512

################################################################################

Check Mounted FS

---------------

192.168.168.[128-130] (3)

---------------

Type     Size Use% Mounted on

ext3      35G  48% /

ext3     976M  19% /boot

overlay   35G  48% /var/lib/docker/overlay2/e97afe00b9dd9b5e64cf488dcc11c577108bdd29ad45ce9a2648d07a54cf19cc/merged

overlay   35G  48% /var/lib/docker/overlay2/56d7fbbec49e318ab3c859de21e99844b67adf29b6b48fb9bbc18a4dd37d62fa/merged

overlay   35G  48% /var/lib/docker/overlay2/f5f43c4d506c6a2a918c69144921bbe59820a8bb78a6d63ce5016d4a00961181/merged

################################################################################

Check for nosuid and noexec mounts

---------------

192.168.168.[128-130] (3)

---------------

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

gvfsd-fuse on /run/user/0/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

proc on /run/docker/netns/6d9ca88a673a type proc (rw,nosuid,nodev,noexec,relatime)

proc on /run/docker/netns/09102f7d799a type proc (rw,nosuid,nodev,noexec,relatime)

proc on /run/docker/netns/e07f715c6b89 type proc (rw,nosuid,nodev,noexec,relatime)

################################################################################

Check for /tmp permission

---------------

192.168.168.[128-130] (3)

---------------

1777

################################################################################

Check for tmpwatch on NM local dir

---------------

192.168.168.[128-130] (3)

---------------

grep: /etc/cron.daily/tmpwatch: No such file or directory

Not in tmpwatch: /tmp/hadoop-mapr/nm-local-dir

################################################################################

Java Version

---------------

192.168.168.[128-130] (3)

---------------

openjdk version "1.8.0_161"

OpenJDK Runtime Environment (build 1.8.0_161-b14)

OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

---------------

192.168.168.128

---------------

Loaded plugins: fastestmirror, langpacks

Existing lock /var/run/yum.pid: another copy is running as pid 20051.

Another app is currently holding the yum lock; waiting for it to exit...

  The other application is: yum

    Memory :  30 M RSS (349 MB VSZ)

    Started: Thu Jul 12 10:15:40 2018 - 00:01 ago

    State  : Running, pid: 20051

Installed Packages

copy-jdk-configs.noarch                 2.2-5.el7_4                     @updates

java-1.7.0-openjdk.x86_64               1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.7.0-openjdk-headless.x86_64      1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.8.0-openjdk.x86_64               1:1.8.0.161-0.b14.el7_4         @updates

java-1.8.0-openjdk-headless.x86_64      1:1.8.0.161-0.b14.el7_4         @updates

javapackages-tools.noarch               3.4.1-11.el7                    @base  

python-javapackages.noarch              3.4.1-11.el7                    @base  

tzdata-java.noarch                      2018c-1.el7                     @updates

---------------

192.168.168.129

---------------

Loaded plugins: fastestmirror, langpacks

Existing lock /var/run/yum.pid: another copy is running as pid 20051.

Another app is currently holding the yum lock; waiting for it to exit...

  The other application is: yum

    Memory :  38 M RSS (356 MB VSZ)

    Started: Thu Jul 12 10:15:40 2018 - 00:01 ago

    State  : Running, pid: 20051

Existing lock /var/run/yum.pid: another copy is running as pid 20044.

Another app is currently holding the yum lock; waiting for it to exit...

  The other application is: yum

    Memory :  36 M RSS (355 MB VSZ)

    Started: Thu Jul 12 10:15:40 2018 - 00:03 ago

    State  : Running, pid: 20044

Installed Packages

copy-jdk-configs.noarch                 2.2-5.el7_4                     @updates

java-1.7.0-openjdk.x86_64               1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.7.0-openjdk-headless.x86_64      1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.8.0-openjdk.x86_64               1:1.8.0.161-0.b14.el7_4         @updates

java-1.8.0-openjdk-headless.x86_64      1:1.8.0.161-0.b14.el7_4         @updates

javapackages-tools.noarch               3.4.1-11.el7                    @base  

python-javapackages.noarch              3.4.1-11.el7                    @base  

tzdata-java.noarch                      2018c-1.el7                     @updates

---------------

192.168.168.130

---------------

Loaded plugins: fastestmirror, langpacks

Installed Packages

copy-jdk-configs.noarch                 2.2-5.el7_4                     @updates

java-1.7.0-openjdk.x86_64               1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.7.0-openjdk-headless.x86_64      1:1.7.0.161-2.6.12.0.el7_4      @updates

java-1.8.0-openjdk.x86_64               1:1.8.0.161-0.b14.el7_4         @updates

java-1.8.0-openjdk-headless.x86_64      1:1.8.0.161-0.b14.el7_4         @updates

javapackages-tools.noarch               3.4.1-11.el7                    @base  

python-javapackages.noarch              3.4.1-11.el7                    @base  

tzdata-java.noarch                      2018c-1.el7                     @updates

---------------

192.168.168.[128-130] (3)

---------------

JDK not installed

################################################################################

Hostname IP addresses

192.168.168.129: 192.168.168.130 192.168.168.128 192.168.168.129 192.168.122.1 172.17.0.1

192.168.168.128: 192.168.168.130 192.168.168.128 192.168.168.129 192.168.122.1 172.17.0.1

192.168.168.130: 192.168.168.130 192.168.168.128 192.168.168.129 192.168.122.1 172.17.0.1

################################################################################

DNS lookup

clush: 192.168.168.128: exited with exit code 1

clush: 192.168.168.130: exited with exit code 1

clush: 192.168.168.129: exited with exit code 1

192.168.168.129: Host MapR-VCLEhost not found: 2(SERVFAIL)

192.168.168.128: Host MapR-VCLEhost not found: 2(SERVFAIL)

192.168.168.130: Host MapR-VCLEhost not found: 2(SERVFAIL)

################################################################################

Reverse DNS lookup

clush: 192.168.168.129: exited with exit code 1

clush: 192.168.168.128: exited with exit code 1

clush: 192.168.168.130: exited with exit code 1

192.168.168.129: ;; connection timed out; no servers could be reached

192.168.168.128: ;; connection timed out; no servers could be reached

192.168.168.130: ;; connection timed out; no servers could be reached

################################################################################

Check for root ownership of /opt/mapr

clush: 192.168.168.[128-130] (3): exited with exit code 1

---------------

192.168.168.[128-130] (3)

---------------

stat: cannot stat ‘/opt/mapr’: No such file or directory  --issued stat /opt/mapr and got some output ??

################################################################################

Check for mapr login

---------------

192.168.168.[128-130] (3)

---------------

mapr account for MapR Hadoop

mapr:x:5000:5000::/home/mapr:/bin/bash

################################################################################

Check for mapr user specific open file and process limits

---------------

192.168.168.[128-130] (3)

---------------

Open process limit(should be >=32K): 4096

---------------

192.168.168.[128-130] (3)

---------------

Open file limit(should be >=32K): 1024

################################################################################

Check for mapr users java exec permission and version

---------------

192.168.168.[128-130] (3)

---------------

Java version: openjdk version "1.8.0_161"

OpenJDK Runtime Environment (build 1.8.0_161-b14)

OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

################################################################################

---------------

192.168.168.[128-130] (3)

---------------

Locale setting(must be en_US): LANG=en_US.UTF-8

################################################################################

Check for mapr passwordless ssh (only for MapR v3.x)

192.168.168.130: ls: cannot access /home/mapr/.ssh/authorized_keys: No such file or directory

192.168.168.129: ls: cannot access /home/mapr/.ssh/authorized_keys: No such file or directory

192.168.168.128: ls: cannot access /home/mapr/.ssh/authorized_keys: No such file or directory

clush: 192.168.168.[128-130] (3): exited with exit code 2

################################################################################

Check for system wide nproc and nofile limits

---------------

192.168.168.[128-130] (3)

---------------

*          soft    nproc     4096

root       soft    nproc     unlimited

---------------

192.168.168.[128-130] (3)

---------------

#        - nofile - max number of open file descriptors

#        - nproc - max number of processes

#@student        hard    nproc           20

#@faculty        soft    nproc           20

#@faculty        hard    nproc           50

#ftp             hard    nproc           0

################################################################################

You have new mail in /var/spool/mail/root

[root@node1 ~]#

 

  1. /root/labfiles/pre-install/network-test.sh | tee network.log

 

Uneven IP address count, removing extra client IP

extraip: 192.168.168.130

1 Servers have been launched

1 Clients have been launched

Waiting for client PIDS: 2030

Wait over, post processing

Measuring extra IP address

connect failed: Connection refused

Extra IP address 192.168.168.130 done.

 

Concurrent network throughput results

Outcomes