informix – index creation and problem with space

  212: Cannot add index.

   28: No space left on device
Error in line 2

This was matter of

informix@server://usr/informix/etc >cat onconfig | grep DBSPACETEMP
DBSPACETEMP     temp_space

If you have on temporary dbspace in your config, Informix will use /tmp filesystem by default so make sure you have one temp dbspace big enough and restart Informix to load onconfig with defined setting. To add temp dbspace…

onspaces -c -t -d temp_space -p /data/chunks/temp_c1 -o 0 -s 20971520

clear out vios attention led

$ lssyscfg -r sys -F name
Server-vmv13g-SNSNSNSN
$ lsled -m Server-vmv13g-SNSNSNSN  -r sa -t phys
state=on
$ chled -m  Server-vmv13g-SNSNSNSN-r sa -o off -t virtualsys
[VIOSE0000-0009] Parameter sa is not valid.

$ chled -m  Server-vmv13g-SNSNSNSN -r sa -o off -t virtualsys
$ lsled -m Server-vmv13g-SNSNSNSN  -r sa -t phys
state=off

WannaCry – some links

There are enough articles about it in Web right now, just saving some useful links for future

# PS script to scan AD
https://github.com/kieranwalsh/PowerShell/blob/master/Get-WannaCryPatchState/Get-WannaCryPatchState.ps1
# Nmap module (remove space, WordPress is automatically expanding link?)
https://gist.github.com/Neo23x0/ 60268852ff3a5776ef66bc15d50a024a

# Python tool using massscan
https://github.com/countercept/doublepulsar-detection-script

# Windows XP Patch (PL)
http://download.windowsupdate.com/d/csa/csa/secu/2017/02/windowsxp-kb4012598-x86-custom-plk_05bec673af4dad0a111aacb89fe2c463539c010e.exe

# Windows 2003 Patch (PL)
https://download.microsoft.com/download/4/9/0/490F6C3B-76EA-4D37-AD44-017B5245A2B2/WindowsServer2003-KB4012598-x86-custom-PLK.exe

# Fixing SMB
https://community.tenable.com/thread/11156

oracle – insufficient number of disks

-bash-4.1$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Thu Jan 11 08:49:23 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Automatic Storage Management option

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

This is caused by reconfigured Oracle ASM and missing diskstring parameter

SQL> show parameter disk;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string
asm_diskstring                       string

Let’s fix it…

SQL> create pfile='/tmp/pfile_asm.ora' from memory;

File created.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Automatic Storage Management option
-bash-4.1$ vi /tmp/pfile_asm.ora
-bash-4.1$ tail -n 2 /tmp/pfile_asm.ora
asm_diskstring='/dev/oracleasm/disks/*'
asm_diskgroups='DATA'
SQL> create spfile from pfile='/tmp/pfile_asm.ora';

File created.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Automatic Storage Management option
-bash-4.1$ srvctl stop asm
-bash-4.1$ srvctl start asm
-bash-4.1$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Thu May 11 09:00:23 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Automatic Storage Management option

SQL> show parameter disk;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string      DATA
asm_diskstring                       string      /dev/oracleasm/disks/*
-bash-4.1$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    307196   223644                0          223644              0             N  DATA/

reconfigure oracle has after hostname change

[root@oranode01 zabbix]# cd /u01/app/grid/product/asm/crs/install
[root@oranode01 install]# ./roothas.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
PRKO-2573 : ONS daemon is already stopped.
CRS-2673: Attempting to stop 'ora.asm' on 'oranode01'
CRS-2677: Stop of 'ora.asm' on 'oranode01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oranode01'
CRS-2677: Stop of 'ora.cssd' on 'oranode01' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oranode01'
CRS-2673: Attempting to stop 'ora.evmd' on 'oranode01'
CRS-2677: Stop of 'ora.evmd' on 'oranode01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oranode01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack

[root@oranode01 install]# ./roothas.pl
Using configuration parameter file: ./crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oranode01 successfully pinned.
Adding Clusterware entries to upstart

oranode01     2016/01/01 08:44:28     /u01/app/grid/product/asm/cdata/oranode01/backup_20170511_084428.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
[root@oranode01 install]# su - oracle
-bash-4.1$ . .grid_profile
-bash-4.1$ crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
-bash-4.1$ crs_stat
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on oranode01

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE

-bash-4.1$ crsctl start resource "ora.cssd"
CRS-2672: Attempting to start 'ora.cssd' on 'oranode01'
CRS-2672: Attempting to start 'ora.diskmon' on 'oranode01'
CRS-2676: Start of 'ora.diskmon' on 'oranode01' succeeded
CRS-2676: Start of 'ora.cssd' on 'oranode01' succeeded
-bash-4.1$ vi /u01/app/grid/product/asm/network/admin/listener.ora
-bash-4.1$ srvctl add asm
-bash-4.1$ srvctl status asm
ASM is not running.
-bash-4.1$ srvctl start asm
-bash-4.1$ srvctl status asm
ASM is running on oranode01

scp – transfer stalled

I’ve had problems with copying files over network, transfer was being stalled, what has helped me were setting system settings (TCP Selective Acknowledgment) and -l option in scp

sudo sysctl -w net.ipv4.tcp_sack=0
scp -l 64192 

rundeck and ldap – freeipa

Documentation is lacking details for Rundeck unfortunately… Here’s what to do if you plan onto granting access to Rundeck via LDAP/FreeIPA. Note that I’m not using any bind user to verify credentials, also had to change userObjectClass and roleBaseDn attributes. Group “admins” is group inside FreeIPA.

[root@itops01 rundeck]# cat /etc/rundeck/jaas-ldap.conf
ldap {
    com.dtolabs.rundeck.jetty.jaas.JettyCachingLdapLoginModule required
      debug="true"
      contextFactory="com.sun.jndi.ldap.LdapCtxFactory"
      providerUrl="ldap://server:389"
      authenticationMethod="simple"
      forceBindingLogin="true"
      userBaseDn="cn=users,cn=accounts,dc=poland,dc=com"
      userRdnAttribute="uid"
      userIdAttribute="uid"
      userPasswordAttribute="userPassword"
      userObjectClass="inetorgperson"
      roleBaseDn="cn=groups,cn=accounts,dc=poland,dc=com"
      roleNameAttribute="cn"
      roleUsernameMemberAttribute="memberUid"
      roleMemberAttribute="memberUid"
      roleObjectClass="posixGroup"
      cacheDurationMillis="300000"
      supplementalRoles="admins"
      reportStatistics="true"
      timeoutRead="10000"
      timeoutConnect="20000"
      nestedGroups="false";
};

Now onto ACLs, by default there’s only admin group, I edited that to admins

[root@itops01 rundeck]# cat /etc/rundeck/admin.aclpolicy
description: Admin, all access.
context:
  project: '.*' # all projects
for:
  resource:
    - allow: '*' # allow read/create all kinds
  adhoc:
    - allow: '*' # allow read/running/killing adhoc jobs
  job:
    - allow: '*' # allow read/write/delete/run/kill of all jobs
  node:
    - allow: '*' # allow read/run for all nodes
by:
  group: admins

---

description: Admin, all access.
context:
  application: 'rundeck'
for:
  resource:
    - allow: '*' # allow create of projects
  project:
    - allow: '*' # allow view/admin of all projects
  project_acl:
    - allow: '*' # allow admin of all project-level ACL policies
  storage:
    - allow: '*' # allow read/create/update/delete for all /keys/* storage content
by:
  group: admins

Also editing profile as documentation says

[root@itops01 rundeck]# cat /etc/rundeck/profile
RDECK_INSTALL="${RDECK_INSTALL:-/var/lib/rundeck}"
RDECK_BASE="${RDECK_BASE:-/var/lib/rundeck}"
RDECK_CONFIG="${RDECK_CONFIG:-/etc/rundeck}"
RDECK_CONFIG_FILE="${RDECK_CONFIG_FILE:-$RDECK_CONFIG/rundeck-config.properties}"
RDECK_SERVER_BASE="${RDECK_SERVER_BASE:-$RDECK_BASE}"
RDECK_SERVER_CONFIG="${RDECK_SERVER_CONFIG:-$RDECK_CONFIG}"
RDECK_SERVER_DATA="${RDECK_SERVER_DATA:-$RDECK_BASE/data}"
RDECK_PROJECTS="${RDECK_PROJECTS:-$RDECK_BASE/projects}"
RUNDECK_TEMPDIR="${RUNDECK_TEMPDIR:-/tmp/rundeck}"
RUNDECK_WORKDIR="${RUNDECK_TEMPDIR:-$RDECK_BASE/work}"
RUNDECK_LOGDIR="${RUNDECK_LOGDIR:-$RDECK_BASE/logs}"
RDECK_JVM_SETTINGS="${RDECK_JVM_SETTINGS:- -Xmx1024m -Xms256m -XX:MaxMetaspaceSize=256m -server}"
RDECK_TRUSTSTORE_FILE="${RDECK_TRUSTSTORE_FILE:-$RDECK_CONFIG/ssl/truststore}"
RDECK_TRUSTSTORE_TYPE="${RDECK_TRUSTSTORE_TYPE:-jks}"
JAAS_CONF="${JAAS_CONF:-$RDECK_CONFIG/jaas-loginmodule.conf}"
LOGIN_MODULE="${LOGIN_MODULE:-RDpropertyfilelogin}"
RDECK_HTTP_PORT=${RDECK_HTTP_PORT:-4440}
RDECK_HTTPS_PORT=${RDECK_HTTPS_PORT:-4443}


# If no JAVA_CMD, try to find it in $JAVA_HOME
if [ -z "$JAVA_CMD" ] && [ -n "$JAVA_HOME" ] && [ -x "$JAVA_HOME/bin/java" ] ; then
  JAVA_CMD=$JAVA_HOME/bin/java
  PATH=$PATH:$JAVA_HOME/bin
  export JAVA_HOME
elif [ -z "$JAVA_CMD" ] ; then
  JAVA_CMD=java
fi

# build classpath without lone : that includes .
for jar in $(find $RDECK_INSTALL/cli -name '*.jar') ; do
  CLI_CP=${CLI_CP:+$CLI_CP:}$jar
done
for jar in $(find $RDECK_INSTALL/bootstrap -name '*.jar') ; do
  BOOTSTRAP_CP=${BOOTSTRAP_CP:+$BOOTSTRAP_CP:}$jar
done


#RDECK_JVM="-Djava.security.auth.login.config=$JAAS_CONF \
#           -Dloginmodule.name=$LOGIN_MODULE \

RDECK_JVM="-Djava.security.auth.login.config=/etc/rundeck/jaas-ldap.conf \
           -Dloginmodule.name=ldap \
           -Drdeck.config=$RDECK_CONFIG \
           -Drundeck.server.configDir=$RDECK_SERVER_CONFIG \
           -Dserver.datastore.path=$RDECK_SERVER_DATA/rundeck \
           -Drundeck.server.serverDir=$RDECK_INSTALL \
...

Now restart service and try logging in!

varnish – basics

Varnish is a cache server for static content, that is supposed to take off load from servers by serving same, non-changed content from cache.
Base config file lives in

/etc/varnish/default.vcl

Varnish has it’s own configuration language, VCL – Varnish Configuration Language and lot’s of place for tweaking, using administrative tool – varnishadm. It has two modes – interactive and standalone

18:46:21 environments/varnish [~d47zm3@sh3llsh0ck~] [master●] » docker exec -it varnish bash
# standalone
[root@49b8fdeea91e /]# varnishadm backend.list
Backend name                   Refs   Admin      Probe
default(172.17.0.2,,80)        1      probe      Healthy (no probe)
# cli
[root@49b8fdeea91e /]# varnishadm
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.10.0-19-generic,x86_64,-smalloc,-smalloc,-hcritbit
varnish-4.0.4 revision 386f712

Type 'help' for command list.
Type 'quit' to close CLI session.

help
200
help []
ping []
auth 
quit
banner
status
start
stop
vcl.load  
vcl.inline  
vcl.use 
vcl.discard 
vcl.list
param.show [-l] []
param.set  
panic.show
panic.clear
storage.list
vcl.show [-v] 
backend.list []
backend.set_health  
ban    [&&   ]...
ban.list

vcl.list
200
active          0 boot

Basic config looks like this

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "apache";
    .port = "80";
}

You can add a lot of options to it, so don’t hesitate checking them out, timeouts, health checks etc.

Let’s set up quickly some Apache Server and Varnish using Docker

#!/bin/bash

# include my bash framework
. ~/.bash_framework

apache_container="apache"
varnish_container="varnish"

decho "Stopping and removing all containers..."
# stop all containers and remove them 
docker stop $(docker ps -a | egrep -v "CONTAINER" | awk '{print $1}') > /dev/null
docker rm $(docker ps -a | egrep -v "CONTAINER" | awk '{print $1}') > /dev/null

decho "Running new set of containers..."
docker run --name "${apache_container}" -d  eboraas/apache-php > /dev/null
docker run -itd --link ${apache_container}:apache-web --name "${varnish_container}" million12/varnish > /dev/null

decho "Put Varnish config file in place..."
docker cp default.vcl ${varnish_container}:/etc/varnish/default.vcl
decho "Restart Varnish to refresh settings..."
docker restart ${varnish_container} > /dev/null

decho "Testing..."
apache_ip=$( docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${apache_container} )
varnish_ip=$( docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${varnish_container} )
sleep 5
decho "Apache..."
curl -s -i "http://${apache_ip}" | grep "HTTP/"
decho "Varnish..."
curl -s -i "http://${varnish_ip}" | grep "HTTP/"

Running it returns

[18:45:57] Stopping and removing all containers...
[18:46:07] Running new set of containers...
[18:46:08] Put Varnish config file in place...
[18:46:08] Restart Varnish to refresh settings...
[18:46:10] Testing...
[18:46:15] Apache...
HTTP/1.1 200 OK
[18:46:15] Varnish...
HTTP/1.1 200 OK

A tool that might help you to check performance of WebServer

19:40:00 environments/varnish [~d47zm3@sh3llsh0ck~] [master●] » httperf --hog --server=172.17.0.3 --port=80 --num-conn 1000
httperf --hog --client=0/1 --server=172.17.0.3 --port=80 --uri=/ --send-buffer=4096 --recv-buffer=16384 --num-conns=1000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 1000 requests 1000 replies 1000 test-duration 0.247 s

Connection rate: 4049.7 conn/s (0.2 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 0.2 avg 0.2 max 1.3 median 0.5 stddev 0.1
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 4049.7 req/s (0.2 ms/req)
Request size [B]: 63.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 0.2 transfer 0.0
Reply size [B]: header 341.0 content 10701.0 footer 2.0 (total 11044.0)
Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.03 system 0.22 (user 11.3% system 87.5% total 98.8%)
Net I/O: 43921.6 KB/s (359.8*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Everything is available at my GitHub, later on I plan to add some heavy webapp with a lot of static content to test out Varnish influence.

sealert – another handy tool for selinux

[root@dc-zabbix01 ~]# sealert -a /var/log/audit/audit.log
100% done
found 1 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /usr/sbin/logrotate from write access on the directory logs.

*****  Plugin catchall_labels (83.8 confidence) suggests   *******************

If you want to allow logrotate to have write access on the logs directory
Then you need to change the label on logs
Do
# semanage fcontext -a -t FILE_TYPE 'logs'
where FILE_TYPE is one of the following: NetworkManager_log_t, abrt_var_cache_t, abrt_var_log_t, acct_data_t, afs_logfile_t, aide_log_t, amanda_log_t, antivirus_log_t, apcupsd_log_t, apmd_log_t, asterisk_log_t, auth_cache_t, bacula_log_t, bitlbee_log_t, boinc_log_t, brltty_
...
Then execute:
restorecon -v 'logs'


*****  Plugin catchall (17.1 confidence) suggests   **************************

If you believe that logrotate should be allowed write access on the logs directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'logrotate' --raw | audit2allow -M my-logrotate
# semodule -i my-logrotate.pp


Additional Information:
Source Context                system_u:system_r:logrotate_t:s0-s0:c0.c1023
Target Context                system_u:object_r:var_t:s0
Target Objects                logs [ dir ]
Source                        logrotate
Source Path                   /usr/sbin/logrotate
Port                          
Host                          
Source RPM Packages           logrotate-3.8.6-12.el7.x86_64
Target RPM Packages
Policy RPM                    selinux-policy-3.13.1-102.el7_3.13.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     
Platform                      Linux 
                              3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18
                              13:06:36 UTC 2017 x86_64 x86_64
Alert Count                   1
First Seen                    2017-04-10 03:38:02 CEST
Last Seen                     2017-04-10 03:38:02 CEST
Local ID                      5935f971-70c9-4d10-8fcd-5e4154796a7b

Raw Audit Messages
type=AVC msg=audit(1491788282.38:800089): avc:  denied  { write } for  pid=16994 comm="logrotate" name="logs" dev="dm-4" ino=14307 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_t:s0 tclass=dir


type=SYSCALL msg=audit(1491788282.38:800089): arch=x86_64 syscall=rename success=no exit=EACCES a0=1de3e50 a1=1decc70 a2=1de3e50 a3=e items=0 ppid=16992 pid=16994 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=25451 comm=logrotate exe=/usr/sbi
n/logrotate subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null)

Hash: logrotate,logrotate_t,var_t,dir,write

aix, ssh access denied

I was trying to SSH root user after changing it’s shell in /etc/passwd, could not log in with all the time permision denied message, debugging SSH daemon revealed issue

root@host://var/adm >
root@host://var/adm >stopsrc -s sshd
0513-044 The sshd Subsystem was requested to stop.
root@host://var/adm >
root@host://var/adm >
root@host://var/adm >
root@host://var/adm >/usr/sbin/sshd -dd
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 161
...
debug1: KEX done
debug1: userauth-request for user root service ssh-connection method none
debug1: attempt 0 failures 0
User root not allowed because shell /usr/bin/bash  does not exist
input_userauth_request: invalid user root
debug2: input_userauth_request: try method none
debug2: monitor_read: 6 used once, disabling now

There was white space after shell in /etc/passwd.