UPDATE 1: See “Successful Attempt” paragraph. Thanks to Clinton Mills’ comment for pointing out that the previous solution was not perfect. I have a fix for that.
I will show you a very nice and non-intrusive way how to log bash history to a syslog. You may wonder what problem I was trying to solve? Right, the issue is that I want to log root users bash history from multiple servers to a central syslog box. I also want to preserve root users history on each server for convenience purposes.
But first let’s talk about my failed attempts to do so.
Failed attempts
- Hacking bash source code
Really? We are talking about a server farm, who would like to maintain custom built bash packages, make sure that security fixes and bugs are also pushed to your custom bash sources etc etc? – No one! Unless you have nothing else to do.
- Using trap
If you google for a solution you will come across this blog post. It sounds like a feasible idea until you actually try it. Using bash built-in trap command which allows you to basically catch users input and then pipe it to a logger command.
There are few problems taking this approach, first user input does not get logged on logout, second, if you press enter multiple times the previous command gets logged multiple times, third, how do you log shell’s PID?
- Using script (typescript)
Sounds like a good idea too, but first script logs input and output what happens in the terminal. What is wrong with that? Well try to tail the same file you’re logging to – you’ll see what I mean. :-)
Successful attempt
There is an updated trick below which logs to syslog as well as writes commands to .bash_history file so you do not lose your bash history.
PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'
Is that simple? – Yes, it is!
You can put the command (not the whole line, but just the command) into /etc/sysconfig/bash-prompt-default and make it executable. This will log all users bash history system wide to a syslog.
Or you can add it to ~/.bashrc file per user – it’s up to you.
Surely, it’s possible to tell the logger to send messages to specific log facility and so on, but that’s out of the scope of this blog post.
The way I do, I put that into /root/.bashrc, configure rsyslog to log to central log server, which then writes messages to a separate file.
Please don’t say – “oh that’s easy to bypass, this is another failed attempt”. Yes it is easy to bypass, but that’s not the thing I am trying to achieve. At the end of the day root can bypass pretty much everything. This is a solution if you want to have some kind of audit trail what happens on your systems when you have multiple sys admins.
Your input is always welcome in the comments section below.
PS: Thanks to Steve Harris for an awesome brainstorming session which helped me to come up with this idea.
there is also a library intended to do this.
snoopy logger, can easily be preloaded on linux to make it work without changing the existing applications
https://github.com/renard/snoopylogger
Check out rootsh.
Pingback: .:[ d4 n3wS ]:. » Log Bash History to Syslog
Although this is a valid solution for some, it is easily circumvented by a user creating their own .bashrc, or .bash_profile that reset the variable or not source the initial file to begin with, let alone if bash is even their shell.
I have always found it to be a better solution by patching bash, a quick google search for “bash-syslog.patch” shows quite a few results for a bunch of varying versions of bash. I believe this is a form of the one I have been using:
https://github.com/migrantgeek/bash-syslog
Doesn’t syslog work over port 161 UDP?
Wouldn’t this mean the commands could be out of order on the other side?
Why would they? Because UDP is unreliable protocol?
You can always have syslog over TCP.
Nice. There is also snoopy logger: http://sourceforge.net/projects/snoopylogger/ I haven’t played with it though.
Pingback: Logging bash activity « McGowanFam
export HISTFILE= ???
I like this one, better than compiling some software for 100-200 servers. And MUCH better than the trap solution. Thank you!
Mhhh, i dont know why, but with this i have Problems with LVM.
Here is the error:
File descriptor 63 (pipe:[26807]) leaked on lvs invocation. Parent PID 3754: -bash
Pingback: Linux 如何將資料寫到 Syslog | Tsung's Blog
Vaidas – Adding the following entry “history -a >(logger -t “$USER[$$] $SSH_CONNECTION”)” in /etc/sysconfig/bash-prompt-default does not seem to take effect for my sessions. I am running CentOS with centrify (to authenticate to LDAP [AD]). Any suggestions? Thank you
What are the permissions of /etc/sysconfig/bash-prompt-default file?
Works fine,
My output is :
Nov 29 09:26:55 templaterhel root[8536] 10.180.207.138: #1322566015
Nov 29 09:26:55 templaterhel root[8536] 10.180.207.138: ll
Nov 29 09:26:56 templaterhel root[8536] 10.180.207.138: #1322566016
Nov 29 09:26:56 templaterhel root[8536] 10.180.207.138: qq
Nov 29 09:26:58 templaterhel root[8536] 10.180.207.138: #1322566018
Nov 29 09:26:58 templaterhel root[8536] 10.180.207.138: tail /var/log/messages
How can i remove the unix timestamp in logs?
Thanks
That’s a bit weird. What syslog daemon do you use? Can you show us your $PROMPT_COMMAND ?
you may type this to disable timestamps:
export -n HISTTIMEFORMAT
unset HISTTIMEFORMAT
I was using CentOS and the bash-prompt-default was not working. I ended up using /etc/bashrc and put in export PROMPT_COMMAND=’history -a >(logger -t “$USER[$$] $SSH_CONNECTION”)’ at the bottom of the file
Hi Clinton,
Did you make bash-prompt-default file executable?
Thanks,
zooz
Here is a question: How do you write to the syslog server and the local bash file at the same time? One of the issues I see with this setup is that after you logout you can no longer use the up arrow or ctrl r to find previous commands.
Also I setup ‘readonly PROMPT_COMMAND’ so the user can not change the path
Thanks for pointing this out. I have found a way how you can both log to syslog and write to a .bash_history at the same time. See the update on the post.
And, yeah, defining the variable as readonly is a pretty neat trick too.
Thanks,
Vaidas
This worked perfect! Thanks so much, this makes a big difference!
For some reason mine doesn’t log any root command after i use su to switch from a user to root?
Make sure you use “su -l username”. See: “man su” for what “-l” does.
thx you Vaidas, your idea with history -a >(…) is great!
i used a mix of this and of the trap DEBUG solution, which offers the advantage that ‘su’ or ‘sudo’ commands are logged immediately and not after logouts, and also the ‘cd’ command will be logged with the correct ‘pwd’, the reason is that PROMPT_COMMAND is executed after the bash command while trap DEBUG is done before the bash command.
#prompt & color
#http://www.pixelbeat.org/docs/terminal_colours/#256
#http://www.frexx.de/xterm-256-notes/
_backnone=”\e[00m"
_backblack="\e[40m"
_backblue="\e[44m"
_frontred_b="\e[01;31m"
_frontgreen_b="\e[01;32m"
_frontgrey_b="\e[01;37m"
_frontgrey="\e[00;37m"
_frontblue_b="\e[01;34m"
PS1="\[${_backblue}${_frontgreen_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgreen_b}\] ”
#’history’ options
declare -rx HISTSIZE=500000
declare -rx HISTFILESIZE=500000
declare -rx HISTCONTROL=”"
declare -rx HISTIGNORE=”"
if groups | grep -q root; then declare -x TMOUT=3600; fi
declare -rx AUDIT_LOGINUSER=”$(who -mu | awk ‘{print $1}’)”
declare -rx AUDIT_LOGINPID=”$(who -mu | awk ‘{print $6}’)”
declare -rx AUDIT_USER=”$USER” #defined by pam during su/sudo
declare -rx AUDIT_PID=”$$”
declare -rx AUDIT_TTY=”$(who -mu | awk ‘{print $2}’)”
declare -rx AUDIT_SSH=”$(echo $SSH_CONNECTION | awk ‘{print $1″:”$2″->”$3″:”$4}’)”
trap ‘echo -ne “${_backnone}${_frontgrey}” && history -a >(tee -a ~/.bash_history | logger -p user.info -t “[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$SSH_CONNECTION] $PWD”) ‘ DEBUG
let me know where to put this script.
i had a problem with piped bash commands that hangs… i found a workaround using a subshell, but this caused the ‘history -a’ to not refresh the history outside the subshell scope…
finally the solution was to use a function that re-read the history after the subshell execution.
it works as i wanted. see it at http://superuser.com/a/386811/117132
as Vaidas wrote, more easy to deploy than patching the bash in C (i did that too in the past).
but there is some performance drop while re-reading each time the history file and doing a disk ‘sync’…
Awesome stuff. :-)
thx Vaidas for this page and for showing the trick using traps ;-)
after some try&err&tuning i wrote a blog entry here http://blog.pointsoftware.ch/index.php/howto-bash-audit-command-logger/#more-223 to explain the whole stuff in detail.
the method works by running this at the start of a bash session:
declare -rx HISTCONTROL=”" #does not ignore spaces or duplicates
declare -rx HISTIGNORE=”" #does not ignore patterns
declare -rx AUDIT_LOGINUSER=”$(who -mu | awk ‘{print $1}’)”
declare -rx AUDIT_LOGINPID=”$(who -mu | awk ‘{print $6}’)”
declare -rx AUDIT_USER=”$USER” #defined by pam during su/sudo
declare -rx AUDIT_PID=”$$”
declare -rx AUDIT_TTY=”$(who -mu | awk ‘{print $2}’)”
declare -rx AUDIT_SSH=”$([ -n "$SSH_CONNECTION" ] && echo “$SSH_CONNECTION” | awk ‘{print $1″:”$2″->”$3″:”$4}’)”
declare -rx AUDIT_STR=”[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$AUDIT_SSH]”
set +o functrace #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
shopt -s extglob #enable extended pattern matching operators
function audit_DEBUG() {
if [ "$BASH_COMMAND" != "$PROMPT_COMMAND" ] #avoid logging unexecuted commands after ‘ctrl-c or ‘empty+enter’
then
local AUDIT_CMD=”$(history 1)” #current history command
if ! logger -p user.info -t “$AUDIT_STR $PWD” “${AUDIT_CMD##*( )?(+([0-9])[^0-9])*( )}”
then
echo error “$AUDIT_STR $PWD” “${AUDIT_CMD##*( )?(+([0-9])[^0-9])*( )}”
fi
fi
}
function audit_EXIT() {
local AUDIT_STATUS=”$?”
logger -p user.info -t “$AUDIT_STR” “#=== bash session ended. ===”
exit “$AUDIT_STATUS”
}
declare -fr +t audit_DEBUG
declare -fr +t audit_EXIT
logger -p user.info -t “$AUDIT_STR” “#=== New bash session started. ===” #audit the session openning
#when a bash command is executed it launches first the audit_DEBUG(),
#then the trap DEBUG is disabled to avoid a useless rerun of audit_DEBUG() during the execution of pipes-commands;
#at the end, when the prompt is displayed, re-enable the trap DEBUG
declare -rx PROMPT_COMMAND=”trap ‘audit_DEBUG; trap DEBUG’ DEBUG”
declare -rx BASH_COMMAND #current command executed by user or a trap
declare -rx SHELLOPT #shell options, like functrace
trap audit_EXIT EXIT
Pingback: HOW TO : Log all commands issued in shell to syslog | Kudithipudi.Org
Pingback: Log Bash History | TechByTom
Thanks for your post, I have a couple of questions.This solution is perfect for me except that it shows in the process list when the user uses
ps x
dan 29787 0.0 0.1 3796 580 pts/0 S 10:54 0:00 tee -a /home/dan/.bash_history
dan 29788 0.0 0.1 3780 604 pts/0 S 10:54 0:00 logger -t dan[29740] … 3821 …
I am using this to see if the more advanced users are doing anything sketchy. Any advanced user is going to see that process running under their user and kill it.
1) Is there anyway to hide that process from the user’s process list?
2) Also is there a way to set this so it will log to a separate file, say /var/log/global_bash_history or something similar?
Thanks
Hi Dan,
Like I mentioned in my post (if I remember correctly now), that the bash logging trick is not to snoop on someone, it’s to not intrusively log one’s bash activity. Because this script goes into .bashrc one way or another, so a user can just remove it if you will.
Yeah, that’s pretty simple, you can easily filter logs with rsyslog or syslog-ng depending on what your distribution uses.
Pingback: How To Log Your Bash History With Syslog | Backdrift
I see the same as reported – if the command is put into /etc/sysconfig/bash-prompt-default (or xterm) it doesn’t work and yes, it’s executable. If I set the value directly to PROMPT_COMMAND it works as expected. Anyone know what the difference is?
Well, I kept working on this as I wanted a way to distribute it to several servers without overwriting existing configurations (bashrc, bash_profile and like). I ended up putting the command into /etc/profile.d/history2log.sh and appended a “readonly” before PROMPT_COMMAND. That worked as /etc/bashrc couldn’t overwrite it anymore. So I ended up with this commandline:
readonly PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'Finally something that works out of the box!
Thanks for this!
I know the feeling too :-) Finally the hard work was pulled in mainstream. But that’s what open source is all about ;-)
I solved “multiple log lines” problem using trap. And instead DEBUG, I used ERR and put a return 255 at last line of function >:D
Plus just create “log2syslog.sh” file with content bellow and copy it to /etc/profile.d/
—————————————-
function log2syslog
{
declare COMANDO
declare HOSTN
COMANDO=$(fc -ln 0)
HOSTN=$(hostname -s)
nc -w0 -u 192.168.0.10 514 <<< "$HOSTN bash: ($USER) : $COMANDO”
return 255
}
trap log2syslog ERR
—————————————-
Sorry, somehow I got a false-positive here, firstly I got logged all commands, but after login/logoff, now it just got logged if command returns sometype of error :/
readonly PROMPT_COMMAND=’history | tail -1 | cut -c 1-7 –complement | logger -t “$USER[$$] ${SSH_CONNECTION:-$HOSTNAME}”‘
is another solution, that doesn’t show up in the process-tree. ;)
featured by killermoehre and me
in bash, exec command below
$ history -a >(tee -a ~/.bash_history | logger -t “$USER[$$] $SSH_CONNECTION”)
return
-bash: logger: command not found
$ which logger
/usr/bin/logger
and,
$ history -a >(tee -a ~/.bash_history | /usr/bin/logger -t “$USER[$$] $SSH_CONNECTION”)
-bash: /usr/bin/logger: No such file or directory.
what should i do ?
please tell me.